← all tasks

hopper-brisk-robust

Hopper (MuJoCo Hopper-v5) · solved 3 of 11 agent attempts pass

The task. Open, and meant to be: speed and robustness together.

more

Every controller measured so far has one or the other: the parity designs (18-19 m) and TD3 (20 m) fall on every seed under -30 % friction or -20 % motor gain; the robust designs survive all four perturbations at 7-8 m. The oracle is the robust Opus design from hopper-robust (7.8 m, suite >= 0.95), which fails the brisk rung.

Gate. per-seed success is the `brisk` rung of this benchmark's ladder ['alive', 'forward', 'brisk', 'fast'] (`--success-mode brisk`), not the evaluator's default flag; the task is solved when the success rate over the grader's held-out seeds is `>= 0.9`; the same gate must also hold on the sibling scenario(s) `heavy_torso_20`, `low_friction_30`, `weak_motors_20`, `gravity_high_10` (`--scenarios`); the aggregate `total_reward_mean` over the grader's held-out seeds (base scenario) must be `>= 2500` (rung `metric:total_reward_mean`)

Axes: plant:hopper, scenario:standard_v0+physics-suite, gate:brisk_10m, return:2500, robustness:torso-friction-motors-gravity

Each video re-runs the attempt's final controller through the task's own evaluator on a public seed (never the grader's held-out ones). The table under each video is the official grade on the held-out seeds.

Best attemptOpus 5.5 (claude-code, 30 min) · standard_v0

Graded on the held-out seeds: passes every gate on the grader's held-out seeds.

criteriongradedneeded
base:brisk1.000≥ 0.900
scenario:heavy_torso_201.000≥ 0.900
scenario:low_friction_301.000≥ 0.900
scenario:weak_motors_201.000≥ 0.900
scenario:gravity_high_101.000≥ 0.900
metric:total_reward_mean2652≥ 2500

How the agent got there: 30 min · 86 tool calls · 1 edit-tool edits · 60 runs of its own Python (tests, experiments, rewrites) · no tuner runs · official verifier 4× (success rates in order: 0.80, 1.00, 1.00, 1.00).

The agent's final message

That stopped background task was an old wait loop, polling the log of an earlier parameter search that I had already killed. Nothing depended on it, so I'm not restarting it. The submitted `/app/work/controller.py` is unchanged and still passes the full gate on the visible seeds.

Controller family: other: phase-machine PD servo with ankle leg-angle regulation

A three-phase contact state machine drives knee/ankle thrust for hopping while the hip PD-servos torso pitch and the stance ankle regulates leg angle and speed, with gains found by CMA-ES.

Shown: public seed 0 (standard_v0); this episode passes the brisk rung. This episode: episode_steps 1000, x_final 13.1, min_height 1.08, max_abs_pitch 0.137, n_flights 12, total_reward 2641.

controller.py · agent transcript · trial hopper-brisk-robust__r8oRaae

Best attemptOpus 5.5 (claude-code, 30 min) · heavy_torso_20

Graded on the held-out seeds: passes every gate on the grader's held-out seeds.

criteriongradedneeded
base:brisk1.000≥ 0.900
scenario:heavy_torso_201.000≥ 0.900
scenario:low_friction_301.000≥ 0.900
scenario:weak_motors_201.000≥ 0.900
scenario:gravity_high_101.000≥ 0.900
metric:total_reward_mean2652≥ 2500

How the agent got there: 30 min · 86 tool calls · 1 edit-tool edits · 60 runs of its own Python (tests, experiments, rewrites) · no tuner runs · official verifier 4× (success rates in order: 0.80, 1.00, 1.00, 1.00).

The agent's final message

That stopped background task was an old wait loop, polling the log of an earlier parameter search that I had already killed. Nothing depended on it, so I'm not restarting it. The submitted `/app/work/controller.py` is unchanged and still passes the full gate on the visible seeds.

Controller family: other: phase-machine PD servo with ankle leg-angle regulation

A three-phase contact state machine drives knee/ankle thrust for hopping while the hip PD-servos torso pitch and the stance ankle regulates leg angle and speed, with gains found by CMA-ES.

Shown: public seed 0 (heavy_torso_20); this episode passes the brisk rung. This episode: episode_steps 1000, x_final 14.3, min_height 1.08, max_abs_pitch 0.135, n_flights 12, total_reward 2790.

controller.py · agent transcript · trial hopper-brisk-robust__r8oRaae

FailureOpus 5.5 (claude-code, 56 min) · heavy_torso_20

Graded on the held-out seeds: passes the nominal plant but misses the gate under heavy_torso_20.

criteriongradedneeded
base:brisk1.000≥ 0.900
scenario:heavy_torso_200.700≥ 0.900
scenario:low_friction_301.000≥ 0.900
scenario:weak_motors_200.975≥ 0.900
scenario:gravity_high_101.000≥ 0.900
metric:total_reward_mean2408≥ 2500

How the agent got there: 56 min · 54 tool calls · 1 edit-tool edits · 24 runs of its own Python (tests, experiments, rewrites) · no tuner runs · official verifier 5× (success rates in order: 0.00, 1.00, 0.90, 0.60, 0.60).

The agent's final message

The 60 minutes are up and the controller does **not** pass the gate, so it will score zero. On the visible seeds 0–9 it fails two of the requirements: | Requirement | Result (seeds 0–9) | |---|---| | Nominal, reaches 10 m in 8 s (≥ 0.9 of seeds) | 1.00 ✅ (never falls) | | Low friction | 1.00 ✅ | | Weak motors | 1.00 ✅ | | High gravity | 1.00 ✅ | | Heavy torso | **0.60 ❌** | | Nominal mean return ≥ 2500 | **2398 ❌** (about 11.6 m) | **What's in `/app/work/controller.py`:** a linear state-feedback law, `u = clip(K @ s, -1, 1)`. `s` is the 11-value Hopper observation, each value scaled by a typical size and clipped, plus a foot-contact flag and a constant. That makes it two control laws, one for stance and one for flight, with no timing clock. The 3×13 gain matrix `K` was found by CMA-ES, a search that uses no gradients and no neural network. Each candidate was scored on all five plants (nominal plus the four changes). The docstring explains what each group of gains does. **How I got here:** 1. **Hand-built hopping logic, tuned automatically:** the search always settled on standing still (return about 1050, no forward motion). 2. **Why that design fails:** the hip and knee can only rotate backward, and the hip sits at its 0 limit in stance. So the knee can barely crouch, and the crouch-then-thrust stance I'd designed can't work. 3. **Linear law, first stage:** I left out most of the per-step survival bonus, so the search was paid for forward progress rather than for standing.…

Controller family: linear policy tuned by search

A 3x13 gain matrix maps normalized Hopper observations plus contact flag and bias to motor torques, clipped to [-1,1], with the gains found by CMA-ES over nominal and perturbed plants.

Shown: public seed 4 (heavy_torso_20); this episode fails the brisk rung. This episode: episode_steps 996, x_final 14, min_height 0.926, max_abs_pitch 0.205, n_flights 16, total_reward 2741. It was the first failing one of 5 public seeds tried.

controller.py · agent transcript · trial hopper-brisk-robust__rFNQ2jz

FailureFable 5.1 (claude-code, 44 min) · low_friction_30

Graded on the held-out seeds: passes the nominal plant but misses the gate under low_friction_30.

criteriongradedneeded
base:brisk1.000≥ 0.900
scenario:heavy_torso_201.000≥ 0.900
scenario:low_friction_300.000≥ 0.900
scenario:weak_motors_200.000≥ 0.900
scenario:gravity_high_101.000≥ 0.900
metric:total_reward_mean2673≥ 2500

How the agent got there: 44 min · 98 tool calls · 1 edit-tool edits · 77 runs of its own Python (tests, experiments, rewrites) · repository tuner 6× · official verifier 2× (success rates in order: ?, 1.00).

The agent's final message

The final controller is in place at `/app/work/controller.py`, verified with the official evaluator on the same file. **Outcome.** The gate is not fully met. The submission passes the nominal plant and two of the four sibling scenarios, and fails weak motors and low friction. | Scenario (seeds 0-4, official evaluator) | brisk rate | return mean | |---|---|---| | standard_v0 | 1.00 | 2672 (gate 2500: pass) | | heavy_torso_20 | 1.00 | | | gravity_high_10 | 1.00 | | | weak_motors_20 | 0.00 | | | low_friction_30 | 0.00 | | On seeds 0-9 the same three scenarios are 10/10, with nominal distance about 13.4 m. **What the controller does.** A stance/flight state machine with no learning. In stance the hip holds torso pitch with a PD loop, the knee compresses then thrusts with a constant torque, and the ankle balances the thigh angle over the foot and adds a toe-down push when the hopper exceeds its target speed. In flight the hip holds attitude, the knee pre-bends for landing, and the ankle sets the sole parallel to the ground with a toe-up offset. The docstring explains each gain. **What I found along the way.** The foot sits behind the hip whenever the torso leans forward, positive ankle torque rocks the body onto its heel, and the long foot acts as a third leg segment. The heavy leg means the hip cannot reposition the foot in flight, so leg placement comes from stance-end ankle pushes. The resulting gait is very parameter sensitive: a multi-scenario evolution strategy over 18 …

Controller family: lookup table / heuristic rules

A touchdown/liftoff state machine runs PD torso-pitch control on the hip, a soft knee spring then constant-torque thrust in stance, an ankle that balances the leg over the foot and regulates speed, and PD attitude, knee pre-bend and foot-flat control in flight, with parameters found by a multi-scenario (1+1) evolution strategy.

Shown: public seed 0 (low_friction_30); this episode fails the brisk rung. This episode: episode_steps 159, x_final 2.09, min_height 0.691, max_abs_pitch 0.171, n_flights 3, total_reward 419. It was the first failing one of 1 public seeds tried.

controller.py · agent transcript · trial hopper-brisk-robust__kmoWXqh

FailureOpus 5.5 (claude-code, 52 min) · standard_v0

Graded on the held-out seeds: fails the main gate badly (success rate below 0.5).

criteriongradedneeded
base:brisk0.000≥ 0.900
scenario:heavy_torso_200.000≥ 0.900
scenario:low_friction_300.000≥ 0.900
scenario:weak_motors_200.375≥ 0.900
scenario:gravity_high_100.000≥ 0.900
metric:total_reward_mean2027≥ 2500

How the agent got there: 52 min · 94 tool calls · 1 edit-tool edits · 59 runs of its own Python (tests, experiments, rewrites) · no tuner runs · official verifier 2× (success rates in order: 0.50, 0.00).

The agent's final message

The controller in `/app/work/controller.py` does not pass the gate, and this submission will score zero. On visible seeds 0–4 it stays upright on 4 of the 5 scenarios but reaches only about 8.3 m on the nominal plant (return 2032 against the 2500 needed). The gate needs 10 m on 90% of seeds in every scenario. **Official evaluator results (seeds 0–4, brisk mode):** | Scenario | Alive | Brisk (≥10 m) | |---|---|---| | standard_v0 | 5/5 | 0/5 (~8.3 m, return 2032) | | heavy_torso_20 | 5/5 | 0/5 | | low_friction_30 | 5/5 | 0/5 | | weak_motors_20 | 3/5 | 3/5 | | gravity_high_10 | 5/5 | 0/5 | **How it works:** each joint gets a sine-wave oscillator (about 1.5 Hz) plus feedback: - **Hip:** a pitch PD keeps the torso inside ±0.2 rad. In stance, pushing the thigh backward both rights the torso and drives the body forward. - **Knee:** held mostly straight, so the leg acts like a stiff pogo stick. - **Ankle:** the oscillation gives a toe-down push that launches each hop, and a speed term leans the body forward when it is going too slowly. The docstring explains every gain. I tuned the 18 gains with a black-box search (CMA-ES) across all five scenarios; no network or policy was trained. **What went wrong along the way:** - Every search that rewarded survival settled on the robot standing still (return about 1040). That was true for the contact-switching controller and for a plain linear feedback law. - Scoring distance only, with the oscillator structure, produced real hopping withi…

Controller family: CPG / clocked oscillator

A free-running phase oscillator generates sinusoidal hip, knee, and ankle commands with tuned offsets, amplitudes, and phase shifts, plus pitch PD feedback on the hip and knee and speed feedback on the ankle, with gains found by CMA-ES.

Shown: public seed 0 (standard_v0); this episode fails the brisk rung. This episode: episode_steps 1000, x_final 8.51, min_height 1.08, max_abs_pitch 0.192, n_flights 19, total_reward 2062. It was the first failing one of 1 public seeds tried.

controller.py · agent transcript · trial hopper-brisk-robust__kebbpnN