← all tasks

humanoid-robust

Humanoid (MuJoCo Humanoid-v5) · solved 6 of 9 agent attempts pass

The task. The humanoid forward gate plus the four physics perturbations as a sibling suite, each gated at 0.9.

more

Opus 5.5's forward design (the humanoid-forward oracle) survives the heavier torso, weak motors and stronger gravity but falls on 27/40 seeds with -30 % friction; re-tuned by the repository tuner for all four perturbations at once (robust_v3) it is alive on >= 95 % of seeds under each and clears the suite (held-out 0.975) at 0.25 m/s: robustness bought with speed, the slick floor the hard case (forward 0.875 on 40 non-task seeds). The same structure tuned for speed (fast_v1, 1.04 m/s) falls under every perturbation. See docs/humanoid_fast_2026-10-24.md.

Gate. per-seed success is the `forward` rung of this benchmark's ladder ['alive', 'forward', 'brisk', 'fast'] (`--success-mode forward`), not the evaluator's default flag; the task is solved when the success rate over the grader's held-out seeds is `>= 0.9`; the same gate must also hold on the sibling scenario(s) `heavy_torso_20`, `low_friction_30`, `weak_motors_20`, `gravity_high_10` (`--scenarios`)

Axes: plant:humanoid, scenario:standard_v0+physics-suite, gate:forward_3m, robustness:torso-friction-motors-gravity

Each video re-runs the attempt's final controller through the task's own evaluator on a public seed (never the grader's held-out ones). The table under each video is the official grade on the held-out seeds.

Best attemptOpus 5.5 (claude-code, 15 min) · standard_v0

Graded on the held-out seeds: passes every gate on the grader's held-out seeds.

criteriongradedneeded
base:forward1.000≥ 0.900
scenario:heavy_torso_201.000≥ 0.900
scenario:low_friction_301.000≥ 0.900
scenario:weak_motors_201.000≥ 0.900
scenario:gravity_high_101.000≥ 0.900

How the agent got there: 15 min · 28 tool calls · 1 edit-tool edits · 19 runs of its own Python (tests, experiments, rewrites) · no tuner runs · official verifier 2× (success rates in order: 1.00, 1.00).

The agent's final message

The walking controller is done and passes the gate on every scenario I tested. On 120 seeds per scenario, none of them used for tuning, all 600 episodes stayed upright for the full 15 s and every one passed the 3 m gate. | scenario | seeds 30–149 | shortest distance (3 m needed) | |---|---|---| | standard_v0 | 120/120 | 9.25 m | | heavy_torso_20 | 120/120 | 8.92 m | | low_friction_30 | 120/120 | 10.31 m | | weak_motors_20 | 120/120 | 9.48 m | | gravity_high_10 | 120/120 | 9.34 m | The official evaluator on seeds 0–4 gives success 1.0 on all five scenarios. It also reaches `brisk` (≥ 8 m) every time, though not `fast` (15 m). **How it works** (`/app/work/controller.py`): - **Start:** both legs on the ground for 0.3 s while the robot settles. - **Stepping:** the legs swap every 0.105 s, a fixed rhythm that doesn't wait for foot contact. The result is a quick shuffle at about 0.6–0.7 m/s. - **Foot placement:** the swing leg is aimed using how far the body is from the standing foot and how fast it is moving. This is what keeps it balanced forwards and sideways. - **Torso:** the standing leg's hip holds the torso slightly upright and pointed straight ahead. - **Knees:** nearly straight on the standing leg, bent to lift the swing foot. - **Abdomen and arms:** held stiffly in a neutral position. - **Weaker motors:** commands are scaled by each motor's actual strength, which offsets the weaker motors up to the ±0.4 limit. The hand-set version stepped every 0.36 s and fell within …

Controller family: SIMBICON

A timed two-state SIMBICON machine swaps stance leg every ~0.105 s, placing the swing hip at world-frame angles from COM offset and velocity feedback, while the stance hip applies torso pitch/roll/yaw PD torque and the knees and abdomen are held by PD targets.

Shown: public seed 0 (standard_v0); this episode passes the forward rung. This episode: episode_steps 1000, x_final 9.49, min_height 1.27, max_abs_pitch 0.0877, n_steps 124, total_reward 5574.

controller.py · agent transcript · trial humanoid-robust__LggMRom

Best attemptOpus 5.5 (claude-code, 15 min) · heavy_torso_20

Graded on the held-out seeds: passes every gate on the grader's held-out seeds.

criteriongradedneeded
base:forward1.000≥ 0.900
scenario:heavy_torso_201.000≥ 0.900
scenario:low_friction_301.000≥ 0.900
scenario:weak_motors_201.000≥ 0.900
scenario:gravity_high_101.000≥ 0.900

How the agent got there: 15 min · 28 tool calls · 1 edit-tool edits · 19 runs of its own Python (tests, experiments, rewrites) · no tuner runs · official verifier 2× (success rates in order: 1.00, 1.00).

The agent's final message

The walking controller is done and passes the gate on every scenario I tested. On 120 seeds per scenario, none of them used for tuning, all 600 episodes stayed upright for the full 15 s and every one passed the 3 m gate. | scenario | seeds 30–149 | shortest distance (3 m needed) | |---|---|---| | standard_v0 | 120/120 | 9.25 m | | heavy_torso_20 | 120/120 | 8.92 m | | low_friction_30 | 120/120 | 10.31 m | | weak_motors_20 | 120/120 | 9.48 m | | gravity_high_10 | 120/120 | 9.34 m | The official evaluator on seeds 0–4 gives success 1.0 on all five scenarios. It also reaches `brisk` (≥ 8 m) every time, though not `fast` (15 m). **How it works** (`/app/work/controller.py`): - **Start:** both legs on the ground for 0.3 s while the robot settles. - **Stepping:** the legs swap every 0.105 s, a fixed rhythm that doesn't wait for foot contact. The result is a quick shuffle at about 0.6–0.7 m/s. - **Foot placement:** the swing leg is aimed using how far the body is from the standing foot and how fast it is moving. This is what keeps it balanced forwards and sideways. - **Torso:** the standing leg's hip holds the torso slightly upright and pointed straight ahead. - **Knees:** nearly straight on the standing leg, bent to lift the swing foot. - **Abdomen and arms:** held stiffly in a neutral position. - **Weaker motors:** commands are scaled by each motor's actual strength, which offsets the weaker motors up to the ±0.4 limit. The hand-set version stepped every 0.36 s and fell within …

Controller family: SIMBICON

A timed two-state SIMBICON machine swaps stance leg every ~0.105 s, placing the swing hip at world-frame angles from COM offset and velocity feedback, while the stance hip applies torso pitch/roll/yaw PD torque and the knees and abdomen are held by PD targets.

Shown: public seed 0 (heavy_torso_20); this episode passes the forward rung. This episode: episode_steps 1000, x_final 9.18, min_height 1.26, max_abs_pitch 0.0943, n_steps 125, total_reward 5549.

controller.py · agent transcript · trial humanoid-robust__LggMRom

FailureSonnet 5.5 (claude-code, 55 min) · weak_motors_20

Graded on the held-out seeds: passes the nominal plant but misses the gate under weak_motors_20.

criteriongradedneeded
base:forward1.000≥ 0.900
scenario:heavy_torso_201.000≥ 0.900
scenario:low_friction_300.900≥ 0.900
scenario:weak_motors_200.875≥ 0.900
scenario:gravity_high_101.000≥ 0.900

How the agent got there: 55 min · 32 tool calls · 0 edit-tool edits · 14 runs of its own Python (tests, experiments, rewrites) · no tuner runs · official verifier 6× (success rates in order: 0.00, 0.90, 0.80, 1.00, 0.87, 1.00).

The agent's final message

`/app/work/controller.py` walks forward and stays up on most seeds. It does not clearly clear the 0.9 gate, so the task may not count as solved. I tested it on 15 seeds (10–24) that the parameter search never used: | Scenario | `forward` success rate | |---|---| | `standard_v0` | 1.00 | | `heavy_torso_20` | 1.00 | | `low_friction_30` | 1.00 | | `weak_motors_20` | 0.80 | | `gravity_high_10` | 1.00 | `weak_motors_20` is below the 0.9 gate on this sample. With 15 seeds a single failure moves a rate by about 0.07, so the true rate could land on either side of 0.9. I also tried a later search candidate. It was no better: the weak-motors rate rose to 0.93 but `gravity_high_10` fell to 0.80. I kept the first one. On the visible seeds the controller reaches about 6 m of forward progress per episode and returns around 5400 on `standard_v0` seed 0. It walks well past the 3 m `forward` rung, but I did not measure it against `brisk` (8 m) or `fast`. **How it works.** It is a hand-structured stance/swing state machine in the style of SIMBICON, with no learning. The docstring in `controller.py` explains each gain. - **State machine:** the stance leg alternates by timer or by swing-foot touchdown. - **Stance leg:** it keeps the torso upright by feeding torso pitch and roll back to the stance hip torque. - **Swing leg:** the swing foot is placed from the centre-of-mass offset and velocity error. - **Other joints:** the abdomen and hip yaw hold heading, and the arms are held in a fixed …

Controller family: SIMBICON

A foot-contact-timed two-state machine alternates stance and swing legs. The stance hip balances the torso, the swing hip places the foot from COM offset and velocity error, and all joints are PD servos with clipped, gear-scaled torques.

Shown: public seed 4 (weak_motors_20); this episode fails the forward rung. This episode: episode_steps 387, x_final 1.94, min_height 0.999, max_abs_pitch 0.296, n_steps 32, total_reward 2027. It was the first failing one of 5 public seeds tried.

controller.py · agent transcript · trial humanoid-robust__ip9hfqS

FailureFable 5.1 (claude-code, 40 min) · standard_v0

Graded on the held-out seeds: fails the main gate badly (success rate below 0.5).

criteriongradedneeded
base:forward0.000≥ 0.900
scenario:heavy_torso_200.000≥ 0.900
scenario:low_friction_300.000≥ 0.900
scenario:weak_motors_200.000≥ 0.900
scenario:gravity_high_100.000≥ 0.900

How the agent got there: 40 min · 107 tool calls · 1 edit-tool edits · 86 runs of its own Python (tests, experiments, rewrites) · repository tuner 11× · official verifier 5× (success rates in order: 0.00, 0.00, 0.00, 0.00, 0.00).

The agent's final message

The 60-minute budget is nearly spent (started 23:33, now 00:12 with the last tuning and variant rounds showing only noise-level changes), so I'm stopping here with an honest report. **Outcome: the submission does not pass the gate.** The final controller at `/app/work/controller.py` keeps the humanoid stepping for about 4 s on average and up to 9 s, but never the full 15 s, so the `forward` rung is 0/5 on every scenario. Final evaluator numbers on seeds 0 to 4, standard plant: | metric | value | |---|---| | alive / forward rate | 0 / 0 | | mean episode length | 274 steps (4.1 s) | | mean mass-centre progress | 1.3 m | | flight fraction | 10 % | The four sibling scenarios also fail 5/5. **What the controller is.** A hand-designed Raibert-style stepping controller, no learning. A step clock alternates stance and swing legs. The stance leg holds the pelvis attitude through its hip torques and supports the weight with a gravity feed-forward plus a soft knee PD. The swing leg is placed by a capture-point law relative to the mass centre, computed in the body's heading frame, with the swing hip position estimated from the stance foot and leg forward kinematics and converted to hip angles by a knee-compensated virtual-leg inverse kinematics. Twelve gains were searched with the tune tool over ten rounds. The docstring documents every constant. **Findings that mattered, in order of impact.** - The feet are spheres with no ankle, so standing is impossible and balance comes only fr…

Controller family: Raibert / foot placement

A clocked, contact-aware stepping controller alternates stance and swing legs: the stance leg acts as a strut while hip torques regulate pelvis attitude, and the swing leg lifts and places the foot relative to the center of mass using a capture-point law converted to hip angles by virtual-leg IK.

Shown: public seed 0 (standard_v0); this episode fails the forward rung. This episode: episode_steps 148, x_final 1.13, min_height 0.984, max_abs_pitch 0.668, n_steps 9, total_reward 804. It was the first failing one of 1 public seeds tried.

controller.py · agent transcript · trial humanoid-robust__xr9LRa5