When Reward Chasing Meets Robustness

In the previous post, I described a small heuristic-learning control harness: let an LLM inspect experiments, read trajectory diagnostics, edit controller code, tune exposed parameters, and keep a compact “whiteboard” between attempts.

However, the LunarLander result there was incomplete. The harness had found a small handwritten controller that could land reliably, but it was still behind a pretrained SB3 PPO policy on Gym reward. A follow-up question would be:

Can the same harness push a non-neural LunarLander controller toward PPO-level reward, and if it can, how about robustness of the controller?

Short answer: yes, it can chase nominal reward surprisingly well. But when I moved robustness into the loop, the story became more interesting: the harness recovered much of the lost robustness, then exposed a very specific remaining failure mode.

Starting point

The parent controller was candidate_0009 from the previous post . It was already a real controller, not a toy. On the original run it reached terminal landing success with zero crashes:

That was a good control result, but not yet PPO-level behavior. The PPO reference I used, from the Stable-Baselines3 LunarLanderContinuous checkpoint, scored about 271.30 mean reward on the fixed standard-v0 seed set.

Visually, candidate_0009 landed, but it was conservative and slow:

The PPO reference was cleaner and faster:

Adding a performance mode

The harness previously had two success modes:

For this stage, I added a third mode:

For the LunarLander refinement, the frontend configuration was:

Env: lunar_lander
Scenario: standard_v0
Resume candidate: candidate_0009, then later candidate_0007
Success mode: performance
Reference score metric: total_reward_mean
Reference score target: 271.3
Reference score tolerance: 5.0
Reference label: sb3_ppo_lunar_lander_continuous_v2
Tuning trials: 120
Tuning workers: 8

The important part is that PPO was not used as a teacher policy. The harness only saw a scalar reference target: “get close to this reward while preserving terminal success and no crashes.” The controller remained a small Python function.

The whiteboard guide for the next attempt was roughly:

Preserve:
- terminal_success_rate = 1.0
- crash_rate = 0.0
- clean final landing geometry

Improve:
- total_reward_mean toward the PPO reference
- reduce hover time
- reduce fuel use and side-engine chatter
- reduce final lateral drift and touchdown speed

Avoid:
- redesigning the controller family from scratch
- accepting higher reward at the cost of crashes

This is the useful part of the harness design: the next attempt is not just “maximize reward.” It is “hold the behavioral contract, then improve reward.”

The controller that passed

The accepted controller was candidate_0018.

On the synthesis seeds it passed the performance gate:

On the holdout seeds it still passed:

Because we have the gate as 271.3 - 5.0, this controller got accepted. The controller cleared it, barely but legitimately.

The behavior looks a little shaking. The controller uses high-gain lateral and attitude correction, plus a relatively hard side-engine deadzone:

CANDIDATE_0018_PARAMS = {
    "k_ang_vel": 24.0,
    "k_angle": 22.0,
    "k_vy": 45.0,
    "k_x": 2.0,
    "k_x_dot": 50.0,
    "main_power_scale": 0.4,
    "side_deadzone": 0.7,
    "side_min_mag": 0.75,
    "target_vy_descend": -2.5,
    "target_vy_soft": -0.08,
}

In other words, it is:

1. Descend quickly when high above the pad.
2. Switch to a soft vertical target near the ground.
3. Use a PD-like main engine throttle for vertical speed.
4. Use lateral position, lateral velocity, angle, and angular velocity to fire side thrusters.
5. Reduce side thrust after one leg touches, and stop after stable two-leg contact.

That is an inspectable controller. It is also a clue about the later failure: the policy is thresholded, high-gain, and tuned to a narrow landing envelope.

The robustness test

After that, I froze both controllers:

• candidate_0018 fixed, no more LLM edits.
• SB3 PPO fixed, no fine-tuning.
• zero controller included as a sanity baseline.

Then I evaluated all controllers on the same ten seeds, across a small perturbation suite:

standard_v0
gravity_low_10
gravity_high_10
gravity_low_20
gravity_high_20
main_weak_10
main_weak_20
main_strong_10
main_strong_20
side_weak_20
side_strong_20
heavy_weak_main_v0
light_strong_main_v0
weak_attitude_control_v0

These perturbations are simple: change gravity, main engine power, side engine power, or combine heavier effective dynamics with a weaker main engine. The controller does not observe these perturbation values. It only sees the standard LunarLander observation vector.

The result was not close:

The robustness score below is:

mean terminal success - 0.25 * crash rate - 0.10 * nominal regression penalty

Some representative rows:

There is an uncomfortable detail here: on the larger frozen seed set, candidate_0018 even regressed on standard_v0, with terminal success 0.8 and mean reward 146.25. That does not contradict the performance gate above. But rather says the gate was too small. The synthesis holdout was enough to catch many bad controllers, but not enough to certify broad seed robustness.

This is the kind of thing a harness should make visible.

What changed

The performance pass was real. The robustness failure was also real.

My reading after candidate_0018 was:

1. candidate_0009 from previous blog was safer and slower.
2. candidate_0018 learned to spend reward budget more aggressively.
3. The reward improvement came from faster descent and stronger attitude/lateral correction.
4. That made the controller narrower.

The wobbly GIF is the visual symptom of the same thing. The controller is actively trading angle, lateral velocity, and main-engine usage around a tuned descent schedule. It can look efficient on the sampled seeds and still be fragile when the effective physics change.

The spec ladder is easier to see if the three key controllers are put side by side:

For me, this is the warning sign: high-gain switching logic often gives crisp nominal behavior before it gives robust behavior.

Putting robustness into the loop

So I added another success mode:

This was intentionally not a full robust-control certificate. It was a cheap smoke test inside the harness loop:

Nominal:
- terminal_success_rate must stay high
- crash_rate must stay low
- total_reward_mean must stay within 5 points of the PPO reference

Robust smoke suite:
- standard_v0
- gravity_high_20
- main_weak_20
- heavy_weak_main_v0

Robust gates:
- average terminal success >= 0.8
- worst-scenario terminal success >= 0.5
- average crash rate <= 0.2

I also made the whiteboard more explicit. Instead of saying “be more robust”, it now summarized the parent candidates by role:

Best nominal parent:
- high reward on standard LunarLander

Best robust parent:
- keeps success across the smoke perturbation suite

Failure mode to fix:
- heavy_weak_main_v0 collapse

Do not:
- hard-code scenario names
- branch on unobserved perturbation values
- trade away the nominal reward just to hover forever

The best result in this branch was not the highest-reward controller. It was candidate_0008, which was more conservative than candidate_0018 but kept the robustness gates alive.

On the main run:

The candidate search curve is the most useful visual from this phase. The top panel shows nominal reward climbing toward the PPO reference. The bottom panel shows why reward alone is not enough: several later candidates reached high nominal reward while the robustness probe collapsed.

This is the picture I would use to explain the harness loop. It is not a smooth gradient descent curve, but it is an optimization trace in code space: each point is a candidate controller, and the whiteboard compresses the previous failures into the next edit request.

This did not beat PPO’s nominal reward. But the gap was small enough to matter differently: it was now a question of tradeoffs, not failure. The controller landed reliably, remained interpretable, and behaved well under the smoke perturbations.

Frozen comparison, again

Then I repeated the frozen full robustness comparison, now with candidate_0008:

candidate_0008 fixed, no more LLM edits
SB3 PPO fixed, no fine-tuning
zero controller as sanity baseline
10 seeds: 0 1 2 3 4 5 6 7 8 9
suite: standard + gravity + engine + combo

The headline is more nuanced than the candidate_0018 failure. PPO still wins on average, but not by a lot.

Some representative rows:

The same data is also useful as a small Pareto-style frontier. Here the x-axis is nominal reward and the y-axis is worst-case terminal success over the smoke robustness probe.

That plot is the compact version of the story: candidate_0018 chased reward and lost robustness, while candidate_0008 gave up some nominal reward to recover the robustness gate. The PPO checkpoint is still the stronger full-suite baseline, but the handwritten controller is no longer obviously out of the game.

The full-suite mean reward gap is about 6.86 reward points, or roughly 2.6% of PPO’s mean. That is smaller than I expected for a compact handwritten controller. PPO is still more efficient on most rows, especially the nominal and actuator-weak cases, but candidate_0008 has a different robustness profile.

The most interesting row is the stress case:

candidate_0008 / heavy_weak_main_v0: terminal=0.90, reward=245.38
SB3 PPO        / heavy_weak_main_v0: terminal=0.80, reward=221.47

That is an important scenario: heavier effective dynamics plus weaker main-engine authority. Under that perturbation, the small controller is better than the PPO checkpoint on both terminal success and reward.

The price is also clear: candidate_0008 burns more fuel and lands more conservatively. In the matrix, its fuel-per-step is usually higher than PPO. It is less elegant, less efficient, and more cautious. But it is not a failed controller. It is a small inspectable controller with a slightly lower average reward and one very interesting robust-control-shaped win.

Why this matters

This experiment clarified a distinction I had been blurring:

Those are not the same target.

The progression is the useful part:

That last row is the actual research direction. The harness is no longer just asking the LLM to write a clever controller. It is asking the LLM to move through a sequence of specifications:

land -> land efficiently -> land efficiently under uncertainty

This starts to look like a lightweight, program-search version of robust control. It is not H-infinity synthesis, tube MPC, or a formal min-max controller. But the shape is familiar: define an uncertainty set, preserve nominal performance, and improve worst-case behavior without using the hidden disturbance as an observation.

What this does not prove yet

There is one claim I do not want to overstate. The plots above show that the full harness can move through a useful sequence of controller specifications. They do not yet prove, by themselves, that the whiteboard is the causal reason.

Historically, the early LunarLander attempts before the environment digest, trajectory diagnostics, and robust whiteboard were much harder to steer. That is useful evidence from the lab notebook, but it is not a controlled ablation: the prompt, diagnostics, objective, and harness code were all changing at the same time.

The controlled ablation I would run next is straightforward:

The figure to report would be the same code-search curve above, repeated for each condition: candidate iteration on the x-axis, reward and robustness on the y-axis. If the full-whiteboard curve reaches the robust-performance gate in fewer candidates, then the whiteboard stops being just a nice engineering convenience and becomes load-bearing evidence.