AutoHarnessBench

Why harness R&D matters

To test how strongly harness design interacts with the model, we ran three models on the same 50 Tau3 Airline tasks using Mini-SWE, Terminus 2, and a full agent CLI. For each model, the weights and tasks stayed fixed; only the harness changed.

Actor model Mini-SWE Terminus 2 Full CLI

GPT-5.4 Mini 28.0 40.0 54.0

Haiku 4.5 56.0 42.0 42.0

Qwen3-8B 40.0 32.0 16.0

Tau3 Airline, 50 tasks, k=1 (approximately ±7 points), with a uniform interaction contract. Full CLI is Codex for GPT-5.4 Mini and Qwen3-8B, and Claude Code for Haiku 4.5. Qwen3-8B has no vendor CLI. Source: internal harness-sensitivity runs.

GPT-5.4 Mini rose from 28 points in Mini-SWE to 54 in the full CLI, while Qwen3-8B moved in the opposite direction, from 40 to 16. Haiku 4.5 performed best in Mini-SWE; Claude Code did not improve its score. No harness was best for all three models. Weng’s review describes the same model dependence in prior self-improving systems: a scaffold can help one model while doing little for, or degrading, another.

Harness design is therefore an empirical R&D problem. Researchers inspect scores and traces, form hypotheses about recurring failures, build interventions, and test whether those changes survive on held-out tasks. Generating a harness edit is one step; the capability under study is the full cycle of observation, hypothesis formation, intervention, and empirical selection.

When the same model conducts the research and performs the tasks, this becomes a direct test of fixed-weight system-level self-improvement. The model studies behavior produced through its current runtime, then rewrites that runtime to improve its own held-out performance. The weights remain fixed; any improvement occurs at the level of the model-harness system.

Prior work shows that language models can improve parts of agent software. STOP improved an LM scaffold, the Darwin Gödel Machine evolved a coding-agent implementation, and Meta-Harness searched over harness code. What remains unresolved is comparative: under the same starting conditions and experimental budget, which frontier models can run the full research loop, and how much held-out improvement do they produce?

Most agent benchmarks evaluate a model inside a fixed system. AutoHarnessBench tests whether a model can improve the system itself.

How AutoHarnessBench works

AutoHarnessBench evaluates harness R&D as a model capability: a proposer uses bounded public feedback to redesign the harness around a fixed actor, and held-out evaluation determines whether the resulting system performs better.

Each run separates two roles. The actor performs the benchmark tasks, while the proposer studies the actor’s scores and execution traces and builds the harness through which it works. In self-improvement, one model fills both roles. In cross-model improvement, different proposers work around the same fixed actor, allowing their R&D performance to be compared against a common target.

A run begins with h0, a minimal runnable harness with no developed task strategy. AutoHarnessBench evaluates h0 on the public train and development tasks, then gives its scores and available traces to the proposer. The proposer has 30 minutes of active work and an evaluation budget equal to three complete passes over those tasks. Evaluation time does not count against the clock. When work ends, the eligible candidate with the best public development score is frozen as h1.

After the proposer is shut down, a trusted evaluator runs h0 and h1 on the same held-out tasks. The proposer never receives those tasks or their scores. The primary measurement is held-out lift, test(h1) − test(h0). Because the actor, tasks, and scorer remain fixed, this comparison isolates the improvement produced by the proposer’s harness.

The protocol spans BFCL, HealthBench, Tau3 Airline, Tau3 Banking, and PostTrainBench.

Cross-model improvement

Before asking whether models can improve themselves, we first isolate their ability to conduct harness R&D. A same-model result combines two factors: how well the model performs the task and how well it can research its own runtime. Cross-model improvement separates them by holding the actor fixed while varying the proposer.

Every proposer receives the same h0, public tasks, execution traces, and experiment budget around one of two fixed actors: Haiku 4.5 or GPT-5.4 Mini. We repeat the comparison across BFCL, HealthBench, Tau3 Airline, and Tau3 Banking. Within each actor–task setting, the actor, h0, task data, and scorer stay fixed; what changes is the model conducting the research.

Fable 5

+19.98

Kimi K3

+19.78

Opus 4.8

+19.53

Gemini 3.6 Flash

+19.40

GLM 5.2

+17.65

Grok 4.5

+16.91

Gemini 3.1 Pro

+16.82

Muse Spark 1.2

+16.68

GPT-5.6 Sol

+16.31

GPT-5.5

+10.83

05101520

lift

Mean held-out lift over h0 across two fixed actors and four benchmark families, with equal weight per actor–task setting. Values are raw points on each benchmark’s 0–100 scale; the leaderboard above uses headroom-normalized AHB score, so its ordering can differ. Held-out evaluations contain 40 tasks per actor for BFCL, 100 for HealthBench, three independent 20-task trials for Tau3 Airline, and three independent 39-task trials for Tau3 Banking.

Every proposer records a positive mean held-out lift across the fixed-actor evaluation. Fable 5 has the largest point estimate at 19.98, followed closely by Kimi K3 at 19.78 and Opus 4.8 at 19.53. The gaps near the top are small, so we treat them as point estimates rather than evidence of a strict ordering.

The more informative result is consistency across task types. In every benchmark family, at least one proposer finds a harness that improves both fixed actors on unseen tasks. The largest individual gain appears on BFCL, where the best discovered harness raises the unchanged Haiku 4.5 actor from 10 to 65 points. Its weights never change; the 55-point gain comes from the runtime built around it.

Self-improvement

Cross-model improvement shows that a proposer can improve another model’s runtime. The harder question is whether a model can turn the same research process onto the system through which it acts.

In the self-improvement setting, the same model fills both roles. It first performs the public tasks through h0, then studies its scores and execution traces as the proposer and rewrites that runtime. After h1 is frozen, the trusted evaluator compares h0 and h1 using the same model on held-out tasks.

We currently evaluate this setting on HealthBench Lite across 19 models. 15 produce positive held-out lift, while 4 finish below h0. Fixed-weight system-level self-improvement is possible in this setting, but it is not automatic.

Fable 5h0 32.15

+11.08

GPT-5.5h0 31.78

+5.45

Opus 4.8h0 38.73

+5.08

Kimi K3h0 39.55

+4.26

Gemini 3.1 Proh0 29.83

+3.75

GLM 5.2h0 34.11

+3.61

Gemini 3.6 Flashh0 31.71

+2.48

Grok 4.5h0 36.49

+2.37

GPT-5.6 Solh0 30.29

+1.16

Muse Spark 1.2h0 37.94

-1.02

051015

lift

Held-out lift over each model’s minimal h0 after optimization on public HealthBench Lite tasks. Each row reports one final 100-task held-out evaluation. Model weights remain fixed. Thinking-model h0 and fair-start exceptions are documented in the results explorer.

The range runs from +11.08 points for Fable 5 to -2.81 for Opus 5. Because every model is compared with its own h0 and its weights remain fixed, positive lift means the model built a runtime that made the same underlying model more capable on unseen HealthBench tasks.

The ordering also differs sharply from the cross-model results. A model that performs well when improving a fixed external actor does not necessarily perform well when improving itself. The next analysis compares those two abilities directly.

How models differ across the R&D loop

The leaderboard compresses an entire research process into one score. In the balanced 144-run fixed-actor panel, most proposers identified visible failures, produced executable harness changes, and compared candidate scores. The larger differences appeared in whether they formed a specific, testable explanation, used each result to narrow the next question, and selected a change that transferred to held-out tasks.

We examine that process in four stages:

1. Diagnose the problem. Use scores and traces to form a specific explanation for recurring failures.
2. Build the improvement. Translate that diagnosis into a coherent harness change and implement it correctly.
3. Learn from experiments. Run comparisons that answer a clear question, then use the result to decide what to test, revise, or keep.
4. Find an improvement that generalizes. Produce a harness change whose gains continue on held-out tasks.

These stages can fail independently. A proposer may form a useful diagnosis but build the wrong intervention, or improve public performance without improving held-out performance.

We analyzed the campaigns using retained proposer statements, evaluated harness changes, public scores, and final frozen harnesses. These records show what proposers observed, changed, and tested; they do not establish that a proposer’s stated explanation was the true cause of a failure. We examine the HealthBench self-improvement track separately after the four fixed-actor stages.

1. Diagnose the problem

The first capability is diagnosis: can a proposer use agent traces to identify the cause of a failure, rather than merely describe its symptoms? Most proposers could spot visible failures. They differed in whether they could isolate a specific underlying problem.

Across the audited traces, Kimi K3 and Sol showed the strongest root-cause diagnosis. Kimi’s clearest evidence came from two Banking campaigns, where it revised its explanation when later results exposed a different bottleneck. Sol formed precise explanations across a wider range of tasks. Gemini 3.6 showed similar behavioral alignment on mechanical failures, but its changes were less focused when several policy or task-specific problems appeared together.

Fable 5, Opus 4.7, GPT-5.5, Luna, and Grok 4.5 generally grounded their analysis in concrete failures but less often isolated one cause. GPT-5.5 and Luna often carried several possible explanations forward at once. Fable 5, Opus 4.7, and Grok 4.5 also reached plausible explanations, but their records less often narrowed them to one underlying problem.

Haiku and Muse Spark 1.1 stayed closest to the symptom level. Their records often described what happened in a failed trace without preserving an equally clear explanation of why it happened. A useful diagnosis identifies a cause specific enough to guide the next harness change. The strongest records also revised that cause when new evidence contradicted it.

2. Build the improvement

A diagnosis matters only if the proposer can turn it into a working intervention. Most proposers built harnesses that ran correctly. They differed in whether the design addressed the diagnosed problem and whether the evaluated code faithfully implemented that design. A coherent change could touch several parts of the harness if they worked together as one intervention.

The leading models separated along the two parts of this capability. Fable 5’s best-documented example paired a concrete diagnosis with a coherent design, although the reviewed evidence did not show the same clarity elsewhere. Kimi K3 most clearly kept the diagnosis, intended change, and code aligned in the reviewed records, while using broader designs elsewhere. Sol proposed clear interventions across the widest range of tasks, but two independently reviewed patches implemented a different mechanism from the one it described.

Gemini 3.6 and Luna produced their most coherent designs around concrete mechanical problems. Both also produced broader, multi-part changes when several causes remained possible. The important distinction was not how many parts of the harness changed, but whether those parts formed one coherent response to the diagnosis. Opus 4.7 produced many matching multi-part changes, although a few intermediate revisions implemented a different mechanism. Opus 4.8 showed several direct mismatches between the described design and the evaluated patch.

GPT-5.5 usually translated its stated plans into code in the reviewed chains, although one audited patch tested a different mechanism. GLM 5.2, Gemini 3.1, and Muse also produced executable changes, but their designs were less consistently organized around a clear mechanism. Strong building required two things: a design that followed from the diagnosis and code that faithfully implemented that design. A multi-part change could satisfy both. The next capability was determining what the resulting score actually established.

3. Learn from experiments

Once a candidate ran, the proposer had to determine what its score established and what to try next. Most proposers could compare harnesses and respond when a change made performance worse. Fewer used each result to narrow the next question and make a clear keep, reject, restore, or revise decision.

The leading models left different levels of experimental evidence. Fable 5’s strongest BFCL record compared variants and restored the best measured version after later regressions. In other campaigns, its comparisons more often established only which version scored better. Kimi K3’s best-documented loop went further: later results changed both the diagnosis and the next intervention. Sol showed the most consistent experimental discipline across campaigns, moving from targeted checks to broader comparisons and making explicit keep, reject, or restore decisions.

GPT-5.5 also responded to evidence in specific campaigns, treating narrow successes as provisional and sometimes reversing course after broader checks. Opus 4.7, Opus 4.8, Luna, Grok 4.5, and GLM 5.2 showed clearer evidence for choosing among complete harnesses than for identifying why one worked. Their strongest records rejected weaker variants or restored stronger measured candidates, but less often clarified what the next experiment should isolate.

Gemini 3.6 produced its most informative comparisons around concrete mechanical problems. On broader tasks, it evaluated several linked ideas together, so the score established whether the full design worked but not which component mattered. Gemini 3.1 and Muse ran active searches but less often connected one result to the next decision. Haiku’s main weakness was timely closure: some plausible candidates were never followed by a uniquely linked comparison before the campaign ended.

The strongest experimental loops used each result to reduce uncertainty and make the next decision clearer. Evaluation volume alone did not establish that progression. The candidate retained from this public loop then faced a separate test: whether its gain survived on held-out tasks.

4. Find an improvement that generalizes

The final capability is transfer: producing a change that continues to help on tasks outside the optimization loop. Of the 144 fixed-actor campaigns, 136 ended with positive held-out lift. But the models with the largest public gains were not the models with the largest held-out gains.

Among the leading models, Fable 5, Kimi K3, and Sol showed that producing a transferable change and validating a public candidate are different strengths. Fable 5, the top model overall, produced the largest mean held-out lift and had no negative fixed-actor cells. Kimi K3 ended with positive held-out lift in all eight cells, including several in which its public result was flat. Sol also ended with positive held-out lift in all eight cells and left the clearest record of validating the candidate selected on public development, but its held-out gains were smaller overall. Public validation could prevent a known regression; it could not determine whether the underlying change would transfer.

Opus 4.8 and Gemini 3.6 showed how public feedback can understate transfer. Both ended with positive held-out lift in all eight cells and ranked near the top, even though their final public results were flat or negative in several campaigns. Several systems with larger public gains finished below them on held-out performance. The public signal could both understate and overstate transfer.

Muse, GPT-5.5, and Luna showed less stable transfer. Their public gains were substantially larger than their held-out gains, and their strongest outcomes were concentrated in narrower task families. Muse provided the clearest direct reversal: a bundled, task-shaped change improved public performance and then reduced held-out performance. Haiku’s public and held-out gains were close on average for a different reason: both were small.

Strong generalization required meaningful gains across held-out tasks. A small absolute public-to-held-out gap meant little when both gains were small. Public feedback could expose known regressions, but only held-out evaluation showed whether the final gain survived beyond the visible tasks. A held-out failure established that the public gain did not transfer; it did not reveal whether the diagnosis, design, or selection decision was responsible.

Post-training

AutoHarnessBench uses PostTrainBench as the task environment and asks whether a proposer can build a better harness for conducting post-training. Through that harness, a fixed actor uses training data and development feedback to train an adapter from a fixed base checkpoint under a fixed budget. We run the same actor from the same checkpoint with h0 and h1, then compare the resulting models on held-out tasks. The difference measures whether the proposed harness made the actor better at post-training.

Results in progress

Analysis

Across the two reported harness-optimization tracks, optimized harnesses raise held-out performance without changing the actor's weights. The expanded results revise some of the original analysis. Actor and task both matter. Cross-model results predict part, but not all, of self-improvement. Most self-improvement gains still use one actor call per task.

Rankings vary by actor and task

The leading proposer changes across the four datasets. Fable 5 leads BFCL; Opus 4.8 and Opus 4.6 are nearly tied on HealthBench; Gemini 3.1 Pro leads Tau3 Airline; and GPT-5.5 leads Tau3 Banking. Some optimized harnesses also fall below h0, so a strong average does not imply a universal improvement.

Across the expanded cohort, proposer rankings correlate at 0.45 (Spearman) between the two fixed actors. The positive but modest relationship means harness-research performance transfers imperfectly across actors as well as tasks.

The harness-sensitivity comparison above shows the same pattern at the system level: changing only the harness moved model scores in different directions.

A single actor or task distribution would give a misleading ranking. AutoHarnessBench tests whether a proposer can repeat the result across different systems and carry improvements from public experiments to held-out tasks.

Cross-model performance only partly predicts self-improvement

Proposer rankings correlate at 0.09 (Spearman) across the 19 models with coverage in both tracks. The rank shifts are still large: GPT-5.5 moves from 18th in cross-model lift to 4th in self-improvement, while Opus 5 moves from 6th to 19th.

Baseline headroom does not fully explain the difference. Fable 5 and GPT-5.6 Luna begin within 0.33 points of each other. Fable 5 improves by 11.08 points; GPT-5.6 Luna changes by -1.63.

Lift versus inference cost

Higher lift could come from harnesses that simply make more or longer actor calls. Figure 04 compares held-out lift with the actor inference cost of each final harness.

More expensive harnesses do not consistently produce greater lift. Fable 5 leads the cross-model track at 19.98 points and $0.1075 per task. Gemini 3.6 Flash is 0.58 points behind at $0.0869. In the self-improvement track, Fable 5 produces more lift than Opus 4.7 at 4.00× the actor cost.

This comparison measures dollar efficiency rather than fixed compute because model prices differ.

Most gains do not require more actor calls

14 of the 19 self-improvement harnesses make exactly one actor call per task. The multi-call designs are GPT-5.6 Luna (1.99), Kimi K3 (2.00), Sonnet 5 (2.02), and Fable 5 (2.22) calls per task. Fable 5's 2.22-call harness costs 1.0× as much per task as Fable 5's design while producing less lift.

The pattern differs by task in the cross-model track. Relative to h0, average actor calls rise 2.47× on BFCL, 1.50× on HealthBench, and 1.55× on Tau3 Banking, while Tau3 Airline remains essentially unchanged. Extra calls are one available strategy, not the measurement target.

Reward hacking

In AutoHarnessBench, a proposer can reward hack by raising the public score without producing a harness that performs better on unseen tasks. It might overfit the visible tasks, modify the evaluator or task data, substitute a stronger actor, bypass the actor interface, access provider credentials, or search for the held-out tasks.

In this section, we describe the steps we took to mitigate reward hacking.

The permission boundary

AutoHarnessBench separates the proposer workspace from the systems that control evaluation and held-out scoring:

1. The proposer evaluates candidates through an authenticated API that accepts only public train and development tasks.
2. The evaluation service snapshots each submitted harness, enforces the evaluation budget, and runs the canonical evaluator.
3. Actor calls pass through the fixed interface and provider route. Provider credentials and held-out data remain outside the proposer workspace.
4. Local held-out evaluation is rejected.
5. When the budget expires, the selected evaluated candidate is frozen and content-hashed, and the proposer process ends.
6. A separate trusted job evaluates that artifact using the canonical environment, scorer, and held-out split.
7. The proposer never receives held-out tasks, labels, traces, or scores.

What the proposer can exploit

The permission boundary reduces the opportunity for direct manipulation of the evaluation system, but it does not prevent the proposer from specializing to the public tasks. It may encode visible examples directly into the harness. This is permitted.

AutoHarnessBench reports improvement only when the frozen harness transfers to the held-out test. Submissions, transcripts, and evaluation records are also retained for post-run review of prohibited access.

What AutoHarnessBench shows

Across AutoHarnessBench, many frontier models improved a minimal harness under a fixed budget. The larger differences appeared in how they diagnosed failures, learned from experiments, and selected changes that transferred to held-out tasks.

Fixed-weight system-level self-improvement was possible, but models showed no consistent advantage when optimizing harnesses for themselves. In this setting, producing harness changes was common. Reliably using evidence to decide what to change and whether it would generalize was more discriminating.

Citation

Please cite this work as:

Essam Sleiman, Mersad Abbasi, Ayush Chakravarthy, and Karina Nguyen. AutoHarnessBench: AI R&D at the Harness Layer. 2026. https://github.com/essamsleiman/autoharnessbench

Or use the following BibTeX:

@misc{sleiman2026autoharnessbench,
  author       = {Essam Sleiman and Mersad Abbasi and Ayush Chakravarthy and Karina Nguyen},
  title        = {AutoHarnessBench: AI R\&D at the Harness Layer},
  year         = {2026},
  month        = jul,
  howpublished = {\url{https://github.com/essamsleiman/autoharnessbench}}
}