AutoHarnessBench

Harnesses are a test bed for capability development

We ran three models on the same 50 Tau3 Airline tasks using Mini-SWE, Terminus 2, and a full agent CLI. For each model, the weights and tasks stayed fixed; only the harness changed.

Actor model Mini-SWE Terminus 2 Full CLI

GPT-5.4 Mini 28.0 40.0 54.0

Haiku 4.5 56.0 42.0 42.0

Qwen3-8B 40.0 32.0 16.0

Tau3 Airline, 50 tasks, k=1 (approximately ±7 points), with a uniform interaction contract. Full CLI is Codex for GPT-5.4 Mini and Qwen3-8B, and Claude Code for Haiku 4.5. Qwen3-8B has no vendor CLI. Source: internal harness-sensitivity runs.

GPT-5.4 Mini rose from 28 points in Mini-SWE to 54 in the full CLI, while Qwen3-8B moved in the opposite direction, from 40 to 16. Haiku 4.5 performed best in Mini-SWE; Claude Code did not improve its score. No harness was best for all three models. Weng’s review describes the same model dependence in prior self-improving systems: a scaffold can help one model while doing little for, or degrading, another.

With model weights fixed, researchers can implement a candidate behavior in the harness, inspect where it helps or fails, and test whether the gain survives on held-out tasks. A gain that transfers shows that adding the behavior makes the system more capable even though the weights have not changed. This suggests a broader role for harness R&D as a capability-development test bed: behaviors validated at the system layer may become targets for internalization into model weights through methods such as post-training.

System-level self-improvement

Harness improvement is empirical R&D. Researchers decide which failures matter, form hypotheses about their causes, choose changes to test, and use each result to decide what to try next.

We ask whether a model can carry out the same process. Generating a harness edit is one step. The capability under study is the full research cycle: observation, hypothesis formation, intervention, and empirical selection.

When the same model conducts the research and performs the tasks, this becomes a direct test of fixed-weight system-level self-improvement. The model studies behavior produced through its current runtime, then rewrites that runtime to improve its own held-out performance. The weights remain fixed; any improvement occurs at the level of the model-harness system.

Prior work shows that language models can improve parts of agent software. STOP improved an LM scaffold, the Darwin Gödel Machine evolved a coding-agent implementation, and Meta-Harness searched over harness code. What remains unresolved is comparative: under the same starting conditions and experimental budget, which frontier models can run the full research loop, and how much held-out improvement do they produce?

Most agent benchmarks evaluate a model inside a fixed system. AutoHarnessBench tests whether a model can improve the system itself.

Introducing AutoHarnessBench

AutoHarnessBench evaluates harness R&D as a model capability: a proposer uses bounded public feedback to redesign the harness around a fixed actor, and held-out evaluation determines whether the resulting system performs better.

Each run separates two roles. The actor performs the benchmark tasks, while the proposer studies the actor’s scores and execution traces and builds the harness through which it works. In self-improvement, one model fills both roles. In cross-model improvement, different proposers work around the same fixed actor, allowing their R&D performance to be compared against a common target.

A run begins with h0, a minimal runnable harness with no developed task strategy. AutoHarnessBench evaluates h0 on the public train and development tasks, then gives its scores and available traces to the proposer. The proposer has 30 minutes of active work and an evaluation budget equal to three complete passes over those tasks. Evaluation time does not count against the clock. When work ends, the eligible candidate with the best public development score is frozen as h1.

After the proposer is shut down, a trusted evaluator runs h0 and h1 on the same held-out tasks. The proposer never receives those tasks or their scores. The primary measurement is held-out lift, test(h1) − test(h0). Because the actor, tasks, and scorer remain fixed, this comparison isolates the improvement produced by the proposer’s harness.

The protocol spans BFCL, HealthBench, Tau3 Airline, Tau3 Banking, and PostTrainBench.

Cross-model improvement

Cross-model improvement isolates harness-R&D capability: we hold the task-performing actor fixed, vary the proposer conducting the research, and measure the held-out gain produced by each proposer’s harness.

We evaluate proposers around Haiku 4.5 and GPT-5.4 Mini across BFCL, HealthBench, Tau3 Airline, and Tau3 Banking. Within each actor–task setting, every proposer receives the same h0, public tasks, execution traces, and experiment budget. The actor, h0, task data, and scorer remain fixed.

Fable 5

+19.98

Kimi K3

+19.78

Opus 4.8

+19.53

Gemini 3.6 Flash

+19.40

GLM 5.2

+17.65

Grok 4.5

+16.91

Gemini 3.1 Pro

+16.82

Muse Spark 1.2

+16.68

GPT-5.6 Sol

+16.31

GPT-5.5

+10.83

05101520

lift

Mean held-out lift over h0 across two fixed actors and four benchmark families, with equal weight per actor–task setting. Values are raw points on each benchmark’s 0–100 scale; the leaderboard above uses headroom-normalized AHB score, so its ordering can differ. Held-out evaluations contain 40 tasks per actor for BFCL, 100 for HealthBench, three independent 20-task trials for Tau3 Airline, and three independent 39-task trials for Tau3 Banking.

Every proposer shown has a positive mean held-out lift. Fable 5 has the largest point estimate at 19.98, followed by Kimi K3 at 19.78 and Opus 4.8 at 19.53.

Self-improvement

Self-improvement tests whether a model can improve the runtime through which it performs tasks.

The same model serves as actor and proposer. It performs the public HealthBench Lite tasks through h0, studies its scores and execution traces, and rewrites the harness. After h1 is frozen, a trusted evaluator compares h0 and h1 using the same model on held-out tasks.

Across 19 models, 15 produce positive held-out lift and 4 finish below h0. Fixed-weight system-level self-improvement is possible in this setting, but it is not automatic.

Fable 5h0 32.15

+11.08

GPT-5.5h0 31.78

+5.45

Opus 4.8h0 38.73

+5.08

Kimi K3h0 39.55

+4.26

Gemini 3.1 Proh0 29.83

+3.75

GLM 5.2h0 34.11

+3.61

Gemini 3.6 Flashh0 31.71

+2.48

Grok 4.5h0 36.49

+2.37

GPT-5.6 Solh0 30.29

+1.16

Muse Spark 1.2h0 37.94

-1.02

051015

lift

Held-out lift over each model’s minimal h0 after optimization on public HealthBench Lite tasks. Each row reports one final 100-task held-out evaluation. Model weights remain fixed. Thinking-model h0 and fair-start exceptions are documented in the results explorer.

Fable 5 has the largest gain at +11.08, while Opus 5 has the largest decline at -2.81. Every model is compared with its own h0 while its weights remain fixed, so the measured change comes from the harness around the model.

Post-training

AutoHarnessBench uses PostTrainBench as the task environment and asks whether a proposer can build a better harness for conducting post-training. Through that harness, a fixed actor uses training data and development feedback to train an adapter from a fixed base checkpoint under a fixed budget. We run the same actor from the same checkpoint with h0 and h1, then compare the resulting models on held-out tasks. The difference measures whether the proposed harness made the actor better at post-training.

Results in progress

How models conduct harness R&D

The leaderboard compresses an entire research process into one score. To understand what separates successful campaigns, we examined what proposers observed, changed, tested, and retained across 185 runs.

A successful campaign requires four capabilities:

1. Diagnose the problem. Use scores and traces to identify recurring failures and their underlying cause.
2. Build the improvement. Translate that diagnosis into a coherent harness change and implement it correctly.
3. Learn from experiments. Run comparisons that answer a clear question, then use the result to decide what to test, revise, or keep.
4. Find an improvement that generalizes. Produce a harness change whose gains continue on held-out tasks.

These capabilities can fail independently. A proposer may diagnose the right problem but build the wrong intervention, or improve public performance without improving held-out performance.

How models differ across the R&D loop

We analyzed each campaign using its retained proposer statements, evaluated harness changes, public scores, and final frozen harness. For cross-model comparisons, the first four sections use the balanced 144-run fixed-actor panel: eight actor–benchmark cells for each of 18 proposer configurations. We then examine the HealthBench self-improvement track separately and compare it with the cross-model results.

1. Diagnose the problem

The first capability is diagnosis: can a proposer use agent traces to identify the cause of a failure, rather than merely describe its symptoms? Most proposers could spot visible failures. They differed in whether they could isolate a specific underlying problem.

Across the audited traces, Kimi K3 and Sol showed the strongest root-cause diagnosis. Kimi’s clearest evidence came from two Banking campaigns, where it revised its explanation when later results exposed a different bottleneck. Sol formed precise explanations across a wider range of tasks. Gemini 3.6 showed similar behavioral alignment on mechanical failures, but its changes were less focused when several policy or task-specific problems appeared together.

Fable 5, Opus 4.7, GPT-5.5, Luna, and Grok 4.5 generally grounded their analysis in concrete failures but less often isolated one cause. GPT-5.5 and Luna often carried several possible explanations forward at once. Fable 5, Opus 4.7, and Grok 4.5 also reached plausible explanations, but their records less often narrowed them to one underlying problem.

Haiku and Muse Spark 1.1 stayed closest to the symptom level. Their records often described what happened in a failed trace without preserving an equally clear explanation of why it happened. A useful diagnosis identifies a cause specific enough to guide the next harness change. The strongest records also revised that cause when new evidence contradicted it.

2. Build the improvement

A diagnosis matters only if the proposer can turn it into a working intervention. Most proposers built harnesses that ran correctly. They differed in whether the design addressed the diagnosed problem and whether the evaluated code faithfully implemented that design. A coherent change could touch several parts of the harness if they worked together as one intervention.

The leading models separated along the two parts of this capability. Fable 5’s best-documented example paired a concrete diagnosis with a coherent design, although the reviewed evidence did not show the same clarity elsewhere. Kimi K3 most clearly kept the diagnosis, intended change, and code aligned in the reviewed records, while using broader designs elsewhere. Sol proposed clear interventions across the widest range of tasks, but two independently reviewed patches implemented a different mechanism from the one it described.

Gemini 3.6 and Luna produced their most coherent designs around concrete mechanical problems. Both also produced broader, multi-part changes when several causes remained possible. The important distinction was not how many parts of the harness changed, but whether those parts formed one coherent response to the diagnosis. Opus 4.7 produced many matching multi-part changes, although a few intermediate revisions implemented a different mechanism. Opus 4.8 showed several direct mismatches between the described design and the evaluated patch.

GPT-5.5 usually translated its stated plans into code in the reviewed chains, although one audited patch tested a different mechanism. GLM 5.2, Gemini 3.1, and Muse also produced executable changes, but their designs were less consistently organized around a clear mechanism. Strong building required two things: a design that followed from the diagnosis and code that faithfully implemented that design. A multi-part change could satisfy both. The next capability was determining what the resulting score actually established.

3. Learn from experiments

Once a candidate ran, the proposer had to determine what its score established and what to try next. Most proposers could compare harnesses and respond when a change made performance worse. Fewer used each result to narrow the next question and make a clear keep, reject, restore, or revise decision.

The leading models left different levels of experimental evidence. Fable 5’s strongest BFCL record compared variants and restored the best measured version after later regressions. In other campaigns, its comparisons more often established only which version scored better. Kimi K3’s best-documented loop went further: later results changed both the diagnosis and the next intervention. Sol showed the most consistent experimental discipline across campaigns, moving from targeted checks to broader comparisons and making explicit keep, reject, or restore decisions.

GPT-5.5 also responded to evidence in specific campaigns, treating narrow successes as provisional and sometimes reversing course after broader checks. Opus 4.7, Opus 4.8, Luna, Grok 4.5, and GLM 5.2 showed clearer evidence for choosing among complete harnesses than for identifying why one worked. Their strongest records rejected weaker variants or restored stronger measured candidates, but less often clarified what the next experiment should isolate.

Gemini 3.6 produced its most informative comparisons around concrete mechanical problems. On broader tasks, it evaluated several linked ideas together, so the score established whether the full design worked but not which component mattered. Gemini 3.1 and Muse ran active searches but less often connected one result to the next decision. Haiku’s main weakness was timely closure: some plausible candidates were never followed by a uniquely linked comparison before the campaign ended.

The strongest experimental loops used each result to reduce uncertainty and make the next decision clearer. Evaluation volume alone did not establish that progression. The candidate retained from this public loop then faced a separate test: whether its gain survived on held-out tasks.

4. Find an improvement that generalizes

The final capability is transfer: producing a change that continues to help on tasks outside the optimization loop. Of the 144 fixed-actor campaigns, 136 ended with positive held-out lift. But the models with the largest public gains were not the models with the largest held-out gains.

Among the leading models, Fable 5, Kimi K3, and Sol showed that producing a transferable change and validating a public candidate are different strengths. Fable 5, the top model overall, produced the largest mean held-out lift and had no negative fixed-actor cells. Kimi K3 ended with positive held-out lift in all eight cells, including several in which its public result was flat. Sol also ended with positive held-out lift in all eight cells and left the clearest record of validating the candidate selected on public development, but its held-out gains were smaller overall. Public validation could prevent a known regression; it could not determine whether the underlying change would transfer.

Opus 4.8 and Gemini 3.6 showed how public feedback can understate transfer. Both ended with positive held-out lift in all eight cells and ranked near the top, even though their final public results were flat or negative in several campaigns. Several systems with larger public gains finished below them on held-out performance. The public signal could both understate and overstate transfer.

Muse, GPT-5.5, and Luna showed less stable transfer. Their public gains were substantially larger than their held-out gains, and their strongest outcomes were concentrated in narrower task families. Muse provided the clearest direct reversal: a bundled, task-shaped change improved public performance and then reduced held-out performance. Haiku’s public and held-out gains were close on average for a different reason: both were small.

Strong generalization required meaningful gains across held-out tasks. A small absolute public-to-held-out gap meant little when both gains were small. Public feedback could expose known regressions, but only held-out evaluation showed whether the final gain survived beyond the visible tasks. A held-out failure established that the public gain did not transfer; it did not reveal whether the diagnosis, design, or selection decision was responsible.

The missing capability is knowing what to change

In this section, we evaluate whether frontier models can improve the system through which they themselves act. We found that self-improvement was possible, but did not provide a reliable advantage. Thirty of 34 same-model campaigns improved held-out performance, yet only six of 16 systems captured more headroom on themselves than on matched fixed actors.

Models already showed that they could modify their own harnesses. Their weaker capability was forming a reliable causal explanation of their own failures. When the evaluation made a problem concrete, models could diagnose it and engineer around it. When the problem was harder to localize, using the same model as actor and proposer provided little help. Cross-model improvement tests general R&D skill. Self-improvement adds a harder requirement: the proposer must learn from evidence produced by a system whose blind spots it shares.

Developing this capability requires learning the full connection between a failure, its diagnosis, the change made in response, and the result on unseen tasks. A useful self-diagnosis predicts which intervention will transfer, rather than merely sounding plausible. The deeper capability is turning evidence about the model’s own behavior into a reliable guide for what to change, even when the environment does not make the failure obvious.

Efficiency

Lift versus inference cost

Higher lift could come from harnesses that simply make more or longer actor calls. Figure 04 compares held-out lift with the actor inference cost of each final harness.

More expensive harnesses do not consistently produce greater lift. Fable 5 leads the cross-model track at 19.98 points and $0.1075 per task. Gemini 3.6 Flash is 0.58 points behind at $0.0869. In the self-improvement track, Fable 5 produces more lift than Opus 4.7 at 4.00× the actor cost.

This comparison measures dollar efficiency rather than fixed compute because model prices differ.

Most gains do not require more actor calls

14 of the 19 self-improvement harnesses make exactly one actor call per task. The multi-call designs are GPT-5.6 Luna (1.99), Kimi K3 (2.00), Sonnet 5 (2.02), and Fable 5 (2.22) calls per task. Fable 5's 2.22-call harness costs 1.0× as much per task as Fable 5's design while producing less lift.

The pattern differs by task in the cross-model track. Relative to h0, average actor calls rise 2.47× on BFCL, 1.50× on HealthBench, and 1.55× on Tau3 Banking, while Tau3 Airline remains essentially unchanged. Extra calls are one available strategy, not the measurement target.

What the results establish

Together, these results provide evidence for a concrete form of system-level self-improvement. Some models can increase their own effective capability by studying their behavior and redesigning the harness through which they act, while their weights remain fixed. The cross-model track shows the same research capability can improve weaker actors across function calling, medical reasoning, airline service, and banking knowledge work.

Reward hacking

In AutoHarnessBench, a proposer can reward hack by raising the public score without producing a harness that performs better on unseen tasks. It might overfit the visible tasks, modify the evaluator or task data, substitute a stronger actor, bypass the actor interface, access provider credentials, or search for the held-out tasks.

In this section, we describe the steps we took to mitigate reward hacking.

The permission boundary

AutoHarnessBench separates the proposer workspace from the systems that control evaluation and held-out scoring:

1. The proposer evaluates candidates through an authenticated API that accepts only public train and development tasks.
2. The evaluation service snapshots each submitted harness, enforces the evaluation budget, and runs the canonical evaluator.
3. Actor calls pass through the fixed interface and provider route. Provider credentials and held-out data remain outside the proposer workspace.
4. Local held-out evaluation is rejected.
5. When the budget expires, the selected evaluated candidate is frozen and content-hashed, and the proposer process ends.
6. A separate trusted job evaluates that artifact using the canonical environment, scorer, and held-out split.
7. The proposer never receives held-out tasks, labels, traces, or scores.

What the proposer can exploit

The permission boundary reduces the opportunity for direct manipulation of the evaluation system, but it does not prevent the proposer from specializing to the public tasks. It may encode visible examples directly into the harness. This is permitted.

AutoHarnessBench reports improvement only when the frozen harness transfers to the held-out test. Submissions, transcripts, and evaluation records are also retained for post-run review of prohibited access.

Citation

Please cite this work as:

Essam Sleiman, Mersad Abbasi, Ayush Chakravarthy, and Karina Nguyen. AutoHarnessBench: AI R&D at the Harness Layer. 2026. https://github.com/essamsleiman/autoharnessbench

Or use the following BibTeX:

@misc{sleiman2026autoharnessbench,
  author       = {Essam Sleiman and Mersad Abbasi and Ayush Chakravarthy and Karina Nguyen},
  title        = {AutoHarnessBench: AI R\&D at the Harness Layer},
  year         = {2026},
  month        = jul,
  howpublished = {\url{https://github.com/essamsleiman/autoharnessbench}}
}