AutoHarnessBench

Can frontier models automate agent harness R&D?

Each model is given a minimal harness scaffold, a task suite, and a 30-minute budget. It iteratively constructs and refines the harness by executing tasks and inspecting the resulting traces. The final harness is frozen and evaluated on held-out tasks.

View results Read the blog Paper (soon)

Track

Benchmark

Actor

0% 5% 10% 15% 20% 25%

24.1%

23.4%

22.7%

22.5%

21.8%

21.7%

21.0%

20.4%

20.3%

18.8%

18.7%

18.5%

14.8%

13.8%

12.4%

6.5%

#1 · Fable 5

#2 · Kimi K3

#3 · Opus 4.7

#4 · Opus 4.8

#5 · Gemini 3.6 Flash

#6 · Gemini 3.1 Pro

#7 · Opus 5

#8 · GLM 5.2

#9 · Sonnet 5

#10 · Grok 4.5

#11 · GPT-5.6 Sol

#12 · Opus 4.6

#13 · Sonnet 4.6

#14 · Muse Spark 1.2

#15 · GPT-5.6 Terra

#16 · GPT-5.6 Luna

#17 · Muse Spark 1.1

#18 · GPT-5.5

#19 · Haiku 4.5

Detailed data

Benchmark suites

Harnesses are tested across four environments spanning function calling, expert-judged reasoning, tool use, retrieval, and simulated user interaction.

Function calling

BFCL Multi-Turn

Handle evolving requests by selecting and sequencing functions with valid arguments across multi-turn conversations.

Function calling 40 trusted tasks per actor

Healthcare reasoning

HealthBench Lite

Produce safe, useful healthcare responses graded against expert-written rubrics.

Rubric judged 100 trusted tasks Cross-model + self-improvement

Airline customer support

Tau three Airline

Resolve airline support requests using tools, policy, and a simulated customer.

Tool use 3 × 20-task trusted trials

Banking customer support

Tau three Banking

Handle banking requests using tools and retrieval over a 698-document knowledge base.

Retrieval + tool use 3 × 39-task trusted trials

Cite AutoHarnessBench

@misc{sleiman2026autoharnessbench,
  author       = {Essam Sleiman and Mersad Abbasi and Ayush Chakravarthy and Karina Nguyen},
  title        = {AutoHarnessBench: AI R\&D at the Harness Layer},
  year         = {2026},
  month        = jul,
  url          = {https://github.com/essamsleiman/autoharnessbench},
  note         = {GitHub repository}
}