Coding Agents: Harness Study Ranks 176 Setups
A new arXiv paper takes the coding-agent harness apart and compares 176 matched settings across four models and two benchmarks, component by component.
In short
Measured component by component across 176 settings, the harness pays off most where the context window is tight, while planning mainly buys cost savings once the model is already strong.
At a glance
- 176 matched settings, five context-management strategies, four context-window budgets.
- Two benchmarks: SWE-bench Verified and Terminal-Bench 2.1, run across four models.
- Most efficient recipe per the abstract: rule-based trimming first, model summarization second.
- Bash-capable models run on a bash-only interface at substantially lower cost.
- Preprint dated September 17, 2026; arXiv 2609.20804; 43 pages; nine authors.
Most coding-agent numbers are joint numbers: a model plus whatever scaffolding happened to be wrapped around it. A preprint posted as arXiv 2609.20804 pulls that scaffolding apart and measures the pieces separately, across 176 matched settings, four models and two benchmarks.
The layer nobody isolates
A harness decides what lands in the context window, whether the agent drafts a plan before acting, and which tools it may call at all. Leaderboards score the bundle, so a harness tweak and a model upgrade look identical from the outside. The authors therefore build a deliberately small harness of their own, one where a single component can be swapped without moving anything else.
How it was measured
The abstract describes five context-management strategies and four context-window budgets, plus ablations covering planning and the action space. Evaluation runs on SWE-bench Verified and Terminal-Bench 2.1. The paper carries the date September 17, 2026 and runs to 43 pages.
Four results
- Scarcity makes context management pay: the tighter the budget, the more it helps, largely by keeping runs from dying on context overflow.
- Ordering beats method: trimming by rule first and only then summarizing with a model gave the strongest efficiency among the strategies compared.
- Planning changes job: a scaffold that props up accuracy on weaker models turns into a cost saver on stronger ones, with accuracy moving little either way.
- Fewer tools, lower bills: models comfortable with a shell work fine on a bash-only interface, and do so far more cheaply on command-line-centric work.
What this article does not verify
The reporting here rests on the paper's abstract page, not the full text, so the size of the reported cost gaps sits in the PDF and is unchecked. The companion site at harnesstax.github.io loads its content via script and returned only the heading “HarnessTax: How Much Does the Harness Matter for Coding Agents?” when fetched, so none of its figures or tables could be inspected. And a preprint has not been peer reviewed.
The practical reading still holds up: treat context management, planning and tool surface as three separate dials rather than one setting called “the harness.” What tunes well under a weak model is not what the paper suggests you want under a strong one.
FAQ
What is a harness in a coding agent?
The software layer around the model. It controls what enters the context window, whether the agent plans before acting, and which tools it is allowed to call.
Which benchmarks does the harness study use?
Per the abstract, SWE-bench Verified and Terminal-Bench 2.1, evaluated over four models in 176 matched settings.
Does planning make coding agents more accurate?
On weaker models the abstract describes planning as an accuracy scaffold; on stronger models it mainly saves cost, with little change in accuracy.
