What a coding agent's harness parts really do
A new study splits the harness into planning, action space and context management, then measures 176 matched setups on two coding benchmarks.
In short
The value of a harness sits in its individual components rather than the bundle: context management pays off mainly when the context window is tight, while planning props up accuracy for weaker models and mostly cuts cost for stronger ones.
At a glance
- 176 matched configurations, run with four models on SWE-Bench Verified and Terminal-Bench 2.1.
- The execution loop stays fixed; three components vary: planning, action space, context management.
- Under comparison: five context-management strategies and four context-window budgets.
- Most efficient per the abstract: rule-based elision staged before LLM summarization.
- Preprint dated September 17, 2026, nine authors, 43 pages, arXiv 2609.20804v1.
Harnesses are usually scored as whole systems, which tells you which one won but not which part did the work. A preprint posted on September 17, 2026 takes the other route: it holds the execution loop fixed and varies only planning, action space and context management. That grid comes to 176 matched configurations, measured with four models on SWE-Bench Verified and Terminal-Bench 2.1.
The experimental grid
The authors deliberately use a lightweight harness so the moving parts are the only thing that moves. Five context-management strategies are compared against four context-window budgets, with targeted ablations for planning and for the action space on top. Because the loop underneath never changes, a score difference can be attributed to a component instead of to a whole framework.
Four findings
Context management matters more as the context-window budget tightens, and the abstract attributes most of that benefit to avoiding context-overflow failures rather than to smarter recall. On strategy design, staging cheap rule-based elision ahead of LLM summarization came out strongest on overall efficiency. Making the elided text recoverable added machinery the models rarely reached for and bought no accuracy.
Planning changes job depending on the model: a scaffold for accuracy on weaker models, a cost saver on stronger ones, with little movement in accuracy either way. The action-space result is similarly conditional. Predefined tools help models with weaker bash skills, whereas bash-capable models run fine on a bash-only interface at substantially lower cost, most visibly on command-line-centric tasks.
What we could not verify
The abstract page does not name the four models, so the exact families and versions live only in the 43-page full text. The companion project page at harnesstax.github.io returned nothing but its title and a loading placeholder when we fetched it, with no readable tables or figures. We therefore cannot back any per-configuration score, dollar cost or ranking of harness variants, and we report the four findings only as strongly as the abstract states them.
FAQ
What is a coding harness?
It is the layer between the model and the codebase: which tools the agent may call, whether it plans first, and how much prior history survives in the context window. This study fixes the execution loop and varies only those three components.
Which benchmarks were used?
SWE-Bench Verified and Terminal-Bench 2.1. Across both, the paper reports 176 matched configurations covering four models, five context-management strategies and four context-window budgets.
Is a bash-only tool interface enough for a coding agent?
Per the abstract, it depends on the model. Models with weaker bash proficiency do better with predefined tools, while bash-capable models work effectively with bash alone and at substantially lower cost.
