ContextBench

A benchmark for context retrieval in coding agents.Measuring how coding agents find and use code context.
Compare task success, retrieval quality, and efficiency.

Read the paper Dataset

TasksTasks: 1,136
ReposRepositories: 66
LanguagesLanguages: 8

Leaderboard

Results updated 14 Sept 2026

Submit result

Agent–model combinations, including mini SWE-agent baselines. Agents use ContextBench-specific adaptations.

Systems: 15Agent systems
Best Pass@1: 60.4%openJiuwen-JiuwenSwarm + DeepSeek-V4-Pro
Best context F1: 0.376Agentless + GPT-5

Sort by

Agent system rankings by Recall. Ranks are preserved when reversing order or searching. Equal scores share a rank.
Rank	Agent + model
1	openJiuwen-JiuwenSwarm + DeepSeek-V4-ProSubmission & logs	0.753	60.4%	0.257	0.642	$0.57
2	mini SWE-agent + GPT-5	0.606	47.2%	0.312	0.591	$0.45
3	mini SWE-agent + Claude Sonnet 4.5	0.588	53.0%	0.344	0.658	$0.76
4	Prometheus + GPT-5	0.584	51.2%	0.231	—	—
5	SWE-agent + GPT-5	0.476	49.0%	0.208	—	—
6	OpenHands + GPT-5	0.472	49.0%	0.130	—	—
7	Agentless + GPT-5	0.461	45.2%	0.376	—	—
8	mini SWE-agent + Devstral 2	0.404	40.2%	0.332	0.616	$0.91
9	mini SWE-agent + Gemini 2.5 Pro	0.313	36.4%	0.311	0.529	$0.38
10	mini SWE-agent + DeepSeek-V4-Pro	0.291	57.5%	0.338	—	—
11	mini SWE-agent + GLM-5.1	0.236	51.4%	0.306	—	—
11	mini SWE-agent + MiniMax-M2.5	0.236	49.0%	0.292	—	—
13	mini SWE-agent + Qwen3.7-Max	0.185	48.4%	0.257	—	—
14	mini SWE-agent + Qwen2.5-32B	0.076	5.7%	0.108	0.171	—
15	mini SWE-agent + Qwen3-8B	0.048	4.0%	0.076	0.077	—

Agent + modelRecall ↓

Showing 15 of 15 systems · Higher Recall is better.

— Not reported · Equal scores share a rank

Evaluation notes & result history

Dataset totals above describe ContextBench as a whole. Run-specific task coverage, dataset revisions, and configurations are available with submission artifacts when provided; historical entries may not include them.

Rank numbers reflect performance on the selected metric. Reversing the display order or searching does not change those ranks. Recall and Context F1 use line-level retrieval scores on a 0–1 scale; Pass@1 is shown as a percentage and costs are in USD.

Evaluation code & documentation Result history Submission & review process

Key Findings

Echoing "The Bitter Lesson": More scaffolding does not mean better context retrieval.

Even frontier LLMs struggle to retrieve precise code context.

LLMs favor recall over precision, introducing substantial noise.

Balanced retrieval achieves higher accuracy at lower cost.

Retrieved context is often not used in final solutions.

Abstract

LLM-based coding agents have shown strong performance on automated issue resolution benchmarks, yet existing evaluations largely focus on final task success, providing limited insight into how agents retrieve and use code context during problem solving.

We introduce ContextBench, a process-oriented evaluation of context retrieval in coding agents. ContextBench consists of 1,136 issue-resolution tasks from 66 repositories across eight programming languages, each augmented with human-annotated gold contexts. We further implement an automated evaluation framework that tracks agent trajectories and measures context recall, precision, and efficiency throughout issue resolution.

Using ContextBench, we evaluate four frontier LLMs and five coding agents. Our results show that sophisticated agent scaffolding yields only marginal gains in context retrieval ("The Bitter Lesson" of coding agents), LLMs consistently favor recall over precision, and substantial gaps exist between explored and utilized context. ContextBench augments existing end-to-end benchmarks with intermediate gold-context metrics that unbox the issue-resolution process, offering valuable signals for guiding LLM reasoning in software tasks.

Construction Pipeline

An overview of the ContextBench construction pipeline. ContextBench is curated through three key steps: Task Deduplication, Task Selection, and Expert Annotation.

1Task Deduplication

Removes exact and near-duplicate tasks from multiple issue resolution benchmarks using rule-based and embedding-based detection.

2Task Selection

Identifies challenging tasks based on agent solvability and the scope and dispersion of edits in ground-truth patches.

3Expert Annotation

Employs expert developers to trace code dependencies to construct gold contexts, validated through LLM-based patch generation.

Dataset Statistics

A repository-level benchmark spanning 8 programming languages and introducing human-verified gold contexts to expose intermediate context retrieval signals missing from final task resolution rate evaluation.

Language	#Repo	#Task	#File	#Block	#Line
Python	20	512	1,520	6,714	115,122
Java	6	57	262	3,030	49,057
JavaScript	9	153	819	3,949	87,907
TypeScript	8	119	537	1,106	40,621
Go	7	104	679	3,000	71,596
Rust	9	63	272	1,842	50,402
C	3	68	250	1,591	62,300
C++	4	60	209	1,884	45,110
Total	66	1,136	4,548	23,116	522,115

How to Cite

If you use ContextBench, please cite:

@misc{li2026contextbenchbenchmarkcontextretrieval,
      title={ContextBench: A Benchmark for Context Retrieval in Coding Agents},
      author={Han Li and Letian Zhu and Bohan Zhang and Rili Feng and Jiaming Wang and Yue Pan and Earl T. Barr and Federica Sarro and Zhaoyang Chu and He Ye},
      year={2026},
      eprint={2602.05892},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2602.05892},
}

Authors

Han Li¹ Letian Zhu^1* Bohan Zhang^1* Rili Feng^1* Jiaming Wang¹ Yue Pan² Earl T. Barr² Federica Sarro² Zhaoyang Chu^2† He Ye^2†

1Nanjing University 2University College London

*These authors contributed equally as co-second authors.

†Corresponding authors. Email: {he.ye, zhaoyang.chu.25}@ucl.ac.uk