SWE-bench repository
Official benchmark code, data, and evaluation documentation.
← Back to Compute College
Compute College
What is SWE-bench?
Learn what SWE-bench measures, why it matters for AI coding agents, and how software-engineering benchmarks connect to AI compute demand.
Plain-English definition
SWE-bench is a software-engineering benchmark that evaluates whether AI systems can resolve real GitHub issues by producing changes to real repositories that satisfy evaluation tests.
Memory trick: SWE-bench is closer to “fix this repo issue” than “write this function.”
Why it matters
Repository-level repair is closer to deployed coding-agent work than short completions. If such workflows become reliable, developers can generate longer, repeated inference demand for debugging, patching, and validation.
- Repository-level repair mirrors deployed coding-agent work more than short completions do.
- Reliable repair workflows generate repeated inference for debugging, patching, and validation.
- That repetition is what turns a benchmark gain into sustained token demand.
Simple example
A task can give an agent a code repository plus an issue description, then evaluate whether the submitted patch resolves the problem under its tests. Tool access and agent scaffold affect both score and cost.
- A task supplies a repo plus an issue and checks whether the submitted patch passes tests.
- Tool access and agent scaffold change both the score and the compute consumed.
- The same model can post very different numbers under different scaffolds.
Example figures are illustrative calculations, not current quoted market prices.
Current example
Primary source
The official SWE-bench repository describes a benchmark for resolving real-world GitHub issues and its variants, including SWE-bench Verified. Its September 2026 update also made SWE-bench Multimodal v2 available as open source. Last checked: Sep 16, 2026.
No leaderboard performance claim is made here; consult the official benchmark configuration before comparing systems.
Common mistake
Do not compare SWE-bench values without checking the subset, scaffold, tools, test-time compute, and evaluation date.
Practical takeaway
What you can do with this
Use SWE-bench as capability evidence, then measure your own repository tasks by cost per accepted patch and engineer review burden.
- Confirm the subset (such as Verified), scaffold, tools, and date before comparing values.
- Treat the public number as capability evidence, not a cost estimate.
- Measure cost per accepted patch and reviewer burden on your own issues.
Decision check: do two SWE-bench numbers you are comparing share subset, scaffold, tools, and test-time compute?
Compute College
Follow model releases as market signals
Follow model releases as AI compute market signals in the ComputeTape Market Brief.
Compute College track
Model Benchmarks & AI Compute Economics
Step 11 of 25: What is swe bench