What to compare
- benchmark version and split
- model and harness revision
- task population and exclusions
- resolved tasks and denominator
- cost and tool budget
Read the sources
SWE-bench ↗
Coding-agent evaluations over repository issues, with separate benchmark variants and harness settings.
Read the official leaderboard and evaluation documentation; this directory does not mirror its results. Pin the evaluation split, harness and result source. Check each code and dataset licence before reuse.
Stanford HELM ↗
Reproducible evaluation framework with scenario, metric and result references.
The official project links its evaluation suites and published results. Pin the suite, model version and evaluator. Check dataset licences independently from the framework licence.
When to look again
Before changing the coding model or harness, or after a relevant evaluation or repository requirement changes.
Give your agent this question
Prepare a source-linked comparison for the same evaluation setting. Separate publisher-reported results from results we reproduced; leave incompatible settings side by side without a combined rank.