What to compare
- corpus and query split
- embedding and reranker revision
- retrieval metric and cutoff
- answer metric and judge
- latency and failure population
Read the sources
MTEB ↗
Embedding evaluations across retrieval tasks, languages and modalities.
The project documents evaluation code and links its leaderboard. The code licence does not settle the licences of every benchmark dataset or model.
BEIR ↗
Retrieval evaluation across heterogeneous datasets, including lexical, dense and reranking methods.
Use the official project documentation and dataset references. Inspect each dataset's use conditions; retrieval scores are not end-to-end answer-quality results.
Ragas ↗
Methods for evaluating retrieval, grounded answers and agent workflows on a chosen test set.
Public documentation; running a model-based evaluator needs separate model access and a budget. Pin the metric implementation and judge configuration. A judge score is not an independent human finding.
When to look again
Before redesigning retrieval, after a corpus or query change, and when observed answer quality declines.
Give your agent this question
Propose a retrieval comparison on a fixed corpus, then a separate answer-quality check. Explain which public benchmark resembles the task and which differences need a local test.