General LLM & JudgeMeasures: Knowledge and reasoning across 57 subjects, from elementary maths to professional law.
Scoring: Multiple-choice accuracy against fixed answer keys.
Best use: A quick, comparable check of broad knowledge — never a product-quality claim on its own.
Watch out: Heavily reused and likely contaminated in training data; some items are erroneous, and scores are near saturation for frontier models.
General LLM & JudgeMeasures: Many scenarios and metrics together — accuracy, robustness, fairness, bias, toxicity, efficiency.
Scoring: Standardised runs across scenarios, reported as a matrix rather than one number.
Best use: Comparing models when you care about trade-offs, not just headline accuracy.
Watch out: Broad is not the same as relevant — pick the HELM scenarios that match your users, and ignore the rest.
General · ConversationMeasures: Multi-turn chat quality across writing, reasoning, maths, coding, and roleplay — 80 questions.
Scoring: LLM-as-judge (originally GPT-4) scoring or pairwise comparison.
Best use: A fast judge-based smoke test for a chat model, and the reference design for judge prompts.
Watch out: Only 80 items — far too small for fine-grained ranking. Inherits every judge bias in §2.
Agents & Tool UseMeasures: Real assistant questions needing reasoning, web browsing, multimodality, and tool use — 466 questions.
Scoring: Exact-match answers to questions with unambiguous, verifiable answers.
Best use: Testing whether an agent can actually complete open-ended assistant work, not just chat about it.
Watch out: Small and hard by design — report level-by-level results, and expect browsing / tool setup to dominate scores.
Agents & Tool UseMeasures: LLM-as-agent across 8 environments — operating systems, databases, web, games, and household tasks.
Scoring: Task success in each environment.
Best use: A first cross-environment picture of agent capability before you build product-specific agent evals.
Watch out: Environments are simulated and dated — a good AgentBench score does not predict success in your own tool stack.
Agents · Safety & XPIAMeasures: Prompt-injection attacks and defences for agents — 97 tasks, 629 security cases — with utility measured alongside security.
Scoring: Attack success rate plus benign task utility in the same environment.
Best use: The regression gate for an agent that reads untrusted content. See §4 for the full method.
Watch out: Results are environment-specific — extend it with your own tasks instead of comparing raw scores across setups.
Safety & XPIAMeasures: Indirect prompt injection via email, web pages, documents, code, and QA contexts.
Scoring: Attack success rate by surface and attack family.
Best use: Assistants that ingest other people's content — the closest match to a multi-party Copilot surface.
Watch out: Mostly text output harm — pair it with InjecAgent or AgentDojo if your assistant can call tools.
Safety & XPIA · AgentsMeasures: Indirect injection in tool-integrated agents — 1,054 cases covering data stealing and direct-harm attacks.
Scoring: Attack success rate for tool misuse.
Best use: When the risk is a bad action, not just bad text — tool calls, sends, and deletions.
Watch out: Your agent harness, prompts, and tool permissions change the result as much as the model does. Report all three.
Safety & XPIANotInject
Measures: Benign inputs that look like injections — the over-defence side of XPIA.
Scoring: False-refusal / false-positive rate on benign look-alike cases.
Best use: A smoke test run alongside every ASR suite, so a “more secure” model is not just a model that refuses everything.
Watch out: Small set — a signal, not a verdict. Grow it with your own product's benign edge cases.
Safety & XPIAMeasures: Probe-based vulnerability scanning — jailbreaks, leakage, toxicity, hallucination, and injection probes. A harness, not a fixed benchmark.
Scoring: Probe pass / fail rates by vulnerability family.
Best use: A starting sweep when you inherit a new model or system and need to find where to dig.
Watch out: A clean probe run is not a safety case. Follow up every hit with a product-specific regression suite.
CodingMeasures: Resolving real GitHub issues — 2,294 issues from 12 popular Python repositories.
Scoring: Fail-to-pass and pass-to-pass unit tests after the model's patch is applied.
Best use: The reference for real software-engineering agent capability, far beyond single-function generation.
Watch out: Python-only and repo-specific; scaffold and compute budget move scores a lot. Use the verified / lite splits carefully and say which you ran.
CodingMeasures: Generating standalone Python functions from docstrings — 164 hand-written problems.
Scoring: pass@k against unit tests.
Best use: A quick historical baseline for code generation — useful mainly for comparison with older results.
Watch out: Small, saturated, and widely contaminated. It says almost nothing about working in a real codebase — use SWE-bench for that.
HealthcareMeasures: Medical knowledge and clinical reasoning in the style of USMLE exam questions.
Scoring: Multiple-choice accuracy.
Best use: A first check of medical knowledge — a floor, not evidence of clinical safety or usefulness.
Watch out: Exam questions are not patients. Real clinical eval needs workflow tasks, uncertainty handling, and clinician-adjudicated cases.
Healthcare · RAGMeasures: Answering biomedical research questions (yes / no / maybe) from PubMed abstracts.
Scoring: Accuracy against expert-annotated labels, with a larger automatically labelled set alongside.
Best use: Testing whether a system can ground an answer in a specific paper's abstract — a small, clean RAG-style task.
Watch out: Abstracts only, and “maybe” is doing real work — do not collapse it away when you report results.
Finance & LegalMeasures: Answering questions about real public-company financial documents — 10-Ks, 10-Qs, earnings materials.
Scoring: Human-judged correctness on 150 open-book questions with documented evidence.
Best use: A reality check for document-grounded finance QA and retrieval — models that ace generic QA often fail here.
Watch out: Only 150 questions, so treat it as a diagnostic, not a ranking. Verify the evidence span, not just the final number.
Finance & LegalMeasures: Legal reasoning across 162 tasks built with legal professionals — classification, entailment, and rule application.
Scoring: Task-specific metrics, mostly accuracy / F1 against lawyer-curated labels.
Best use: Seeing where a model handles legal reasoning, not just legal vocabulary.
Watch out: US-law centred and task-fragmented — report per task family, and never read it as readiness for legal advice.
MultimodalMeasures: College-level multimodal reasoning — 11.5K questions with images, charts, diagrams, and tables across 30 subjects.
Scoring: Multiple-choice / short-answer accuracy.
Best use: Testing whether a vision-language model can reason with an image, not just describe it.
Watch out: Exam-style again — pair it with your own product images and failure review before drawing conclusions.
MultimodalMeasures: Mathematical reasoning in visual contexts — charts, plots, diagrams, and natural images; 6,141 examples.
Scoring: Answer accuracy, typically extracted and judged automatically.
Best use: A focused probe of chart / figure reading — a common silent failure in document and dashboard products.
Watch out: Automatic answer extraction adds its own error — audit a sample of “wrong” answers by hand.
RAG & RetrievalMeasures: Zero-shot retrieval across 18 datasets — search, QA, fact-checking, citation, and more.
Scoring: nDCG@10 and related ranking metrics per dataset.
Best use: Choosing and regression-testing the retriever underneath a RAG system, separately from the generator.
Watch out: Good retrieval scores do not guarantee grounded generation — evaluate retrieval, grounding, and citation as separate stages.
RAG & RetrievalMeasures: Whether a model repeats common human falsehoods — 817 questions across 38 categories.
Scoring: Truthfulness and informativeness, originally human-judged with automatic judges alongside.
Best use: A hallucination / misconception probe, and a reminder that a fluent answer can be confidently wrong.
Watch out: Small and adversarial by design — a diagnostic slice, not a general accuracy measure.
ConversationMeasures: Live human preference between two anonymous models on real user prompts.
Scoring: Pairwise votes aggregated into ratings (Bradley–Terry / Elo-style).
Best use: An external, real-user preference signal — and a design reference for running your own pairwise human eval.
Watch out: Preference rewards style and confidence as well as correctness, and the voting population is not your user base.
No benchmarks in that domain yet.
How to choose: match the benchmark's task and ground truth to your product's riskiest decision, run it as a regression gate with slice reporting, and add your own product-specific cases — every public benchmark above is a starting point, not a substitute for your own golden set (§1.2).