How to evaluate an LLM beyond public leaderboards

A public leaderboard can help shortlist models. It cannot, by itself, select an LLM for a business process. A score depends on the tasks, data, prompt, parameters, model version and grading method. It rarely measures your most expensive errors, your complete architecture or future behaviour in production.

The right unit of evaluation is therefore not just the model. It is the system that will actually be deployed: model, prompt, data, retrieval, tools, validation rules, interface and human oversight.

Short answer

To evaluate an LLM beyond public leaderboards:

1. define the business decision and unacceptable errors;
2. build a representative, versioned test set;
3. measure several dimensions rather than one average;
4. compare systems under identical conditions;
5. combine automated metrics, human review and adversarial tests;
6. estimate uncertainty and analyse failures by segment;
7. measure latency and cost per successful task;
8. repeat the evaluation after every change and in production.

Nexxom framework for evaluating an LLM system in its intended context.

What a leaderboard actually measures

A benchmark is a defined experiment. It connects a set of questions or tasks to a protocol, configuration and metric. The result can be useful, but it becomes interpretable only when those conditions are known.

In February 2026, NIST published AI 800-3 to distinguish, among other concepts, accuracy observed on benchmark items from performance generalised to a population of comparable items. The distinction is a reminder that a number obtained on one sample is not automatically the system's future performance.

HELM, Stanford's framework for holistic model evaluation, emphasises broad coverage and multidimensional measurement. The project publishes scenarios, prompts, outputs and metrics to make comparisons more transparent and reproducible.

The operational conclusion is straightforward: a leaderboard supplies an observation within a particular frame. A business decision requires evidence that the observation transfers to the actual task.

Start with the decision, not the metric

Before choosing an indicator, describe what the system must accomplish and what happens when it is wrong.

For a document assistant, an incomplete but properly sourced answer may be acceptable. A fabricated reference may be critical. For invoice extraction, writing style matters little, while an error in an amount or supplier could trigger an incorrect payment. For an agent, response quality is not enough. Tool selection, parameters and action authorisation must also be evaluated.

An evaluation brief should specify:

• user populations and languages;
• documents, tools and data available to the system;
• expected outputs and formats;
• tolerable, major and critical errors;
• cases that require abstention or human approval;
• latency, volume and cost objectives;
• required security and traceability.

Build a representative test set

A good test set does not contain only frequent cases. It covers the distribution of real work and the situations that expose system limits.

Separate at least a development set, used to improve the system, from a final validation set kept outside the optimisation loop. Otherwise, the team may adapt prompts and rules to known examples without improving generalisation.

Version every item, its origin, difficulty, expected answer or behaviour, and grading rule. Remove or protect personal and confidential data as required by the context.

Measure multiple dimensions

One average hides trade-offs. The NIST AI RMF recommends quantitative, qualitative or mixed methods that match the deployment context and are documented. Six measurement families are usually necessary for an LLM system.

1. Task quality

The metric should match the job: accuracy of structured fields, presence of required elements, faithfulness to sources, usefulness of a recommendation or compliance with a format. Lexical metrics alone rarely work well for open-ended answers.

2. Critical errors

Calculate errors that could cause harm, a wrong decision or a violation separately. A system with a 95% success rate may still be unusable if the remaining 5% covers the highest-risk cases.

Use a severity matrix. A stylistic flaw, a recoverable omission and the disclosure of sensitive data should not carry the same weight.

3. Robustness and consistency

Rephrase requests, change information order, introduce realistic noise and repeat selected tests. A useful system should remain sufficiently stable when the form changes without altering meaning.

4. Security and safe behaviour

Test prompt injection, unauthorised information access, dangerous outputs, tool calls and the ability to refuse or request approval. For RAG and agent systems, the evaluation must cross component boundaries.

5. Latency and availability

Measure the distribution of response times rather than the mean alone. High percentiles reveal what users experience under difficult conditions. Include rate limits, provider errors and fallback behaviour.

6. Cost per successful task

Price per token does not describe process cost. Include retries, retrieval, tool calls, hosting, human review and error handling. The useful denominator is a task completed at the required quality level.

Automation, humans and LLM judges

No grading method fits every criterion.

An LLM judge must itself be evaluated. Compare its ratings with a human-annotated sample, use an explicit rubric, hide model identities and monitor disagreements. It should not be the sole arbiter of a sensitive decision.

Make the comparison fair

Compare candidates inside the same system and document every variable: model version, date, prompt, parameters, tools, document repository, context limit and safety policy. If a model requires a specific adaptation, report that scenario too, but do not merge its results with the common configuration.

Report results by segment and with an indication of uncertainty where possible. A small difference between two models may not be meaningful given the size or variability of the test set.

Do not select the model on the final set. Optimise on development data, freeze the configuration, and then run a final comparison. Retain raw outputs so failures can be audited.

Move from evaluation to a production threshold

A useful evaluation ends with an explicit decision, not a score table.

Example release rules include:

• no unmitigated critical error in the final set;
• minimum performance for every priority segment, not only the average;
• acceptable abstention rate when information is missing;
• latency and cost compatible with the process;
• security controls tested;
• fallback and human approval available;
• a named owner for monitoring outcomes.

The threshold depends on risk. An internal writing assistant and a system influencing financial decisions should not share identical criteria.

Continue measuring in production

Deployment changes the distribution of requests. Users write differently, documents evolve and providers may update a model. NIST recommends testing before deployment and regularly during operation.

Monitor reported failures, human escalations, refusals, incidents, latency, cost and drift by segment. Add confirmed new cases to the regression corpus without exposing sensitive data. Rerun the suite before every model, prompt, RAG or tool change.

A reproducible decision method

The best LLM is not the model on the first line of a general leaderboard. It is the model that, within a defined architecture, meets the use case's quality and safety thresholds at an acceptable cost and latency.

A strong evaluation makes trade-offs visible. It also enables model replacement, decision justification and regression detection. That measurement capability, more than the provider's name, turns a prototype into a governable system.

Primary sources

• NIST AI RMF Core, Measure function, principles for measurement, documentation and evaluation in deployment context.
• NIST AI 800-3: Expanding the AI Evaluation Toolbox with Statistical Models, benchmark accuracy, generalisation and uncertainty.
• HELM: Holistic Evaluation of Language Models, Stanford's transparent and multidimensional evaluation framework.
• HELM official website, reproducible scenarios, metrics and results.

Sources verified on 6 August 2026.