Evaluating a multilingual LLM in the enterprise: metrics that reveal gaps
A global average hides language gaps. Learn how to build an actionable multilingual evaluation for an enterprise use case.
Short answer
An LLM can post an excellent average while failing on the language, register or terminology used by your customers. A multilingual evaluation compares the same intent across languages, measures answer quality and checks gaps in safety, latency and cost. Machine-translating an English test set is not multilingual coverage.
Start with real cases and observable criteria. Combine a public benchmark, a private business set, deterministic checks and local human review. For some teams, two languages carry most of the value. For others, a lower-resource language creates the highest risk and needs a dedicated sample.
Why a global average misleads
Benchmarks often aggregate different languages and tasks. A highly represented language can dominate the score. A model may get the meaning right while using the wrong register, local units or legal terminology.
The HELM framework promotes transparent, multi-scenario and multi-metric evaluation. NIST explains that measures must be tied to context and document limitations on its AI measurement and evaluation page. The NIST AI 800-3 report explores statistical models to characterize variance and stability across languages.
A comparison protocol
Nexxom diagram: keep the same intent, measure each language and connect evidence to a decision.
- Map languages. List interface languages, document languages, regional variants and expected proficiency.
- Sample intents. Create semantically equivalent pairs or triplets, then add cases that cannot be translated literally.
- Set criteria. Measure accuracy, source faithfulness, format, tone, safety and the ability to ask for clarification.
- Separate sets. Keep development, blind control and incident-driven regression sets apart.
- Test robustness. Vary spelling, accents, code-switching, length, information order and document noise.
- Review locally. Annotators need language and domain expertise. Measure inter-annotator agreement before interpreting model scores.
- Decide by slice. A critical gap in one language must not be averaged away.
| Layer | Question | Example evidence |
|---|---|---|
| Understanding | Is the intent recognized? | Intent label and rationale |
| Generation | Is the answer accurate and useful? | Criterion-level annotation |
| Localization | Are register and conventions appropriate? | Local review |
| Safety | Are refusals and safeguards consistent? | Adversarial cases per language |
| Operations | Are cost and latency acceptable? | Measures by language and region |
Choose the right metrics
Lexical overlap metrics such as BLEU or ROUGE can add context, but cannot measure faithfulness or usefulness alone. For free-form answers, combine fact checks, business rules, a calibrated LLM evaluator and human review. For structured extraction, validate schema, types and values independently of prose.
| Dimension | Possible measure | Limitation to state |
|---|---|---|
| Accuracy | Correct answers per language | Depends on annotation |
| Faithfulness | Entailment or citation checks | Reference may be incomplete |
| Robustness | Gap after perturbations | Perturbations need documentation |
| Fairness | Performance gap between groups | Does not prove no bias |
| Efficiency | Latency and tokens per request | Changes with routing |
The FLORES-101 benchmark helped standardize low-resource translation evaluation. Recent work warns that even a multilingual benchmark can underrepresent communities (Languages Still Left Behind). Treat it as a comparison point, not a replacement for business cases.
Measure gaps without hiding them
Report one row per language and intent. Add an interval or dispersion when the sample allows it. A two-point gap means something different on 20 cases and 2,000 cases. Keep failure examples and severity, not only aggregates.
Nexxom Plotly matrix: a 0 to 5 heuristic for trade-off discussion, not a model ranking.
| Report view | Publish | Do not conclude |
|---|---|---|
| By language | Score, variance, examples and cost | That an average proves parity |
| By intent | Successes, failures and severity | That a language is homogeneous |
| By version | Delta from the previous release | That a global gain applies everywhere |
| By mode | Text, RAG, tools and refusals | That text testing covers agents |
Translation and code-switching
Do not mechanically translate every instruction. Users mix languages, acronyms, names and units. Include cases where the question is in one language and the source in another. Check that the model cites the right sentence and does not turn a legal nuance into a claim.
For RAG, test retrieval, generation and citation separately. Measure whether relevant documents are found in each language, then whether the answer remains faithful. Our article on hybrid RAG covers lexical and semantic trade-offs.
Human calibration and governance
Train annotators with examples and counter-examples. Have two people review a subset and arbitrate disagreements. Document dialect range, professional register and terms that must not be translated.
The NIST AI RMF recommends context-specific measures, documented uncertainty and oversight. The Measure Playbook calls for tracking errors, feedback and corrective action. Use a decision matrix with per-language thresholds instead of one global score.
| Situation | Recommended decision | Why |
|---|---|---|
| Small gap, low stakes | Deploy with monitoring | Reversible risk |
| Large gap in a strategic language | Limit or improve | Direct user impact |
| Localized safety failure | Block the slice | Average cannot compensate |
| Little human data | Enrich the sample | Uncertainty is too high |
Monitor in production
Instrument detected language, requested language, source language, latency, cost, refusals and human corrections. Do not infer language from the browser alone: one account may work in several languages. Provide a route to report misleading translation or an inappropriate register.
For some teams, a monthly language dashboard is enough. For others, daily checks of high-risk languages are required. Define a drift threshold and an action. A model, tokenizer or corpus update should trigger a targeted re-evaluation.
Decision checklist
- Do cases represent languages and variants actually used?
- Are intents equivalent without being simple translations?
- Do metrics separate accuracy, safety, localization and operations?
- Are blind and regression sets available for each language?
- Do annotators understand language, domain and register?
- Do results show dispersion, examples and severity?
- Are RAG, tools and formats tested separately?
- Can a per-language threshold block release?
Evaluating a multilingual LLM means measuring usage conditions rather than seeking a universal score. A defensible decision connects each language to evidence, limits and a monitoring plan. For further reading, see our evaluation beyond public leaderboards and our method for LLM-as-a-judge.
Make the evidence durable
Store prompts, model versions, translation instructions and annotation decisions with each run. Keep a changelog for tokenizer, retrieval index and routing updates. When a language changes score, this record helps distinguish a model regression from a data or evaluation change. It also gives procurement, product and compliance teams a common basis for deciding whether to expand, limit or pause a deployment.
Document difficult cases
Difficult cases have disproportionate value. An answer that is correct in English but unsafe in French, a misread acronym or a wrongly converted unit can cause an incident without moving the average. Mark each example with severity and expected action. Replay it after every model change and check that a fix does not degrade another language.
Test data also needs a privacy policy. Pseudonymize tickets, limit access and retain translation provenance. When annotators are unavailable for a language, state the limitation and reduce deployment scope instead of inventing certainty. This transparency makes the decision stronger for legal teams and users.
Connect measurement to operations
An evaluation table should end in an operational choice: enable a language, require human review, route to a specialist model or wait for more data. Assign an owner, review date and withdrawal threshold. Evaluation then becomes a continuous control alongside latency and availability.
Keep a language-specific release note. Record changed prompts, retrieval sources, tokenizers, safety policies and routing logic. When a result moves, the team can explain whether the cause is model capability, data coverage or infrastructure. This prevents silent regressions from reaching customers and gives leadership a clear go, limit or pause decision.
Add a cross-language consistency check. Present the same policy, table or instruction in every version and verify that obligations, negations and conditions remain intact. A fluent answer that drops an important exception is a faithfulness failure even when its grammar is perfect.
Include terminology governance. Maintain a small glossary for product names, legal terms, units and sensitive phrases. Review additions with domain owners and test both the preferred term and plausible alternatives. This makes quality measurable and prevents a translation shortcut from changing the business meaning.
Treat low-resource languages explicitly
Do not hide a small sample behind a confident percentage. Report the number of cases, the confidence interval or qualitative uncertainty, and the reason for any missing annotation. Consider routing high-impact requests in a low-resource language to a human or a specialist workflow until evidence improves. This is a product decision, not a judgement about the value of that language.
When collecting more data, prioritize real failure modes rather than random volume. Ask support teams for anonymized examples, invite local reviewers to identify idioms and test the same policy across formal and informal registers. Record consent and provenance so that future evaluations can reproduce the selection process.
