AI evaluation is the structured measurement of how an AI system behaves — how accurate, safe, fair and useful its outputs are — against defined criteria and ground truth. For LLMs and AI agents it has become the discipline that decides whether a system is fit to ship, not just whether it works in a demo.
Classical machine learning evaluation asked a narrow question: how often is the model right? Generative and agentic systems make that question insufficient. There is rarely one correct answer, the output is open-ended, and the same model can be accurate and still be unsafe, off-brand, culturally wrong or exhausting to use.
Modern AI evaluation therefore measures behaviour across multiple dimensions at once. It asks whether an answer is grounded in fact, whether it treats users equally, whether it resists manipulation, whether it discloses what it is, and — increasingly — whether the user actually got what they came for and how much effort it cost them.
Evaluation is also a governance artefact. Regulatory regimes such as the EU AI Act and a growing set of sector-specific rules expect documented, repeatable measurement of high-risk AI systems, produced with enough independence to be credible to an auditor.
AI evaluation is the repeatable measurement of an AI system's behaviour against explicit criteria — risk, quality and user experience — so that decisions about shipping, fixing and certifying it rest on evidence rather than impression.
The distinction matters because teams routinely mistake one for the other. Observability tells you a conversation happened, how long it took and how many tokens it burned; it does not tell you whether the answer was true, fair or useful. Evaluation answers that, but only on the inputs you tested. The mature pattern is a loop: monitoring surfaces suspicious real traffic, that traffic feeds new evaluation cases, and evaluation results set the thresholds monitoring alerts on.
Fixed datasets with known ground truth, run repeatedly to compare model versions. Reproducible and cheap, but limited to what the benchmark happens to contain.
Scoring samples of real traffic, A/B tests and live experiments. Reflects genuine user behaviour, at the cost of control and of exposing users to failures.
Trained reviewers scoring outputs against rubrics, ranking responses side by side and adjudicating disagreements. Still the gold standard for subjective quality.
A model scores outputs against a rubric at scale. Fast and consistent, but must itself be validated against human labels or it quietly inherits the failure it is meant to detect.
Deliberate attempts to break the system: jailbreaks, prompt injection, unsafe requests, manipulation and edge-case abuse, executed as a structured suite rather than ad hoc.
Judging a whole multi-step flow — tool calls, decisions, recoveries — instead of a single response. The trajectory itself becomes the object of measurement.
Evaluating retrieval, routing or a single prompt in isolation localises defects; end-to-end evaluation is the only thing that predicts what users will experience.
An LLM plays realistic users — personas, languages, moods, misspellings — to generate large volumes of controlled conversations without waiting for production traffic.
Assessment run by an external party rather than the team that built the system. Marking your own homework is not a credible compliance posture, and auditors treat it as such.
Useful evaluation is dimensional: instead of one score, a system is measured on a set of named properties that can each pass or fail independently. Those dimensions fall into two broad families — risk, which asks whether the system is safe to deploy, and performance, which asks whether it is any good to use.
Whether the system states things that are not supported by its sources or by fact. Measured against grounded references, not plausibility.
Whether outcomes or tone shift unfairly across demographics, languages or dialects when the underlying request is the same.
Harmful, abusive or unsafe content, including the subtler culturally specific forms that keyword filters miss.
Resistance to adversarial pressure: prompt injection, jailbreaks, social engineering and red-team attack patterns.
Whether the system can explain its reasoning and disclose its limits and its non-human nature when it should.
Whether personal data is requested, retained, repeated or leaked at any point in the interaction.
How much friction the user experienced getting what they needed. The strongest leading indicator of abandonment of automated channels.
The optimal path versus the path actually taken across a multi-step agent flow. Hard to judge without human review — and highly diagnostic.
Whether the conversation actually accomplished the user's goal, paired with first-contact resolution and escalation rate.
The slope of sentiment across a conversation rather than its average — catching exchanges that start neutral and end badly.
Whether the agent correctly understood what the user wanted before acting on it.
How often users have to rephrase themselves to be understood. A friction signal that needs no survey.
Capitalisation, profanity, repeated requests and explicit attempts to escape to a human.
Per-tool reliability in agentic flows that call functions, APIs or downstream systems.
The proportion of agent actions users had to undo or correct — the clearest harm signal in transactional agents.
Some dimensions belong to both families. Disengagement, conversational inconsistency and context misalignment read as risk indicators to a compliance team ("users abandoning the assistant signals a trust problem") and as experience indicators to a product team ("users abandoning the assistant signals friction"). The measurement is identical; only the framing and the decision it drives change.
Decide what must be measured and what counts as a pass, before any data is collected. Undefined thresholds turn results into opinion.
Curated cases with ground truth, red-team suites, and representative coverage of the languages, dialects and user situations the system will actually meet.
Replay real traffic, or use simulated users to produce controlled multi-turn conversations across personas and adversarial conditions at volume.
Apply deterministic metrics where a ground truth exists and rubric-driven model judging where it does not — with the judge calibrated against human labels.
Native-market reviewers adjudicate subjective, cultural and high-stakes cases, and audit a sample of automated scores to keep the judge honest.
Publish per-dimension scores with evidence, fix what failed, and re-run the same suite so improvements and regressions are directly comparable.
Underneath the classical metrics — accuracy, precision and recall, F1, ROUGE and BLEU for generated text, mean average precision for retrieval, perplexity for language models — the same principle holds: a metric is only as trustworthy as the ground truth behind it. See our guide to AI evaluation metrics for how to choose between them.
Token counts and latency charts say nothing about truthfulness, fairness or usefulness. Observability is necessary and not sufficient.
An LLM judge that has never been checked against human labels can be confidently wrong in exactly the places that matter most.
A single blended score hides the dimension that is failing. Report per dimension, per language and per segment.
Systems that behave well in English routinely degrade in other languages and dialects. Coverage has to match the deployment population.
Every individual response can be acceptable while the overall conversation still fails the user. Agentic systems must be judged end to end.
Models, prompts, tools and user behaviour all change. Evaluation is a cadence, not a launch gate you pass once.
Sigma AI runs independent evaluation programs for LLMs and AI agents: simulated users, calibrated model judging, curated and red-team datasets, and human-in-the-loop review by native-market experts across hundreds of languages and dialects. Risk and experience are measured by the same engine, so compliance teams and product teams work from one set of evidence.
Tell us what you're deploying, where, and to whom — we'll scope the evaluation with you.