Written by Technical Team | Last updated 20.08.2026 | 21 minute read
One of the most dangerous questions in an AI project is also one of the most cos. Someone opens a prototype, types in a few sensible questions and receives impressive responses. A document is summarised accurately. An AI assistant retrieves the right policy. An agent successfully updates a record in a business system. The demonstration works, stakeholders are impressed and the temptation is to move towards deployment.
But a successful demonstration is not evidence that an AI system is ready for real users.
Generative AI systems are probabilistic. Give the same system slightly different wording, an unusual document, incomplete context or an unexpected combination of instructions and its behaviour can change significantly. A workflow that succeeds nine times may fail on the tenth. A chatbot that performs brilliantly on straightforward questions may confidently invent an answer when confronted with ambiguity. An AI agent may choose the correct tool but pass the wrong parameter. A retrieval system may find a relevant document while missing the paragraph that actually answers the question.
Traditional software testing is still essential, but it does not answer all of these questions.
This is where AI evaluations, or “evals”, become critical.
An eval is a structured way of measuring whether an AI system performs as intended. Rather than relying on demonstrations, intuition or occasional manual testing, evaluations turn expectations about AI behaviour into repeatable tests.
That sounds simple. In practice, it represents a fundamental change in how organisations need to think about AI quality.
The objective is not to prove that an AI system is perfect. Perfect performance is usually impossible. The objective is to understand where the system succeeds, where it fails, how frequently those failures occur, how serious they are and whether the resulting level of performance is acceptable for the intended business use.
An AI system answering internal questions about office procedures can tolerate a very different error profile from one recommending financial actions, extracting information from clinical documents or automatically making changes to customer accounts.
“Good enough” therefore cannot be defined by the model provider, a benchmark score or the enthusiasm generated by a prototype.
It has to be defined by the organisation deploying the system.
The AI industry has become accustomed to benchmark scores.
New models are routinely compared on reasoning, mathematics, coding, knowledge, multimodal understanding and other capabilities. These benchmarks are useful when comparing the general abilities of models, but they answer a different question from the one most businesses actually need answering.
A benchmark might tell you that one model performs better than another on a standard reasoning dataset.
It does not tell you whether your customer support assistant correctly interprets your returns policy.
It does not tell you whether your retrieval system selects the right sections of your technical documentation.
It does not tell you whether an agent reliably obtains approval before issuing a refund.
It does not tell you whether the system behaves appropriately when a user provides conflicting instructions.
And it certainly does not tell you whether the application generates enough business value to justify its cost.
This is why it is useful to distinguish between model evaluation and system evaluation.
Model evaluation measures the underlying model. System evaluation measures the application you have actually built around it.
For most organisations deploying AI, the second matters considerably more.
Consider a relatively straightforward AI knowledge assistant. The user asks a question. Before the language model responds, the application may identify the user’s permissions, rewrite the query, search a vector database, retrieve several documents, rank those documents, construct a prompt, send that prompt to a language model, apply output rules and finally display the answer.
If the answer is wrong, the model may not be the problem.
The correct information may never have been retrieved. An old version of the document may have outranked the current version. Metadata may have been missing. The prompt may have presented conflicting information. The user may not have had permission to access the source needed to answer the question. Or the model may simply have interpreted perfectly good evidence incorrectly.
Evaluating only the final model output hides these distinctions.
A mature evaluation framework therefore looks at AI as a system.
Depending on the application, that might mean evaluating factors such as correctness, completeness, relevance, groundedness, retrieval quality, tool selection, tool parameters, instruction following, safety, security, latency, cost and user satisfaction.
For an agentic system, the evaluation problem becomes more complex again.
Imagine an AI agent responsible for processing a supplier enquiry. It may need to interpret an email, search an ERP system, inspect previous orders, check stock availability, calculate a response, create an internal note and draft a reply.
There are multiple ways for that workflow to fail even if the final email looks plausible.
The agent could have searched for the wrong supplier. It could have selected the wrong product record. It could have read outdated stock information. It could have skipped an approval step. It could have taken five unnecessary actions when two would have been sufficient. It could have reached the right answer through an unsafe process.
This leads to one of the most important principles in AI evaluation:
Do not only evaluate whether the system reached the right answer. Evaluate how it got there.
That is particularly important as organisations move from AI systems that generate content towards AI systems that take actions.
A poor answer from a chatbot is inconvenient. A poorly evaluated agent with authority to modify business systems can create a real operational incident.
The hardest part of AI evaluation is rarely writing the test.
It is deciding what should be tested.
Many teams start from the technology. They ask which evaluation framework they should use, what metrics they should monitor or whether they should use an LLM as a judge.
Those are useful implementation questions, but they come later.
The first question should be:
What would have to be true for us to trust this system in the job we are asking it to perform?
The answer needs to be specific.
Suppose a business is developing an AI system to extract information from incoming supplier invoices. Saying that the system should be “accurate” is not sufficiently precise.
Which fields matter?
Is an incorrect invoice number equally serious as an incorrect bank account number?
Can the system leave a field blank when uncertain?
Should uncertain cases automatically be sent to a person?
What proportion of invoices need to be processed automatically for the project to be worthwhile?
How quickly must processing happen?
What does it cost per invoice?
A useful evaluation framework starts converting these operational expectations into measurable criteria.
For example, the organisation might decide that supplier name, invoice number and invoice date need extremely high extraction accuracy; bank details must never be changed automatically without validation; low-confidence invoices must be escalated; processing should complete within a defined time; and the overall cost per successfully processed invoice must remain below a specified threshold.
The evaluation now reflects the real job.
This becomes even more important for conversational AI because “good” is often multidimensional.
A customer service answer might be factually correct but unnecessarily verbose. It might accurately describe a company policy but fail to answer the user’s actual question. It might be helpful but reveal information the user should not have access to. It might give the right answer while using an inappropriate tone. It might answer perfectly but take 25 seconds to do so.
There is no single number called “AI quality” that captures all of this.
Instead, organisations need an evaluation scorecard.
The exact criteria vary by system, but it is useful to think in several layers:
Not every AI application needs every dimension.
The important point is that the criteria should come from the intended use of the system rather than from whatever metrics happen to be easiest to collect.
The next step is defining acceptable thresholds.
This is where organisations often discover that AI evaluation is as much a business and risk-management discipline as an engineering discipline.
Consider an AI assistant that drafts replies for employees.
Perhaps 90% high-quality drafts are perfectly acceptable because a person reviews every response before sending it.
Now imagine the same system automatically sending those replies to customers.
The model has not changed. The consequence of failure has.
The acceptable threshold should therefore change.
This is why human oversight should never be treated as a vague statement that “a person remains in the loop”. It changes the risk model of the system and should be incorporated explicitly into the evaluation.
A useful principle is that the amount of autonomy an AI system receives should be proportional to the evidence you have that it can exercise that autonomy safely.
If performance is uncertain, keep a human approval step.
If evaluations demonstrate strong performance across relevant scenarios, some decisions may be automated.
If production monitoring later shows deterioration, autonomy can be reduced again.
Evaluation therefore becomes more than a quality-control exercise. It becomes one of the mechanisms through which organisations can safely decide how much responsibility to give AI.
Once “good” has been defined, the next challenge is creating tests that tell you whether the system actually meets that standard.
A useful starting point is an evaluation dataset: a collection of representative examples against which different versions of the AI system can be tested.
Sometimes these examples have objectively correct answers. If an AI system extracts an invoice total, for example, the expected value can be recorded in advance and compared directly with the output.
Other tasks are inherently subjective.
There may be several excellent ways to summarise a document. A customer support response can be phrased differently while remaining equally useful. A research assistant may reach the same conclusion through different evidence.
That means AI evaluation usually requires a mixture of evaluation techniques.
Deterministic evaluation works when something can be checked using explicit rules. Did the system return valid JSON? Did it include the correct reference number? Did an agent call the permitted API? Was the response generated within the latency limit? Did the answer contain information from an unauthorised document?
These tests are fast, repeatable and relatively objective.
Human evaluation is valuable when quality depends on professional judgement. Subject matter experts can assess correctness, usefulness, tone, nuance or whether a recommendation would genuinely be acceptable in practice.
Human evaluation remains extremely important, but it does not scale indefinitely. It is expensive, slow and subject to differences between evaluators.
This has led to increasing use of LLM-as-a-judge evaluation, where one language model assesses the output of another against defined criteria.
For example, an evaluator model can be asked to determine whether an answer is supported by supplied evidence, whether it follows a policy, whether it fully answers a question or which of two candidate responses is better.
This can make qualitative evaluation significantly more scalable.
But an AI judge is still an AI system.
It can be inconsistent. It can misunderstand ambiguous criteria. It can favour particular writing styles. It can sometimes reward confident but incorrect responses. A poor evaluation prompt can produce misleading scores with extraordinary efficiency.
LLM judges should therefore be treated as measurement instruments that themselves require evaluation.
A sensible approach is to create a set of responses assessed by trusted human reviewers, then compare the automated judge’s decisions against those human judgements. Where they disagree, investigate why. Refine the rubric, examples or evaluator until its decisions align sufficiently well with the judgement you actually care about.
The goal is not to eliminate humans. It is to use human expertise where it adds the most value and automation where repetition makes it useful.
The quality of the evaluation dataset matters just as much as the evaluator.
A dataset containing 200 near-identical, straightforward questions can create the illusion of reliability while telling you almost nothing about how the application handles difficult situations.
Good evaluation datasets deliberately represent the variety and messiness of the real operating environment.
They should include normal cases, difficult cases, edge cases and failure cases.
For a knowledge assistant, that could include clearly answered questions, ambiguous questions, questions where relevant information appears across several documents, questions based on outdated terminology, questions for which no answer exists, conflicting documents, spelling mistakes and attempts to access information outside the user’s permissions.
For an AI agent, the dataset might include unavailable tools, malformed API responses, missing information, duplicate records, contradictory instructions, unexpected changes halfway through the workflow and requests requiring escalation to a person.
One particularly valuable source of evaluation examples is production failure.
Every meaningful failure discovered after launch should provoke a simple question:
Could we turn this into a test so that this exact class of failure cannot silently return?
When the answer is yes, the example should be added to the evaluation suite.
This creates an AI equivalent of regression testing. The system does not merely get fixed; the organisation accumulates institutional knowledge about the ways in which it can fail.
Over time, the evaluation dataset becomes a strategic asset.
It captures something no general model benchmark can capture: the organisation’s definition of acceptable AI behaviour in its own environment.
This also prevents one of the most common problems in AI development: improving one thing while accidentally making something else worse.
Perhaps a new prompt increases answer completeness but makes responses unnecessarily long. A new model reduces hallucinations but doubles cost. A new retrieval strategy improves complex questions while reducing performance on simple ones. A new agent instruction improves successful task completion but results in too many unnecessary tool calls.
Without a repeatable evaluation suite, teams tend to assess the latest change using whichever examples prompted the change in the first place.
That encourages local optimisation.
With evals, versions can be compared systematically across the broader set of behaviours that matter.
The need for deeper evaluation becomes particularly clear with retrieval-augmented generation, or RAG.
A RAG application combines information retrieval with generation. When a user asks a question, the system finds relevant information and provides it to a language model, which generates the answer.
A weak evaluation might simply ask: was the final answer correct?
A better evaluation separates the pipeline.
Did retrieval find the necessary source?
Were the most relevant sources ranked highly enough to be included?
Did irrelevant documents contaminate the context?
Was the answer actually grounded in the retrieved evidence?
Did the system appropriately refuse to answer when sufficient evidence was unavailable?
This decomposition is powerful because it changes debugging from guesswork into diagnosis.
If retrieval recall is poor, changing the language model may achieve very little. If retrieval is excellent but groundedness is weak, the problem may sit in the prompt or generation layer. If both are good but users remain dissatisfied, the problem may be the experience itself.
AI agents require an even richer set of evaluations because their outputs include actions as well as words.
Consider an agent that can search customer records, amend CRM fields, generate quotations and schedule follow-up activity.
You might evaluate whether it ultimately completed the user’s task, but that alone is insufficient.
The agent also needs to be tested for whether it chose appropriate tools, supplied accurate parameters, performed steps in a sensible order, avoided unnecessary actions, respected permissions, recognised when approval was required and stopped when it encountered conditions outside its authority.
For high-impact workflows, the distinction between outcome correctness and trajectory correctness becomes important.
Outcome correctness asks whether the final result was right.
Trajectory correctness asks whether the route taken to produce that result was acceptable.
Imagine an agent asked to change a customer’s correspondence address.
It identifies the correct new address and ultimately updates the record correctly. On outcome alone, the task passes.
But suppose the agent first attempted to modify a different customer’s record before correcting itself.
The final result is right. The trajectory is unacceptable.
The more authority AI receives, the more evaluation must examine process as well as output.
This also means test environments matter. AI agents should not be evaluated for the first time against unrestricted production systems. Organisations need ways to test tool use, permissions and transactional behaviour in controlled environments where failures cannot create real-world damage.
And evaluations cannot stop at deployment.
Pre-release testing is necessarily based on a sample of expected situations. Real users will eventually produce inputs the development team never considered.
They will use different language, omit information, combine tasks, misunderstand what the system can do and occasionally try to break it. Business data will change. Documents will be replaced. APIs will be updated. Models may change. User behaviour may shift as people become more familiar with the application.
This is why mature AI systems use both offline evaluations and online evaluations.
Offline evaluation happens against controlled datasets. It is useful during development, before a release and whenever prompts, models, retrieval strategies, tools or workflows are changed.
Online evaluation examines real production behaviour.
That does not mean manually reading every conversation. Instead, organisations can evaluate samples of production interactions, automatically score certain dimensions, monitor operational signals and flag unusual cases for human review.
Useful production signals might include:
Production monitoring performs another valuable function: discovering the unknown unknowns.
An offline test can only measure scenarios someone thought to include.
Real usage reveals things nobody anticipated.
The best teams therefore create a feedback loop between production and evaluation. Interesting failures become new test cases. New test cases improve development. Development produces a new version. The new version runs through the broader evaluation suite before deployment.
The evaluation framework becomes a living system rather than a one-off launch gate.
There is a natural temptation to treat AI evaluation as a technical specialism.
It involves datasets, metrics, automated judges, tracing, experiments and statistical comparison, so it can easily become the domain of developers and machine learning engineers.
But the most important evaluation decisions cannot be made by engineers alone.
Someone has to decide which mistakes matter.
Someone has to define acceptable risk.
Someone has to determine whether shaving two seconds from response latency matters more than improving accuracy by one percentage point.
Someone has to decide whether a particular class of decision can be automated or requires human approval.
Someone has to decide whether the business value generated by the system justifies its operational cost.
Those are organisational decisions.
For that reason, the strongest AI evaluation programmes bring together technical teams, domain specialists, risk or compliance stakeholders and the people who actually perform the work the AI is being introduced to support.
The domain experts are especially important.
An AI engineer can measure whether an answer resembles a reference answer. An experienced claims handler, accountant, clinician, engineer or customer service adviser can often tell you whether that answer would actually be useful in the real workflow.
This is also why generic claims about AI accuracy should be treated cautiously.
“95% accurate” sounds reassuring until you ask what the remaining 5% contains.
If those failures are evenly distributed minor imperfections, the system may be excellent.
If nearly every failure involves one rare but financially significant scenario, the same headline number might conceal an unacceptable risk.
Aggregate metrics can hide important failure modes.
Evaluation results should therefore be segmented.
How does the system perform on easy versus difficult tasks? New customers versus existing customers? Long documents versus short ones? Different product categories? Different languages? High-value transactions? Requests involving incomplete information?
The useful question is not simply “What is our score?”
It is “Where does the system fail, and does that failure matter?”
This becomes particularly important when comparing models.
The most capable model is not automatically the best model for a particular application.
Imagine Model A achieves a 94% task success rate at a relatively high cost and Model B achieves 92% at one fifth of the cost with substantially lower latency.
Which is better?
There is no universal answer.
If the system handles a high-risk decision where those two percentage points represent meaningful errors, Model A may be the obvious choice.
If the system generates low-risk first drafts which are always reviewed by an employee, Model B may create much better economics.
Evaluation makes those trade-offs visible.
The same applies to system complexity.
Teams building AI applications can easily add more retrieval stages, more agents, more prompts, more tools and more sophisticated orchestration because each additional component appears to solve a problem.
But complexity has a cost.
Every additional component can introduce latency, expense and another failure mode.
A well-designed evaluation suite allows teams to ask whether the additional complexity genuinely improves performance.
If a four-agent architecture performs no better than a single well-designed workflow, the simpler system may be superior.
This is one of the less obvious strategic benefits of evals: they help organisations avoid building AI by intuition.
Instead of debating whether a larger model, more elaborate architecture or new framework “feels better”, teams can test whether it actually produces an improvement against criteria that matter.
That changes the nature of AI development.
Prompt engineering becomes test-driven improvement rather than experimentation by anecdote.
Model selection becomes an evidence-based decision rather than a comparison of vendor marketing.
Agent design becomes measurable rather than mysterious.
Governance becomes linked to observed system performance.
And conversations with senior stakeholders become more useful because teams can discuss AI in terms of measurable capability, risk, cost and business value.
Ultimately, there is no universal evaluation score at which an AI system becomes “good enough”.
A brainstorming assistant and an autonomous payment agent should not be held to the same standard.
The right threshold depends on the task, the consequences of failure, the availability of human oversight, the ability to detect and reverse mistakes, the value created by automation and the organisation’s tolerance for risk.
What matters is that the threshold is deliberate.
Without evals, organisations are effectively making deployment decisions based on impressions.
The prototype looked good.
The model is highly rated.
The team tried twenty examples and most seemed fine.
No one has complained yet.
Those signals may feel reassuring, but they provide little protection once AI begins operating at scale.
A system used ten times in a demonstration can hide a one-in-a-hundred failure.
A system used 100,000 times cannot.
And as AI moves deeper into operational workflows, that distinction becomes increasingly important. The question is no longer simply whether a model can produce an impressive answer. It is whether the complete system can behave reliably enough, safely enough and economically enough to perform a particular role within the organisation.
That is what AI evals are ultimately for.
They provide a disciplined way to replace “it seems to work” with evidence.
They allow teams to define the behaviour they actually want, measure it repeatedly, detect regressions, understand failure modes and improve performance over time.
Perhaps most importantly, they provide the foundation for increasing AI autonomy responsibly.
If an organisation cannot explain how it knows its AI system is performing adequately, it probably does not yet have enough evidence to give that system more responsibility.
The organisations that become successful with AI will therefore not simply be the ones with access to the best models.
They will be the ones that become exceptionally good at defining what good looks like, measuring it and continuously improving the systems they put into production.
Because the real competitive advantage is not having an AI system that can do something impressive once.
It is having an AI system you can trust to do something valuable again and again.
Is your team looking for help with AI development and consultancy? Click the button below.
Get in touch