Testing LLMs: Engineering Confidence Without Certainty

Software testing has always relied on a fragile assumption: that we can enumerate test cases that represent production behavior. We test specific inputs, verify outputs, and trust that production will behave similarly. This works until it doesn't. Search ranking degrades as user behavior evolves. Heuristics fail when traffic patterns shift. Data pipelines break when upstream APIs change. Even supposedly deterministic systems depend on environmental stability that we rarely measure or guarantee.

Large language models make this assumption untenable. They violate both prerequisites for traditional testing: determinism and bounded inputs. The same prompt produces different outputs across runs. Natural language creates infinite combinations of tasks, phrasings, and contexts that no test suite can cover. The gap between test and production, always present in classical systems, becomes qualitatively different and unmanageable with traditional methods.

To deploy ML systems, we need to move from a brittle notion of correctness to engineered bounds on risk. That shift rests on three pillars: constrain what the system is allowed to do, gather probabilistic evidence over representative scenarios, and analyze the space of fixes with a realistic view of what each lever can and cannot achieve.

There is also a third challenge that matters in practice. The ability to "fix" errors is limited because the core model largely sits outside the usual edit-compile-test loop. One cannot patch a function and be done. There are levers but often no precise patches.

What the two core changes imply

Non-determinism. Reproducibility weakens, and a single golden output is often the wrong oracle. Two consequences follow. First, evaluation benefits from properties that any acceptable answer should satisfy, rather than from one target string. Second, decision procedures can incorporate aggregation and checks: for example, agreement among multiple samples, or agreement between an answer and evidence that was retrieved for it. These mechanisms do not remove randomness; they use it to expose instability.

Unbounded inputs. Enumeration is impossible. Coverage becomes a statement about scenarios rather than about individual examples. A scenario is defined along axes such as task type, domain, language, input length, retrieval quality, and adversarial pressure. Evidence then concerns how well the system behaved in each scenario and how often real traffic occupies those scenarios. When the mix shifts, the original claims degrade in a known way, not an invisible way.

Constraints and properties: what counts as correct

When correct outputs cannot be enumerated for unbounded inputs, correctness shifts from point specifications to property classes. This requires a compositional approach where different property types guard against different failure modes.

The specification hierarchy. Properties form a natural hierarchy from syntactic to semantic:

1. Structural constraints (contracts). These enforce output well-formedness independent of input content. Schema validation, type constraints, and budget limits provide the narrowest but most reliable bounds. They catch malformed JSON, type violations, and resource abuse, but say nothing about semantic correctness.
2. Invariance properties (metamorphic relations). These specify how outputs should change—or not change—under input transformations. Key classes include:These properties test semantic stability without requiring ground truth for every input.
   • Equivalence invariants: Paraphrases should yield identical classifications
   • Monotonicity invariants: Adding gold evidence should not degrade answer quality
   • Consistency invariants: Formatting changes should not alter semantic outputs
3. Robustness properties (placebo/ablation tests). These verify that the model ignores irrelevant information and attends to relevant information:These properties directly probe for spurious correlations and over-reliance on superficial cues.
   • Placebo resistance: Adding distractor passages should not change answers
   • Sufficiency tests: Removing irrelevant context should preserve correct outputs
   • Necessity tests: Removing critical context should trigger abstention

Evidence under uncertainty: probabilistic claims and scenario coverage

With stochastic outputs, acceptance becomes probabilistic. A typical claim has the form: for a given scenario, the failure rate is at most $p$ with confidence $(1-\alpha)$. The required sample size follows from standard bounds. For instance, with zero observed failures, the so-called rule of three gives an approximate 95 percent upper bound of $3/N$. Targeting 1 percent at 95 percent confidence suggests on the order of 300 independent trials. Tighter targets require more samples. When failures are observed, interval estimates such as Wilson or Clopper–Pearson communicate uncertainty more honestly than point estimates.

Independence and multiple comparisons. Two statistical issues threaten validity:

1. Dependence between trials inflates effective sample size. Near-duplicate inputs, temporal correlation, and shared random seeds violate independence assumptions. Mitigate by varying prompts, seeds, time windows, and users.
2. Multiple testing degrades system-level confidence. Testing $m$ scenarios at individual confidence $(1-\alpha)$ yields system confidence of at most $(1-\alpha)^m$. Apply Bonferroni correction, false discovery rate control, or hierarchical testing to maintain overall error rates.

Scenario coverage through grids. Since exhaustive testing is impossible, partition the input space into scenarios along interpretable axes: task type, domain, language, input length, retrieval quality, and adversarial pressure. This scenario grid serves three purposes:

1. Targeted sampling: Populate each cell with diverse test cases including paraphrases and adversarial variants
2. Production alignment: Track what fraction of real traffic maps to each scenario
3. Drift detection: Monitor when traffic distribution shifts across scenarios

Maintain a repository of production incidents as ground truth. Include red team inputs proportional to their expected occurrence: jailbreaks, prompt injections, confusable characters, tool-abuse patterns, and long-context stressors.

Quantifying and pricing risk

Risk is the product of frequency and cost. The analytical move is to separate routine loss from tail risk and to make both explicit.

1. Taxonomy and costs. A failure mode can be described by severity, detectability, and expected remediation. Examples include unsupported factual claims that slip past verifiers, harmful or non-compliant content, correct-looking but wrong summaries, tool misuse by an agent, and excessive abstention that degrades experience. Catastrophic modes invite strong constraints such as human approval, reduced privileges, deterministic decision rules, or prohibition.
2. Targets from tolerance. If a failure mode has average cost $C$ and acceptable expected loss over a period is $L$, then a target failure probability for that period is $L/C$. The evaluation plan then seeks to show, per scenario, that the upper bound on the failure rate is below that target at a chosen confidence level.
3. Tail constraints. Expected loss is appropriate for routine error. Rare, high-severity events are better handled by hard constraints and fences: scoped credentials, limited budgets, and multi-person approval for irreversible actions.
4. Abstention as a priced action. Abstaining is neither free nor a last resort. If $C_{\text{error}}$ is the cost of a wrong answer and $C_{\text{abstain}}$ is the cost of declining to answer, then a confidence threshold can be chosen to minimize $p_{\text{error}}(\tau) \cdot C_{\text{error}} + p_{\text{abstain}}(\tau) \cdot C_{\text{abstain}}$. The optimal threshold depends on the scenario mix and should be revisited as that mix changes.
5. Confidence and calibration. Raw model "confidence" is not reliably calibrated. Practical proxies include agreement across multiple samples, agreement with an independent verifier, the presence and quality of grounded citations, entropy where available, and similarity between retrieved evidence and the answer. A simple calibrator, such as logistic or isotonic regression, can map these signals to empirical probabilities within each scenario. Post-deployment calibration curves then indicate whether the system's probability statements match observed frequencies.

For high-risk features, some teams adopt a safety case: a short argument that states the claims, links them to evidence, and lists the controls and playbooks that apply in production. The value is not ceremony but clarity about what is being claimed, about where the evidence lives, and about how the system behaves when assumptions fail.

Intervention strategies when the model is largely immutable

The immutability of the base model forces a fundamental question: where in the system can we intervene to address different error types? The answer depends first on understanding the causal mechanisms behind failures, then mapping those mechanisms to appropriate intervention points.

Errors in LLM systems stem from distinct causes. Some are merely structural—correct content in wrong formats, malformed JSON, inconsistent schemas. Others reflect missing context: the model lacks information that exists but wasn't retrieved or was poorly framed. A third class involves task misspecification where the model misunderstands what's being asked. Deeper still are knowledge errors where the model's parametric knowledge is wrong or absent, reasoning errors where it makes invalid inferences despite having correct information, and calibration errors where confidence systematically misaligns with accuracy. Finally, alignment errors violate intended policies or safety constraints.

These error classes map naturally to a hierarchy of intervention points, with a key trade-off: interventions closer to the output are safer but less powerful, while those closer to the model are more powerful but riskier. At the surface, output constraints (schemas, grammars) can enforce structural correctness but cannot touch semantic errors. Moving inward, context manipulation (retrieval, prompts) addresses information gaps and task misunderstanding, but these fixes prove brittle—what works for today's prompt distribution may fail tomorrow. External validators can detect and reject any error type they're programmed to recognize, but they cannot generate correct answers, only identify wrong ones. At the core, model adaptation (fine-tuning, RLHF) can shift fundamental behavior, but with risks of regression, reward hacking, and lost calibration.

This structure suggests a selection principle: use the minimal intervention that can address the error's causal mechanism. Format violations need only output constraints. Hallucinations in the presence of available evidence point to retrieval or verification fixes. Systematic overconfidence may require fine-tuning with calibration data. The principle is conservative by design—deeper interventions carry compound risks that accumulate across the system.

The practical tool is a fixability matrix mapping error types to intervention points, scored by effectiveness, cost, and risk. This matrix reveals an uncomfortable truth: many errors have no clean fix at any layer. The immutability constraint forces a separation between detecting errors (possible through validation for any error type) and correcting them (requires matching intervention to cause). We're not debugging in the traditional sense but building compensating controls around a black box we cannot directly modify.

What emerges is a pragmatic philosophy: treat the model as a powerful but flawed component, surround it with the minimal machinery needed to bound its failures, and maintain clear documentation about which errors can be fixed, which can be detected, and which must be accepted as inherent limitations.