Operating a Retrieval-Augmented Classifier

Consider a design: a query gets embedded, retrieval pulls the nearest labeled examples from a vector store, and an LLM assigns a label from a fixed set. Mechanically this is kNN classification with an LLM head: the store is the training set, retrieval finds the neighbors, and the LLM replaces majority vote. The pattern shows up wherever a system classifies into a known taxonomy, and the framing carries a practical consequence: everything the instance-selection literature learned about maintaining kNN training sets applies to the store.

The design makes three assumptions, one about the query, one about the labels, one about the encoder. The query assumption: the text determines the label. A review that says only "not what I expected" could be about sizing, quality, or shipping, and no classifier recovers information that never made it into the text. The label assumption: the classes are well-defined, meaning trained humans shown the same query agree. Agreement also caps what you can measure, because the holdout is labeled by those same humans. Measure agreement first, and treat any accuracy target above it as a target on noise. The encoder assumption: proximity tracks the label, so neighbors in embedding space share classes. This one is only approximately true, and how true it is in your space is measurable.

Under those assumptions, a good row is definable by borrowing the conditions for a valid instrument, the checklist rather than the statistics. Relevance: queries of the row's class retrieve it. Exclusion: queries of other classes do not, and when they do, the row does not move their decisions. Exogeneity: the row is retrieved for its class content, not for surface attributes it happens to share with queries. None of the three can be observed directly: no log line says a row failed exclusion. A violation surfaces downstream, in the neighborhood some query received, so the evidence arrives as outcomes while the fixes happen at components. The essay follows the evidence: what broken neighborhoods look like, then the workups that trace each back to a broken condition, then what changes when the loop runs for months instead of once.

Symptoms: how failure presents

Retrieval hands the LLM a neighborhood, and a neighborhood can fail three ways.

Empty. The query is far from everything in the store. Retrieval returns rows, because it always returns rows, but none of them is close, and the LLM is effectively deciding without examples. The cause is one of three things: the query belongs to a class the store has never seen, it belongs to a known class phrased in a way no row matches, or it is off-topic traffic the system was never meant to handle. Which of the three it is determines the fix, and the triage below separates them.

Conflicted. The query is close to plenty of rows, but the rows disagree with each other about the label. Nothing is missing; the store just fails to pin down the boundary between confusable classes in this region. This is not a "no match" problem and adding more generic rows for either class does not fix it. It needs rows chosen to sit on the right side of the boundary.

Wrong. The query is close to rows that agree with each other and are incorrect for this query: a homogeneous neighborhood carrying the wrong label. Isolation looks fine, entropy looks fine, the decision is confidently wrong; nothing observable at decision time flags it, so only a labeled holdout catches it.

For each symptom the job in this layer is the same: build an instrument you can trust, localize what it finds, and route to the right workup. The fixes live with the components.

Empty neighborhoods first. An empty flag has two possible sources: the query really is far from everything, or the measurement manufactured the flag. "Empty" is nothing but a score plus a cutoff, and both can lie, so the first job is building both so that a flag means the first thing. Note where this work belongs: it is instrumentation, not retrieval. The retriever uses similarity ordinally, ranking rows within a single query, and any monotone rescaling leaves the neighborhood unchanged; the monitor uses it cardinally, comparing distances across queries, classes, and encoder versions, and cardinal use is where the raw number lies.

The score lies when it is raw cosine similarity. Similarities compress into a narrow band that differs by encoder and by class density: a dense class's typical max similarity might sit near 0.85 while a sparse class's sits near 0.65, so a single cutoff manufactures flags for the sparse class and misses real gaps in the dense one. The fix is to score by percentile. Distance to the kth nearest row is a standard out-of-distribution score (Sun et al., 2022), and referencing it against a holdout of queries the system handled correctly makes it comparable: a query at the 2nd percentile is more isolated than 98 percent of queries that worked, whatever the raw cosine reads. Keep the reference class-agnostic, because calibrating on the class the LLM assigned trusts an assignment that a genuinely novel query breaks. The residual cost, over-flagging sparse classes, comes back as labeling volume rather than wrong decisions, since the triage below sends those queries cleanly back to their class.

The cutoff lies when it is arbitrary. A convention like "bottom 5 percent" flags a fixed share of traffic whether or not that share correlates with anything going wrong. Tie it to outcomes instead: bin holdout queries by percentile, find where accuracy degrades, and put alpha there, because isolation is a risk proxy and its link to error is a fact about your deployment, not a theorem. One over-reading to resist: the percentile is not a per-query test with error control. It resembles a conformal p-value (Bates et al., 2023), but that guarantee assumes exchangeability, and drift, the thing being monitored, is exchangeability breaking. What the cutoff supports is a control chart: with a frozen reference, the flag rate sits near alpha while nothing changes and climbs when traffic moves away from the store.

With both built, a flag means real isolation, and the question becomes what to do with it. A flagged query on its own could be anything; a cluster of them is a gap. So cluster the flagged queries from a lookback window by their embeddings (HDBSCAN fits, since you do not know how many gaps exist and uncovered queries form variable-density clusters), discard singletons as noise, and rank surviving clusters by traffic-weighted size. Then triage each cluster by forcing its members through the LLM with the full set of classes available and reading the distribution of returned labels. Labels concentrating on one existing class with high confidence: a new phrasing of that class, routed to the auto-ingestion pipeline. Labels scattering with low confidence: off-topic traffic; track its volume, do not absorb it. Labels concentrating while the cluster's central queries fit no existing class on inspection: a candidate new class, and because a new class touches every system that uses the taxonomy, this bucket ends at a human, not a pipeline.

Conflicted neighborhoods need no new instrument. Label entropy across top-k and the margin between the top two classes are already relative quantities; threshold them by the same outcome-binning, finding the entropy above which holdout accuracy degrades. Localization is the pair: aggregate the flags by which two classes are contending, and the recurring pairs are the targets. The route is the separation workup, which builds rows for the boundary and tests whether rows can fix the pair at all or the encoder is the limit.

Wrong neighborhoods are what the first two checks cannot see, and the gap is structural: a novel or misfit query that a crude encoder places inside a close, confident neighborhood shows no isolation and no disagreement, just a wrong answer. Only ground truth catches it. Compute per-class recall on the labeled holdout, and within each low-recall class, pull the misclassified queries whose isolation and entropy both looked healthy. Those queries retrieved confident, coherent, incorrect neighborhoods, and they localize the problem: the rows they retrieved. The route runs through what those rows show: systematically mislabeled rows are a bad-rows problem; correctly labeled rows sitting where another class's queries land are a separation problem.

Hold the routes loosely, because the mapping is many-to-many: conflicted can also come from redundancy suppressing the entropy that should have flagged it, and wrong can come from confounded similarity rather than from any single bad row. And one component never appears in the routes: the head. Wrong decisions appearing across many classes at once while store signals stay healthy point at a prompt or model regression, visible as a jump in the LLM-neighborhood disagreement rate, and the fix is a rollback confirmed by replay, not anything done to the store. The workups follow: rows first, then geometry together with the encoder that shapes it, then selection policy.

Component workups

Each workup runs the same sequence: define the failure, detect it, act on the diagnosis, verify the action.

Bad rows

Rows break the setup's checklist in two ways, and the familiar failure modes sort into the two families. Exclusion failures are rows that push decisions toward wrong labels: mislabeled rows, overclaiming rows (the text genuinely maps to more than one class, and the single-label schema forces it to assert one, a deterministic claim the text cannot support), and stale rows (correct when added, wrong now). Same defect, three etiologies. Relevance failures are rows that do no work or crowd out work: orphans (never retrieved, zero relevance) and redundant rows (near-duplicates that add no marginal relevance, and under fixed-k retrieval do worse than nothing, as below). A toxic centroid is not a third family; it is either family plus high retrieval frequency in a dense region, a severity modifier that sets priority, not diagnosis.

The two families need different instruments, and the instruments sort by what data they consume.

Exclusion failures are caught by correctness-linked instruments, and those rest on one piece of infrastructure: for every production decision, log the query, the top-k row IDs retrieved, the LLM decision, and an eventual correctness signal (a labeled holdout match, a user correction, a downstream escalation flag). With the logs, the primary instrument is the damage score. For each row, compute retrieval count over a lookback window and conditional correctness, the fraction of decisions correct given the row appeared in top-k. Negative lift against the base rate means the row harms decisions on average; lift magnitude times retrieval count is the damage score, and sorting by it puts toxic centroids at the top regardless of etiology. The damage score is silent about why a row is bad; it identifies which rows are doing the most harm.

One store-side check catches exclusion failures before they fire, for rows too rarely retrieved to accumulate damage evidence, and it runs in two modes. The question is the same in both: does the row's neighborhood agree with its label? The cheap mode counts: if most of the row's nearest store neighbors carry a different label, the row is an island (rare phrasing of a real class) or a mislabel. The expensive mode asks: run the row's own text through the system with the row excluded, and treat LLM disagreement with the stored label as a flag.

Relevance failures need no correctness signal at all; retrieval statistics suffice. Orphans are rows with zero retrievals over a long enough window, read straight off the logs. Duplicates are found geometrically: pairwise cosine similarity within each class, flagging pairs above a threshold (0.95 to start; the principled calibration is the similarity at which two rows produce identical top-k membership across production queries, below which they are functionally one row). Cross-class near-duplicates, the same text under two labels, are not redundancy but a labeling conflict, detected the same way and remediated differently: resolve the taxonomy or relabel, do not thin.

Action follows diagnosis, family by family.

On the exclusion side, when the damage score and both modes of the store-side check agree, the row is confidently mislabeled: relabel if the right label is determinable, otherwise remove. Overclaiming rows take one more step, because their harm depends on who retrieves them. A generic text under one label is harmless if only equally generic queries retrieve it; the label works as a base-rate prior. The harm is spillover: generic short texts sit centrally in embedding space, so "not what I expected" is close to "not what I expected, runs two sizes small" and "not what I expected, box arrived crushed," queries whose class is knowable from their own words, and the generic row injects its coin-flip label into decisions that had better evidence. The damage condition is query specific, row generic.

So test who retrieves it: force-label those production queries with full class options. Labels concentrating on specific classes mean the queries carry their own signal and the row is pure noise; split it into versions that each pass exclusion ("not what I expected, runs small" under sizing, "seams tore in a week" under quality, "arrived damaged" under shipping), or just remove it, or store the honest soft mapping if the schema allows one. There is no right single label to find for a 60/40 text. Labels scattering with low confidence mean the queries themselves are generic: people really do write "not what I expected" and stop, and the next generic query arrives whatever you do to the store. The work moves off the store: have the LLM flag detected ambiguity (neighborhood label entropy is the signal), and let the deployment decide what a flag triggers, a clarifying question, a review queue, or a standing pseudo-class for the recurring cases.

Stale rows get maintenance machinery rather than a one-time fix: a timestamp and label-set-version tag on every row, re-checks when the taxonomy changes, and change-point detection on each row's "LLM agrees with this row's label" rate, since a sharp recent drop means the world moved on the row even if the taxonomy did not. Re-verify against current product state, relabel or retire.

On the relevance side, the remediation for duplicates is to thin, and the reason to bother is that under fixed-k retrieval duplicates actively distort through two mechanisms. Crowding out: k is a budget, so three near-copies in a top five leave the LLM three distinct rows of evidence rather than five, and the displaced rows may have carried the disagreement or the boundary information. Pseudo-replication: the head reads agreement among neighbors as independent corroboration, but three copies agreeing is one observation wearing three costumes, the effective sample size sits far below k, and the decision comes out more confident than the evidence. Duplicates also corrupt the diagnostics themselves: copies suppress the label entropy that detects conflicted neighborhoods, so a duplicated wrong row makes a wrong neighborhood look confident. That is a reason to dedupe even in stores where accuracy currently looks fine. Thin each duplicate cluster to its best row, merge the provenance onto the survivor, and keep the door shut going forward with a tight coverage-gain check at ingestion.

Orphans are the mildest case: tolerate them (cheap, keeps optionality if traffic shifts back) or drop them (saves space). The aggregate is more informative than any single row: a rising orphan count means traffic has moved away from part of the store, the same drift that a falling coverage rate shows from the other side.

Whatever the family, confirm before scaling. Leave-one-out replay pulls the suspect row, re-runs the decisions where it appeared in top-k, and checks whether accuracy improves. It is expensive, and it is the only test that directly answers whether removing the row helps. Run it fully for small suspect sets; sample it for large ones.

Poor separation

Neighborhood and bad-row work are about which rows are in the store. Separation is about how those rows lay out geometrically, and in the setup's terms it is where exclusion gets built; because the geometry is downstream of the encoder, this is also where the encoder gets audited. A class with one canonical, central row and many varied edge cases retrieves reliably. A class with only edge cases retrieves unreliably because its neighborhood is sparse and noisy. The remedy is deliberate seeding: prototypes that are unambiguously class A (strong relevance for A, clean exclusion for everything else), prototypes that are unambiguously class B, and discriminative pairs that enforce exclusion right where it is hardest, at the boundary between the two. Boundary rows buy more accuracy per row than anything else added to a store, because they resolve decisions the store was previously agnostic about. Production traffic supplies relevance, since rows mined from it match how queries are actually phrased. Seeding supplies exclusion. A store built only from traffic drifts toward ambiguous phrasings because that is what real queries look like; a store built only from prototypes has perfect exclusion and no relevance. You need both.

Four measurements, the first three of which measure exclusion and the fourth of which asks whether exclusion is achievable at all:

Per-row silhouette: compare each row's mean distance to other rows of its own class against its distance to the nearest row of a different class. Positive silhouette means the row is on the right side of the boundary. Negative silhouette means it is geometrically closer to another class than to its own and is mislabeled or an ambiguous boundary case. Aggregated by class, mean silhouette tells you which classes are well-separated.

k-NN label purity: for each row, look at its k nearest store neighbors and compute the fraction sharing its label. Per-class mean purity is a separation metric.

Inter-class confusion from retrieval: on a labeled query set, when a query of class A retrieves rows, what fraction of those rows are also class A? Off-diagonal terms identify class pairs bleeding into each other. The pairs with the largest mutual confusion are the priority targets for discriminative seeding.

Encoder ceiling: train a linear classifier on the embeddings of labeled data and evaluate on holdout. Its accuracy is roughly the best any store can do in this embedding space. A low ceiling means the encoder cannot separate the classes as defined, and the move is a better encoder or a finetune, not seeding. A high ceiling with low separation metrics means headroom, and seeding pays. This gap is what decides where to invest.

A crude encoder introduces a subtler failure than a low ceiling: confounded similarity. Embeddings represent class content and surface attributes together, so a query and a row can be similar because they share length, register, language, anger, or channel boilerplate rather than anything class-relevant. Retrieval then fires through the nuisance channel: the angry query pulls the angry row whatever its class, the query that opens with quoted forwarded text pulls rows that do too. This is the exogeneity condition from the setup failing, and the omitted-variable logic says when it bites: nuisance-driven retrieval biases decisions only if the nuisance also predicts labels among the rows. If every class has rows across registers, languages, and lengths, an angry query retrieves angry rows from several classes and the nuisance cancels in the neighborhood. The bias needs both legs, and stores acquire both by accident all the time: one class seeded from a single source inherits its boilerplate, another is disproportionately angry because that is the mood of the texts it was mined from.

The test is direct. Fit a shallow probe from surface features (length, language, sentiment, boilerplate markers) to row labels in the store. Accuracy meaningfully above chance is confounding exposure, and the informative features name the confounder. The row-level version: a row whose retrieval set is more homogeneous in some surface attribute than its class's traffic is being retrieved for the wrong reason. The fixes are to break either leg: balance the nuisance across classes, or remove it from the representation by normalizing text before embedding (strip IDs, salutations, boilerplate) or finetuning the encoder so the nuisance directions collapse. Correlated errors that cluster on a surface attribute rather than spreading as uniform noise are the production symptom that says this test is overdue.

Candidate sources for seeding: mine production cases the system handled correctly with high LLM-neighborhood agreement, since those are clean anchors. Ask the LLM to generate prototypical phrasings per class and have a domain reviewer keep the good ones. For confusable pairs specifically, ask the LLM to generate phrasings that are clearly class A and not class B (plus the mirror) and audit. And vary the surface attributes deliberately within each class (register, length, channel), because the confounding analysis above turns that from a nicety into a requirement: a class seeded in one voice is a class whose voice becomes its label.

Verify by re-measuring: silhouette, class purity, and the relevant per-class F1 should all move. If none does, the seeding was either redundant or below the encoder ceiling.

Selection policy

Everything above treats top-k by cosine as fixed. It is not; the neighborhood handed to the LLM is a policy with knobs, and each alternative retriever people reach for is a response to one of the failure types already described. The organizing rule for when to turn a knob: read-time policy masks, write-time hygiene fixes. A selection trick hides a store problem from the neighborhood without removing it from the index, the diagnostics, or the label risk, so policy is for the problems hygiene cannot reach.

Hybrid retrieval (lexical search fused with dense) answers encoder crudeness on rare tokens. Product codes, names, and domain jargon that an embedding smears into one region are exactly what exact match catches, so hybrid raises relevance where classes are distinguished by specific vocabulary. It buys nothing where classes are distinguished by phrasing and register, which is where dense retrieval earns its keep.

Cross-encoder reranking answers encoder-limited boundaries. A bi-encoder scores query and row independently and cannot model their interaction; rescoring the top candidates with a cross-encoder sharpens the boundary, raising the effective encoder ceiling for the neighborhood without retraining anything. Reach for it when the separation section's tests say the confusable pairs are at the bi-encoder's limit, and test it directly: replay the holdout with reranking on and check whether the confusable-pair errors move. The cost is latency on every query, paid forever, which is why it is a targeted tool and not a default.

Diversity selection (MMR, determinantal point processes) answers redundancy at read time and is the mask the rule warns about: it keeps duplicates out of the neighborhood while they continue to bloat the index, suppress the entropy diagnostics, and multiply label risk. Use it as a stopgap while the dedup runs, not instead of it.

Stratified retrieval, top-m per candidate class instead of global top-k, changes the head's task from reading the vote of a neighborhood to comparing the best case for each contender. In conflicted regions this is the difference between a decision made by local density, whichever class happens to dominate the area, and a decision made by explicit contrast with both sides' strongest exemplars present. It pairs naturally with boundary seeding: the seeded discriminative rows are exactly the strong cases stratification surfaces.

Operating over time

The two layers above describe one pass of a diagnostic loop: observe symptoms, work up components, apply fixes. Time turns the single pass into a different problem. The ground shifts under the system in four ways, each shift demands a response, the responses form a continuous stream of changes to the store, and the stream itself becomes a thing to govern: every change needs verification, and the add side grows past what hands can curate. Problems first, then the discipline, then the scaling.

Drift

Not every change is yours to schedule. Four kinds arrive on their own, and each has a response. Traffic drift is the chronic one: the query distribution moves as products, seasons, and users change, and coverage decays even while the store stands still. The response is the symptom layer run continuously; the isolation flag rate is the alarm, and the empty-neighborhood triage generates the rows to add, which is the stream the rest of this section governs. Taxonomy drift: classes get split, merged, renamed, or retired. Tag every row with its label-set-version and re-check on changes. Encoder drift: a new embedding model produces a different geometry under the same rows. Tag every row with its encoder version, and treat encoder changes as full migrations with offline replay on a fixed query set; hybrid embedding spaces produce nonsense similarities. Language drift: same text, different right answer over time. Catch it with change-point detection on per-row "LLM-agrees-with-this-row's-label" rates, the same signal used for staleness in the bad-rows workup.

Updates as experiments

Any store update is a change to the system, whether that change is a single human-labeled row or a batch auto-promotion of probationary rows. Replay a frozen evaluation set before merging. For changes large enough to matter, shadow the new store against live traffic for a day before flipping. Most teams skip the replay step and then spend the next standup debating whether quality moved. That debate is irrecoverable without a fixed reference. With one, the conversation is brief.

Scaling the adds

Traffic drift generates a standing queue of candidate rows: new phrasings from the empty triage, boundary candidates from conflicted regions, corrections from users. There is a ladder for processing it. Manual curation works at low volume and stops scaling quickly. Assisted curation, where the LLM proposes a label and a human approves, scales an order of magnitude further and is where many systems should stop. Auto-ingestion, a pipeline that labels and inserts with minimal human intervention, is the desired state, and it is only feasible under conditions worth stating plainly: a correctness signal has to exist (user corrections, a refreshing holdout) or probation is theater; the validation checks below have to be computable; and the insertion rate has to stay low enough to replay. Missing any of these, assisted is the honest ceiling.

The reason the conditions are strict is that an automated pipeline amplifies its own errors. If the LLM mislabels a query and you auto-insert, the next ambiguous query retrieves this bad row, gets the same wrong label, gets auto-inserted, and the error compounds. Feedback loops are real. The pipeline has to be designed against them.

The right framing is instance selection: the literature on modifying kNN training sets has been working on exactly this problem for fifty years, and the classical algorithms map directly. Edited Nearest Neighbor (Wilson, 1972) removes a point whose k nearest neighbors mostly disagree with its label, which is the neighborhood-coherence check from the bad-rows section. Condensed Nearest Neighbor (Hart, 1968) keeps only the points needed to correctly classify the rest, which is deduplication formalized. Tomek links flag pairs of opposite-class points that are each other's nearest neighbor, marking boundary cleanup candidates. The auto-add side should run on the same machinery as the pruning side.

The pipeline has six stages.

Trigger detection picks candidates from production logs. Several sources. Queries whose isolation percentile falls below alpha and whose force-labeling concentrates on a single existing class (the new-phrasing candidates from the triage above). Queries with high top-k label entropy where the LLM nevertheless reached a confident decision (boundary candidates). Queries flagged by user correction or downstream escalation (highest-quality candidates regardless of isolation).

Labeling assigns a class. Three modes. LLM auto-label with a high confidence threshold (cheap, fast, can be wrong). LLM auto-label queued for sampled human review (mid-cost, catches systematic errors). Direct human review (expensive, accurate, used for high-stakes classes or ambiguous candidates).

Validation applies the kNN checks, which are the setup's conditions made testable: coverage gain tests marginal relevance, and local purity and class margin test exclusion. A candidate query and its proposed class have to pass four checks before insertion.

Coverage gain: the candidate should not look just like a row that is already in the store under the same class. If it does, it adds nothing. Compute its max similarity to existing rows of the proposed class; the lower this is, the more new ground the candidate covers.

Local purity: the candidate's k nearest neighbors in the full store should mostly carry the proposed class. If they do not, either the auto-label is wrong or the candidate is genuinely a boundary case that should not be auto-added. Threshold around 0.6 and tune from there.

Class margin: the candidate should be measurably more similar to rows of its proposed class than to rows of any other class. This is the Tomek-link check from the other side: it makes sure the candidate does not sit right on the boundary between two classes.

Spread cost: adding the candidate should not stretch the proposed class's geometry past a threshold. If a class's 90th-percentile pairwise distance jumps when you add the row, the class is getting smeared across the embedding space and you are trading discrimination for coverage.

Damage simulation sits on top of the four: replay a labeled holdout twice, once with the candidate added and once without, and compare accuracy. Expensive but the only direct measure.

Provenance is metadata. Every inserted row carries its source (production-mined, human-labeled, LLM-generated, hand-seeded), labeling confidence, timestamp, and encoder version. Without provenance the store is unauditable.

Probationary period quarantines auto-inserted rows. They are flagged for some window during which their conditional correctness rate is watched. Promote to "full" status if they perform well, demote or remove if they cause damage. The probation prevents one mislabeled candidate from poisoning the store.

Safety limits cap the rest. Rate-cap auto-ingestion per day. Preserve class balance: refuse to auto-insert if it would skew the store toward one class. Dedupe before insertion. Replay a fixed evaluation set before any batch activation.

Auto-add and bad-row pruning are two sides of the same instance-selection loop: adding rows expands coverage, pruning preserves discrimination, and neither alone maintains both. An aggressive add policy needs an aggressive prune policy to match.

Metrics

The detection metrics named in the sections above are the diagnostic layer. Most of them do not need to be on a dashboard; they need to be computable on demand. The topline is a tighter set, chosen so that each metric catches a distinct failure and the dashboard stays small enough to read.

Macro-F1 on the rolling labeled holdout catches quality regressions, and because it weights every class equally, a collapsing rare class moves it where raw accuracy would not. Read it against inter-annotator agreement: the gap between F1 and agreement is the improvable part, and movement above the agreement line is noise.

Coverage rate, the share of queries whose isolation percentile clears alpha, falls when traffic drifts away from the store or the store ages away from traffic.

Out-of-distribution rate, the share of queries more isolated than even the uncovered tier (a second, stricter alpha on the same isolation percentile), rises when the system starts getting asked things outside its scope.

LLM-neighborhood disagreement rate, the fraction of decisions where the LLM picks something other than the top-k majority, rises slowly when the store gets noisy and jumps when a prompt or model change regresses.

Cost per decision, broken down at the source by embedding, retrieval, and LLM but reported as one number, catches spend creep before finance does.

P95 end-to-end latency catches the slow tail that p50 hides; reserve p99 for incidents.

Per-class F1 explodes the dashboard, but a "worst-performing class" line that names the current laggard belongs on it. Each topline metric needs a target, an alert threshold, and an owner; without a target, nobody can say whether the number is good.

If an executive view is needed, three of the six carry it: macro-F1, coverage rate, and cost per decision.

Closing

The failures described here share one property: none announces itself. A toxic centroid degrades a few hundred decisions a week in one region of traffic. A stale row keeps answering for a taxonomy that has moved on. An empty region of the store shows up as a slightly worse quarter, attributed to nothing. The store does not break; it erodes, and erosion is invisible without instruments. The instruments are cheap: a decision log with retrieved row IDs, a labeled holdout that refreshes, a replay harness, and a handful of per-row and per-class statistics computed from them. What they buy is the ability to locate a problem instead of debating it.