One Record, Many Assertions

Agent evaluations tend to acquire a new expectation type for every capability: messages, tool calls, state changes, delegations, UI actions, memory, artifacts, audit logs. The result is usually a fragmented test system in which closely related requirements are represented and evaluated differently.

A cleaner design separates three artifacts:

trial record   what happened
contract       what must hold
result         what the evaluator concluded

The trial record should be independent of the particular assertions that will later inspect it. A useful core is:

S0   relevant state before the run
T    timestamped events during the run
Sf   relevant state after the run
D    normalized changes from S0 to Sf

D is derived, but worth materializing. Many agent requirements are frame conditions: the requested change occurred and nothing else changed. Querying a normalized delta is simpler and more reliable than reconstructing that claim separately in every assertion.

The trace is a timestamped event log. Each event has a common envelope and a typed payload:

event:
  event_id: ...
  wall_time: ...
  monotonic_time: ...

  actor_ref: ...
  surface: ...
  audience: ...
  turn: ...
  phase: ...

  kind: message | tool_call | delegation | mutation | ui_event | artifact | audit
  event_type: ...

  entity_refs: [...]
  operation_ref: ...
  correlation_id: ...
  caused_by_event_id: ...

  status: ...
  payload: ...

The envelope supports selection and attribution. The payload contains the domain-specific content of a message, tool call, UI event, or mutation.

Several fields do more than provide metadata. Monotonic time supports reliable ordering and elapsed-time checks. Actor, surface, and audience distinguish what different participants did and could observe. Logical entity references tie events to the business object in the scenario rather than to a fixture-specific identifier. Operation and causal references connect related events exactly.

That last point matters for audit evaluation. If a successful mutation requires an audit event, the evaluator should verify that both records name the same operation, actor, and resource. A semantic judge should not be asked whether two loosely similar descriptions “seem related.” Correlation IDs group a workflow; causal or operation references establish attribution.

The record should also model UI state semantically. Initial UI state belongs in S0, interactions in T, final UI state in Sf, and normalized changes in D. Assertions can then address element identity, role, value, visibility, enabled state, ordering, and accessibility attributes. Geometry or screenshots remain available when visually relevant, but they are poor default representations of UI behavior.

The contract remains separate from this evidence. Its stable outer form is:

select → extract → test

with an optional condition controlling when the assertion applies.

In practice, important assertions often relate more than one selection. A message may need to agree with final state; an audit event may need to correspond to a mutation; a fallback may need to follow a denial within a time bound. Named bindings make that relation explicit:

assertion:
  id: ...

  when: ...

  bind:
    <name>:
      from: trace | initial_state | final_state | delta
      where:
        actor_ref: ...
        surface: ...
        audience: ...
        turn: ...
        phase: ...
        kind: ...
        event_type: ...
        entity_ref: ...
        operation_ref: ...
        correlation_id: ...
      extract: <path> | count | first | last | set(<path>)

  require:
    quantifier: every | some | none
    op: equals | contains | matches_schema |
        count_at_most | precedes | follows_within |
        has_matching | matcher
    left: <binding or value>
    right: <binding or value>
    on: [...]
    within: ...
    matcher: exact | structural | semantic |
             grounded_semantic | custom
    grounding: [...]

  severity: critical | high | medium | low

The schema is richer on the evidence side than on the assertion side. That is deliberate. The record must represent what a real agent system does; the assertion language only needs to select evidence and state the relation that should hold.

A relatively small set of relations covers the common cases:

• predicates over initial state, final state, or delta;
• existence, absence, counts, and frame conditions;
• equality and consistency between selected values;
• precedence, duration, and response within a time bound;
• structural validation of outputs and artifacts;
• semantic judgment grounded in other recorded evidence;
• universal claims such as “every successful mutation has a matching audit event.”

For example, an audit requirement can be expressed structurally:

bind:
  mutations:
    from: trace
    where:
      kind: mutation
      status: succeeded

  audits:
    from: trace
    where:
      kind: audit

require:
  quantifier: every
  op: has_matching
  left: mutations
  right: audits
  on: [operation_ref, actor_ref, entity_ref]
  within: 2s

Other audit requirements operate on the same evidence. The audit surface may prohibit card numbers, access tokens, or other sensitive content even when the corresponding internal operation legitimately handled them. A stronger check can use the trace as grounding and ask whether the audit record contains enough information for a reviewer to reconstruct why an action was taken or refused.

No separate audit assertion framework is needed. Nor does UI, memory, delegation, or artifact generation require one. New capabilities extend the trial record; reusable requirements may add operators over that record.

The result is a third artifact rather than an annotation written back into either the record or the contract:

assertion_result:
  assertion_id: ...
  status: pass | fail | not_applicable | invalid | evaluator_error

  observed: ...
  expected: ...
  evidence_refs: [...]

  evaluator:
    type: deterministic | semantic | custom
    version: ...
    score: ...

  explanation: ...

The statuses should remain distinct. fail means the evidence violates the contract. not_applicable means the assertion’s condition was false. invalid means the assertion could not be evaluated, perhaps because the harness did not record a required surface. evaluator_error means the judge or custom matcher failed.

An empty selection should not silently pass unless emptiness is precisely what the assertion requires. Every verdict should point back to the event IDs, state paths, or delta entries used to produce it. A semantic judge is itself part of the evaluator and should carry a versioned model, prompt, threshold, and evidence rendering.

This architecture does not solve every evaluation problem. It cannot assert evidence that was never recorded. Counterfactual claims require another execution or an explicit model of what would have happened. Pass rates, latency distributions, pass@k, and release gates summarize multiple trials and therefore belong above the single-run contract. Some complex requirements will still need custom code.

Those limits do not weaken the common grammar. They locate the missing piece. A requirement that cannot be expressed usually points to one of three gaps: the harness did not record the evidence, the schema cannot select it, or the evaluator lacks the required predicate.

The design principle is therefore not that every future requirement can be anticipated. It is that messages, actions, state, time, UI, delegations, artifacts, and audit data can share one evidence model—and that requirements over them can share one assertion grammar.