Skip to content

ADR-201: Findings assembled as classifier-ready assessment records

Summary

  • Decided: ways-audit assembles each finding and never decides it. For each claimed pair of way and control, it writes a record with the claim's criterion (satisfied_when), the firing evidence (fire count, session ids that point to transcripts, first and last seen), a tier (process when the claim has no criterion, outcome when it has one), and who assembled the record and when. The determination field, and the fields that record who filled it, when and on what basis, stay empty, and no code path in ways-audit writes them. A separate classifier, which this record neither designs nor constrains, fills them later. The ledger is append-only: each run appends a fresh snapshot of the rows.
  • Trades away: The finding record carries more structure than a bare verdict, and its schema must stay stable enough to serve as a dataset over time. Findings have no determinations until a separate effort supplies a classifier. The record rejected letting the tool write the determination, requiring a person to decide every finding, and storing only the verdict.
  • One-way? No. The record does not discuss reversal. It fixes a record shape and a boundary in ways-audit, and it leaves the classifier and any compaction of the ledger to later work.
  • Probes: Not confident (no-determination): Should ways-audit stay unable to write a determination? Yes or no. Not confident (append-only): Should the ledger keep appending snapshots instead of updating rows? Yes or no.
  • Inversion: At one end, the tool decides each finding itself. At the other, a person reads every transcript and decides. This sits between them: the tool gathers the inputs into a record with the decision left empty, for a classifier outside this design to fill.

Context

ADR-200 separated a compliance claim (a control-design assertion; SOC 2 Type I; OSCAL Component Definition) from an assessment finding (session-derived evidence with a determination of satisfied / other than satisfied; NIST SP 800-53A; OSCAL Assessment Results). It also required, in §1, that every claim carry a determination criterion — the observable session behavior that would let an assessor decide the finding. ADR-151 packages the tool that produces findings: ways-audit.

ADR-200 left one thing unresolved: who fills the determination, and when. Two tempting answers are both wrong.

  • The tool determines inline. If ways-audit reads a claim and its firing evidence and writes satisfied itself, it manufactures the very evidence ADR-200 forbids — a model grading whether a model followed model-injected guidance, stamped as fact. This recreates the ADR-013 overclaim at the finding layer.
  • A human always determines. If every finding must wait on a person to read the transcript and rule, the assessment surface is an unbounded labor sink. It does not scale past a handful of ways, and the corpus is designed to hold hundreds (ADR-200 §7).

The honest position is neither. The tool should assemble the finding — gather the inputs and leave the determination unset — producing a record that some other system will classify. That classifying system is deliberately out of scope: we do not design it, ship it, or constrain what it is. In NIST 800-53A the assessor is a role, not a person; generalize it to a classifier and leave it entirely external. ways-audit's whole responsibility is to make the record cleanly classifiable — well-structured and easily accessible — and then stop.

Decision

1. ways-audit assembles findings; it never determines them

The finding pipeline reads (claim + its determination criterion + firing events + transcript pointers) and writes a finding record with the determination slot empty. The tool's output is an assessable record, not an assessment. No code path in ways-audit writes a determination value — that boundary is the load-bearing honesty of this ADR.

2. An assembled finding is a classifier-ready record

Frame the finding record as one row of a supervised-classification dataset — features (the observable inputs) and a label (the determination) filled later:

Role Field Source
way, control the claimed (way, control) this row assesses claim sidecar (ADR-110)
feature criterion — the claim's satisfied_when (null if none) claim sidecar
feature evidence — total fire count, sessions (each id is a transcript pointer), first_seen / last_seen firing-event log
feature tier — process when there is no criterion, outcome when there is (ADR-200 §3) criterion presence
label determination — satisfied / other-than-satisfied, null until labeled a classifier
provenance assessed_by, assessed_at, basis — null until labeled the classifier, when it labels
provenance assembled_at, assembled_by — who built the row, when the assembler

Informally: the ledger is a dataset with an empty label column. The framing is not decorative — it is why the record stores the criterion and the raw evidence beside the empty determination, so a finding is classifiable and auditable, not a bare verdict.

The record also carries empty provenance slots — assessed_by, assessed_at, basis — so that when something does fill the label, what filled it and on what are recorded next to it. ways-audit writes those slots empty and never fills them.

3. The classifier is out of scope — a different, undesigned system

We do not design, ship, or constrain the system that classifies these records. It is a separate concern for a later day, and possibly a different tool entirely. This ADR fixes only the record and the boundary: ways-audit produces clean, accessible, classifiable rows and stops. Whatever eventually reads them — a deterministic check for a process-tier criterion, a human reviewer, a trained model for an outcome-tier one (ADR-200 §3) — is not our design problem here, and nothing in this decision presumes which it will be. The one guarantee we make is structural: the dataset is easy to get at and unambiguous to label.

Consequences

Positive

  • Resolves ADR-200's open question without either overclaim (tool-determines) or an unbounded human labor sink (human-only).
  • The determination boundary is enforced in code, not just prose: ways-audit has no path that writes a determination.
  • The scope is small and honest: the tool ships a dataset, not an assessment engine. Classification is someone else's problem, on someone else's schedule.
  • The ledger is dual-purpose at no extra authoring cost — an audit trail and a labeled- when-classified dataset — and stays useful even if the classifier is never built.

Negative

  • The finding record carries more structure (criterion + evidence + empty label and provenance slots) than a bare verdict, and the schema must stay stable enough to be a dataset over time.
  • A dataset with a perpetually-empty label column is inert until something classifies it; this ADR deliberately does not deliver that something, so findings have no determinations until a separate effort supplies a classifier.

Neutral

  • The classifying system is entirely out of scope — not designed, not stubbed, not constrained here. This ADR fixes only the record shape and the assemble-never-determine boundary.
  • satisfied_when moves from ADR-200 §1's "proposed, not yet in schema" note into the actual claim schema (ADR-151 §3) as part of realizing this ADR.
  • The ledger is append-only, like the firing-event log it is built from: each assemble --write appends a fresh snapshot of the rows (their assembled_at and evolving firing evidence distinguish them), rather than mutating prior rows in place. It is therefore a time series of assemblies, not a unique-keyed table; a consumer that wants current state reduces by (way, control) to the latest assembled_at. This keeps the writer a trivial, honest append and defers any compaction policy to the — out of scope — consumer.

Alternatives Considered

  • Tool writes the determination inline. Rejected: manufactures evidence; the exact error ADR-200 was written to stop, relocated one layer down.
  • Determinations are human-only. Rejected: an unbounded labor sink that cannot cover a corpus of hundreds; also forecloses the deterministic process-tier check that needs no human at all.
  • No stored criterion/evidence — store only the verdict. Rejected: a bare verdict is neither auditable back to its basis nor usable as classifier training data, discarding the record's second purpose for nothing saved.

References

  • ADR-200 — the claim/finding model this refines (§1 assessability, §2 findings, §3 the two evidence tiers).
  • ADR-151 — the ways-core claim schema and the ways-audit binary that assembles findings.
  • ADR-110 — the provenance.yaml claim sidecar the criterion is stored in.
  • NIST SP 800-53A — assessment findings and the satisfied / other than satisfied determination; the assessor as a role.
  • OSCAL Assessment Results — the machine-readable finding layer this record shape anticipates.