ADR-201: Findings assembled as classifier-ready assessment records¶
Summary¶
- Decided:
ways-auditassembles each finding and never decides it. For each claimed pair of way and control, it writes a record with the claim's criterion (satisfied_when), the firing evidence (fire count, session ids that point to transcripts, first and last seen), a tier (process when the claim has no criterion, outcome when it has one), and who assembled the record and when. Thedeterminationfield, and the fields that record who filled it, when and on what basis, stay empty, and no code path inways-auditwrites them. A separate classifier, which this record neither designs nor constrains, fills them later. The ledger is append-only: each run appends a fresh snapshot of the rows. - Trades away: The finding record carries more structure than a bare verdict, and its schema must stay stable enough to serve as a dataset over time. Findings have no determinations until a separate effort supplies a classifier. The record rejected letting the tool write the determination, requiring a person to decide every finding, and storing only the verdict.
- One-way? No. The record does not discuss reversal. It fixes a record shape and a boundary in
ways-audit, and it leaves the classifier and any compaction of the ledger to later work. - Probes: Not confident (no-determination): Should
ways-auditstay unable to write a determination? Yes or no. Not confident (append-only): Should the ledger keep appending snapshots instead of updating rows? Yes or no. - Inversion: At one end, the tool decides each finding itself. At the other, a person reads every transcript and decides. This sits between them: the tool gathers the inputs into a record with the decision left empty, for a classifier outside this design to fill.
Context¶
ADR-200 separated a compliance claim (a control-design assertion; SOC 2 Type I;
OSCAL Component Definition) from an assessment finding (session-derived evidence
with a determination of satisfied / other than satisfied; NIST SP 800-53A; OSCAL
Assessment Results). It also required, in §1, that every claim carry a determination
criterion — the observable session behavior that would let an assessor decide the
finding. ADR-151 packages the tool that produces findings: ways-audit.
ADR-200 left one thing unresolved: who fills the determination, and when. Two tempting answers are both wrong.
- The tool determines inline. If
ways-auditreads a claim and its firing evidence and writessatisfieditself, it manufactures the very evidence ADR-200 forbids — a model grading whether a model followed model-injected guidance, stamped as fact. This recreates the ADR-013 overclaim at the finding layer. - A human always determines. If every finding must wait on a person to read the transcript and rule, the assessment surface is an unbounded labor sink. It does not scale past a handful of ways, and the corpus is designed to hold hundreds (ADR-200 §7).
The honest position is neither. The tool should assemble the finding — gather the
inputs and leave the determination unset — producing a record that some other
system will classify. That classifying system is deliberately out of scope: we do
not design it, ship it, or constrain what it is. In NIST 800-53A the assessor is a
role, not a person; generalize it to a classifier and leave it entirely external.
ways-audit's whole responsibility is to make the record cleanly classifiable —
well-structured and easily accessible — and then stop.
Decision¶
1. ways-audit assembles findings; it never determines them¶
The finding pipeline reads (claim + its determination criterion + firing events +
transcript pointers) and writes a finding record with the determination slot empty.
The tool's output is an assessable record, not an assessment. No code path in
ways-audit writes a determination value — that boundary is the load-bearing honesty of
this ADR.
2. An assembled finding is a classifier-ready record¶
Frame the finding record as one row of a supervised-classification dataset — features (the observable inputs) and a label (the determination) filled later:
| Role | Field | Source |
|---|---|---|
way, control |
the claimed (way, control) this row assesses |
claim sidecar (ADR-110) |
| feature | criterion — the claim's satisfied_when (null if none) |
claim sidecar |
| feature | evidence — total fire count, sessions (each id is a transcript pointer), first_seen / last_seen |
firing-event log |
| feature | tier — process when there is no criterion, outcome when there is (ADR-200 §3) |
criterion presence |
| label | determination — satisfied / other-than-satisfied, null until labeled |
a classifier |
| provenance | assessed_by, assessed_at, basis — null until labeled |
the classifier, when it labels |
| provenance | assembled_at, assembled_by — who built the row, when |
the assembler |
Informally: the ledger is a dataset with an empty label column. The framing is not decorative — it is why the record stores the criterion and the raw evidence beside the empty determination, so a finding is classifiable and auditable, not a bare verdict.
The record also carries empty provenance slots — assessed_by, assessed_at,
basis — so that when something does fill the label, what filled it and on what are
recorded next to it. ways-audit writes those slots empty and never fills them.
3. The classifier is out of scope — a different, undesigned system¶
We do not design, ship, or constrain the system that classifies these records. It is a
separate concern for a later day, and possibly a different tool entirely. This ADR fixes
only the record and the boundary: ways-audit produces clean, accessible,
classifiable rows and stops. Whatever eventually reads them — a deterministic check for a
process-tier criterion, a human reviewer, a trained model for an outcome-tier one
(ADR-200 §3) — is not our design problem here, and nothing in this decision presumes
which it will be. The one guarantee we make is structural: the dataset is easy to get at
and unambiguous to label.
Consequences¶
Positive¶
- Resolves ADR-200's open question without either overclaim (tool-determines) or an unbounded human labor sink (human-only).
- The determination boundary is enforced in code, not just prose:
ways-audithas no path that writes a determination. - The scope is small and honest: the tool ships a dataset, not an assessment engine. Classification is someone else's problem, on someone else's schedule.
- The ledger is dual-purpose at no extra authoring cost — an audit trail and a labeled- when-classified dataset — and stays useful even if the classifier is never built.
Negative¶
- The finding record carries more structure (criterion + evidence + empty label and provenance slots) than a bare verdict, and the schema must stay stable enough to be a dataset over time.
- A dataset with a perpetually-empty label column is inert until something classifies it; this ADR deliberately does not deliver that something, so findings have no determinations until a separate effort supplies a classifier.
Neutral¶
- The classifying system is entirely out of scope — not designed, not stubbed, not constrained here. This ADR fixes only the record shape and the assemble-never-determine boundary.
satisfied_whenmoves from ADR-200 §1's "proposed, not yet in schema" note into the actual claim schema (ADR-151 §3) as part of realizing this ADR.- The ledger is append-only, like the firing-event log it is built from: each
assemble --writeappends a fresh snapshot of the rows (theirassembled_atand evolving firing evidence distinguish them), rather than mutating prior rows in place. It is therefore a time series of assemblies, not a unique-keyed table; a consumer that wants current state reduces by(way, control)to the latestassembled_at. This keeps the writer a trivial, honest append and defers any compaction policy to the — out of scope — consumer.
Alternatives Considered¶
- Tool writes the determination inline. Rejected: manufactures evidence; the exact error ADR-200 was written to stop, relocated one layer down.
- Determinations are human-only. Rejected: an unbounded labor sink that cannot cover a corpus of hundreds; also forecloses the deterministic process-tier check that needs no human at all.
- No stored criterion/evidence — store only the verdict. Rejected: a bare verdict is neither auditable back to its basis nor usable as classifier training data, discarding the record's second purpose for nothing saved.
References¶
- ADR-200 — the claim/finding model this refines (§1 assessability, §2 findings, §3 the two evidence tiers).
- ADR-151 — the
ways-coreclaim schema and theways-auditbinary that assembles findings. - ADR-110 — the
provenance.yamlclaim sidecar the criterion is stored in. - NIST SP 800-53A — assessment findings and the satisfied / other than satisfied determination; the assessor as a role.
- OSCAL Assessment Results — the machine-readable finding layer this record shape anticipates.