ADR-108: Embedding-Based Way Matching with all-MiniLM-L6-v2¶
Summary¶
- Decided: Way matching uses the all-MiniLM-L6-v2 sentence embedding model instead of BM25. Way descriptions are embedded once when the corpus is built and their vectors stored in
ways-corpus.jsonl. At run time oneway-embedprocess, built on llama.cpp's GGML library, embeds the prompt and compares it with each stored vector. The model file, about 21 to 44 MB, downloads to the XDG cache directory and is checked against a committed SHA-256 sum. BM25 stayed as a fallback at the time; ADR-125 later removed it and made the embedding model required. - Trades away: Users download a model file of 22 to 44 MB that git does not track, and building the binary depends on llama.cpp. The model's understanding is fixed by its training and may miss project-specific senses of words such as "way", which BM25's explicit vocabulary handled. Thresholds need recalibration to the cosine scale. The record rejected larger models, ONNX Runtime, Python inference, fine-tuning for now, and hybrid scoring.
- One-way? No. The record does not discuss reversing it. At the time BM25 remained as a fallback and the corpus format was additive, so the earlier path stayed available until ADR-125 removed it.
- Probes: Not confident (model-download): Is a separate download of about 20 to 40 MB acceptable for every install? Yes or no. Not confident (off-the-shelf): Should the project keep the stock model rather than fine-tune it on way-matching data? Yes or no.
- Inversion: At one end matching uses word overlap with no download and no model. At the other it uses a large model that understands meaning well. This decision sits near the small-model end: the smallest sentence model the record found to separate meanings, in a single static binary.
Amendment note (2026-04-17): ADR-125 removed BM25 entirely. The "BM25 remains as a fallback" decision below is historical — the embedding model is now the sole retrieval tier. Sections describing the engine-selection cascade (
way-embed→way-match→ NCD), BM25 fields in the corpus, and the "fallback chain is automatic" behavior no longer apply.
Context¶
ADR-014 introduced BM25 for semantic way matching. ADR-107 Phase 1 shipped an external corpus for correct IDF computation across 58 ways. A vocabulary audit removed 7 ambiguous terms and reduced false positives from 8 to 4 on a test prompt ("write about Dwarf Fortress and AI agent simulation").
The remaining 4 false positives expose a fundamental BM25 limitation: bag-of-words matching operates on stems, not meaning. "Agent" in SSH (ssh-agent) and "agent" in AI (autonomous agent) share the same stem. "Document" in docstrings and "document" in "write a document" are identical tokens. No amount of vocabulary tuning can fix this — the model has no concept of word sense.
The ways system now has 62 ways across 5 domains (softwaredev, itops, meta, writing, research). As the corpus grows, BM25 vocabulary collisions will increase. Each new domain introduces common terms that overlap with existing ways.
Meanwhile, the prompt-evaluation loop must remain imperceptible to the user. The current BM25 path spawns 58 processes per prompt (~120ms). Any replacement must be faster, not slower.
Evidence¶
Tested 2026-03-21 in an empty directory with a creative writing prompt:
| State | Ways fired | True positives |
|---|---|---|
| Before corpus + audit | 8-9 | 1 |
| After corpus + audit | 4 | 1 |
The 3 remaining false positives (docs, docstrings, subagents) all match on stem collisions that BM25 cannot disambiguate.
Decision¶
Replace BM25 with embedding-based semantic matching using all-MiniLM-L6-v2 as the primary way-matching engine. BM25 initially remained as a zero-dependency fallback; ADR-125 subsequently removed BM25 entirely, making the embedding model the sole retrieval tier.
Model choice: all-MiniLM-L6-v2¶
- Parameters: 22M (6 transformer layers, 384-dim embeddings)
- License: Apache 2.0
- GGUF size: ~44MB (F16), ~22MB (Q5_K_M), ~21MB (Q4_K_M)
- Why this model: Smallest viable sentence embedding model with strong semantic discrimination. Battle-tested in production search systems. Pre-converted GGUF available on HuggingFace (second-state/All-MiniLM-L6-v2-Embedding-GGUF).
Architecture: pre-compute + single-spawn runtime¶
Way embeddings are computed at authoring time and stored in ways-corpus.jsonl. At runtime, only the user prompt is embedded — one forward pass, then cosine similarity against pre-computed vectors.
Authoring time (like generate-corpus.sh today):
Runtime (called by match-way.sh, replaces 58 way-match pair calls):
Timing budget¶
| Operation | BM25 today (58 spawns) | Embedding (1 spawn) |
|---|---|---|
| Process spawns | 58 × ~2ms = ~116ms | 1 × ~2ms |
| Model load | N/A | ~5ms (GGUF mmap) |
| Scoring | <1ms each (in-process) | ~12ms forward pass + <1ms cosine sims |
| Total | ~120ms | ~22ms |
The embedding path is 5-6x faster than the current BM25 path while providing dramatically better discrimination. (Measured on Linux x86_64, 2 threads.)
Binary packaging¶
Build on llama.cpp's GGML library (pure C tensor computation, no dependencies). Two files:
bin/way-embed— static binary (~5MB), built with cosmocc for cross-platform APE- Model weights downloaded to
${XDG_CACHE_HOME}/claude-ways/user/(~44MB F16, ~22MB Q5_K_M)
The binary loads the model via mmap (no full read into RAM). The model is distributed via GitHub Release artifact or direct HuggingFace download, verified against a committed SHA-256 checksum. It lives in the XDG cache directory, not in the git repo (too large at 44MB).
Future option: llamafile-style single binary with model weights concatenated into the executable. Deferred until cosmocc builds of llama.cpp stabilize.
Corpus format evolution¶
ways-corpus.jsonl gains an embedding field:
{
"id": "writing",
"description": "Content creation — documents, presentations...",
"vocabulary": "write draft compose...",
"threshold": 2.0,
"embedding": [0.023, -0.041, 0.118, ...]
}
BM25 fields (threshold, tokenized vocabulary) were retained in the corpus at the time of this ADR to serve the fallback path. ADR-125 removed the BM25 engine; these fields are no longer read at runtime.
Configuration (superseded)¶
Superseded by ADR-125: embedding is the sole tier; no fallback cascade.
Originally, the scanner detected which engine was available:
- If
bin/way-embedexists and model file present → use embedding - Else if
bin/way-matchexists → use BM25 with corpus - Else if gzip + bc available → use NCD (legacy fallback)
Per ADR-125, the embedding model is a hard dependency. If the embedding engine is unavailable, matching does not degrade — it errors. This surfaces setup problems early rather than silently returning degraded results.
Security boundary¶
Same constraint as ADR-107: runtime scanners never write to ~/.claude/. The model file and pre-computed embeddings are authoring-time artifacts committed to the repo. The runtime binary only reads.
The model file is a published, checksummed artifact from HuggingFace. It can be verified against known hashes. It does not execute arbitrary code — GGUF is a tensor format, not an executable format.
Consequences¶
Positive¶
- Solves the stem-collision problem that BM25 cannot fix. "SSH agent" and "AI agent" will have distant embedding vectors despite sharing a stem.
- 15x faster than current 58-spawn BM25 path. Imperceptible to the user.
- Pre-computed embeddings mean runtime cost is independent of corpus size. 200 ways would cost the same as 58.
- Multilingual potential without per-language stemmers or vocabulary files. MiniLM handles many languages out of the box (trained on multilingual data). ADR-107 Phase 3 locale work becomes simpler.
- BM25 vocabulary tuning becomes unnecessary. New ways need only a good description — no manual keyword curation.
Negative¶
- 44MB model file download (F16) or ~22MB (Q5_K_M). Not git-tracked — distributed via GitHub Release or HuggingFace. Users run
make modelto download. - llama.cpp / GGML dependency for building the binary. More complex build than the current single-file
way-match.c. Though the binary itself remains dependency-free once compiled. - Model quality is fixed at MiniLM's training — it may not perfectly capture domain-specific semantics (e.g., "way" as a concept in this system vs. "way" as a path). BM25's explicit vocabulary handles this better for known edge cases.
- Quantization (Q4) trades quality for size. Need to validate that Q4 embeddings still discriminate well enough on the test fixture corpus.
Neutral¶
way-matchbinary and BM25 scoring initially remained in the repo as the fallback. ADR-125 subsequently removed both, as the fallback was rarely exercised once the multilingual model was reliably distributed.- The
ways-corpus.jsonlformat was initially additive — embedding vectors sat alongside BM25 fields. Per ADR-125, the BM25 fields are no longer read at runtime; the corpus format will drop them when convenient. - The
/ways-testsskill needs to learn to score with embeddings (cosine similarity thresholds are on a different scale than BM25 scores). - Threshold values will need recalibration. BM25 thresholds (1.8-2.5) don't apply to cosine similarity (0.0-1.0). The test fixture corpus provides the calibration data.
Alternatives Considered¶
- Larger embedding models (Snowflake Arctic, Qwen3-0.6B): Better quality but 1.2GB+ model files. Overkill for matching 60 short descriptions against short prompts. MiniLM's 22M parameters are sufficient for this task scale.
- ONNX Runtime instead of GGML/llama.cpp: Would require shipping a shared library (~15MB onnxruntime.so). Less portable than a static binary. GGML is pure C with no dependencies, matching our existing build pattern.
- Python-based inference (fastembed, sentence-transformers): Adds a Python runtime dependency. Non-starter for a system that runs on bash + coreutils + static binaries.
- Fine-tuning MiniLM on way-matching data: Would improve domain-specific discrimination (training data exists: 74 test fixtures + 58 way descriptions). Deferred — try the off-the-shelf model first. Fine-tuning is ~$5-20 on cloud GPU if needed.
- Hybrid scoring (BM25 + embedding combined): Run both engines, combine scores. More complex, harder to calibrate, and the embedding model alone should handle all cases BM25 handles plus the ones it can't. Keep it simple — one engine primary, one fallback.
- Keep BM25 and accept the false positives: The vocabulary audit reduced false positives from 8 to 4, but the remaining 3 FPs are structural BM25 limitations. As the corpus grows past 100 ways, vocabulary collisions will worsen. The problem doesn't stabilize — it compounds.