Back to Research Notes

Code-brick contracts and a read-only Document Atlas

Between October 12 and October 24, 2026, I tested Consumption Engine code bricks with executable test oracles in place of prose project rules. On ApprovalBench, source-blind rebuilds passed 15 of 15 held-out checks, against 9 of 15 with documentation alone. A read-only Document Atlas mapped a large private file collection.

Kind
Result, measured by the author
Published

On October 7, 2026, a 37-task study on real project records produced a negative result for prose project rules: a compiled prose guide (48.8%) and library retrieval tools (49.0%) did not beat an ordinary agent workflow (53.8%, p = 0.0177). Worse, when an agent read stale project documents on disk after a correction, its accuracy dropped from 88.2% to 35.3%.

Between October 12 and October 24, 2026 (private repository, commit 2694413), I worked with the team (the AI agents I direct) to test a different representation: replacing prose project summaries with AST-mapped code bricks paired with executable test oracles, alongside a read-only Project and Document Atlas across a large private file collection. All commit hashes, SHA-256 digests, and denominators are published in consumption-october-2026-summary.json.

What a code brick changes

A prose rule says what a module is supposed to do in English, but it cannot execute a test when code changes, and it cannot stop an agent from reading an older file on disk. A code brick binds four machine-checkable pieces into a single record (BRICKS.jsonl, schema code-brick-v1):

  1. Verified AST spans (UNITS.jsonl): exact file paths, line ranges, and SHA-256 span hashes (100% verified against the source tree).
  2. Feature boundary (FEATURE_MAP.json): which symbols belong to a feature (U_f), which belong to shared contracts (Shared), and which belong to sibling features (0 sibling leaks allowed).
  3. Executable test oracle (ORACLES/<feature>/): pytest checks synthesized without reading existing tests/, mutants/, or canaries/, graded by mutation-kill parity against the shipped test suite.
  4. Semantic staleness classification: when a span changes, the engine runs the brick’s executable oracle to distinguish cosmetic edits (COSMETIC_EDIT) and safe refactors (REFACTOR_PRESERVED) from behaviour-changing edits (BEHAVIOUR_CHANGED).
Prose project rules (October 7) versus executable code-brick contracts (October 24) On October 7, prose project rules scored 48.8% and 49.0% versus 53.8% for a plain file reader across 37 tasks. Between October 15 and October 24, executable code-brick contracts scored 15/15 vs 9/15 on ApprovalBench and 22/22 vs 11/22 on qudbud. Prose project rules (Oct 7, 37 real tasks) Full engine: 48.8% | Rules file: 49.0% Plain file reader baseline: 53.8% (p = 0.0177) Failure mode: agents still read stale files on disk (accuracy fell from 88.2% to 35.3% on dirty trees) No executable test oracle attached to claims Result: NULL / negative on open-ended prose Code bricks + test oracles (Oct 15-24) ApprovalBench (1 pkg): 15/15 vs 9/15 docs qudbud (second repo): 22/22 vs 11/22 docs Blind oracle kill parity: 100% on both Self-maintenance: 10/10 BC flagged, 0/10 false-stale Result: WIN on bounded code reconstruction
Comparison of the October 7 prose-rule study against the October 15-24 executable code-brick reconstruction experiments across two repositories.

Results across two codebases (P1a through P1C7)

I tested the code-brick pipeline with the team across two Python repositories:

Codebase & Phase Structure Blind Feature Mapping Blind Oracle Kill Parity Source-Blind Rebuild (Bricks vs Docs-Only) Self-Maintenance (BC / Cosmetic / Refactor)
ApprovalBench (P1a-P1c, 75261d7) 28 files (1 package), 3 features Jaccard = 1.000 (4/4 features) 37/37 (100.0% across 3 features) 15 / 15 (100%) vs 9 / 15 (60%) 10/10 BC caught, 0/10 cosmetic false-stale, 0/5 refactor false-broken
qudbud (P1C7, 68cd88a) 12 .py files, 115 AST units Jaccard = 1.000 (115/115 units) 42/42 (100.0% across 3 features) 22 / 22 (100%) vs 11 / 22 (50%) 10/10 BC caught, 0/10 cosmetic false-stale, 0/5 refactor false-broken

Mapping 883,486 local files

Alongside code bricks, the team and I ran a read-only Project and Document Atlas across 883,486 local files (84.5 GB, 122 top-level entries, 24,513 duplicate SHA-256 groups representing 33.74 GB reclaimable, 66,530 candidate docs, 34,394 deduplicated docs) while refusing secret and key paths before file open (never_read_opened = 0). A 5-project end-to-end intake (P-mvp1 to P-mvp4, commit 05f5d02) ingested 699 keeper files (69 exact duplicates skipped, 125 singletons included), verified 8,013/8,013 (100%) span SHA-256 hashes, and passed ce_storage verify (811 objects, 8,180 records, 0 errors).

Three other approaches missed their pre-registered gates. They stay in the record as negative findings:

  • Unsupervised TF-IDF + LSA topic clustering (P-docatlas3, commit a7bfae0, NULL near miss): Across 34,394 deduplicated documents (k = 30), clustering passed G1 (14 giant clusters <= 5%) and G2 (median cohesion = 0.209 >= 0.15), but scored 6/10 (z = 1.53) against the 8/10 blinded intruder-detection bar (G3). I rejected unsupervised topic labels in favour of deterministic path/project taxonomy plus SQLite FTS5 search.
  • Word-only feature location on unseen C and assembly (P-extract1, LOSS): Scored 0.0 line F1 across 5 unseen features because identifier vocabulary differed between prose specifications and implementation symbols.
  • RAM-only binary labelling (P-rom1, LOSS): Scored 3/28 (0.107) recall against a 0.50 bar, whereas structural routine matching (0.955) supported a 16/16 oracle-checked subsystem transplant (P-game1).

Limits and open questions

  • Storage expansion (2.316x): In the 5-project intake, 11.91 MB of raw source expanded to 27.58 MB of BRICKS.jsonl and 147.54 MB of derived CATALOGUE.sqlite indices.
  • Zero external reproductions (0 logged): All runs were executed on my own machine, in a private repository. Only the synthetic v1.0.0 fixture and October 2026 summary JSON are public.
  • Open prose-task parity: Code bricks establish a clear advantage on test-backed code reconstruction, while open-ended prose tasks remain bounded by the October 7 37-task result (48.8%/49.0% vs 53.8%) and +89% token overhead (September 30 study).

Inspect the evidence

File details on this page

Format, size and a short SHA-256 identifier, measured when this page was built.