Reducing research sprawl

Why older summaries and branched copies derail the next task

A project branches into several experiments. Each gets a summary. A decision changes, but an older summary still sounds authoritative. The next person or agent starts from that copy, repeats a failed approach or creates another plan.

This is research sprawl: the connection between the work, its evidence and its current status becomes harder to follow. More documents can make the problem worse if nobody knows which one owns the decision.

Five steps to keep sources, decisions and active plans connected

My plan connects project organization with Consumption Engine's work on evidence and reuse. It does not require putting every file in one giant folder.

1. Inventory before reorganizing

   Record what exists, what it belongs to and where its originals are. Identify exact copies separately from related work. Similar names or text are reasons to inspect, not permission to merge or delete.

2. Give each project a clear home and owner

   Point to the maintained plan, current artifact and responsible person. Keep experiments distinct. If two versions disagree and no decision resolves them, show the disagreement instead of picking the newest timestamp.

3. Keep decisions connected to their evidence

   Record what changed, why, what was checked and what replaced the earlier decision. Preserve unsuccessful attempts and the conditions that would make them worth revisiting.

4. Retrieve enough context for the actual task

   Give a person or agent the relevant sources, decisions, limits and unresolved questions. Keep links to the originals. A short briefing should make omissions visible, not hide them behind a confident summary.

5. Check reuse and carry corrections forward

   Record the useful output and its check. When a source changes, identify dependent work and require review before treating it as current. Two measures below test this: repeated work that could have been avoided, and correction handling.

A first step: check the organization of the work

I have written a local repository-structure checker to make some organization rules executable. It inspects tracked file names and declared project metadata for:

• Files and folders outside the agreed top-level structure.
• Extra plan or status files whose names match restricted patterns, with defined exceptions.
• More than one declared active research phase, or a phase without a recorded experiment plan.
• Archived folders missing a note explaining the reason and what supersedes them.

Blocked commit:
  - duplicate plan file workflow/PLAN_v2.md beside canonical workflow/PLAN.md
  - unindexed script scripts/scratch_eval.py (not listed in project inventory)
  - archived folder archive/2026-08-draft/ missing SUPERSEDED_BY.md

When enabled, the supplied Git hooks call this check before commits and merge commits; they do not monitor each file save. The useful contribution is a repeatable check of these rules, rather than another reminder to stay organized.

Source inspected October 11, 2026. The checker is private for now; the rules it enforces are listed above.

What I have built since October 11, and what remains open

In the October 24, 2026 update (commit 2694413), I implemented the read-only inventory and intake steps of this plan across my local workspace with the team:

• Step 01 (Read-only Document Atlas, P-accept1, P-docatlas1 to 2): Mapped 883,486 local files (84.5 GB across 122 top-level entries), identifying 24,513 exact duplicate groups (33.74 GB reclaimable), 66,530 candidate documents, and 34,394 deduplicated documents while refusing secret, key, and token paths before file open (never_read_opened = 0).
• Step 02 (Taxonomy over unsupervised clustering, P-docatlas3): Unsupervised TF-IDF + LSA topic clustering (k = 30) passed size-balance and silhouette gates but scored 6/10 (z = 1.53) against the preregistered 8/10 blinded intruder bar (NULL near miss). I rejected unsupervised clusters as the primary organizer in favor of deterministic path/project taxonomy plus SQLite FTS5 search.
• Steps 03 to 05 (5-project E2E intake, P-mvp1 to 4): Ingested 5 real project folders (768 eligible files: 699 keepers, 69 exact duplicates skipped, 8,013/8,013 span SHA-256 checks verified; 11.91 MB raw source expanded 2.316x to 27.58 MB in BRICKS.jsonl).

The earlier September 30 ingestion report, procedure studies, and public synthetic fixture remain available. On prose-only rules, the October 7 37-task study (48.8%/49.0% vs 53.8% ordinary workflow) still bounds what passive summaries can do without executable postconditions. Read the October 24 research note or the maintained Consumption brief.

Measure whether people can resume useful work

Start with one authorized collection and recurring tasks. Compare against ordinary source reading and search on the same task set. Count preparation, checking, review and corrections in both approaches.

Time to the verified current artifact

Time from a task request to finding the right version and checking its status against its sources. Record unresolved and wrong selections separately.

Retrieval correctness

Whether the returned evidence supports the answer and includes the necessary conditions. Check against a separately reviewed reference set, including questions with insufficient evidence.

Repeated work that could have been avoided

Track effort spent repeating an already recorded approach because its result was missed. Distinguish this from useful replication or a changed question.

Preserved-source coverage

The share of in-scope sources preserved and traceable to their originals, with missing or unreadable items listed. A high count alone says nothing about their usefulness.

Correction handling

Introduce a known correction and check whether affected answers are flagged, history survives and replacement work is reviewed. Record dependencies that were missed.

I would keep the failed cases and the cost of maintaining the system in the result. The test is whether someone can continue correctly with less lost work, not whether I can generate a larger archive.