Throughput is not progress
An author-run census of 1,979 agent-landed pull requests in a single-operator fleet, measuring what kind of bytes landed, who the record says wrote them, and who reviewed them.
Status: Result. Author-run census; dataset public.
Question
Between July and August 2026 I ran a private GitHub repository as a factory. One human, me, operated roughly a hundred git identities for six weeks, with AI seats claiming work by pushing branches and landing it through pull requests. Around 2,300 merged PRs and 50 or so landings a day is a good-looking dashboard. I wanted to know what those PRs actually were.
The question is not whether agent PRs get merged. Existing work already measures acceptance across many projects. My question is narrower: on one pinned repository, what was the composition of the PR stream, whose identity survived the merge, and who reviewed it? The estimand is composition and identity, not merge rate, and none of this is a causal claim.
Object and method
The object is thisisntjon/poketcg, a private repository that stays private, at pin 9522a8a37078d00f46b99a586b825b789b01387d. I took the first-parent history at that pin and called a commit PR-linked if its subject carried a PR number in the squash or merge form. Files and churn come from git log --numstat. Each PR-linked commit gets one file-shape label from the codebook: docs_only if every changed path is .md or .txt, has_code if at least one path is a source-code suffix, other if neither, and small if additions plus deletions are under 100. These labels describe bytes, not value. A stratified sample of 80 PRs, oversampling code, was then inspected by one person for intent, and a seeded sample of 40 PRs had its GitHub reviews pulled through the API. The kill bar was pre-registered: if the file list covered less than 80% of GitHub-merged PRs, the instrument was invalid. Observed coverage was 1,979 / 2,338 = 84.6%, so the instrument stands. The 15.4% gap is not imputed. The instrument, the pre-registration, and the full dataset are public in the seed-protocol repository under workflow/research/2026-08-24-pr-case-study/.
Findings
| Measure | Value | Qualifier |
|---|---|---|
| PR-linked first-parent commits | N = 1,979 | 84.6% of 2,338 GitHub-merged PRs |
docs_only | 855, 43.2% | prior lore ~2026-08-16: 44.7% |
has_code | 761, 38.5% | |
other | 362, 18.3% | |
| churn < 100 | 786, 39.7% | prior lore: 39.6% |
Bound if every missing PR were has_code | 855/2,338 = 36.6% docs-only | adversarial fill |
| Peak week, 2026-07-08 | 652 landings, 58.4% docs-only, 29.8% has_code | |
| Week of 2026-08-12 | 79 landings, 6.3% docs-only, 77.2% has_code | |
Unique names, git shortlog -sn --all | 103 | seat names live only off main |
| Unique names on PR-linked squash commits | 2 | |
| Independent human GitHub review, not author | 0/40 | Clopper-Pearson 95% interval 0 to 8.8% |
| PRODUCT in the 80-PR intent sample | 2, 2.5% | code-oversampled; an upper-ish glimpse |
The composition result is the plain one. On this pin, 43.2% of PR-linked commits changed only markdown or text files, and 39.7% had churn under 100 lines. Two figures that had circulated as campaign lore, 44.7% and 39.6%, reproduce. Churn is heavy-tailed: p50 = 146, p90 = 1,349, p99 = 8,879, max = 135,280. A mean of lines per PR is not an estimand here.
The time series is the part I find most useful. The figure below shows the weekly file-shape mix.
Source: artifacts/summary.json field weekly. Object: poketcg (private repository) at pin 9522a8a37078d00f46b99a586b825b789b01387d. Labels: docs_only = every changed path is .md/.txt; has_code = at least one .py/.go/.js/.ts/.rs/.java/.c/.cpp/.h; n = PR-linked first-parent commits that week (CODEBOOK.md). Author-run, not independently reproduced.
The week with the most landings, 652 in the week of 2026-07-08, was 58.4% docs-only and 29.8% code. The week of 2026-08-12 had 79 landings and was 6.3% docs-only and 77.2% code. The highest-volume week is the worst code-share week. When volume fell by an order of magnitude, code share rose. That is the measurable form of the title: throughput and construction moved in opposite directions.
Identity collapses at the merge. Across all refs, git shortlog -sn --all shows 103 names. On the PR-linked squash commits that make up main, there are 2. The seat names are real as protocol; the AI seats really did work under them. They are erased at squash and never appear on the PR’s GitHub user field. A miner reading GitHub identity would report a solo developer, and would be describing the same repository.
Review is the same story. In the seeded sample of 40 merged PRs, 17 had any review at all, 16 of those came from a bot connector, and the single human login was the PR author. Independent human review on GitHub was 0/40, with a Clopper-Pearson 95% interval of 0 to 8.8%. My own doctrine for the fleet was author XOR verifier. GitHub cannot operationalize that doctrine once squash and owner login are the public objects.
Finally, the intent sample. Of 80 inspected PRs, 2 changed the playing agent, the thing the whole campaign existed to improve. Evidence receipts were 29/80, governance plus ceremony were 30/80, and playing-agent edits were 2/80. Because the sample oversampled has_code, 2.5% PRODUCT is an upper-ish glimpse of population product share, not a lower bound. Even inside the code stratum, most of the Python was harness and evaluation, not the agent.
What it does not show
It does not show that documentation is waste. The playbook that came out of this campaign was captured as markdown, and a short conformance audit can be load-bearing. The veto is on volume as a proxy for strength, not on writing things down.
It does not show that the fleet had 103 independent agents, or that it had one. Both counts are artifacts of where you look.
It does not show that the PRs went unverified. It shows that verification, where it happened, left no trace in the GitHub review graph. Receipts and second-seat authors on branches are not this graph.
It does not show that merge volume implies, or fails to imply, product progress in general. It shows one factory where the dashboard went up while the product stood still, with the mechanism visible: a docs-majority firehose, identity collapse, bot reviews, harness code mistaken for agent code.
It does not support a higher or lower merge rate than anyone else’s. The estimand is different.
Limitations
This is n = 1 repository, n = 1 principal. It is a mechanism study, not a treatment effect, and it says nothing about external validity beyond this factory.
It is author-run, not independently reproduced. I built the instrument, ran it on my own repository, and inspected my own sample. An independent rerun of measure_pr_census.py on the same SHA is the next scientific bar, and until it happens this note should be read as a claim I am making about my own data.
The intent labels come from one inspector on n = 80. Exact agreement between the automatic pass and the human pass was 50/80 = 62.5%. A second coder could move PRODUCT by a few PRs. Given the file-shape census, they cannot move it to a majority.
Coverage is 84.6%, and the missing 15.4% of GitHub-merged PRs is not imputed. Under the adversarial assumption that every missing PR had code, docs-only would still be 855/2,338 = 36.6%.
The source mill, thisisntjon/poketcg, is a private repository and will stay that way. The public dataset is the artifacts/ folder in seed-protocol, pinned to 9522a8a37078d00f46b99a586b825b789b01387d. New merges on the source will move the rates. Cite the pin, not live HEAD.
Data
All files are in the public seed-protocol repository under workflow/research/2026-08-24-pr-case-study/.
- PAPER.md, the full case study with methods, tables, and threats to validity
- CODEBOOK.md, the label rules
- artifacts/summary.json, machine-readable results including the weekly series
- artifacts/pr_index.jsonl, one row per PR-linked commit