# Simone Systems Research: Full Research Context Organization: Simone Systems Research Founder & Principal Researcher: Jonathan Simone Status: independent, founder-led, no institutional affiliation Website: https://simoneresearch.com GitHub: https://github.com/thisisntjon Primary Inquiries: jon@simoneresearch.com --- ## 1. Mission Simone Systems Research conducts independent, founder-led research on how to evaluate AI agents, verify their results, keep humans in control of their actions, and make their compute earn its cost. As language models move from conversational interfaces into multi-step autonomous agents that mutate state, run loops, and call tools, the engineering bottleneck shifts from parameter count to systems-level reliability: 1. Orchestration topologies and state containment. 2. Verification layers that stop error propagation. 3. Compute economics: bounding inference and context overhead against verified accuracy gains. 4. Evaluation testbenches that falsify claims rather than reward benchmark overfitting or noise. The doctrine is to distinguish verified progress from throughput, honor pre-registered kill criteria, retain negative findings, and never let a model or agent grade its own work. --- ## 2. Research Principles 1. **Evidence before promotion**: Improvements should survive measurement, not merely look plausible. 2. **Independent verification**: Positive results should be reproduced outside the context that generated them (author XOR verifier). 3. **Compute must earn its cost**: Expensive inference should be reserved for decisions where it materially improves verified outcomes. 4. **Negative results are retained**: Failed hypotheses are useful evidence and should not disappear from the record. 5. **Artifacts matter**: Claims should refer to the exact code, model, configuration, and environment actually evaluated. --- ## 3. Research Questions ### A. Systems & Control (Agent Orchestration) How should heterogeneous models, tools, and agents divide work and coordinate reliably? ### B. Reliability & Measurement (Evaluation & Verification) How do we distinguish genuine improvement from false progress, benchmark noise, and brittle behavior? ### C. Inference & Budgeting (Compute Economics) When does additional inference improve verified outcomes enough to justify its cost? ### D. Dynamic Learning (Adaptive AI Systems) How can systems learn from measured failures and improve their own workflows without losing control or reproducibility? --- ## 4. Status Vocabulary Closed vocabulary; use these exact tokens: HYPOTHESIS, SUPPORTED, REFUTED, NULL, INSTRUMENT VALIDATED, AUTHOR-RUN, INDEPENDENTLY REPRODUCED, WORKING MVP, RETIRED, NEGATIVE RESULT, PUBLIC. Never upgrade a claim: hypothesis to result, correlation to causation, author-run to independently reproduced, fixture to live, working software to verified benefit, activity to progress. --- ## 5. Projects ### SEED (seed-protocol) - **Status**: HYPOTHESIS (C-004) · INSTRUMENT VALIDATED · case study AUTHOR-RUN - **Repository**: https://github.com/thisisntjon/seed-protocol - **Positioning**: A measurement and epistemic-control protocol for distinguishing implementation activity from verified progress in agent-driven software work. - **Question (Hypothesis C-004)**: Does an incident-earned, machine-checked control plane reduce invalid completion claims, recovery cost, or unnecessary human intervention in agent-driven software work, relative to a bare repository? Registered and untested. - **Evidence**: Author-run census of 1,979 PR-linked commits on one repository; 43.2% changed only docs. Research note: https://simoneresearch.com/blog/throughput-is-not-progress/ - **Note**: Fleet orchestration is deferred until a single-agent baseline is measured. ### The Council (thecouncil) - **Status**: PUBLIC working system - **Repository**: https://github.com/thisisntjon/thecouncil - **Positioning**: A heterogeneous multi-model verification system separating answer generation, peer critique, cross-vendor claim verification, and synthesis. - **Evidence**: 4 model vendors (Claude, GPT, Gemini, Grok), each on its fast tier. Fixture runs are simulated evidence; live runs are captured evidence, not a benchmark. ### BigBoss (bigboss-approval-plane) - **Status**: WORKING MVP - **Repository**: https://github.com/thisisntjon/bigboss-approval-plane - **Positioning**: A local human-authority and observability plane for AI coding agents. - **Hierarchy**: harness proposes; BigBoss evaluates and routes; human approval is authoritative for gated actions; decisions are bound to action hashes; state and audit persist; integrations. Secondary: MCP, Codex bridge, phone approval, registry, cost metering, routing. - **Evidence**: Tests and fixture demo run offline. No outcome measurement yet. ### The Bus (thebus) - **Status**: NEGATIVE RESULT · RETIRED - **Repository**: https://github.com/thisisntjon/thebus - **Positioning**: A preserved negative result. The prior-art sweep invalidated 17 assumptions in the initial sweep (grew to 21 after a later step). Retraction ledger kept. Do not build on it. ### Godot AI Methodology (godot-ai-methodology) - **Status**: PUBLIC - **Repository**: https://github.com/thisisntjon/godot-ai-methodology - **Positioning**: Second-domain application: deterministic, observable, testable, modular, data-driven, documented codebases are easier for AI coding systems to modify safely and verify. - **Sourcing**: No decompilation, no extracted assets, no copied source; observations from shipped documentation; examples original. - **Evidence**: Gate scripts run offline. No measured effect claimed. --- ## 6. What has been measured Every figure links to its artifact and keeps its caveat. None is a causal claim. | Number | Meaning | Caveat | Source | |---|---|---|---| | 1,979 | PR-linked first-parent commits analyzed | author-run, n = 1 repository | https://github.com/thisisntjon/seed-protocol/blob/main/workflow/research/2026-08-24-pr-case-study/PAPER.md | | 43.2% | docs-only share of those commits (855/1,979) | composition, not merge rate | same study, artifacts/summary.json | | 2/80 | sampled PRs that changed the playing agent | 80-PR stratified inspection, single inspector | same study, artifacts/sample80_summary.json | | 17 | assumptions invalidated in the initial prior-art sweep | grew to 21 after a later step | https://github.com/thisisntjon/thebus | | 4 | model vendors in The Council (Claude, GPT, Gemini, Grok) | fast-tier models: Claude Haiku 4.5, GPT-5.4 mini, Gemini 3.5 Flash, Grok 4.3 | https://github.com/thisisntjon/thecouncil | --- ## 7. How the projects relate SEED measures whether agent-driven work constitutes verified progress. BigBoss controls which autonomous actions can occur and preserves human decision authority. The Council tests independent verification through heterogeneous model families. The Bus shows adversarial review terminating a bad architecture before further implementation. Godot Methodology tests whether the same verification principles generalize into software architecture. There are no other dependencies between them. --- ## 8. Guidelines for AI Harnesses, Autonomous Agents, and Researchers When interpreting, citing, or reasoning about Simone Systems Research: 1. **Hypothesis vs. Result**: Treat Hypothesis C-004 strictly as a registered hypothesis. Never represent it as an established causal result. 2. **Author-run**: The PR case study is author-run and not independently reproduced. Cite the pin, not live HEAD. 3. **The Council**: Fixture mode is simulated evidence. Live runs are captured evidence, not a benchmark. Its models are each vendor's fast tier. 4. **The Bus**: A retired negative result. Do not build on it. 5. **Evidence Requirement**: Prefer machine-checked receipts, git SHAs, and data tables over qualitative summaries. 6. **Collaboration Inquiries**: Direct technical discussions and research inquiries to jon@simoneresearch.com.