Agent security research | Simone Systems Research

Four risks and the work investigating them

Each example connects a practical risk to software or research that investigates it, with an explanation of what the evidence supports and what remains unresolved.

Where the four studies sit across an agent workflow, with each study's pinned artifact and boundary.

Summary of investigated failure modes, mechanisms, public artifacts, and scope limits.
Failure Mode	Mechanism	Public Evidence	Scope Boundary
Unapproved action drift	Bind approval to SHA-256 `action_hash` in SQLite	BigBoss v0.1.0 (399 tests, 4 CI jobs)	Requires compliant adapter; not an OS sandbox
Unreviewed agent PRs	Pre-registered PR commit and reviewer census	1,979 / 2,338 PRs (84.6% coverage, `9522a8a`)	One repository, one operator; 15.4% gap not imputed
Self-grading & shared error	Separate claim extraction and cross-provider re-derivation	Recorded physics run (`af8643c`, `1.24 keV` to `1.5 eV`)	Single question and run; single-provider falls back to author
Desktop host ACL failure	Add/remove/restore Windows profile ACL entries	Event 208 reversal (`0x80070005`)	n = 1 Windows 11 workstation; not independent replication

Agent actions that were never specifically approved

A blanket “yes” does not survive the agent changing its command afterward. BigBoss makes approval refer to one specific proposal: the proposed action, its workspace, and the policy version are hashed into an action_hash, and the adapter contract requires the executed action to match the hash that was returned. Policy routing decides what is automatically allowed, what enters review, and what is blocked: structured read-only actions can proceed, while file changes and shell commands enter review under the default policy. Decisions and events persist in SQLite, so the record outlives the terminal session that produced it.

Scope: enforcement depends on an integrated adapter honoring the contract. BigBoss is an approval workflow, not a complete sandbox for arbitrary processes on the machine. It is a prototype with one human authority per instance; release v0.1.0 passed 399 tests on Ubuntu and Windows.

Policy implementation approval store read the case study

Agent-written code that reaches a codebase without independent human review

To argue about review coverage for agent-written code you need a record of what landed and who signed off. Throughput is not progress is my census of 1,979 PR-linked commits on one repository at a fixed version, recording file composition, whose identity survived the merge, and which pull requests carried a GitHub review. The minimum coverage threshold was set before the study: below 80% coverage of GitHub-merged pull requests the measurement method would have been rejected. Observed coverage was 1,979 / 2,338 = 84.6%, meeting that threshold, and the 15.4% gap is not imputed.

Scope: the study measures the kinds of files changed and the author identities recorded, not merge rate. One repository, one operator, one pinned commit (9522a8a).

Codebook dataset and pre-registration read the study

A model that grades its own output

Repeated answers can share a mistake, so agreement alone does not settle correctness. The Council separates answering, peer critique, claim checking, and synthesis across model providers, and keeps each stage inspectable. Verifier selection prefers an author different from the one that produced the claim.

Scope: in a single-provider configuration, verification falls back to the same author. Peer ratings are model judgments, not measured correctness. The recorded live example illustrates the process; it is one question and one run.

Application repository a recorded run read the case study

Desktop agents and host access control

Desktop AI tools depend on Windows permissions. I ran two studies on one Windows 11 workstation, examining launch failures while two profile permission entries were added, removed and restored. They recorded whether the application opened after all its processes had stopped, along with Windows AppModel-Runtime events: Claude Desktop with permissions present, removed, then restored; removing them produced Event 208 with 0x80070005 at LaunchProcess, and ChatGPT Desktop, where an observed failed baseline preceded the controlled sequence.

Scope: n = 1 machine and one package each. Same-host consistency across two packages is not independent replication. The ACL command is an experimental intervention, not a validated general repair.

anthropics/claude-code#94384 openai/codex#45511 raw reversal record

What I still don’t know

Four things I can’t answer yet. Each says how to find out.

How much of what a connected coding tool does reaches BigBoss?
How to find out: list every action one coding tool can take, then count how many reach BigBoss.
Is agent code merged without independent human review followed by more bugs or reverts?
How to find out: use a repository where some agent changes get independent human review and some do not, and where later bugs and reverts can be seen. Follow each merged change.
How often does checking with an AI from a different provider change the verdict?
How to find out: run the same fixed questions twice, once checked by a different provider’s AI and once by the AI that wrote the answer, and compare verdicts.
Do two Windows permission entries decide whether these desktop AI apps open on other PCs too?
How to find out: someone other than me repeats the test on other PCs, starting from an app that already fails to open.

How this research connects to the Consumption Engine

I develop these agent-control tools alongside the Consumption Engine. Each study here retains its own scope; the evidence overview distinguishes implemented components, test runs, and open evaluations.

Everything above points at a repository, a dataset, or a public issue.