Give an answer more than one kind of review

Checking claims separately from reviewing answers

A convincing answer can contain a mistake even when its conclusion is right. Asking another model to rate the whole answer may leave that mistake untouched. I wanted an application where reviewing an answer and checking its individual claims are separate jobs.

I built The Council with the team (the AI coding agents I direct): a React interface, an API that coordinates model responses and peer critique, and a separate claim-checking service. The public implementation connects multiple providers and exports the intermediate outputs alongside the synthesis.

My contribution is the application and workflow connecting these stages. The language models and provider SDKs supply the underlying model capabilities. The research question is when the additional checking earns its cost.

Four stages you can inspect

1. Generate answers

   Providers answer the same question. Their responses remain visible separately.

2. Compare and critique

   Peer review surfaces disagreements and evaluates the answers. Its ratings are model judgments, not correctness measurements.

3. Check individual claims

   The verifier breaks answers into claims and routes them for checking. With multiple providers configured, it selects a provider other than the claim’s author.

4. Synthesize with the checks available

   The final stage combines the material. A JSON record preserves answers, critiques, claim verdicts, and the synthesis for inspection.

A useful design decision sits inside claim checking: a calculation needs re-derivation, while a factual claim may need supporting evidence. The claim-routing helper distinguishes those categories. The current verifier falls back to the author when only one provider is configured; the cross-provider design requires multiple configured providers.

A recorded run makes the distinction concrete

The public June 30, 2026 capture asks which has more total energy: a photon or a free electron, each with a wavelength of 1.0 nm. The run is labeled live, with web search off. Here is how one answer moved through the four stages in the recording.

Limits: the overall total-energy conclusion did not change; three of four initial answers already contained ~1.5 eV; the recorded Stage 3 verifier used model reasoning in text rather than an executed numerical test; the offline synthetic fixture (npm run demo:fixture) is distinct from this recorded provider run; and this single trajectory does not establish that verification caused an accuracy improvement.

Try the sample without an API key

The repository includes a deterministic offline replay: npm run demo:fixture. It exercises the pipeline without provider keys or network calls and writes JSON and Markdown reports. It runs every stage on fixture data, so you can read each stage's output.

The live quickstart runs the interface and services with your own provider keys. The recorded live output is a separate artifact, so a visitor can inspect what happened in an actual run without initiating a new one.