Give an answer more than one kind of review

When agreement leaves a wrong step intact

A convincing answer can contain a mistake even when its conclusion is right. Asking another model to rate the whole answer may leave that mistake untouched. I wanted an application where reviewing an answer and checking its individual claims are separate jobs.

I built The Council with AI coding assistance: a React interface, an API that coordinates model responses and peer critique, and a separate claim-checking service. The public implementation connects multiple providers and exports the intermediate outputs alongside the synthesis.

My contribution is the application and workflow connecting these stages. The language models and provider SDKs supply the underlying model capabilities. The research question is when the additional checking earns its cost.

Four stages you can inspect

1. Generate answers

   Providers answer the same question. Their responses remain visible separately.

2. Compare and critique

   Peer review surfaces disagreements and evaluates the answers. Its ratings are model judgments, not correctness measurements.

3. Check individual claims

   The verifier breaks answers into claims and routes them for checking. With multiple providers configured, it selects a provider other than the claim’s author.

4. Synthesize with the checks available

   The final stage combines the material. A JSON record preserves answers, critiques, claim verdicts, and the synthesis for inspection.

A useful design decision sits inside claim checking: a calculation needs re-derivation, while a factual claim may need supporting evidence. The claim-routing helper distinguishes those categories. The current verifier falls back to the author when only one provider is configured; the cross-provider design requires multiple configured providers.

A recorded run makes the distinction concrete

The public June 30, 2026 capture asks which has more total energy: a photon or a free electron, each with a wavelength of 1.0 nm. The run is labeled live, with web search off. Here is one trajectory within that recorded output.

The example shows a useful behavior of the implemented pipeline: a mistake survived whole-answer review and was identified at the claim level. It does not establish an overall accuracy improvement. Different providers can share errors, and model confidence is not a calibrated probability that a claim is correct.

Inspect it without spending on a live run

The repository includes a deterministic offline replay: npm run demo:fixture. It exercises the pipeline without provider keys or network calls and writes JSON and Markdown reports. Its evidence is simulated; use it to understand the application’s structure.

The live quickstart runs the interface and services with your own provider keys. The recorded live output is a separate artifact, so a visitor can inspect what happened in an actual run without initiating a new one.