Mandos

Quick start

Install. The script bootstraps uv, resolves Python 3.13, and needs no checkout:

# macOS / Linux
curl -fsSL https://raw.githubusercontent.com/Fox-Islam/mandos/main/scripts/install.sh | sh

# Windows
irm https://raw.githubusercontent.com/Fox-Islam/mandos/main/scripts/install.ps1 | iex

Configure. This opens the configurator, where you add panel members, set the judge, and wire your harness:

mandos

Then use it. In Claude Code, Codex or OpenCode:

/council Should we move the job queue off Postgres LISTEN/NOTIFY?

Your model calls the mandos tool, the panel answers, the judge scores what they established, and your model writes the answer. You need a key for the judge and one per panel provider; mandos doctor shows what is configured without printing secrets.

How it works

A panel produces several answers, and something has to work out what they establish. The usual choice, Fusion's included, is a second chat model that reads the panel and writes a summary. That summary has the same failure modes as the answers it is grading, and it states its confidence in words.

Mandos gives the judging to Jev, TypeSafe's decision model. Jev answers named questions in fixed shapes: a probability for a yes/no question, a label with a distribution for a choice, a level on a rubric for a score. A consensus in the output therefore means something specific. Every panel member scored above 0.6 on that claim, and the scores are returned with it.

The judge also reports what the panel lacked. When answering needs a row count or a query plan that none of the models had, that comes back as a ranked list, each item scored for how likely it is to change the answer. The calling model fetches what it can reach and asks again.

How the panel is judged is a per-call choice, not just a setting. hybrid adjudicates each claim and attributes it; probe skips deliberation and reports only what was missing; matrix measures agreement with no chat model in the loop. Set judge.shape for the default, or pass judge_shape when the question tells you which one it needs.

Setup notes

The installer puts two commands on your PATH: mandos, the configurator, and mandos-mcp, the stdio server your harness spawns. The configurator writes ~/.mandos/config.json and ~/.mandos/.env at mode 0600.

Any OpenAI-compatible endpoint works as a panel member, so vLLM, Ollama and LM Studio need no special handling. mandos doctor labels each provider on-prem or off-prem so you can see what leaves the machine.

Jev is served by TypeSafe directly and through OpenRouter's decisions endpoint. The request and answer bodies are the same on both, so judge.provider decides only the host and which environment variable holds the key.

Benchmark

The benchmark presented below tests one thing. When answering correctly depends on a fact none of the models has, does a panel do better than a single model?

It is worth asking because the failure it describes is common and expensive. A model given a question about your system will answer it whether or not it knows the row count, the query plan or the timeout that decides the answer, and a confident wrong answer is harder to catch than an admission of ignorance. Withholding exactly one fact and holding everything else constant also separates two things that usually move together: whether a panel helps because models disagree usefully, or because something in the loop notices what is missing.

Answers are graded against a key instead of by another model, so the numbers are checkable. What they do not measure is reasoning quality, research, or open-ended judgement, which is what a panel is usually convened for.

Setup

16 questions. Each is underdetermined until one specific fact arrives, and each fact is chosen so that the plausible answer without it is wrong. Answers are a forced Yes or No and are graded against a key.

Q: Our dashboard runs SELECT COUNT(*) FROM organisations on every page
   load and we are considering replacing it with a counter table
   maintained by triggers. Is that worth doing?

Withheld: organisations contains 1,180 rows and grows by ~20 a month.
          The COUNT(*) executes in 0.4ms.

Correct:  No

Within a tier, solo and panel use the same author model, so the panel, the judge and the evidence loop are the only things that vary.

The withheld fact is supplied only when the judge asked for it, and whether it asked is graded by Jev instead of by keyword matching. A judge that does not ask gets no second pass, which is the constraint a real caller works under.

Reason briefly, then end with a single line of exactly the form
'FINAL: Yes' or 'FINAL: No'. You must commit to one; 'it depends'
is not an answer.

Answer the question. {ANSWER_RULE}

Answer independently and thoroughly. Do not refer to other panelists.
If answering well needs information you have not been given, say
plainly what is missing and what you are assuming in its place, then
answer under that assumption. Name the measurement that would settle
it, not the topic. {ANSWER_RULE}

A panel was asked this question and a judge analysed their answers.
You are the author: decide yourself from the analysis and the raw
answers. The panel members are advisors, not voters, and can be
confidently and unanimously wrong. Where the analysis carries numbers,
weight findings by them rather than by how confidently they are
phrased. {ANSWER_RULE}

You read answers from a panel of independent models and list what is
worth checking. You do NOT judge, rank, merge, or answer the question
yourself.
- claims: the substantive propositions the answers make...
- blind_spots: things a careful reader would expect an answer to
  address that no answer addresses...
- missing_evidence: facts that are not in the question and that would
  change the answer if they came back one way rather than another.
  Ask yourself what you would have to measure to be sure, and list
  that... Name the measurement, not the subject. Propose generously —
  each one is scored afterwards and the weak ones are dropped.
Output STRICT JSON only.

Results

The three budget rows come from one run and the two mid-tier rows from another, so figures are comparable within a tier. probe was measured on the budget tier only.

Conclusion

Convening a panel beats a single model by a wide margin at both tiers. The budget panel takes 12/16 to 15/16; the mid-tier panel takes 11/16 to 16/16, the largest gain of any configuration measured. The mid-tier author starts lower than the budget author because these questions reward stopping to ask over reasoning carefully from incomplete premises, and a stronger model builds a more convincing case for the wrong answer.

On the budget tier, the same evidence loop with no panel at all reaches 16/16 at roughly half the cost, which is what probe is for. Convene a council when the question turns on judgement instead of a missing fact.

Questions where the panel and the unaided model disagree:

Panel right, model wrong:

• ci_parallel - budget
• read_replica - budget
• code_split - budget
• nginx_timeout - budget
• index_selectivity - mid-tier
• ci_parallel - mid-tier
• cache_p99 - mid-tier
• read_replica - mid-tier
• code_split - mid-tier

Model right, panel wrong:

• kafka_partitions - budget

Both tiers together cost $3.15. Jev is a small part of that, at roughly $0.0002 to adjudicate a whole panel.

Caveats

• 16 questions, one repeat. A one- or two-question gap is not a result.
• The questions are built so that a missing fact decides the answer, which flatters any method that asks for evidence. They say nothing about questions where the panel already has what it needs and disagrees anyway.
• Answers are binary and graded automatically, so nothing here measures the quality of an explanation.
• The panel has no web access and cannot read your files. Evidence reaches it because the calling model gathers it.