How Autopilot Picks Models, Strategies and Panel Size

Autopilot plans each run from your question and two settings, speed and cost. This article explains how. It covers what Autopilot asks about a question, the rules that turn the answers into a plan, and the benchmark results that shaped those rules.

Time to read: 6-7 minutes.

How Autopilot turns a question into a plan

Why not always use a panel?

Our benchmark covers 322 evaluated runs across ensemble strategies and model teams. In 61% of them, the best single model matched or beat the panel's final answer.

A panel is worth its cost on hard, open or high-stakes questions. On a simple question it mostly adds cost and waiting time. So Autopilot's first decision is whether a question needs a panel at all.

The same data showed that price and quality are loosely linked for panels:

Team	Average score	Cost per run
Three budget models	9.26	$0.036
Three premium models	8.93	$0.218
Two fast models, nano judge	4.84	$0.003

A well-composed budget panel beat a premium one at about a sixth of the cost. A panel whose final answer was written by a nano model collapsed. Both results shaped the rules below.

What does Autopilot ask about a question?

Autopilot sends the question to Jev, a structured decision model from TypeSafe served through OpenRouter. Jev does not write text. It answers fixed questions with a choice or a score, each with a probability.

It asks:

Category and domain: what the question is about, and whether it belongs to medicine, law, finance or construction.
Difficulty and stakes: each on a 1-5 scale.
Answer shape: a checkable answer, an open question, a decision, a document, or a critique.
Needs: how likely the question needs a panel, the web, careful reasoning or tools.
Risk: how adversarial or ambiguous the question is.

Jev also sees the last few turns of the conversation, the kinds and sizes of files you attach, and which tools you have connected. Answers for the same input are cached for a day.

If Jev is slow or unavailable, a local classifier answers the same questions from keywords. The plan is then marked as a quick local read.

How do the answers become a plan?

A fixed set of rules turns Jev's answers and your settings into a plan. The rules are code, not another model, so the same answers always give the same plan.

One model or a panel. A question goes to one model when a panel is unlikely to help and difficulty is 2 or less. At Fast with Lean cost, anything below high stakes goes to one model. High-stakes questions always get a panel.

A specialist team. A confident medical, legal, financial or construction read with high stakes calls that field's team.

The method. The answer shape picks the strategy:

A decision gets a Debate.
A critique, or an adversarial question, gets Red Team / Blue Team.
A checkable answer gets Chain-of-Thought when it needs reasoning, else Competitive Refinement.
A document gets an Expert Panel, or Hierarchical for large product work at Thorough.
Anything else gets Collaborative Synthesis.

The rounds. Speed sets the limit: one round at Fast, two at Balanced and three at Thorough. Hierarchical is the exception and always gets at least three.

The models. Cost sets the quality band and the panel size. Some strategies need three models, so they never run with fewer. Autopilot spreads a panel across vendors, because models from one vendor tend to share blind spots.

The judge. The model that writes the final answer is not a lightweight model. This rule comes straight from the nano-judge result above. If lightweight models are the only ones open to you, Autopilot uses one and the plan card warns you.

What happens when Autopilot is unsure?

When Jev is unsure of the answer shape, Autopilot steps the cost up one level, from Standard to Max quality. It never steps up an explicit Lean setting. The plan card shows the higher estimate before you run.

When Jev reads a question as very ambiguous, a panel run can start by asking you to clarify.

How does Autopilot keep to my balance?

Before it runs, Autopilot estimates a low and a high cost for the plan. If the high estimate does not fit your balance, it steps the cost down until it fits, and says so on the plan.

The plan card also shows what a Max-quality run of the same shape would cost. That is the figure behind "saved" on each answer. It is always the same team size at the top tier, never an arbitrary maximum.

How do we check the plans?

Autopilot has a shadow mode. It plans alongside every run without changing anything, so its plans can be compared with what people chose. We also replay the benchmark prompts through it.

Every plan is logged with what Jev answered, without the question text itself. The Wrong plan? button on each answer adds labelled examples. We use both to tune the thresholds.