ModelDirector: A Model Selection Engine That Thinks Before It Spends

Every production AI agent eventually hits the same wall. It started simple: one model, one call site, one decision. Then the agent grew. Tasks got more varied — a routing decision here, a code generation step there, a long-context analysis somewhere else. And the model call sites multiplied.

The naive solution is to hardcode. if task == "simple_qa": use_gpt5mini() elif task == "hard_coding": use_opus(). It works until it doesn’t. Until the day a “simple_qa” turns out to need multi-step reasoning, or the “hard_coding” is just a four-line function that Sonnet handles in 400ms for a tenth of the cost.

That’s the problem ModelDirector was built to solve.

The Model Selection Problem

The core issue isn’t model quality — it’s model fit. Every major model is capable enough for most tasks. The question isn’t “can this model do it?” It’s “which model can do it correctly at the lowest cost?”

This sounds like a routing problem, but it isn’t. A router forwards traffic. A gateway proxies requests. ModelDirector does neither. It sits before the model call entirely — it only answers one question: given this prompt and this list of candidate models, which model should I hand this to?

The distinction matters because it means ModelDirector is entirely independent of how your agent actually calls the LLM. It doesn’t execute prompts. It doesn’t hold sessions. It doesn’t sit in the request path. It returns a decision — a model ID, a confidence score, a USD cost estimate, and the reasoning — and your agent decides what to do with it.

What ModelDirector Actually Does

ModelDirector takes a prompt, scores it against every model in a candidate set, and returns the winner according to a configurable policy.

The candidate set is defined in a YAML config. Each model has:

• Capabilities — a structured score across reasoning, coding, and context (0–100)
• Strengths — structured task-type tags like coding, architecture, long_context
• Description — free-form context for the selector LLM
• Cost — USD per 1M input and output tokens
• Priority — a tiebreaker for the cheapest_capable policy

Here’s what a minimal config looks like with three models:

selector:
  provider: openrouter
  model: anthropic/claude-3.5-haiku
  temperature: 0.0

policy:
  type: cheapest_capable
  threshold: 80

models:
  - id: gpt5mini
    name: openai/gpt-4o-mini
    capabilities: { reasoning: 65, coding: 70, context: 70 }
    strengths: [classification, simple_qa]
    cost: { input: 0.15, output: 0.60 }

  - id: sonnet
    name: anthropic/claude-3.5-sonnet
    capabilities: { reasoning: 88, coding: 92, context: 95 }
    strengths: [coding, architecture, long_context]
    cost: { input: 3.00, output: 15.00 }

  - id: opus
    name: anthropic/claude-opus-4
    capabilities: { reasoning: 95, coding: 95, context: 99 }
    strengths: [hard_reasoning, complex_coding]
    cost: { input: 15.00, output: 75.00 }

The selector model — configurable, independent of the candidate set — reads the prompt and evaluates each model. It returns per-model confidence scores across reasoning, coding, and context dimensions, plus a free-form explanation. The policy engine then applies the configured selection logic.

Three policies are built in:

• cheapest_capable — picks the lowest-cost model whose overall confidence exceeds the threshold
• highest_confidence — picks the model with the top confidence score regardless of cost
• best_value — picks the model with the highest confidence / cost.input ratio

The output looks like this:

{
  "selected_model": "sonnet",
  "policy": "cheapest_capable",
  "scores": {
    "gpt5mini": { "overall": 74, "reasoning": 70, "coding": 82, "context": 71 },
    "sonnet":   { "overall": 91, "reasoning": 90, "coding": 95, "context": 93 },
    "opus":     { "overall": 97, "reasoning": 98, "coding": 96, "context": 99 }
  },
  "estimated_cost_usd": {
    "gpt5mini": 0.00010,
    "sonnet":   0.00200,
    "opus":     0.01000
  },
  "input_tokens": 7,
  "reason": "'sonnet' is the first model exceeding the threshold of 80 (overall=91, priority=2)."
}

The estimated_cost_usd field is first-class. Every response shows what each candidate would cost for the actual prompt, not list prices. This makes the trade-off explicit — you’re not guessing whether Sonnet is worth it, you’re seeing the actual USD difference.

The Benchmark That Says the Most

The benchmark runs 30 real tasks — translation, Q&A, code generation, architecture design, prose — through the full pipeline against a three-model candidate set with real USD costs. The selector is anthropic/claude-3.5-haiku (a small, fast, cheap model — the irony of using a small model to pick between models is intentional).

Results from the June 2026 run:

• 30 tasks completed, 0 errors
• Mean cost savings vs always-using-Opus: 100%
• 11 tasks picked gpt5mini (~$0.00003 per 200-token prompt)
• 19 tasks picked sonnet (~$0.0006 per 200-token prompt)
• 0 tasks picked opus

The last line is the most interesting. Sonnet cleared the 80-point threshold for every single task in the battery. The selector correctly identified that Opus’s extra capability wasn’t justified by any of the tasks. Lower the threshold to 60, or tighten the capability profiles, and Opus starts getting picked on the hardest tasks.

Selector latency: p50 3.9s, p95 4.5s, max 5.1s. The overhead is real — you’re burning selector tokens before the actual model call. For a high-volume system making thousands of calls per minute, that overhead compounds. For most agent use cases where each task run is seconds to minutes, it’s negligible relative to the task itself.

The Design Principles That Make It Different

Most model selection code ends up as a tangled if/else tree buried inside an agent. ModelDirector is designed around four principles that keep it from becoming that:

Stateless. No database, no storage, no learning, no telemetry. Every request is independent. There is no state to corrupt, no session to manage, no data to leak. This makes it safe to call from any agent, in any environment, without worrying about what happens to the data.

Model-agnostic. The engine has no hardcoded model names. It doesn’t know what “Sonnet” or “Opus” are — it only knows what the config tells it. This means you can point it at a local Ollama instance, a self-hosted vLLM, a custom fine-tune, or a commercial API, and the engine works the same way. The candidate set is your definition of what the models are.

Cost-first. Cost is a first-class field in every model profile and a first-class output in every response. The estimated_cost_usd field isn’t an afterthought — it’s computed from actual token estimates and the per-1M-token cost you provide. The trade-off is always visible to the caller.

Configuration-first. Everything is in the YAML. No magic defaults that hide cost trade-offs. The threshold, the policy, the model capabilities, the cost figures — all explicit, all auditable, all version-controllable.

Why This Matters for Agent Builders

The ACO System — the multi-agent pipeline Aniket uses for software execution — faces this decision at every stage. The PM stage needs a model that can take a vague requirement and produce structured output. The Architect stage needs reasoning about system design. The Developer stage needs code generation. The QA stage needs evaluation logic.

Hardcoding a single model across all stages means either overpaying on easy tasks or under-performing on hard ones. A model selection layer like ModelDirector lets the pipeline be explicit: for this specific task at this specific stage, what’s the right model?

The policy engine is pluggable, so the selection logic can reflect business constraints — a team might set a budget ceiling that prefers Sonnet over Opus even when Opus scores higher, or a compliance requirement might mandate a specific provider. The policy encodes that logic; the engine enforces it.

The Adapter Surface

The engine is the product. Adapters are thin translation layers on top of it:

• Python SDK — from modeldirector import ModelDirector
• CLI — modeldirector select -c config.yaml -p "..."
• REST — FastAPI server for microservices
• MCP — FastMCP tool for Claude Desktop, Continue, Roo Code, Cline

Adding a new interface — a Slack command, a Discord bot, a custom agent — is a matter of constructing a ModelDirector and translating the input and output. The engine doesn’t change.

The Honest Trade-offs

ModelDirector isn’t the right tool for every situation.

The selector overhead is real. You’re making two LLM calls instead of one — the selector call to pick the model, then the actual model call to do the work. For tasks that take minutes, that’s noise. For high-frequency, low-latency pipelines handling thousands of short requests per second, that overhead becomes the bottleneck.

The selector model introduces a dependency. If the selector model is unavailable or slow, the selection step fails. A degraded selector affects every downstream call. Production deployments need to handle selector failures gracefully — fall back to a default model, or surface the error rather than silently failing.

The capability profiles are hand-defined. ModelDirector doesn’t discover model capabilities — you declare them in the config. That means the selection quality depends on how accurately you profile your models. A wrong capability score produces wrong selections.

These aren’t reasons to avoid ModelDirector. They’re the constraints you design around.

What Comes Next

The benchmark numbers — 30 tasks, zero errors, 100% cost savings vs an always-Opus baseline — are a snapshot of one config against one task mix. Real deployments will have different candidate sets, different thresholds, different cost structures. The value isn’t in the numbers; it’s in the approach.

Model selection deserves its own dedicated layer because the question “which model should I use?” is structurally different from “how do I call a model?” Treating it as a first-class concern — with configurable policies, explicit cost estimates, and auditable reasoning — is what makes agent systems that can flex across providers, across model families, and across cost constraints without rewriting call sites.

The repo is at github.com/aniketkarne/ModelDirector. The engine is MIT-licensed, the adapters cover the major integration points, and the benchmark code is included so you can run it against your own task mix and config.