LLM Model Routing by Task Shape: Optimize Cost per Completed Task

The cheapest model is not the one with the lowest token price. It is the model that produces an accepted result at the lowest total cost after retries, escalation, verification, latency, and human review. That changes model selection from a leaderboard decision into a routing problem.

The stable unit of routing is not the model name. It is the shape of work.

Evidence reviewed: 19 August 2026. OpenRouter usage statistics describe traffic on that platform; Hugging Face download statistics describe Hub activity. Neither represents the entire model market.

Reading time: 8 minutes · About 1,500 words

TL;DR

• OpenRouter's 100-trillion-token study shows that real workloads are becoming longer, more tool-oriented, and highly asymmetric. Average prompts rose from about 1,500 to more than 6,000 tokens, while average completions rose from about 150 to 400.
• A small or low-cost worker is a good default for bounded extraction, summarization, reformatting, classification, and drafting only when its output can be checked cheaply.
• Route on six dimensions: task boundary, answer contract, failure cost, deadline, verifiability, and partial-trajectory signals.
• Optimize expected cost per completed task, not dollars per million tokens.
• Log every route, result, retry, escalation, accepted output, and human minute. Recalibrate the boundary whenever models, prices, prompts, or traffic change.

Why task shape matters now

OpenRouter analyzed more than 100 trillion tokens of real-world interactions. Its report describes a workload shift that makes one global model choice increasingly inefficient.

Average prompt length rose roughly fourfold, from around 1,500 tokens to more than 6,000. Average completion length nearly tripled, from around 150 to 400 tokens. Programming requests routinely exceeded 20,000 input tokens, while other categories remained much flatter. Tool invocation also increased, and the platform observed a growing share of agent-like work.

The asymmetry is the useful signal. A 20,000-token code investigation, a 12-field JSON extraction, and a two-sentence classification may arrive through the same API, but they have different context needs, output contracts, failure costs, and latency budgets.

One frontier model for all three tasks buys simplicity at the expense of cost. One cheap model for all three buys low unit price at the expense of retries and escaped errors. Routing lets each task pay for the capability it needs.

Define low-cost as an operating role

The phrase small model is unstable. OpenRouter's research classifies models below 15 billion parameters as small. A Hugging Face report emphasizes sub-1-billion-parameter downloads. Some widely used API models do not disclose parameter counts at all.

For production routing, use an operational definition:

A low-cost worker is a model tier that is cheaper or faster than the incumbent for a frozen task cell and still meets the cell's acceptance contract.

This definition survives vendor changes. A model may be small by parameters, sparse with many total parameters, distilled, or simply sold at a lower price. The router only cares whether it produces accepted work within the required risk and time budget.

OpenRouter's official subagent pattern follows this logic. A strong orchestrator delegates focused summarization, extraction, reformatting, and drafting to a cheaper worker. The documentation also warns that each delegation adds another model call, with its own cost and latency. The worker is useful only if the saved generation cost exceeds orchestration and verification overhead.

The six-dimensional routing contract

1. Task boundary

Bounded tasks have a clear input, output, and stopping condition. Extract these fields from this document is easier to route than investigate why this service is unstable.

Record domain, tool access, context size, and whether the task can be decomposed without losing critical state.

2. Answer contract

Output shape is a practical routing feature:

• fixed label from a small set;
• schema-constrained JSON;
• short grounded summary;
• bounded transformation;
• open-ended recommendation;
• multi-file artifact with tests.

A short answer does not automatically imply an easy task. A one-word legal or medical conclusion can have extreme failure cost. Answer shape must be combined with risk and evidence.

3. Failure cost

Estimate what happens when a plausible but wrong output passes through. Low-cost tiers are strong candidates for reversible, reviewable work. Security decisions, money movement, destructive tool calls, and externally published claims should route to stronger models or mandatory review even when the answer is short.

4. Deadline and concurrency

Interactive latency, overnight batch completion, and queue throughput create different optimal routes. A slow, cheap model may fit a six-hour batch window and fail a two-second interface. A fast worker may reduce queue age even if its token price is slightly higher.

5. Verifiability

Cheap models become more valuable when acceptance is deterministic: schema validation, exact field checks, unit tests, reference matching, or a known calculation. If quality can only be judged by a senior expert, human verification may erase the price advantage.

6. Partial-trajectory signals

Some tasks reveal their true difficulty only after work starts. A low-cost model can inspect the repository, run a test, or attempt extraction, then expose signals such as missing context, repeated tool errors, uncertainty, or acceptance failure.

SWE-Router formalizes this idea for software tasks: allowing a cheaper model to execute a short exploratory trajectory before deciding whether to continue or escalate can outperform routing from the initial prompt alone. The first attempt becomes a diagnostic probe, not necessarily wasted work.

Calculate cost per completed task

Use an expected-cost model:

expected completed-task cost
= worker inference
+ orchestration
+ verification
+ P(retry) × retry cost
+ P(escalation) × frontier-model cost
+ expected human review time
+ expected escaped-failure cost

Token price appears in the first terms. Production economics often live in the later ones.

T2MO makes the same shift explicit by optimizing expected cost per completed task and routing at task-category by difficulty cells. Its published results show why category-level routing is too coarse: one model could pass easy and medium Git workflow cells while its frontend pass rate fell from 100% on easy tasks to 50% on medium and 25% on difficult tasks.

The router should therefore compare models inside frozen cells such as:

task type × difficulty × context band × output contract × risk tier

A model is eligible only after meeting the cell's quality floor. Among eligible models, select the one with the lowest measured completed-task cost or latency objective.

Start with a simple cascade

The first production version can be small:

1. A deterministic policy sends high-risk or non-verifiable tasks directly to the incumbent.
2. Eligible bounded tasks go to a low-cost worker.
3. The worker must produce the answer plus machine-readable confidence and evidence fields.
4. Acceptance checks run immediately.
5. Failed checks, tool loops, context overflow, or low confidence escalate with the full trajectory.
6. The final accepted result and all incurred costs update the routing dataset.

Avoid asking the worker to decide its own authority. The router applies policy; the worker supplies evidence.

As data accumulates, replace coarse rules with a classifier or learned router. Keep hard risk constraints outside the learned component.

A minimal evaluation matrix

Freeze representative tasks and record:

OpenRouter's rankings page is useful for adoption signals, but it explicitly ranks by prompt plus completion tokens rather than accuracy. Hugging Face downloads are useful ecosystem signals, but downloads can reflect embedding models, CI, containers, and repeated dependency pulls. Neither should set a production route without task-level evaluation.

This is the same discipline behind AI Agent benchmark cells: model, harness, task, budget, and grader form one measured system. One-token behavioral fingerprints can then watch whether a supposedly stable route has silently changed.

Common routing mistakes

Routing only by prompt length. Long context can contain a trivial extraction; a short prompt can require difficult reasoning.

Routing only by self-reported confidence. Confidence must be calibrated against accepted results and failure types.

Ignoring escalation context. Repeating the same request with a stronger model wastes the worker's diagnostic work. Pass tools, errors, evidence, and attempted fixes forward.

Optimizing raw token spend. A route that reduces inference cost while doubling human review is more expensive.

Using one permanent threshold. Model behavior, prices, traffic, prompts, and tools change. Routing is a continuously calibrated policy.

FAQ

What is LLM model routing?

It is a policy that selects a model tier for each request using task features, risk, cost, latency, and observed execution signals rather than sending all traffic to one model.

When should a low-cost model handle a task?

When the task is bounded, cheaply verifiable, reversible, and has a low escaped-failure cost, and when measured completed-task cost is lower after retries and escalation.

Can output length predict the right model?

It is one feature, not a decision rule. Combine the output contract with context, reasoning demand, tool use, risk, deadline, and acceptance evidence.

Should routing use rules, a classifier, or another LLM?

Start with explicit rules and hard risk gates. Add a classifier or learned router after collecting labeled task outcomes. Keep safety and authority constraints deterministic.

How do you know when to escalate?

Escalate on acceptance failure, repeated tool errors, missing context, policy risk, time budget, low calibrated confidence, or trajectory features associated with past failures.

References

• OpenRouter: State of AI 2025
• State of AI: An Empirical 100 Trillion Token Study
• OpenRouter: Building agents with a low-cost subagent worker
• Hugging Face: State of Open Models, Summer 2026
• T2MO: Task-to-Model Optimization for Cost-Efficient LLM Systems
• SWE-Router: Trajectory-Aware Routing for Software Engineering Agents