July 4, 2026 · Thoughts
The Metric That Matters: Cost per Solved Coding Task
AI model pricing invites simple math. Multiply input tokens by one rate, output tokens by another, and crown the cheapest model.
Software work refuses to be that tidy.
OpenAI's current GPT-5.6 pricing spans Luna, Terra, and Sol, while Anthropic's Claude Fable 5 pricing sits higher for its premium long-horizon model. Those numbers matter, but they do not tell you what a finished feature costs.
The useful metric is cost per accepted task.
Count the Whole Loop
Suppose a cheaper model costs $1 for a run but needs four attempts, two hours of redirection, and a cleanup patch. A stronger model costs $6 and returns an acceptable change in one pass.
The first option did not cost $1. It cost $4 plus human time, queue delay, and the risk hidden in repeated partial edits.
Track at least:
- Model and tool charges
- Number of retries
- Wall-clock time
- Human intervention time
- Review and cleanup time
- Test failures introduced
- Percentage of generated code retained
- Whether the task was accepted
Now you can compare outcomes instead of invoices.
Define “Solved” Before the Run
Agents are excellent at producing activity. Your measurement needs a finish line.
A solved task should satisfy written acceptance criteria, pass the relevant checks, avoid known regressions, and produce a reviewable patch. If the agent stops with a plausible explanation but no working change, the task is not solved.
For research or diagnosis, define a different artifact: reproduced failure, evidence-backed cause, and recommended options. Do not force every valuable task into a code-merged metric.
Use the Cheapest Reliable Lane
Routing creates savings when it is based on evidence.
Send repetitive, verifiable work to a fast model. Use a balanced model for conventional implementation. Escalate ambiguous or consequential work to a frontier model. If the cheap lane repeatedly fails a class of tasks, update the policy.
The goal is not to minimize spend on every call. It is to minimize waste across the pipeline.
Cache Stable Context
Large repositories make agents reread the same instructions, schemas, and architectural background. Prompt caching can reduce that cost when context is stable and reused.
OpenAI says GPT-5.6 supports explicit cache breakpoints, a minimum cache life, and discounted cache reads. Anthropic also offers input-token discounts for prompt caching on Fable.
Design context with stable material first and volatile task details later. Do not change a large shared prefix for cosmetic reasons if doing so defeats reuse.
Review Time Is a Product Cost
A model that generates enormous diffs may look productive while transferring the burden to the reviewer.
Measure how long it takes a human to establish confidence. Focused patches with clear summaries and tests can be more valuable than sprawling patches containing more code.
Ask agents to keep changes scoped, explain non-obvious decisions, and report commands run. Reviewability is part of performance.
The Takeaway
Token rates are easy to publish and easy to compare. Accepted engineering outcomes are harder, which is exactly why they matter.
Build a small dashboard or spreadsheet. Track real tasks for a month. Let your routing policy evolve from the data.
The cheapest model is not the one with the lowest rate. It is the one that gets the right work done with the least total waste.
