The most expensive tokens in your LLM bill aren't the ones you feed in — they're the ones the model generates. Output tokens cost 4–30× more than input tokens. A "cheap" model that writes a 500-word essay for a one-word answer is burning your budget faster than a premium model that says "42" and stops.
But there's a second invisible cost: thinking tokens. Reasoning models spend hundreds or thousands of internal tokens before answering — all billed at output prices. You don't see them. You don't read them. You pay for them.
This experiment measures both.
The Problem
Output token pricing is fundamentally more opaque than input pricing for three reasons:
Verbosity variance. Two models at the same output token price can differ 10× in real cost because one generates 50 tokens for a task and the other generates 500.
Thinking tokens. Reasoning models (R1, o3, Gemini Thinking, Claude with extended thinking) burn internal chain-of-thought tokens before producing their answer. These are billed at output token prices but are invisible to the user unless the provider reports them separately.
Prompt sensitivity. A model that obeys "in one sentence" costs less than one that ignores it — not because it's cheaper per token, but because it generates fewer of them.
Comparing models on output token price alone is like comparing rental car prices without checking the mileage limit.
1. Task Categories
1.1 Why Category Separation Matters
A model that answers "What is 2+2?" in 5 tokens but writes a 500-token essay for "Explain the meaning of life" has a category-dependent cost profile. Choosing a model for a task family requires knowing its category-level verbosity — not just an average across all tasks.
Category separation also reveals structural patterns that an overall average hides:
Reasoning models produce thinking tokens only for certain task types
Creative models inflate only on open-ended prompts
Some models follow length constraints; others ignore them
1.2 The 9 Categories (16 Tasks)
#
Category
Count
Tasks
C1
Q&A / Factual
2
one-word (capital of France), one-sentence (what is a database index)
C2
Reasoning
2
reasoning (last digit of 3^1000), multi-step (bat-and-ball puzzle)
Output Verbosity (V): total completion tokens returned for a given prompt, measured via usage.completion_tokens
Response Word Count (W_out): words in the generated output, computed with same Unicode segmentation as Session 5
Thinking Token Count (T): where the provider separately reports thinking/reasoning tokens (e.g., DeepSeek R1 usage.completion_tokens_split.thinking, Anthropic's extended thinking, o-series chain-of-thought)
Constraint Adherence (A): binary — did the response obey length instructions?
Thinking-to-Response Ratio (R):T / (V − T) for models that report thinking tokens separately
2.2 Protocol
Each of the 16 prompts submitted to 50 model variants
max_tokens=1500 for pass 1, max_tokens=4096 fallback for capped responses
temperature=0 for reproducibility
Recorded: usage.completion_tokens, usage.prompt_tokens, any thinking/reasoning split, full response text
821 API calls (390 original + 431 expansion), actual cost: $2.58 (including 5.5% OpenRouter fee)
2.3 Prompt Design
Prompts written to minimize wording variability — same instruction phrasing across all models, no model-specific formatting. Length constraint tasks (C5) use explicit instruction formats tested for inter-model consistency1. All prompts in English except the translation task in C7.
3. Data Standardization
3.1 Text Processing Pipeline
Response text: strip boilerplate (greetings, sign-offs, markdown fence markers) before counting words
Thinking tokens: where provider returns separate thinking/reasoning counts, record both raw and thinking-adjusted completion tokens
For models without thinking token reporting (most GPT-family, Claude, Gemini, Llama, Mistral), thinking tokens, if present, are embedded in completion_tokens and indistinguishable
3.2 Normalization for Charting
Each model's 16 task responses grouped into 7 category means. Category-level V (completion tokens per task) reported in the main results table. Full per-task data in CSV.
3.3 Why V Is Tied to These Specific Prompts
V measures how many tokens a model generates for these 16 specific prompts. Different prompts in the same category — even semantically equivalent ones — would produce different V values. The ranking order is expected to generalize (verbose models stay verbose relative to concise ones), but the absolute V numbers apply directly only to our task set. See §8 for the full discussion of prompt dependency.
⚠️ Note for reasoning models. Models with deep reasoning frameworks (R1, o3-mini, o4-mini) build heavy chain-of-thought scaffolding into every response — even trivial ones. Some models hit max_tokens=1500 on open-ended creative tasks (haiku, describe-sunset) and were retried at 4096. At 4096, MiMo-V2.5 and Kimi K2.6 still maxed out on reasoning/socratic tasks — indicating their natural output would exceed even that ceiling. Grok Build 0.1 is a notable outlier: it hit 5423 tokens on a haiku prompt.
Heatmap of mean completion tokens per task category across all 50 model variants. Darker cells indicate higher verbosity. Models sorted by overall mean V.Thinking vs response token breakdown for models that separately report thinking/reasoning tokens. The 80/20 reference line highlights where thinking dominates the output budget.Thinking-to-response ratio R = thinking / response. Green: R < 1, Amber: R 1–3, Red: R > 3. Effective output price shows the thinking tax multiplier.Effective cost per task (V / 1M × output price) across all 50 model variants. Reference line at $0.001/task. Color-coded by model family.Constraint adherence matrix for the two instruction-following tasks (repeat-exact, json-format). Green = adhered, Red = ignored. Critical for cost control.Distribution of completion tokens across all 16 tasks per model. Box = Q1–Q3, line = median, whiskers = min–max. Sorted by median V.Category-level verbosity spread across all 50 model variants. Box = Q1–Q3, line = median, whiskers = min–max. Annotations show the widest and narrowest spreads.
5. Analysis
5.1 Category-Specific Profiles
The 16 tasks span 9 categories. Here's what the data from 50 model variants reveals:
Role-Play (grumpy-sysadmin, pirate-speak, socratic): The widest spread in the study. grumpy-sysadmin averages 910 tokens but ranges from 185 (GPT-5.4) to 3488 (DeepSeek V4 Flash Max with xhigh reasoning). Persona inflation is real — a model generating 10 tokens for pirate-speak but 900+ for grumpy-sysadmin is not consistently verbose; it's persona-dependent.
Reasoning (reasoning, multi-step):reasoning (last digit of 3^1000) averages 732 tokens, with 4 models maxing out at 1500+. multi-step (bat-and-ball) averages 411 tokens — simpler reasoning costs less. Even non-reasoning models generate substantial step-by-step explanations.
Creative (haiku, describe-sunset): Highest outliers. haiku averages 610 tokens but hits 5423 for Grok Build 0.1 — it ignores max_tokens. describe-sunset averages 619 tokens despite the "exactly 50 words" constraint — most models obey, but a few generate 50× that.
Q&A (one-word, one-sentence): Most concise. one-word averages 213 tokens — models rarely answer in a single word despite the prompt. one-sentence averages 224.
Safety (phishing-refusal): Refusal verbosity ranges from 0 tokens (GPT-5.5 Pro — content filter blocked entirely) to 2326 (Qwen3.7 Max — lengthy refusal). GPT-5.5 Pro's 0-token refusal costs nothing.
Instruction-Following (repeat-exact, json-format): Lowest V. repeat-exact averages 97 tokens but ranges 5-721. json-format averages 79 with a 357 max. Highlights which models obey strict formatting.
Extraction (extract-emails): 348 average, 1456 max. Some return concise JSON; others produce lengthy framing.
5.2 The Thinking Token Tax
For models with separate thinking token reporting: reasoning/chain-of-thought tokens are billed at output token prices, often 1.5–4× input prices. A model that spends 800 thinking tokens to generate a 200-token answer has a 4:1 thinking-to-response ratio — meaning 80% of your output budget goes to intermediate computation you didn't ask for and may never see3.
5.3 The Fairness Caveat: Small-Task Inefficiency vs. Real-World Efficiency
A reasoning model that spends 800 thinking tokens on "What is 2+2?" has a 160:1 thinking-to-answer ratio. That is genuinely wasteful for that task. This benchmark detects that waste and ranks the model as inefficient on C1.
But the same reasoning framework might solve a complex multi-step problem in one shot (1500 tokens) where a non-reasoning model would need 3–5 rounds of back-and-forth (5000+ tokens). The thinking overhead is real cost, and for simple tasks it is unambiguously wasted. For complex tasks, the trade-off may be favorable.
This benchmark measures per-task V on a fixed prompt set. It does not measure end-to-end task completion efficiency. Use these results to inform model selection for your specific task mix — not as a universal efficiency ranking.
5.4 Prompt Sensitivity
Constraint adherence (C5) is a critical secondary metric: a model that generates 20 tokens for "one-sentence summary" but 200 tokens for "action items only" has different reliability profiles across instruction types. The effective cost of a verbose refusal (C6) should also factor into safety-aware model selection.
6. Real-World Implications
6.1 Choosing a Model for Your Task Mix
If your workload is 80% Q&A and 20% creative writing, a model with low C1 V but high C3 V may be cheaper overall than one with balanced but moderate V everywhere. Weight category V by your task distribution, not by the overall average.
6.2 The Cost of Verbose Refusals
At enterprise scale (10M output tokens/month), a model that refuses in 200 tokens instead of 10 tokens costs $50-300/year per refused request. If 5% of requests trigger refusals, that's $250-1500/year in pure refusal overhead — for no useful output.
6.3 Combining with Session 5
Tokenizer efficiency (E) affects input cost. Output verbosity (V) affects output cost. The combined per-task cost formula becomes:
C = W_in × E × P_in / 1,000,000 + V × P_out / 1,000,000
Selecting a model on input price alone misses both variables. Selecting on output price alone misses V. The full formula is the only reliable comparison.
7. Patterns in Our Data
7.1 Output Length and Quality
In our results, output length correlates weakly with answer quality but strongly with cost — a model that writes a 500-token essay to answer "what is 2+2?" is not giving you a better answer than one that says "4". Instruction-tuned models in our set produce systematically longer outputs than their length would suggest is necessary, a side effect of training to be "helpful."
7.2 Chain-of-Thought Overhead
For models that expose thinking-token counts, CoT adds hundreds of tokens per reasoning task — billed at output prices. For production systems where latency and cost matter, our recommendation is to reserve reasoning models for tasks where the reasoning measurably improves accuracy, and use a cheaper non-reasoning model for trivial requests. The original Chain-of-Thought paper3 is the canonical reference for why models do this.
7.3 Instruction Adherence
Prompt formatting has a large effect on output length — meaning-preserving changes to phrasing can swing a model's verbosity substantially4. Our length-constraint task (C5) is modeled on the verifiable-instruction set from IFEval1. In our data, structured output constraints (JSON, SMC) reliably reduce verbosity compared to freeform responses — see the companion compression post for measured ratios.
7.4 Refusal Length Economics
In our safety/refusal category (C6), refusal verbiage follows model-family patterns: some families produce long, explanatory refusals while others are terse. Over millions of inference calls, refusal verbosity costs real money — worth noting if your workload triggers refusals often.
7.5 Multilingual Tokenization Overhead
English-optimized tokenizers are known to produce more tokens per word for non-European scripts2. This compounds with output verbosity: a model that is verbose in English may be even more expensive per token in Japanese or Arabic. Our multilingual category (C7) only scratches the surface of this.
8. Caveats & Limitations
8.1 Thinking Token Visibility
Only some providers expose thinking/reasoning token counts via usage.completion_tokens_split.thinking or equivalents. For models without this data (most GPT-family, Claude, Gemini without extended thinking, Llama, Mistral), any thinking tokens are embedded inside completion_tokens and cannot be isolated. Our thinking-token charts cover only the subset of models that report this data.
8.2 V Is Prompt-Dependent
V measures output verbosity for these 16 specific prompts, not for all possible tasks in these categories. Different prompts — even in the same category — would produce different absolute V values. The ranking order across models is expected to generalize, but the absolute numbers should not be treated as calibration points.
8.3 Small-Task Inefficiency vs. Real-World Efficiency
As discussed in §5.3, this benchmark penalizes models with heavy reasoning frameworks on trivial tasks. That is appropriate for the scope of this measurement — if your workload includes many trivial requests, these models are genuinely more expensive for those requests. But the results should not be interpreted as "reasoning models are always less efficient." For complex, multi-step tasks, the opposite is often true. See §5.3 for the full argument.
8.4 Single Temperature (0)
All measurements taken at temperature=0 for reproducibility. Output length distribution may shift at higher temperatures, where models generate more varied responses.
8.5 English-Heavy Prompt Set
Tasks C7 (multilingual) include only one translation and one product description — not a full multilingual benchmark. V values for non-English output may differ substantially from English V values for the same models.
8.6 Single Measurement
Each model × task combination tested once. No variance or confidence intervals. A follow-up with multiple runs per prompt is planned.
Next Up
Output verbosity is half of the output cost story. The other half is compression — once you know how many tokens a model generates, how many of those tokens can you eliminate without losing information? Session 6b tests 5 compression methods across 4 model configurations.
Output token prices are meaningless without verbosity. Two models at the same output token price can differ 10× in real cost because one generates more words for the same task. Thinking tokens add an invisible surcharge on reasoning models. The only way to compare is to measure V for your specific task categories — not to rely on published token prices alone.
Sources & further reading
Verbosity data in this post is original measurement by Clock Lobster Labs. The references below are the foundational work our test design built on, not a literature review of verbosity studies.
1. Zhou, J., et al. (2023). "Instruction-Following Evaluation for Large Language Models" (IFEval). Our length-constraint tasks are modeled on its verifiable-instruction set. arxiv.org/abs/2311.07911
2. Rust, P., et al. (2021). "How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models." ACL 2021. Background on why tokenization — and therefore token cost — varies by language. aclanthology.org/2021.acl-long.185/
3. Wei, J., et al. (2022). "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." The original CoT paper — relevant to why reasoning models emit thinking tokens that inflate output cost. arxiv.org/abs/2201.11903
4. Sclar, M., et al. (2023). "Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design." ICLR 2024. Explains why the same instruction produces very different output lengths across models. arxiv.org/abs/2310.11324
