Benchmarks
Rigorous, reproducible LLM benchmarks with open datasets and documented methodology. Every result links to raw CSV data so you can verify, extend, or challenge our findings.
Interactive tool
LLM Cost-per-Word Calculator →
Paste your prompt, pick from 31 models, see the real per-word cost. Uses the measured tokenizer-efficiency (E) values from the benchmark below — not the marketing price sheet.
Session 5 — Published 2026-07-09
Tokenizer Efficiency: Why Input Token Prices Are Meaningless Without It
50 model variants tested on code, prose, and blended text. The gap between the most and least efficient tokenizer is 74% — meaning a model with cheaper per-token pricing can cost more per word. Full methodology, charts, and CSV dataset included.
Read the post →
Session 6 — Published 2026-07-09
The 16 Challenges: Why Output Token Prices Hide Thinking Tokens and Efficiency
50 model variants across 16 task types in 9 categories — Q&A, reasoning, coding, creative, role-play, instruction-following, safety, multilingual, and extraction. Why category matters, how thinking tokens inflate cost, and why you can't compare output token prices across models without adjusting for both.
Read the post →
Session 6b — Published 2026-07-09
5 Ways to Compress LLM Output: A Cost-Saving Toolkit for Enterprise
Structured Markdown Compression, JSON Envelope, Diff-Only Delta, Verb-Noun Grammar, and Word Deletion tested across 20 model configs spanning 18 families. Word Deletion cuts output by 49%; Diff-Only Delta by 53%; one method backfires with a negative ratio. Full results, charts, and implementation guidance.
Read the post →