What is a Token?

Priced by the token. Taxed by the token. Thought by the token. Most of the AI economy runs on a unit almost no one can see.

A couple of years ago, if you asked any AI model how many Rs there are in strawberry, the answer was often wrong: usually two, sometimes one. The reason is not that the model is stupid; it is that the model has never seen strawberry. It has seen straw and berry, two separate objects in its vocabulary, and counting letters across the join is not the operation it was built to perform.

To the model, strawberry is not made of letters. It is made of tokens. So is every word it reads, every sentence it generates, every conversation it has ever had. So, increasingly, is every image it sees, every minute of audio it processes, every frame of video, every line of code. Tokens are the unit out of which AI systems build everything they touch. They are also the unit out of which the AI economy is built: priced by the token, throttled by the token, and now, in the reasoning era, increasingly thought by the token. Most operators paying for AI today are paying for something they cannot see, in a unit they have no intuition for.

That has consequences.

A token is not a word

A token is a sub-word unit. When you send the sentence Tokens are not words to a large language model, the model does not receive five words. It receives something like five tokens: [Tokens][ are][ not][ words][.], where the leading space is part of the token and the punctuation is its own object. A different model might split it differently. The split is determined by the model's tokeniser, a fixed lookup table built once, at training time, and frozen for the life of the model.

The algorithm that builds most modern tokenisers is Byte Pair Encoding, or BPE. It was invented by Philip Gage in 1994 as a data compression scheme, described in a short article in The C Users Journal. Gage's idea was simple: find the most frequent pair of bytes in a file, replace it with a new symbol, repeat. After enough rounds, common sequences become single symbols and the file shrinks. Three decades later, the same procedure, adapted by Sennrich, Haddow and Birch in 2015 for neural machine translation, is how most frontier models decide what counts as a unit of meaning.

A typical tokeniser has a vocabulary of between 50,000 and 200,000 tokens. GPT-4 used a vocabulary of around 100,000; GPT-4o and the o-series use a larger one, of roughly 200,000. Claude, Llama, Gemini and DeepSeek each have their own. These vocabularies are not the same. A document tokenised by GPT-4 and the same document tokenised by Claude can produce different counts, sometimes off by 10 to 20 per cent for English, and far more for other languages.

The vocabulary is not a dictionary. It is a frequency artefact of the training data. Common English words tend to be single tokens. Less common ones get fragmented. The is one token. Strawberry is two or three. Antidisestablishmentarianism is six or seven. Made-up words, technical jargon, names and most non-Latin scripts get fragmented further. The tokeniser is, in effect, a compressed record of what the model's builders thought worth compressing.

A useful rule of thumb for English: one token is roughly four characters, or three-quarters of a word. A page of a novel is around 400 tokens. A 300-page book is roughly 120,000.

This is where the strawberry problem comes from. The model is not looking at s, t, r, a, w, b, e, r, r, y. It is looking at straw and berry, two opaque chunks. Asking it to count letters is asking it to inspect the internals of objects it does not have the internals of. It can sometimes get the answer right by reasoning around what it knows about spelling, but it is, fundamentally, working blind.

Fig. 1. Strawberry, as a human reads it and as a model sees it.

Once you see this, a category of model failure stops being mysterious. Strange behaviour around rhymes, anagrams, palindromes, character counts and spelling: all of it falls out of the tokeniser, not the model. The model thinks in chunks. The chunks are not, in any sense your intuition would recognise, words.

The model thinks in chunks. The chunks are not, in any sense your intuition would recognise, words.

Tokens are everything else

The token did not stay a property of text. Every other input a model can take has been forced through the same conceptual machinery.

Images are tokens, in modern vision models. The technique descends from Dosovitskiy and colleagues' 2020 work on the Vision Transformer, which split an image into a grid of small patches (typically 16 by 16 pixels) and treated each patch as a token in a sequence. GPT-4o, Claude, Gemini and Llama all do some version of this. A standard photograph passes through the tokeniser as a few hundred to a few thousand tokens, depending on resolution and how the model chooses to spend its budget. A high-resolution document image can be ten thousand or more.

Audio is tokens. Speech systems like OpenAI's Whisper, and the realtime audio APIs that grew out of them, convert sound into discrete tokens drawn from an audio vocabulary. A second of speech is between roughly five and fifty tokens depending on the codec. A long meeting transcribed and reasoned about by an AI is, at every step, tokens.

Video is the most token-hungry of all. A frame is an image, an image is hundreds of tokens, and a video at thirty frames a second adds up rapidly. Gemini, the leading long-video model, counts a minute of video at a few thousand tokens; longer or higher-resolution content scales accordingly. An hour of video can run to a hundred thousand tokens, which is a hefty chunk of any current context window.

Code is tokens too, with tokenisers tuned to syntax: keywords, operators, indentation and identifiers each get their own representation, often at a higher density than prose. The same function written in two languages can produce different token counts for the same logic.

The implication is rarely stated cleanly. Whatever you feed an AI, it becomes tokens before the model touches it, and it leaves the model as tokens before you see it. A multimodal API call counts image tokens, audio tokens and text tokens in the same currency, on the same invoice. AI strategies that talk about vision, voice and video as separate problems are describing a distinction the model does not recognise. From the inside, it is all tokens.

Fig. 2. Every modality. One stream of tokens.

Tokens are the meter

Sam Altman, OpenAI's chief executive, put the framing plainly at BlackRock's Infrastructure Summit in March 2026.

"We see a future where intelligence is a utility, like electricity or water, and people buy it from us on a meter." Sam Altman, BlackRock U.S. Infrastructure Summit, March 2026

The meter exists already, and has existed since the first API. What is changing is the price per unit and the scale of consumption. The unit has not changed.

API pricing for every major frontier lab is denominated in tokens, with separate rates for input (the tokens you send) and output (the tokens the model generates). Output tokens cost more, usually three to five times more, because they are the expensive side: each generated token requires a full forward pass through the model. As of mid-2026, Claude Sonnet runs in the order of $3 per million input tokens and $15 per million output; Claude Opus at roughly $15 input and $75 output; GPT-4o, OpenAI's mid-tier model, at around $2.50 input and $10 output. The reasoning models cost more again, because they generate enormous numbers of hidden tokens before producing their visible answer.

Context windows are token windows. When a vendor advertises a 200,000-token context, that is not 200,000 words. It is roughly 150,000 English words, about 500 pages of a novel. A two-million-token context, the upper end of what Google currently offers on Gemini, is roughly five novels. The number sounds enormous, and is, in some ways, but every doubling of context window quadruples the compute required under standard attention, which is part of why long contexts remain expensive.

Throughput is measured in tokens per second. Frontier models output at somewhere between fifty and two hundred tokens per second depending on architecture, hardware and load. A human reads at roughly five tokens per second. A state-of-the-art model is generating language about twenty to forty times faster than a human can read it.

These are the developer-facing prices. Above the API sits a second layer, where most people actually encounter AI, and where the unit becomes most thoroughly obscured. Almost no consumer product talks about tokens. Each one invents its own vocabulary to package what is, underneath, a token contract.

Claude Pro at $20 a month sells "messages per 5-hour window". ChatGPT Plus at $20 sells "messages per 3-hour window" and "Thinking messages per week". Google AI Pro at $20 sells "AI credits per generation". ElevenLabs Creator at $22 sells "characters". Midjourney Standard at $30 sells "Fast GPU hours". Lovable Pro at $25 sells "credits per action". Six products, six dialects, for what is, beneath the marketing layer, the same thing.

At the top tier the dialects collapse into a single phrase. Claude Max 20x sells "20x Pro usage" for $200. ChatGPT Pro sells "20x Plus limits" for $200. The convergence is not coincidence. At $200, every major lab is selling the same strategic message: stop worrying about the meter. The phrase "20x" is itself a dialect; the underlying claim is "we will sell you a token allowance large enough that you stop having to think about it."

Nine products. Six dialects. One unit. Verified pricing, May 2026.

Product Price / mo Dialect What it actually delivers Claude Pro $20 messages per 5h window ~45 messages / window; falls back to a lighter model when exceeded ChatGPT Plus $20 messages per 3h window 160 GPT-5 / 3h, plus 3,000 Thinking / week Google AI Pro $19.99 AI credits per generation 1,000 monthly credits, primarily for image and video generation ElevenLabs Creator $22 characters 100,000 characters / month = ~100 minutes of audio Midjourney Standard $30 fast GPU hours 15 fast hours = ~900 images; relax mode unlimited but queued Lovable Pro $25 credits per action 100 / month + 5 / day; a complex feature like auth costs ~1.2 credits Claude Max 20x $200 20x Pro usage ~220k tokens per 5h window; ~$3,650 of API value at heavy use ChatGPT Pro $200 20x Plus limits Effectively unlimited within abuse guardrails; 1M-token context Google AI Ultra $249.99 AI credits 25,000 credits / month; Gemini 3 Pro, Deep Think, Veo 3.1

At $200, every major lab is selling the same strategic message: stop worrying about the meter.

The dialects matter because they are doing real cognitive work. A user burning through "characters" at ElevenLabs and "credits" at Lovable in the same week has no easy way to compare what those two units cost, or to translate between them. The dialects make tokens incomparable across products, which is one of the things they are designed to do. Comparison would expose markup. Different dialects keep products in separate mental categories.

Three observations follow.

Every cost forecast that treats AI as software is wrong. Software has zero marginal cost per call. AI has a marginal cost per token, and the unit cost depends on which model you chose, how long your prompts are, and how much the model is generating. Migrating from one vendor to another can change your bill by a factor of five, even before counting that different tokenisers count tokens differently.

Optimising prompts is mostly optimising tokens. The prompt-caching, batch-API, structured-output and context-compression techniques that have appeared in the last eighteen months are all, underneath, token-economics tools. They reduce the number of input tokens, the number of output tokens, or the number of times the same tokens get re-processed.

Comparing products on their dialects is comparing the wrong thing. "$20 for ChatGPT Plus" and "$22 for ElevenLabs Creator" sound like the same purchase. They are not. They are two different vocabularies for two different token budgets, in two different modalities, with different unit costs underneath. To compare them, you have to translate both back to tokens. Almost no buyer does this.

Tokens are asymmetric

A token is not a neutral unit. It is a frequency artefact of a training corpus, and most training corpora are dominated by English. The consequence is a hidden tax on every language the tokeniser was not optimised for.

In a 2023 paper presented at NeurIPS, Aleksandar Petrov and colleagues at Oxford ran the same multilingual texts through the tokenisers of major language models and measured how many tokens each language required. The asymmetry was severe. Burmese required up to fifteen times as many tokens as English for the same content. Shan and Dzongkha were similar. Most languages of South and Southeast Asia, several African languages, and most non-Latin scripts cost between two and ten times more tokens than English for an equivalent message.

This is not an abstraction. It is a direct, compounded cost. A user writing in Burmese pays roughly fifteen times more, in API fees, for the same conversation. They also get roughly one-fifteenth of the effective context window: a 200,000-token context that would hold a substantial English novel will hold a fraction of that in Burmese. And, because long contexts degrade model performance, they are also more likely to hit the quality floor.

A user writing in Burmese pays roughly fifteen times more, in API fees, for the same conversation.

The asymmetry shows up everywhere downstream. Cheaper models with cheaper tokenisers do not help, because the tokeniser is the problem. Migrating to a different vendor can help slightly, but every major frontier tokeniser was trained on internet text, and the internet is English-heavy. The asymmetry is not a bug that one vendor has fixed. It is a structural feature of how tokenisers are built.

For an operator with global customers, this matters in three ways. AI products priced in dollars per query become more expensive to serve in some markets than others, by amounts that can swing margins. AI products that promise the same context window in every language are not, in any practical sense, offering the same product. And AI products that depend on long-context reasoning are functionally weaker in low-resource languages, in ways no amount of better prompting can fix.

There is a quieter geographic version of the same point. The cost per token at the lab door is set by the cost of compute, which is set by the cost of electricity and silicon, which is set by where the data centre lives. Where the tokens are made matters. Who can afford to make them matters more.

Tokens are cognition now

Until late 2024, tokens were a communication medium. The model received tokens, produced tokens, and the tokens were the answer. Then OpenAI released o1, and Anthropic followed with extended thinking on Claude, and Google with the equivalent on Gemini. With these reasoning models, tokens stopped being only the answer. They became the thought.

A reasoning model, given a hard problem, generates a long internal chain of tokens before producing its visible reply. The chain is not shown to the user, or is shown only in collapsed form. It is, in effect, the model thinking out loud to itself, in tokens, before composing what it wants to say. On a difficult mathematics or coding problem, a reasoning model can generate tens of thousands of hidden tokens to produce a few hundred visible ones. The hidden tokens are billed.

This is a different kind of pricing structure. With a non-reasoning model, you mostly know what you are paying for: input tokens you sent, output tokens you can see. With a reasoning model, you are paying for an amount of cognition that is decided by the model, not by you. Harder problems consume more tokens. Some problems consume orders of magnitude more.

It is also a different kind of capability structure. For most of the history of large language models, capability scaled with training compute: bigger models trained on more data on more GPUs. With reasoning models, capability also scales with inference compute, which is to say, with tokens spent thinking. A model that has been allowed to think for ten thousand tokens before answering is, in measurable ways, a more capable model than the same model answering immediately. The frontier of AI capability is now partly a frontier of how many tokens you are willing to pay for, per question.

The frontier of AI capability is now partly a frontier of how many tokens you are willing to pay for, per question.

Three consequences for any operator using AI.

The unit economics of AI products are now bimodal. Cheap, fast, non-reasoning calls are one product. Expensive, slow, deeply-reasoned calls are another. The same AI feature can vary in cost by two orders of magnitude depending on which mode it is in. Pricing pages that assume a single rate per call will mislead.

Vendor selection now includes a reasoning-budget question. Which vendor lets you cap thinking tokens? Which lets you escalate them? Which charges for hidden reasoning tokens at the same rate as visible output, and which gives a discount? The contracts are diverging.

Test-time compute is the new training compute. The narrative of "bigger model, more parameters" is being partly supplanted by "same model, more thinking tokens". For operators, this means capability gains may now come from spending more per query rather than waiting for the next model. For nations, it means token production capacity, which is to say GPU capacity at the inference layer, is the new strategic bottleneck. There is a reason every major government is now talking about sovereign compute. The unit it is measuring, in the end, is the token.

Coda

The newsletter a few weeks ago argued that, in the agentic era, data is the most important strategic asset a company has. This one argues something narrower and sharper. Data is the substrate; tokens are the form data takes when it moves. They are how data enters a model and how cognition leaves it. They are the meter on every conversation a model has ever had, and the geographic structure of where AI is cheap or expensive to operate.

A token is not a word. It is not a unit of meaning, nor a unit of language, nor a unit of thought in any sense your intuition is preparing you for. It is a frequency artefact of a training corpus, dressed up as a unit of account, doing the quiet work of pricing, throttling and, increasingly, thinking. The AI economy runs on it, asymmetrically, and at scales almost no operator currently models with precision.

Most of the words we use to talk about the modern economy are doing covert work. Token is doing the most concentrated covert work of any word we have looked at so far. It is denominated in cents, but it sets the cost of cognition.

Until next weeek,

Editor, Ideas for a Better World

Sources & further reading

1. Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr and Adel Bibi, Language Model Tokenizers Introduce Unfairness Between Languages, Advances in Neural Information Processing Systems (NeurIPS) 36, 2023. Interactive demo: tokenization-fairness.

2. Alexey Dosovitskiy et al., An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR 2021 (preprint October 2020). The Vision Transformer paper.

3. Alec Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision, OpenAI, September 2022. The Whisper paper, foundational for tokenised audio.

4. Rico Sennrich, Barry Haddow and Alexandra Birch, Neural Machine Translation of Rare Words with Subword Units, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), August 2016.

5. Philip Gage, A New Algorithm for Data Compression, The C Users Journal, vol. 12 no. 2, February 1994. The original BPE paper. See also Wikipedia: Byte pair encoding for context and references.

6. Jessica Rumbelow and Matthew Watkins, SolidGoldMagikarp (plus, prompt generation), LessWrong, February 2023. The anomalous-tokens investigation that made it clear tokens are concrete material objects in the model's vocabulary.

7. OpenAI, Learning to Reason with LLMs, September 2024. The o1 announcement and the inflection point at which test-time compute became a first-class capability axis.

8. Andrej Karpathy, Let's build the GPT Tokenizer, video lecture and accompanying minbpe codebase, 2024. The clearest available walk-through of how BPE actually runs.

9. Sam Altman, remarks at BlackRock's U.S. Infrastructure Summit, March 2026; see also Fortune's coverage of earlier remarks at the Federal Reserve, July 2025, on "intelligence too cheap to meter".