“Send me the token.” “We ran out of tokens.” “It’s a token on Ethereum.”
Three sentences, three unrelated things, one word. Add the two older meanings from linguistics and compilers and you have five. Anyone building with AI agents runs into most of them in the same afternoon, because an agent spends tokens to call an API with a token.
They share one idea: a token is a small thing that stands in for something else. What it stands in for is where they part ways. Each section below has a short animation. They play as you scroll, and the dots let you step through them yourself.
A pass you show at the door
A security token stands in for you. You log in once, the server hands you a string, and from then on the string is shown instead of your password.
The most common shape is a JWT, a JSON Web Token.
- A security token looks like noise. It is one string, sent along with every request.
- Two dots split it into three parts.
- The first two are only base64. Anyone holding the token can read them.
- The third is a signature. The server recomputes it with a secret only it knows, and a match means nobody changed a byte.
- The token carries its own expiry. An hour later the same string opens nothing.
Two things are worth remembering from that.
- It is signed, not encrypted. Anyone who gets hold of the token can read what is in it. Never put a secret in the payload.
- Whoever holds it is you. The server does not ask how you got the string. That is why these expire, and why this is the one kind of token that must never end up in a chat, a log or a screenshot.
Not every security token is a JWT. Many are opaque random strings that the server looks up in a database. The role is the same: a stand-in for a login.
A piece of text with a number
A language model’s token stands in for a fragment of text. Models do not read letters or words. A tokenizer cuts the text into pieces from a fixed vocabulary and hands over the piece numbers.
- To you this is a sentence: six words and a full stop.
- The tokenizer cuts it into pieces it has seen often. Common words survive whole. “Unbelievably” does not.
- Each piece is swapped for a number. This list of numbers is all the model ever sees.
- The same sentence in Norwegian is 14 tokens instead of 8. Models read, remember and bill by the token, so the language you write in changes the price.
Those are real numbers from GPT-4’s tokenizer. Other models cut differently, but the consequences are the same everywhere.
- Tokens are the unit of everything. Context windows, rate limits and invoices are all counted in them.
- A token is not a word. In English it averages about three quarters of one. Rare words, names, code and most other languages cost more.
- It is not a secret. The vocabulary is public, and the same text gives the same tokens every time.
A row in a shared table
A blockchain token stands in for value or ownership. The surprise is where it lives: not in your wallet. The wallet holds a key. The token is an entry in a ledger.
- A blockchain token is not a thing you hold. It is a number next to your address, in a table that thousands of computers keep a copy of.
- To send 30, Alice signs a message saying so. Her private key is the only proof of ownership there is.
- The network puts the message in a block, and every copy of the table adds that block.
- Now the table says 70 and 50. Nothing travelled anywhere. Everyone agrees on new numbers.
A fungible token like a stablecoin is a balance: a number per address. A non-fungible one turns the table around, and maps one item to the address that owns it.
Either way, copying the token is meaningless. There is nothing to copy. What you can lose is the key that signs for it.
One occurrence of a word
This is the oldest of the five. In linguistics a token is one occurrence of a word in a text, as opposed to the word itself, which is a type.
- A linguist's token is one occurrence of a word. This line has eight.
- Colour the repeats, and there are only three different words in it.
- Those are the types. Eight tokens, three types: the ratio is a rough measure of how varied a text is.
It sounds like a footnote, but it is the meaning the AI one grew out of. Counting occurrences is how language was first turned into statistics, and a tokenizer is the same idea with smaller pieces.
The smallest unit of code
A compiler’s token stands in for a slice of source code that means something. Before a program can be understood it is cut into keywords, names, operators and literals. That step is the lexer.
- A compiler reads source one character at a time. To it, this line is 13 characters.
- The lexer groups them into the smallest units that mean something, and throws the spaces away.
- Each token gets a type. From here on the parser only reads the types.
- Hand the same line to GPT-4 and it is six tokens, not five. One of them is a space the compiler threw away.
The last step is the one to keep. The same thirteen characters were cut by two tokenizers into two different sets of tokens, and neither is wrong. A token only means something to the system that cut it.
Side by side
| Stands in for | Made by | If someone copies it | |
|---|---|---|---|
| Security | You, logged in | The login server | They are you until it expires |
| Language model | A fragment of text | The tokenizer | Nothing. The vocabulary is public |
| Blockchain | Value or ownership | A contract on a ledger | Nothing. Only the key can move it |
| Linguistics | One occurrence of a word | Whoever wrote the text | Nothing |
| Compiler | A meaningful slice of code | The lexer | Nothing |
Only one row is a secret. That is the practical point of knowing the difference: when a tool, a colleague or an AI agent asks for “the token”, the first question is which row it is on.
Look inside one
Here is the token from the first animation. Change a character in the payload, or paste one of your own from a test system.
Decoded in your browser. Nothing is sent anywhere, and the signature is shown, not checked: checking it takes the secret.
The header and payload opened without a key. That is the “signed, not encrypted” part, seen first hand.
Further reading
- Introduction to JSON Web Tokens, and OAuth 2.0 for how they are handed out
- OpenAI’s tokenizer, to cut your own text into model tokens
- Token standards on Ethereum, including ERC-20
- Lexical analysis and the type–token distinction on Wikipedia
This post replaces Token Explorer, a small site that used to live at token.kristoffer.dev. That address now lands here.