🧩 What is a 'token' in AI?
The chunks of text a model actually reads — usually smaller than a word.
2 min read
A token is the chunk of text an AI model actually works with. It's often smaller than a word: 'artificial' might split into 'arti' + 'ficial', and a space counts as part of a token too. Roughly speaking, one token is about three-quarters of an English word.
Before a model like ChatGPT reads your message, it breaks the text into tokens. It doesn't see letters or whole sentences — it sees a sequence of tokens, each mapped to a number. That sequence is what it predicts on top of, one token at a time.
Why does this matter to you? Because models are priced and limited by tokens. When a tool says '8,000 token context', it means the model can 'see' about 6,000 words of conversation. When you pay per token, you're paying for how much text you feed in and get back.
You can feel this working: type a word into the tokenizer demo on this page and watch it break into pieces. That's exactly what the model 'sees'.
How text becomes tokens
Type a word and see how a model would split it into "tokens" — the chunks it actually reads. Usually smaller than a word, and spaces count.
3 tokens
Roughly ¾ of an English word per token. This is a simplified demonstration — real models use a much larger vocabulary.
Type any word below and watch it split into the tokens the model actually reads.
🕒 Try it in 60 seconds
- 1Type a long, unusual word into the demo below and watch it split.
- 2Ask an AI: 'About how many tokens is one page of text?'
- 3Then ask: 'What's the difference between a token and a word?'
Want the full picture?
Get the free 5-day "AI in your job" email course — jargon-free, five minutes a day.