Token
Token
A token is a small piece of text that an AI model reads or writes. It may be a word, part of a word or punctuation.
Explained simply
A token is a small piece of text that an AI model reads or writes. It may be a word, part of a word or punctuation.
At a glance
- Category
- Models
- Difficulty
- Beginner
- Introduced
- 2018
Real example
A long report contains many tokens, which affects how much the model can process and sometimes the cost.
Why it matters
Tokens affect context limits, processing time and pricing in many AI services.
Timeline
The ideas behind Token begin developing.
Token becomes a recognised term or technique.
Research and practical applications increase.
Token becomes connected to newer AI systems and products.
Token remains relevant in the models area of AI.
Learn next
Continue with these connected terms:
Simple infographic
A quick visual way to understand Token.
Was this helpful?
Your feedback helps improve this explanation.