Token

The small chunks of text — words or pieces of words — that a language model reads and generates.

A token is the unit of text an AI language model actually works with. Rather than reading whole sentences or single letters, models break text into tokens, which are roughly word-sized pieces (a common rule of thumb is that 100 tokens is about 75 words). The word "unbelievable" might be split into a few tokens, while common short words are usually one.

Tokens matter for two practical reasons: models have a maximum number of tokens they can handle at once (their "context window"), and paid AI services usually charge per token. If a tool says your text is too long, it has hit a token limit — shortening or summarizing the input fixes it.

Related terms

Large Language Model (LLM) · Prompt

Try it with these free tools

← Back to the AI Glossary