Token

A Token is the smallest unit into which a Large Language Model breaks down text. Instead of whole words, LLMs process tokens — word pieces typically 3–4 characters long. "Marketing" becomes "Mark" + "eting," while short words like "the" are a single token. In English, it's about 1.3 tokens per word, in German about 1.5.

Tokens determine both the costs and limits of AI applications. API prices are calculated per million tokens: GPT-5.2 costs $1.75 input / $14 output, Claude Opus 4.6 costs $15/$75, Gemini 3.1 Pro costs $1.25/$5. The context window — how much text a model can process at once — is also measured in tokens.

Tokenization uses algorithms like BPE (Byte Pair Encoding) that merge frequent character sequences into single tokens. Different models use different tokenizers — the same text may have different token counts in GPT vs. Claude. OpenAI's tiktoken library enables exact pre-calculation.

Practical takeaway: long prompts (system + context + question) consume input tokens, the response consumes output tokens (which are more expensive). Efficient Prompt Engineering not only saves costs but also leaves more room in the context window for relevant information.