Token

What it means

Before a model can work with text it splits the text into tokens using a tokenizer. A token is not exactly a word: common words may be a single token while rarer words split into several, and roughly four characters of English map to one token on average. The model reads a sequence of input tokens (the prompt) and produces output tokens (the completion) one at a time. Every model also has a context window, the maximum number of tokens it can consider at once.

Why it matters

Tokens are the unit of both cost and capacity. Providers bill per input and output token, usually at different rates, so token counts directly determine your API spend. They also bound what fits in a request: long documents, chat history and retrieved context all compete for the same context window. Understanding token usage is therefore central to controlling cost, avoiding truncation and diagnosing latency, since generation time scales with the number of output tokens.

In practice

LLM observability tools record token counts on every traced request so you can attribute spend to features, users or prompts. Helicone and Portkey act as gateways that log input and output tokens and roll them into cost dashboards, while Langfuse captures the same on its traces. Watching token trends is how teams spot a prompt that quietly ballooned or a retrieval step stuffing too much context into the window. See how to reduce LLM costs for tactics that start from token usage.