In short
An LLM turns text into tokens, then predicts one token at a time from what is in its context window. It optimises for a plausible continuation, not a verified fact, so production systems need grounding, evaluation sets, structured outputs and per-token monitoring.
Most surprises with LLMs in production come from four mechanics: tokens, next-token prediction, the context window and sampling. Once you see how an LLM produces an answer, the bill, the latency and the occasional confident wrong answer stop being mysterious. They become engineering problems with known fixes.
Text becomes tokens
A model never sees characters or words. A tokenizer splits text into tokens: whole words, pieces of words, punctuation or bytes. Most tokenizers learn subword units with methods such as byte-pair encoding, so common words stay whole and rare ones split into pieces.
An illustrative split: one short sentence, eight tokens, and rare words cost more than common ones.
Everything is counted in tokens: context limits, rate limits and price. The ratio to text is tokenizer-dependent. OpenAI's rule of thumb is about four English characters per token; Anthropic's docs put earlier Claude models at about 3.5, with newer models producing roughly 30% more tokens for the same text. Measure with the provider's token counting tool, not a word count.
One token at a time
An LLM is autoregressive. Given the tokens so far, it outputs a probability for every token in its vocabulary. One token is chosen, appended to the input, and the loop runs again until a stop condition or max_tokens.
Choosing is called decoding. Greedy decoding always takes the top token. Sampling draws from the distribution, and temperature reshapes it first: low temperature sharpens it towards the top choice, high temperature flattens it so less likely tokens get picked more often.
Same prompt, same model: low temperature keeps picking the top token, high temperature spreads the picks.
That is why the same prompt can give different answers. Even temperature 0 was never fully deterministic on Claude's API, and some current models, such as Claude Opus 5.5, have removed temperature, top_p and top_k: the migration guide says to omit them. Design for variation instead of trying to switch it off.
The context window is the only working memory
The model keeps nothing between requests. Everything it can use (system prompt, history, retrieved documents, tool results and its own output) has to fit in the context window, and all of it is sent and billed again on every turn. Prompt caching lowers the price of a repeated prefix, not its size.
A long conversation outgrows the window, and something has to decide what leaves it.
The Claude API does not slide the window for you. An input that is too long is rejected, and a response that hits the limit stops. Your application decides what to drop, summarise or compact. More context is not automatically better either: Anthropic's docs note that recall degrades as token count grows.
Why models hallucinate
Pretraining teaches a model to predict the next token, not to look up facts. When the context lacks the answer, the most plausible continuation can still be fluent, specific and wrong. A 2025 paper by Kalai and colleagues argues that training and benchmark scoring reward guessing over saying "I don't know", which keeps the behaviour alive.
What this means in production
- Grounding. Put the source material in the context and tell the model to answer only from it, with quotes or citations. Choosing how is its own decision: see RAG vs CAG.
- Evaluation sets. Because outputs vary, one good demo proves little. Keep a versioned set of real questions and edge cases, grade it automatically where you can, and rerun it on every prompt or model change.
- Structured outputs. When code consumes the answer, constrain it to a JSON schema (
output_config.formaton the Claude API) instead of parsing prose. Still handle refusals andmax_tokenscut-offs. - Per-token monitoring. Log input, output and cached tokens from the
usagefield, plus time to first token and total latency, per feature. On Claude's price list, output tokens cost more than input tokens, so long answers drive both cost and wait time.
In practice
- Count tokens with the provider's API before you set limits or budgets.
- Set
max_tokensdeliberately and handle every stop reason. - Trim or compact history before it reaches the window, not after an error.
- Ground factual answers in supplied sources and allow "I don't know".
- Gate prompt and model changes on an evaluation set.
- Dashboard cost and latency per token, per feature.
Building something that has to work every time, not just in the demo? See how we build AI systems.
Sources
- Anthropic: Glossary (tokens, temperature, pretraining, context window)
- Anthropic: Context windows
- Anthropic: Claude Opus 5.5 migration guide (sampling parameters)
- Anthropic: Structured outputs
- Anthropic: Reduce hallucinations
- Anthropic: Create strong empirical evaluations
- Anthropic: Pricing
- OpenAI: Key concepts (tokens)
- Hugging Face LLM Course: Byte-Pair Encoding tokenization
- Hugging Face Transformers: Generation strategies
- Hugging Face blog: How to generate text
- Sennrich, Haddow, Birch (2015): Neural Machine Translation of Rare Words with Subword Units
- Holtzman et al. (2019): The Curious Case of Neural Text Degeneration
- Kalai, Nachum, Vempala, Zhang (2025): Why Language Models Hallucinate
