LLM basics: tokens, context, model

LLM, token and context window

An LLM (large language model) is a neural network trained on a large text corpus to predict the next token. It does not "know" facts like a database and does not execute code: at each step it produces a probability distribution over the next token and one is picked. That is both the strength (flexible text understanding) and the weakness (a plausible but wrong answer).

A token is the unit the model reads and writes: a piece of a word, a word, a punctuation mark. For English, 1 token is about 4 characters or ¾ of a word; Ukrainian and code are usually "more expensive" because they split into more tokens. The model does not see letters, which is why it is bad at counting letters in a word or exact arithmetic.

The context window is the maximum number of tokens the model sees in one request: system prompt, conversation history, attached documents, tool results and the answer itself. Whatever did not fit, the model simply does not know. A long context is not free: it is more expensive, slower, and quality often drops for information "in the middle" (lost in the middle).

Context in everyday speech is everything we passed to the model in the request. The model itself is stateless: the "memory" of a conversation is the client re-sending the whole history every time.

Model, version and generation parameters

A model is a specific set of trained weights with a name and a version (for example, a family of large/medium/small models from one vendor). A bigger model is usually smarter but slower and more expensive; a small one is fast and cheap for simple tasks (classification, extraction). For QA this matters: the model version is part of the configuration. A vendor may move an "alias" to a new version and the product behaviour changes without a single commit. That is why tests and production pin a concrete model identifier.

Generation parameters:

• temperature is the "randomness" of token choice. 0 is almost deterministic (the most likely token), higher is more varied and less predictable;
• top_p (nucleus sampling) picks only from the tokens whose cumulative probability is ≤ p;
• max_tokens limits the answer length; when exceeded the answer is cut mid-sentence (finish_reason = length);
• stop sequences are strings at which generation stops;
• seed is not supported by every vendor and does not guarantee full reproducibility.

Important: even with temperature: 0 the answer may differ between runs (parallel GPU computation, model updates, batching). A test must not assume byte-for-byte equal output.

Credits, price and limits

API models are billed per token: input (everything we sent) and output (what the model generated) are priced separately. Output is usually several times more expensive than input. Request cost = input_tokens × input_price + output_tokens × output_price.

Credits are an abstract payment unit: a prepaid balance or "points" in a subscription into which the service converts tokens, requests or agent minutes. When credits run out the service returns an error or disables the feature. For QA this is a separate class of scenarios: what the user sees at a zero balance, whether credits are charged for a failed request, whether the counter is correct.

What else affects cost and speed:

• Context length: every step of a dialogue or agent re-sends the history, so cost grows faster than linearly;
• Prompt caching: a repeated prefix (system prompt, documents) is cached and is cheaper and faster;
• Rate limits: limits on requests and tokens per minute. A 429 response means retry with exponential backoff;
• Latency: time to first token (TTFT) and total time; streaming makes the answer "live" before generation finishes;
• Agents: one user request can trigger dozens of model calls, so a budget and a step limit are needed.