07 · Serving and scale
LLM serving and RAG
3 min read
Large language models use the same serving principles of bounded work, compatible inputs, and safe failure behavior, but generate their output in a sequence of tokens. This makes response length and active requests important to capacity planning. Retrieval-augmented generation, or RAG, also retrieves relevant information for the prompt, so the design must account for both finding evidence and generating an answer from it.
Separate prefill from decoding
A token is a unit of text that the model processes, which may be a word or part of one. During prefill, the model processes the prompt and stores attention keys and values for reuse in a key-value cache, often called the KV cache. During decoding, it generates further tokens while reusing that state, so a long prompt and a long answer affect different portions of the response time.
The KV cache uses memory that depends on the model architecture and the active sequence lengths. More concurrent requests can therefore increase memory demand even though the model weights stay unchanged. Deciding which requests to admit requires accounting for their expected token use and available memory as well as the number already being processed.
Use batching and caching with clear limits
Continuous batching lets a scheduler adapt its active work as requests finish and new ones arrive. Paged KV-memory management reduces waste from allocating large contiguous cache blocks. These techniques improve resource use, but you still need limits on context length, output length, queued work, and per-user demand.
Prefix caching can reuse computation for an identical reusable prompt prefix. It primarily saves prefill work; it does not eliminate the work of generating a new answer. Cache reuse must respect the serving system’s isolation model and compatibility with model and prompt versions. A response cache is a different mechanism and requires appropriate identity, permission, and freshness keys.
Benchmark with the expected distribution of input and output token lengths. A short-prompt throughput number is not a reliable capacity estimate for long document questions. Include queueing, retrieval, safety checks, and cancellation when measuring the user-facing endpoint.
Give RAG a complete data path
The ingestion path parses documents, creates chunks, attaches source and access metadata, computes embeddings, and updates an index. The request path authenticates the requester, retrieves permitted evidence, optionally reranks it, constructs a bounded prompt, and generates an answer with traceable sources. Deletions and permission changes must propagate independently of model retraining.
Chunk size trades context against retrieval precision and prompt cost. Small chunks can lose the explanation around a fact; large chunks can dilute retrieval and consume the context budget. Evaluate using representative questions and known supporting passages rather than selecting a size because it is common in examples.
Evaluate retrieval and generation separately
Measure whether the supporting evidence was retrieved before judging the generated answer. Then assess answer correctness, support from the cited passages, citation accuracy, refusal behavior when evidence is insufficient, latency, and cost. An answer containing a citation is not automatically supported by that citation.
Retrieved text may contain misleading instructions or information the requester is not permitted to access, so the system needs tests for both instruction manipulation and access failures. It also needs an acceptable response when the available evidence cannot support an answer. With the serving paths established, we can now examine how to monitor the complete system and recognize when its data or predictions need attention.