Guided project
Build an Inference Server
Learn what an inference server does, follow the basic steps, then build one in the notebook project. For background, read how the KV cache and PagedAttention work, or practice serving problems in the Cracking Inference study plan.
One inference request
Take a request, generate the answer, and return it to the application.
Request
validate input
Queue
wait for GPU access
Generate
create the answer
Stream
return the response
Request
Input
Generation
Work
Response
Output