Starter guide
What Is an Inference Server?
An inference server is a program that keeps a trained model ready for use. An application sends it a request, and the server runs the model and returns the answer.
The server loads the model once instead of loading it for every request. It also checks inputs, manages several requests, and controls how the hardware is used.
What happens to one request?
The server checks the input and turns the text into small pieces the model can read. The model creates the answer, and the server sends that answer back to the application.
The server can return the whole answer at once. It can also stream the answer in small parts while the model is still working.