Starter guide
How to Build an Inference Server
Begin with one model and one request. Load the model when the server starts, accept an input, run the model, and return the answer.
Once that works, add streaming and a queue. The queue stops too many requests from trying to use the hardware at the same time.
The basic process
Load the model, define the input and output, create one server endpoint, and connect it to the model. Test the full path with one request before adding more features.
Next, stream the answer as it is created. Then add a queue for multiple requests. Test each change before moving to the next one.