- Receive user request.
- Build prompt or input payload.
- Tokenize input.
- Run prefill.
- Generate tokens during decode.
- Stream or return output.
- Validate and post-process output.
- Record logs and metrics.
- Time to first token
- End-to-end latency
- Input tokens
- Output tokens
- Tokens per second
- Requests per second
- GPU memory usage
- GPU utilization
- Queue length
- Error rate
- Cost per request
- Long prompts
- Long outputs
- Too many concurrent requests
- GPU memory exhaustion
- Poor batching
- Slow retrieval
- Cold starts
- Network transfer between GPUs or services
- Reduce prompt size.
- Set output limits.
- Enable streaming.
- Use a serving engine built for LLMs.
- Batch requests where appropriate.
- Quantize only after measuring quality.
- Cache repeated safe work.