NVIDIA Triton documentation
Primary documentation on model serving, batching, concurrency, and inference server behavior.
← Back to Compute College
Compute College
Throughput, concurrency, batching, and queues
Understand how request volume, concurrency, batching, and queueing determine serving capacity.
Plain-English definition
Throughput is the amount of useful work a system completes over time; concurrency is the number of requests in progress; batching groups work to improve hardware utilization; queues hold requests when demand exceeds immediate service capacity. These interact: more batching can improve throughput while increasing waiting time.
Memory trick: Throughput is useful work per time; concurrency is work in flight.
Why it matters
Serving capacity is a balance between utilization and responsiveness. A system with no queue may waste capacity during quiet periods, while a full queue signals demand beyond the service target. Understanding the tradeoff helps teams compare GPU counts and provider capacity honestly.
- Concurrency and arrival bursts determine queue pressure.
- Batching can improve utilization but may add delay or reduce flexibility.
- Throughput should be measured as accepted useful work, not just generated tokens.
Simple example
A service receives 40 requests per second at peak. Each request needs 0.4 seconds of average processing, so a rough concurrency requirement is 16 active requests before headroom, retries, and tail behavior. The real test must measure the chosen model, batch policy, and token shape.
- The simple estimate is a starting point, not a capacity guarantee.
- Long requests can occupy capacity unevenly.
- Queue time and rejection rate reveal when the target is missed.
Example figures are illustrative calculations, not current quoted market prices.
Current example
Primary reference
This source explains the method or risk boundary. It does not replace testing the actual model, data, provider, and workload conditions in production.
Source discipline: examples on this page are illustrative; measure the real workload before making a release or capacity decision.
Common mistake
Theoretical tokens per second are not the same as usable throughput. Real workloads include input processing, variable outputs, queueing, memory limits, and failed or reviewed requests.
Practical takeaway
What you can do with this
Load-test the workload at expected concurrency. Record throughput, queue time, p95 latency, batch size, GPU utilization, tokens, and accepted-task rate.
- Test short and long requests separately.
- Include retries and tool calls in the load shape.
- Leave headroom for bursts and provider degradation.
Decision check: does the claimed capacity meet the user latency target while counting queues, retries, and useful accepted work?
Compute College learning path
AI Engineering
Step 43 of 48: Throughput, concurrency, batching, and queues