NVIDIA inference performance guide
Primary technical reference on inference latency, throughput, batching, and performance measurement.
← Back to Compute College
Compute College
Latency: TTFT, output speed, and end-to-end response time
Separate time to first token, generation speed, and total response time when evaluating AI systems.
Plain-English definition
AI latency is not one number. Time to first token measures how long users wait before output begins; output speed measures how quickly tokens arrive; end-to-end latency includes queueing, retrieval, tool calls, generation, and post-processing. The right metric depends on what the user is trying to do.
Memory trick: Latency is a timeline, not a single speed number.
Why it matters
A fast first token can still hide a slow complete answer, while a fast total response can feel poor if the interface waits silently. Separating stages shows whether the bottleneck is queueing, context processing, model generation, tools, or capacity headroom.
- TTFT shapes perceived responsiveness.
- Output speed and total length shape time to completion.
- Queue and tool time can dominate model execution time.
Simple example
An assistant has 400 ms of queue time, 1.6 seconds to first token, generates 35 tokens per second, and produces 280 output tokens. A dashboard that reports only average API latency hides the queue and the user’s total wait.
- The response budget can be split into measurable stages.
- Shorter output may improve completion time without changing the model.
- Tail latency matters when many users share capacity.
Example figures are illustrative calculations, not current quoted market prices.
Current example
Primary reference
This source explains the method or risk boundary. It does not replace testing the actual model, data, provider, and workload conditions in production.
Source discipline: examples on this page are illustrative; measure the real workload before making a release or capacity decision.
Common mistake
Optimizing average latency can leave users waiting at the tail. A system may look fast in a quiet test and still queue or time out during a traffic burst.
Practical takeaway
What you can do with this
Instrument a representative request from arrival to completion. Set a target for TTFT and total response time, then identify which stage consumes the budget.
- Record token counts beside latency.
- Test at expected peak concurrency.
- Use streaming only when it improves the actual user task.
Decision check: can you name the stage that would need more capacity or a design change if p95 response time misses its target?
Compute College learning path
AI Engineering
Step 42 of 48: Latency: TTFT, output speed, and end-to-end response time