NVIDIA inference performance guide
Primary technical guidance on throughput, latency, batching, and inference workload behavior.
← Back to Compute College
Compute College
AI workload patterns
Compare batch, interactive, streaming, retrieval-heavy, and agentic workload patterns before sizing compute.
Plain-English definition
An AI workload pattern describes how requests arrive and what users expect from the service. Batch jobs can wait to gain throughput; interactive requests need quick responses; streaming produces output as it goes; retrieval-heavy and agent workflows add their own processing and tool calls.
Memory trick: Workload pattern is the shape of demand over time.
Why it matters
Pattern determines the bottleneck. A batch job may benefit from queueing and high utilization, while an interactive assistant may need spare capacity for tail latency. Treating every workload as the same API request leads to poor capacity and cost decisions.
- Arrival shape determines concurrency and queueing.
- Response requirements determine latency and batching tradeoffs.
- Tools, retrieval, and variable steps make per-request cost less predictable.
Simple example
A nightly embedding job can run when GPUs are available, while a customer assistant must start responding quickly during a daytime traffic peak. The first can use queued batch capacity; the second needs a serving pool with headroom and a fallback plan.
- The two workloads may share a model but not a capacity strategy.
- Queueing is acceptable for batch but not necessarily for conversation.
- Headroom protects interactive tail latency.
Example figures are illustrative calculations, not current quoted market prices.
Current example
Primary reference
This source explains the method or risk boundary. It does not replace testing the actual model, data, provider, and workload conditions in production.
Source discipline: examples on this page are illustrative; measure the real workload before making a release or capacity decision.
Common mistake
Average requests per day can hide the capacity an interactive workload needs. Peaks, concurrency, output speed, and retries determine whether users experience a reliable service.
Practical takeaway
What you can do with this
Classify the feature by arrival pattern, latency need, token shape, tool use, and failure tolerance. Build a simple workload table before comparing deployment options.
- Separate batch, interactive, and background traffic.
- Record concurrency and burst behavior.
- Choose a fallback for each pattern rather than assuming one model serves all cases.
Decision check: does your capacity plan match the way requests arrive and the time users have agreed to wait?
Compute College learning path
AI Engineering
Step 39 of 48: Workload patterns