NVIDIA AI inference guide
Technical reference on describing inference workloads through throughput, latency, batching, and hardware behavior.
← Back to Compute College
Compute College
What is an AI workload?
Define an AI workload by its task, traffic, model behavior, latency, reliability, and compute requirements.
Plain-English definition
An AI workload is everything required to produce a result: the task, inputs, model calls, context, tools, traffic, latency, reliability, and resources. Saying “we use an AI model” is not enough for planning; two features can use the same model and still create very different demand and cost.
Memory trick: A workload is the whole job, not just the model.
Why it matters
Workload definition is the bridge between product intent and infrastructure. It determines whether a hosted API, managed deployment, or self-hosted GPUs can meet the requirement, and it gives buyers a common way to compare quotes and capacity.
- Traffic and concurrency shape capacity even when the model stays the same.
- Input/output tokens, tools, and retries shape variable cost and latency.
- Reliability and data boundaries can matter as much as raw model quality.
Simple example
A document assistant handles 10,000 requests per day, averages 4,000 input tokens and 700 output tokens, retrieves two passages, and must answer within 8 seconds at p95. That description is useful for capacity and cost planning; “chat with documents” is not.
- Volume is separated from peak concurrency.
- Token and retrieval assumptions expose the input workload.
- The latency target creates a service requirement beyond model selection.
Example figures are illustrative calculations, not current quoted market prices.
Current example
Primary reference
This source explains the method or risk boundary. It does not replace testing the actual model, data, provider, and workload conditions in production.
Source discipline: examples on this page are illustrative; measure the real workload before making a release or capacity decision.
Common mistake
Model name and GPU count are not enough to size a workload. The same model can be cheap for low-volume batch work and expensive for interactive, high-concurrency serving.
Practical takeaway
What you can do with this
Write a workload card with task, traffic, token ranges, model calls, tools, latency, availability, data boundary, and success criteria. Use it as the input to every provider or deployment comparison.
- Include peak and average traffic.
- Record retries, review, and fallback behavior.
- Update the card when product requirements change.
Decision check: could a provider use your workload card to produce a comparable quote without guessing the important assumptions?
Compute College learning path
AI Engineering
Step 37 of 48: What is an AI workload?