Compute College
Plain-English definitions of AI compute terms — each linked to a full lesson where one exists.
GPU-hours, neoclouds, HBM, spot pricing, PUE, compute futures, and more, defined in one place. Jump to a letter or open the full lesson behind any term.
Learn the Market
A GPU-hour is one graphics processor running for one hour. It is one of the simplest units for comparing AI compute because it turns a messy market into a measurable rental unit. If a provider charges for one H100 running for one hour, that is an H100 GPU-hour. ComputeTape can use GPU-hours to compare on-demand cloud, reserved capacity, neocloud rentals, and spot compute.
Reference guide
This glossary defines the AI compute terms that show up in GPU pricing, model-serving costs, capacity contracts, power deals, and data-center buildouts. Each definition stands on its own, and most terms link to a full Compute College lesson with examples, market signals, common mistakes, and related concepts.
A–Z
An agent loop is the repeated cycle of choosing a next step, using a tool, and checking what happened. The cycle needs a stopping rule so the system knows when the task is done or should be handed back. A research assistant searches, reads a page, and decides whether it has enough evidence or needs another search.
Many GPUs wired together with fast networking so they act as one machine for training or large-scale inference.
The accelerators, memory, networking, power, and cooling that together produce usable AI processing capacity.
The choice between paying per token for a hosted model API and running the model yourself on rented or owned GPUs.
NVIDIA's Blackwell-generation data-center GPU, the successor to the H100 and H200 for AI training and inference.
Electricity generated on-site and used directly by a data center without passing through the public grid connection.
The market where GPU capacity is priced, reserved, and traded as a scarce, time-sensitive resource.
An agreement to buy compute capacity for a future period at a price set today.
Forward-looking pricing for compute capacity delivered later, used to plan or hedge future GPU needs.
Capacity booked in advance so it is guaranteed available when a workload needs it.
Context compaction replaces a long conversation history with a shorter summary of what still matters. It frees room for more work while keeping key decisions, constraints, and unfinished tasks available. An assistant can drop repeated logs while keeping the decision that a particular database migration must wait.
Continuous batching lets an inference server adjust its group of requests while it is running. New requests can join as finished ones leave, rather than waiting for an entire batch to end. One user asks for a sentence and another asks for an essay. When the sentence is finished, its space can be used by a third request.
How much heat a data center can remove per rack, which limits how many high-power GPUs fit in a space.
Cost per completed task is the total spend needed to get one usable result. Count retries, tool calls, and failed attempts along with the final successful model request. If ten accepted results cost $20 across all attempts, each completed task costs $2.
How hosted AI APIs price inference, usually pricing input and output tokens separately; output typically drives the bill.
Curtailable load is electricity demand that can be reduced or disconnected under agreed conditions. A data center accepting those terms may have different access options from one requiring uninterrupted grid service. A facility may agree to reduce its grid draw during system stress, then use backup power or pause suitable work.
A facility that houses servers, power, networking, and cooling — the physical home of AI compute.
The process and capacity needed to connect a data-center site to the power grid.
The waiting line of projects seeking a grid connection, a common bottleneck that delays new AI capacity.
Decode is the stage in which a language model generates its answer, usually one token at a time. It follows the initial reading of the prompt. An answer may start quickly but take a while to finish because it contains thousands of tokens.
Disaggregated inference runs prompt reading and answer generation on separate workers. The workers pass the saved attention data between them, so each group can be sized for its own workload. A service receiving long documents might add prompt-reading workers without adding the same number of answer-generation workers.
An evaluation harness is the software that runs a model or agent through a set of test cases and records the results. It helps repeat the same checks after a change. Before changing models, a team reruns its invoice examples and checks extraction accuracy, cost, and response time.
A view of expected compute price or capacity value across future time periods.
The estimated recurring cost of running a leading AI model to answer users after it has been trained.
The amount of GPU power a cloud can actually deliver to customers, set by hardware, power, and networking.
GPU memory headroom is the memory left after a workload has loaded its model and other required data. Some of that space must remain available as requests and saved attention data grow. A model that just fits on one GPU may leave too little room to serve long conversations at the concurrency you need.
Renting GPU time by the hour or by reservation instead of buying and operating the hardware.
How much useful work a paid GPU actually does; low utilization raises the real cost per result.
Borrowing using GPUs or their rental contracts as collateral, so operators can buy large fleets without paying the full cost upfront.
One GPU made available for one hour — the baseline unit for pricing accelerator rental time.
NVIDIA's Hopper-generation data-center GPU, a widely used baseline for AI training and inference pricing.
An upgraded Hopper GPU with more and faster memory than the H100 for larger models.
Fast memory stacked next to a GPU; its supply and capacity are a key constraint on advanced accelerators.
A high-speed, low-latency network used to link GPUs across a cluster for training.
Inter-token latency is the time between successive output tokens. It describes the pace of a streaming answer after the first token arrives. A reply can start immediately and still feel slow if there are long gaps as the remaining text appears.
The KV cache saves information the model has already calculated about earlier tokens. Reusing it avoids repeating work as the model writes an answer, but the cache takes up memory alongside the model itself. A long chat can fill much more cache than a short question, even when both use the same model.
Cooling GPUs with circulating liquid instead of air, needed as chip power and rack density rise.
Using power (megawatts) as a proxy for how much AI compute a site can run.
How much of a GPU's theoretical compute a job actually uses for useful work; low MFU means paying for compute you are not using.
Model Context Protocol, or MCP, is a shared way for AI applications to connect to tools and data sources. It standardizes the connection; it does not decide which actions a user has authorized. An application can use an MCP connection to read a project tracker instead of building a different custom connector for every client.
The one-time compute expense of teaching a model, driven by GPU-hours, hardware, and run length.
A compute-first cloud operator focused on high-performance AI infrastructure rather than general-purpose services.
The service-level guarantees a neocloud makes on availability, performance, and support for rented capacity.
A rack-scale system that links many GPUs with high-speed interconnect so they act as one large accelerator (scale-up).
NVIDIA's high-speed link connecting GPUs inside a server so they share data quickly.
Paying for GPU capacity as you use it with no commitment — flexible but usually priced higher per hour.
A ratio of total facility power to IT power; lower is more efficient, with 1.0 the ideal.
A long-term contract to buy electricity at agreed terms, used to secure firm power for AI data centers.
Prefill is the work a model does to read the input before it starts writing an answer. Longer prompts usually mean more work at this stage. A question attached to a long report can take longer to start answering than the same question on its own.
A prefix cache hit happens when a serving system reuses saved work for an identical opening part of a prompt. It avoids reading that shared beginning from scratch. Two requests with the same long instructions may reuse the instructions, while their different questions still need fresh processing.
Prompt injection is an attempt to make a model follow instructions hidden in material it was meant to read. The material might be a web page, document, or tool result. A retrieved page tells an assistant to ignore the user and reveal private information. That page is evidence to read, not permission to act.
A reasoning budget limits how much extra work a model can spend working through an answer. Depending on the service, it may be controlled through a token allowance or an effort setting. A simple lookup may need little extra work; a difficult planning problem may justify a larger allowance.
The higher serving cost of models that generate long chains of thought, because output tokens drive inference cost.
GPU capacity committed for a set term in exchange for guaranteed availability and usually a lower rate.
A retry budget limits how many times a system may attempt a failed step again. It can also cap the time or money spent on those attempts. A failed tool call may be tried twice, then sent back to a person instead of being repeated all afternoon.
Scale-up connects accelerators tightly within one system or rack. Scale-out links multiple systems through a larger network. Large AI clusters often use both. GPUs may exchange data over NVLink inside a rack and use another network to communicate with neighboring racks.
AI computing capacity a country controls within its own borders and jurisdiction, reducing dependence on foreign infrastructure.
Speculative decoding uses a cheaper method to draft several tokens, then has the main model check them together. Accepted drafts let the main model make progress with fewer separate generation steps. When many draft tokens are rejected, checking them may save little time.
The price of interruptible GPU capacity sold from spare supply; cheaper but can be reclaimed at any time.
Structured outputs make a model return data that follows a defined format, such as a JSON object with required fields. The shape is controlled, but the values still need checking. An invoice extractor returns amount, currency, and invoice number as separate fields rather than burying them in a paragraph.
Extra compute a model spends while answering — reasoning before it replies — to improve the result.
Time to first token is the wait between sending a request and receiving the first token of the answer. It includes time spent waiting in a queue and processing the prompt. A chat starts replying after two seconds, then takes another eight seconds to finish. Its first-token wait is two seconds.
Time to power is the wait until a data-center project can receive the electricity it needs to operate. Land, servers, and a place in a connection queue do not establish that date. A campus may have its servers ordered while it is still waiting for a substation upgrade and connection approval.
The price per token of model input and output; output tokens usually cost more and dominate serving bills.
Tokens per second per user measures how quickly one user receives generated text. It differs from the total output of a server serving many people at once. A server can produce more tokens overall while each person gets a slower stream as the number of active users rises.
Tool permissions define which actions an AI system is allowed to take through connected software. Reading a record and changing it are different permissions. A support assistant may search tickets but need approval before issuing a refund.
A custom AI accelerator (Google's tensor processing unit) built for specific workloads, trading flexibility for efficiency versus a general-purpose GPU.