Google Cloud Architecture Framework
Primary architecture guidance for reliability, security, performance, cost, and operational tradeoffs.
← Back to Compute College
Compute College
Hosted API vs managed deployment vs self-hosting
Compare three ways to run an AI workload by control, latency, capacity, operations, and total cost.
Plain-English definition
A hosted API runs the model through a provider’s shared service; a managed deployment gives the team a dedicated or configurable serving environment managed by a platform; self-hosting puts more of the model, hardware, and operations under the team’s control. The choice is a workload and risk decision, not just a pricing-page comparison.
Memory trick: Deployment is a bundle of control, capacity, and work.
Why it matters
The options exchange control for operational burden. Hosted APIs can reduce fixed capacity work, while self-hosting may improve control or economics at stable high utilization. Managed deployments sit between them but still depend on region, hardware, quotas, scaling, and support terms.
- Variable API cost can be attractive for uncertain or bursty demand.
- Dedicated or self-hosted capacity can improve control and predictable latency when utilization is high.
- Privacy, portability, support, and failure recovery belong in total cost.
Simple example
A startup begins with a hosted API for uncertain traffic, measures tokens and accepted-task cost, then compares a managed endpoint at expected peak volume. Self-hosting is considered only after the team can staff model serving, monitoring, patching, capacity, and incident response.
- The first decision preserves optionality while demand is uncertain.
- The comparison uses a measured workload rather than a hypothetical GPU count.
- Operations are counted as part of the deployment.
Example figures are illustrative calculations, not current quoted market prices.
Current example
Primary reference
This source explains the method or risk boundary. It does not replace testing the actual model, data, provider, and workload conditions in production.
Source discipline: examples on this page are illustrative; measure the real workload before making a release or capacity decision.
Common mistake
Provider token prices and raw GPU rental rates are not directly comparable. Self-hosting also pays for utilization loss, serving software, power, storage, networking, staff, and failures.
Practical takeaway
What you can do with this
Write a three-column comparison for the current workload. Include cost per accepted task, latency, privacy, capacity risk, operations, exit path, and the utilization level where each option changes rank.
- Use measured traffic and token ranges.
- Price failure and idle capacity explicitly.
- Review the comparison whenever workload volume or data policy changes.
Decision check: does the preferred deployment still win after adding operations, headroom, failure recovery, and the cost of changing providers?
Compute College learning path
AI Engineering
Step 41 of 48: Hosted API vs managed deployment vs self-hosting