GPU Clouds

The ecosystem of compute providers for AI training and inference. Three layers: raw GPU rental (you manage everything), managed inference APIs (you send requests, they serve the model), and hyperscaler AI services (fully managed, enterprise-grade). Choosing the right tier depends on whether you're training, doing large-scale inference, or running API calls.

Raw GPU Rental

You rent an instance with GPUs attached and run whatever you want. Requires managing CUDA environments, model loading, batching, and everything else. Maximum flexibility, maximum ops burden.

Lambda Labs — the developer-friendly option. Clean UI, easy provisioning, stable pricing. H100 SXM5: ~$3.29/hr per GPU. Popular for research and startup training runs. Cloud and cloud-attached storage.

CoreWeave — NVIDIA-focused cloud built specifically for AI. Deep NVIDIA partnership, H100 SXM5 and H200 availability, specialized for large training jobs. Enterprise contracts. More enterprise than Lambda; better for multi-node training runs.

Vast.ai — marketplace model. Individuals and companies list idle GPU capacity; you bid on it. H100 SXM5 spot: $1.50–2.50/hr (vs $3+ on managed providers). Savings come with reliability risk — idle inventory means variable availability, and spot instances can be preempted. Good for interruptible training and batch inference.

RunPod — similar marketplace model to Vast.ai. Serverless mode (pay per second of inference) plus pod mode (always-on instances). Popular for personal use and small teams. Has a simple template system for common setups (Stable Diffusion, Ollama, etc.).

Fluidstack — aggregates datacenter capacity globally. Often cheaper than Lambda for large long-running jobs. Less visibility into hardware provenance.

Paperspace — acquired by DigitalOcean. GPU instances plus Gradient (their managed ML platform). The DigitalOcean parentage means good DevOps integration and pricing transparency.

H100 SXM5 price reference (spot/on-demand, mid-2025 approximate):

Managed Inference APIs

You send prompts, they serve the model. No GPU management, no batching logic. Pay per token or per second of compute. The right layer for most production applications.

Together.ai — fast inference for open models (Llama, Mistral, Qwen, DeepSeek). Competitive pricing, strong throughput. Fine-tuning support. Popular as a drop-in for OpenAI API users who want open-model pricing.

Fireworks.ai — similar positioning to Together.ai. Emphasis on low latency and compound AI (multi-model routing). FireFunctions for function-calling workloads.

Groq — LPU (Language Processing Unit) inference rather than GPU. Groq's hardware is purpose-built for transformer inference: very fast sequential generation (150+ tokens/sec per request on Llama 3 70B). The tradeoff: no CUDA, no fine-tuning, limited model selection. Best for latency-sensitive applications.

DeepInfra — large model catalog including many less-common models, competitive pricing. Simple API.

Baseten — model deployment platform rather than a model catalog. You deploy your own fine-tuned model, they manage serving. Useful for custom models at production scale.

Anyscale — Ray-based serving for custom model deployments. Enterprise-grade, designed for complex multi-model pipelines.

Pricing reference (approximate mid-2025, per million tokens):

Hyperscaler AI Services

Fully managed AI inference from AWS, Google, and Azure. You call an API that routes to managed infrastructure; no model hosting, no hardware. Enterprise SLAs, compliance certifications, existing billing relationships. Highest per-token cost; lowest ops burden.

AWS Bedrock — managed access to Claude (all tiers), Titan, Llama, Mistral, Stable Diffusion, and others via a unified API. The Claude API for enterprise accounts often routes through Bedrock. Deep integration with IAM, CloudWatch, GuardRails for filtering. The enterprise default for companies already on AWS.

Google Vertex AI — managed Gemini (all tiers) plus Google's open model catalog. Strong integration with GCP data infrastructure (BigQuery, Cloud Storage, Dataflow). Vertex AI Workbench for notebook-based ML workflows alongside the API. The enterprise default for companies on GCP.

Azure OpenAI Service — OpenAI models (GPT-5 family, Codex, DALL-E) with Azure enterprise compliance wrapper (SOC 2, HIPAA, FedRAMP). Separate capacity provisioned per customer; not shared infrastructure. The entry point for enterprise Microsoft/OpenAI customers who need data residency and compliance guarantees.

Key distinction from direct APIs: hyperscaler services add compliance overhead (audit logs, data residency, encryption at rest/transit) and legal guarantees (enterprise agreements, data processing addendums) that the direct APIs don't provide at the same tier. They cost more per token; the premium pays for legal and compliance infrastructure.

Choosing the Right Layer

OpenRouter deserves a mention: it routes across ~100+ providers and models under a single API key, with automatic fallback and model selection. Good for applications that want to optimize cost/quality dynamically.

The Cost Architecture of Inference

For high-volume inference (millions of requests/day), raw GPU rental becomes cheaper than managed APIs — but only above a threshold where you're saturating the hardware. Below that threshold, you're paying for idle GPU time, which is expensive.

Rule of thumb:

• Fewer than ~100K tokens/day: use managed API (Together.ai, Groq, Fireworks)
• 100K–10M tokens/day: evaluate based on model size and latency needs
• Above 10M tokens/day per model: probably worth raw GPU or reserved capacity contracts

For local development and personal use, exo and ollama cost zero per-token once the hardware is paid for.

Related

exo · distributed-inference · ollama · agentic-workflows · prompt-caching · mcp

Sources

• Artificial Analysis GPU Price Comparison
• Lambda Labs Cloud
• Together.ai Documentation
• AWS Bedrock Documentation