Use case · LLM inference
LLM inference GPUs: tokens priced like a utility bill
Fixed GPU prices turn $/token into arithmetic: pick the smallest card that holds your model + KV-cache, saturate it with vLLM continuous batching, divide. A RTX 5090 at $0.318/hr serves a 8B model for pennies per million tokens.
The serving cards, by model size
VRAM decides; price-per-VRAM ranks the candidates.
| Tier | GPU | VRAM | On-demand | Interruptible | Why this card | Action |
|---|---|---|---|---|---|---|
| Good | RTX 5090 | 32 GB | $0.318 | $0.159 | 32 GB GDDR7 — the $/token king for 7B–14B FP16 and 32B 4-bit chat models. | Deploy |
| Better | L40S | 48 GB | $0.466 | $0.233 | 48 GB and server-grade cooling for 24/7 endpoints; 32B class and long-context 14B. | Deploy |
| Best | H100 PCIE | 80 GB | $2.147 | $1.073 | 80 GB HBM3 for 70B quantized on one card — or FP8 for maximum throughput per dollar. | Deploy |
Bigger than 70B? Tensor-parallel across 2–8× cards on one machine — the per-GPU price never changes. Sizing tables: VRAM guide.
$/million tokens, computed honestly
Throughput varies with context and batch mix, so we publish the formula, not a marketing number:
$/M tokens = GPU $/hr ÷ (tokens/s × 3600) × 1,000,000
A RTX 5090 at $0.318/hr sustaining ~2,800 tok/s of Llama-3.1-8B under continuous batching → $0.032/M tokens. The same card interruptible halves it.
Benchmark your own model in 10 minutes — the serving guide ships the exact vllm bench command.
Serving patterns
- Reserved + vLLM for the baseline load — −35% on the card that never sleeps.
- Serverless burst on top — scale-to-zero workers absorb the spikes.
- Model library volume mounted read-only — new workers skip the 40 GB download.
- Quantize first — AWQ/GPTQ 4-bit doubles the models a card can hold, rarely hurts chat quality.
Templates: vLLM, Ollama, TGWUI — all OpenAI-compatible out of the box.
LLM inference GPUs: FAQ
What is the cheapest way to serve a 7B–14B model?
A single RTX 5090 at $0.318/hr running vLLM. At typical chat throughput that lands well under $0.15 per million output tokens — an order of magnitude below API-provider pricing for comparable open models.
When do I need an 80 GB card?
Roughly at 70B: a 4-bit 70B needs ~44 GB plus KV-cache, so a single H100 PCIE (80 GB) or 2× L40S with tensor parallelism. FP16 70B wants 2× 80 GB. Below 32B, 32–48 GB cards are the value zone.
Instances or serverless for serving?
Steady saturating traffic → a reserved instance (−35%) you keep hot. Spiky or overnight-idle traffic → serverless with scale-to-zero. Same per-second prices; the difference is who pays for idle.