Three rules before any benchmark
- VRAM decides what runs. Weights plus KV-cache must fit, with headroom: ~2.4 GB per billion parameters in FP16, ~0.62 GB in 4-bit, then add the cache for your context × concurrency. The VRAM guide has the full tables.
- Bandwidth decides how fast. Decoding is memory-bound: tokens per second track GB/s more than TFLOPS. That is why a 1.79 TB/s RTX 5090 beats a 1 TB/s RTX 4090 by more than its FLOPS suggest, and why HBM cards dominate 70B serving.
- Batch decides the bill. Continuous batching (vLLM, SGLang, TGI) turns one card into a dozen concurrent streams. Cost per million tokens is only meaningful at the concurrency you will actually run.
The table: model size → card → $ per million tokens
Indicative aggregate throughput under vLLM continuous batching (roughly 1k tokens in, 1k out, batch 32–64) and the resulting cost at today's on-demand rates — snapshot 2026-09-03. Treat the tokens/s column as order-of-magnitude community numbers and benchmark your own model; the price column is exact.
| Model | Precision / weights | Card | On-demand | ~tokens/s | $ / M tokens | Note |
|---|---|---|---|---|---|---|
| Llama 3.1 8B | FP16, 16 GB | RTX 4090 24 GB | $0.262 | ~2,200 | $0.033 | The default: fits with KV-cache room on 24 GB. |
| Llama 3.1 8B | FP16, 16 GB | RTX 5090 32 GB | $0.318 | ~2,800 | $0.032 | 78% more bandwidth than the 4090 shows up directly in tokens/s. |
| Llama 3.1 8B | 4-bit, ~6 GB | RTX 3060 12 GB | $0.041 | ~900 | $0.013 | The cheapest working endpoint on the sheet. |
| Qwen 2.5 14B | FP16, 28 GB | RTX 5090 32 GB | $0.318 | ~1,500 | $0.059 | 32 GB holds FP16 14B with modest context. |
| Qwen 2.5 32B | 4-bit, ~20 GB | RTX 4090 24 GB | $0.262 | ~700 | $0.104 | AWQ/GPTQ 4-bit on 24 GB; quality within a point of FP16. |
| Qwen 2.5 32B | FP16, 64 GB | H100 PCIE 80 GB | $2.147 | ~2,400 | $0.248 | FP8 on Hopper roughly doubles this again. |
| Llama 3.1 70B | 4-bit, ~40 GB | RTX A6000 48 GB | $0.281 | ~350 | $0.223 | Cheapest single card that holds a 70B. |
| Llama 3.1 70B | 4-bit, ~40 GB | H100 PCIE 80 GB | $2.147 | ~1,200 | $0.497 | One 80 GB card, real batch sizes, FP8 KV-cache. |
| Llama 3.1 70B | FP8, 70 GB | H200 141 GB | $3.058 | ~2,000 | $0.425 | 141 GB leaves room for long contexts and big batches. |
| Llama 3.1 405B | FP8, ~405 GB | H200 141 GB | $3.058 | ~600 | $1.416 | Tensor-parallel across 4× H200 — figure is per GPU. |
Interruptible capacity halves every number in the price column. Formula, if you want to redo it with your own measurement: $/M = $/hr ÷ (tok/s × 3600) × 1,000,000.
7B–14B: consumer cards win
Below 16 GB of weights nothing beats GeForce silicon per dollar. The RTX 5090 ($0.318/hr, 32 GB GDDR7) is the current king of tokens per dollar for 8B FP16 and 14B models; the RTX 4090 ($0.262/hr) is a hair slower and cheaper; the RTX 3090 ($0.104/hr) is the budget pick when throughput matters less than the hourly rate. For always-on endpoints that must survive months of 24/7 duty, the datacenter L4 ($0.225/hr, 72 W) and L40S trade a little speed for passive cooling and ECC.
24B–32B: the 48 GB middle
A 32B model is ~64 GB in FP16 and ~20 GB in 4-bit. Quantized, it runs on a 24 GB card with short contexts; for serious concurrency you want 48 GB. The L40S ($0.466/hr) is the datacenter answer — FP8, 864 GB/s, built for 24/7 — and the RTX A6000 ($0.281/hr) the cheapest 48 GB on the sheet. Compare them in RTX 5090 vs L40S and RTX 6000 Ada vs A6000.
70B and up: HBM or nothing
Seventy billion parameters is ~40 GB in 4-bit and ~140 GB in FP16. One 80 GB card serves the quantized model with real batch sizes: the H100 PCIe ($2.147/hr) with FP8 KV-cache, or the A100 PCIe ($0.662/hr) when budget wins. Full-precision 70B, 100B+ models and long-context serving belong on the H200 ($3.058/hr, 141 GB, 4.8 TB/s) or tensor-parallel across 2–8× H100 SXM on one machine — the per-GPU price does not change with count. Frontier-scale inference with FP4 is where the B200 ($4.204/hr) earns its rate: several times the H100's tokens per second at under 3× the price.
Serving patterns that change the bill
- Quantize first. AWQ/GPTQ 4-bit halves the card class you need and rarely hurts chat quality. FP8 on Hopper/Ada/Blackwell is nearly free quality-wise and 1.5–2× faster.
- Reserve the baseline, burst serverless. A reserved card at −35% carries steady traffic; serverless workers absorb spikes and scale to zero.
- Mount the model library read-only. New workers skip the 40 GB download; cold starts become load-to-VRAM.
- Prefix caching for agents. If prompts share a long system prefix, SGLang can double effective throughput over plain vLLM.
- Measure, then commit. Ten minutes on two candidate cards costs cents; the vLLM guide ships the exact benchmark command.
Put the numbers to work
Every price in this guide is our live rate — fixed, ≥30% under the market median, billed per second. Deploy the exact setup above from the console in about 30 seconds, paid in crypto, no card and no KYC.
