LLM VRAM requirements: how much GPU memory for 7B–405B models

The 60-second rule of thumb

weights_gb  = params_B x bytes_per_param      # fp16 2.0 · int8 1.0 · 4-bit ~0.55
runtime_gb  = weights_gb x 1.1..1.2           # CUDA ctx, activations, fragmentation
total_gb    = runtime_gb + kv_cache_gb        # see below — grows with context

That is genuinely the whole method. The table below applies it to the models people actually deploy, with the cheapest card on our sheet that fits each cell.

The sizing table (inference)

Weights + 15% overhead, excluding KV-cache; 405B rows assume multi-GPU sharding (per-card price shown). Figures are engineering estimates for planning, not benchmarks.

KV-cache: the part everyone forgets

Serving crashes rarely come from weights — they come from context. Every token in flight stores keys and values for every layer:

kv_gb = 2 x layers x kv_heads x head_dim x bytes x context x batch / 1e9

# Llama 3.1 8B (GQA, fp16): ~0.13 GB per 1k tokens per sequence
# 32 concurrent chats x 8k context ≈ 33 GB of cache — MORE than the weights

Levers, in order of cheapness: quantize the cache (FP8 KV halves it), cap concurrent context (vLLM's --max-num-batched-tokens), then buy VRAM. This is why serving pages recommend 32–48 GB cards for "models that fit in 16 GB".

Training & fine-tuning VRAM

Gradient checkpointing trades ~20% speed for ~30% memory and is on by default in axolotl configs — the QLoRA walkthrough shows real consumption at each step.

When one card is not enough

• Tensor parallelism (vLLM --tensor-parallel-size) splits layers across 2–8 GPUs on one machine — NVLink helps but PCIe 4/5 serves fine to 4×.
• Our pricing is linear: 2× RTX 5090 (64 GB total) costs exactly 2× $0.318 — often the cheapest 70B-4bit rig on the sheet.
• Past 8 GPUs, you are in cluster territory: pipeline or FSDP sharding over InfiniBand.

Cross-check any plan against the per-card fit lists on the GPU pages — each fiche computes what it can hold from these same formulas.