The break-even rule
Both cards hold 80 GB, so the choice is never about fit — it is about dollars per finished run. Today the H100 SXM costs $1.587/hr and the A100 SXM4 $0.583/hr: a ratio of 2.72×. The rule that settles every job:
rent the H100 if speedup(H100 / A100) > 2.72, otherwise rent the A100
Speedups below that line make the A100 cheaper per run even though the H100 is faster; above it the H100 is both faster and cheaper. The ratio moves with the weekly price re-check — the number above is live.
What actually differs
| A100 SXM4 80 GB | H100 SXM 80 GB | Why it matters for fine-tuning | |
|---|---|---|---|
| Memory | 80 GB HBM2e | 80 GB HBM3 | Same capacity — identical model fit |
| Bandwidth | 2.0 TB/s | 3.35 TB/s | Optimizer steps and attention are bandwidth-bound: ~1.6× from this alone |
| BF16 tensor (dense) | 312 TFLOPS | 990 TFLOPS | Compute-bound layers scale; small batches rarely saturate either |
| FP8 | — | Transformer Engine | The H100's real weapon: 2.5–3× on full fine-tunes that use it |
| NVLink | 600 GB/s | 900 GB/s | Multi-GPU FSDP all-gathers; matters past 2× cards |
| Price today | $0.583/hr | $1.587/hr | Ratio 2.72× — the break-even line |
Five fine-tuning jobs, costed
Indicative wall-clock on the A100 and the speedup most teams measure on an H100 for the same recipe. The cost columns use today's on-demand prices; interruptible capacity halves both — fine-tunes checkpoint, so use it.
| Job | Setup | A100 hours | H100 speedup | A100 cost | H100 cost | Cheaper |
|---|---|---|---|---|---|---|
| QLoRA, Llama 3.1 8B, 10k pairs | BF16 compute, 4-bit base | 1.5 h | 1.8× | $0.87 | $1.32 | A100 (34% less) |
| LoRA, Llama 3.1 70B (4-bit base) | single card, 48 GB used | 3 h | 2.0× | $1.75 | $2.38 | A100 (27% less) |
| Full fine-tune, 8B, BF16 | 8× node, FSDP | 6 h | 2.2× | $27.98 | $34.63 | A100 (19% less) |
| Full fine-tune, 8B, FP8 (Transformer Engine) | 8× node, FSDP | 6 h | 3.0× | $27.98 | $25.39 | H100 (9% less) |
| DPO / RLHF pass, 8B | policy + reference | 2.5 h | 2.4× | $1.46 | $1.65 | A100 (12% less) |
The pattern is consistent: adapter methods in BF16 favour the A100; anything that engages FP8 or runs long enough for bandwidth to dominate favours the H100. Wall-clock is the tie-breaker — an H100 run that costs 10% more but finishes a day earlier is usually the right call for a deadline.
Decision table
| You are doing | Rent | Because |
|---|---|---|
| QLoRA / LoRA on ≤13B, BF16 | RTX 4090 or A100 PCIe | Fits in 24–80 GB; speedups under the break-even line |
| LoRA on 70B (4-bit base), single card | A100 SXM4 | Bandwidth-bound but 2× speedup stays under 2.7× |
| Full fine-tune, BF16, multi-GPU | A100 SXM4 unless deadline-bound | ≈2.2× speedup is close to the line — pick by wall-clock |
| Full fine-tune with FP8 (TE, NeMo, TorchAO) | H100 SXM | 2.5–3× speedup beats the price ratio outright |
| Sequence lengths ≥32k or big micro-batches | H100 SXM (or H200) | Memory bandwidth and, on the H200, 141 GB |
Making either card cheaper
- Go interruptible. Every job above checkpoints; the flat −50% turns an A100 into $0.291/hr and an H100 into $0.793/hr.
- Enable FP8 before renting an H100. Without it you are paying for bandwidth only; with it the H100 earns its price.
- Tune on 1×, train on 8×. Debug the config on a single card at one-eighth the burn rate, then scale.
- Datasets and checkpoints on a volume at $0.08/GB/month — the trainer instance stays disposable.
- Compare the neighbours. H100 SXM vs A100 SXM4 and the PCIe pair keep live numbers; the QLoRA walkthrough has the exact axolotl config.
Put the numbers to work
Every price in this guide is our live rate — fixed, ≥30% under the market median, billed per second. Deploy the exact setup above from the console in about 30 seconds, paid in crypto, no card and no KYC.
