What is the cheapest way to get an OpenAI-compatible endpoint for my model?

One vLLM template instance on the smallest card whose VRAM fits weights + cache: $0.318/hr for 7B–14B on an RTX 5090. Endpoint, TLS and streaming come with the template — total setup is one deploy form.

How many tokens per second should I expect?

Order of magnitude under continuous batching: 8B on an RTX 5090 ≈ 2,500–3,500 tok/s aggregate; 32B-4bit ≈ 600–900; 70B-4bit on an 80 GB card ≈ 400–700. Single-stream latency is a different number — benchmark with your prompt mix, the guide shows how.

Ollama for one user and instant gratification; vLLM for anything with concurrency — PagedAttention batching is the difference between 10× and 1× hardware utilisation under load.

How to deploy vLLM on a cloud GPU: an OpenAI-compatible endpoint

Deploy in one command

zero to endpoint

powergpu launch --gpu rtx-5090 --template vllm --disk 60 \
  --env MODEL=meta-llama/Llama-3.1-8B-Instruct
# ✓ i-52ab77c1 · https://i-52ab77c1.powergpu.io:8000/v1

curl https://i-52ab77c1.powergpu.io:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"meta-llama/Llama-3.1-8B-Instruct",
       "messages":[{"role":"user","content":"ping"}]}'

The template downloads weights on first boot (billed bandwidth: a 16 GB model costs $0.16). Put HF_HOME on a volume and every future instance skips the download.

Pick the card for the model

Model class	Card	$/hr	Aggregate tok/s ≈	$ / M output tokens ≈
7–8B FP16	RTX 5090	$0.318	2,800	$0.032
14B FP16 / 32B-4bit	L40S	$0.466	900	$0.144
70B-4bit	H100 PCIE	$2.147	550	$1.08

Throughputs are typical continuous-batching aggregates at moderate context; your prompt mix will move them ±40%. That is why the benchmark section exists.

The flags that matter

the 5 flags that do 95% of the work

--max-model-len 8192          # cap context = cap KV-cache = predictable memory
--gpu-memory-utilization 0.92 # default 0.90; raise carefully once stable
--max-num-batched-tokens 8192 # throughput/latency dial — higher = more tok/s
--kv-cache-dtype fp8          # halves cache on Hopper/Blackwell — free VRAM
--tensor-parallel-size 2      # split across 2 GPUs when one is short

Everything else can stay default until the benchmark says otherwise. The single most common OOM cause is leaving --max-model-len at a model's 128k maximum "just in case" — the KV-cache section shows what that costs.

Benchmark before you believe

10 minutes, real numbers

vllm bench serve \
  --base-url https://i-52ab77c1.powergpu.io:8000 \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name random --random-input-len 512 --random-output-len 256 \
  --request-rate 16 --num-prompts 512
# → throughput, TTFT p50/p99, ITL p50/p99

Judge two numbers: aggregate tok/s (your cost) and p99 time-to-first-token (your users). Raise --request-rate until p99 TTFT crosses your budget — that rate is the card's honest capacity for your traffic shape.

Price your tokens

the only pricing formula you need

$/M output tokens = price_per_hour / (tokens_per_sec x 3600) x 1e6

# benchmark said 2,910 tok/s on the 5090:
0.318 / (2910 x 3600) x 1e6 = $0.030 per million output tokens
# reserved (-35%) and cache-hit traffic push it lower still

Production posture

Baseline on reserved (−35%), burst on serverless — one balance, two curves.
Health-check GET /health; restart-on-unhealthy is a template toggle.
Weights on a read-only volume — replacement workers boot in seconds, and a bad deploy can never corrupt the library.
Never serve on interruptible — that mode is for the training side of your pipeline.

Put the numbers to work

Every price in this guide is our live rate — fixed, ≥30% under the market median, billed per second. Deploy the exact setup above from the console in about 30 seconds, paid in crypto, no card and no KYC.