Products · serverless

Serverless GPU endpoints that scale to zero

Ship a model, get a URL, pay per second of actual GPU work. Workers scale with queue depth — from zero to dozens and back — at the same fixed prices as everything else: an L40S worker-second costs exactly $0.466/3600.

Create an endpoint Serving guide

Queue-based autoscale

Workers spawn when requests-per-worker crosses your threshold and drain when the queue empties. You set min/max; the default min is zero.

Cached weights

Model files persist on regional NVMe cache, so scale-up loads from local disk — seconds, not a 40 GB download.

Standard interfaces

vLLM endpoints speak the OpenAI API; anything else is plain HTTP on your container's port, behind TLS we terminate for you.

Per-second economics

Active seconds × fixed price. A burst of 10,000 requests on RTX 5090 workers costs the same whether it takes one worker an hour or twelve workers five minutes.

From weights to URL in one command

Endpoints are described declaratively — model, GPU class, scaling window — and managed like any other resource from the console, CLI or API.

LLM chat & completion — vLLM template, OpenAI-compatible, streaming included.
Image generation — ComfyUI workflows exposed as a JSON API.
Transcription, embeddings, rerankers — Whisper and friends on cheap cards.
Your container — anything that serves HTTP; we handle TLS, scaling and health checks.

serverless — deploy

$ powergpu endpoint create --name llama8b \
    --template vllm --gpu l40s --min 0 --max 8 \
    --env MODEL=meta-llama/Llama-3.1-8B-Instruct
✓ endpoint ep-31c8a2 ready
# https://llama8b-31c8a2.powergpu.io/v1/chat/completions
$ powergpu endpoint stats llama8b
# workers 0→3 (last hour) · p50 342ms · 41,208 req · $1.87

Which GPU class for which model

Serverless supports the same catalogue; these three cover most endpoints.

Worker GPU	VRAM	Good for	Active price	1M tokens ≈
RTX 5090	32 GB	7B–14B chat, embeddings, Whisper	$0.318/hr	$0.05–0.15
L40S	48 GB	14B–32B models, SDXL/Flux, long context	$0.466/hr	$0.10–0.30
A100 SXM4	80 GB	70B quantized, video models, big batches	$0.583/hr	$0.25–0.60

Token costs are illustrative vLLM throughput ranges; benchmark your model with the serving guide.

Serverless GPUs: FAQ

Endpoint reference in the docs and API.

How is serverless billed?

Per second of active GPU time at the same fixed per-model prices as instances, plus the flat storage rate for your model cache. Scale-to-zero means an idle endpoint costs only its cached weights — a 16 GB model cache is about $1.28/month.

What are cold starts like?

Weights stay cached on NVMe next to the GPUs, so cold starts are load-to-VRAM, not download-from-internet: typically 4–15 s for 7B–13B class models, longer for 70B. Set min-workers to 1 to remove them entirely — you pay for that worker only while it exists.

What can I deploy on it?

The vLLM and ComfyUI templates work out of the box (OpenAI-compatible and REST respectively), or bring any Docker image that answers HTTP on a port. Autoscaling watches queue depth per worker.

When should I use instances instead?

Steady, saturating traffic is cheaper on a reserved instance you keep busy. Serverless wins for spiky traffic, many small models, or products that sleep at night — pay for requests, not for idle.