Running in ~30 s
Pre-cached image, ports mapped, TLS terminated — the stack is working before you finish reading this.
Template · LLM serving & chat · CUDA 12.9
Run llama.cpp on a cloud GPU, in 30 seconds
GGUF inference with a built-in server — the lightest way to run quantized models. llama.cpp runs GGUF models with CUDA offload and ships a lightweight OpenAI-compatible server. It is the most memory-frugal way to serve quantized models and the natural home for Q4/Q5/Q6 GGUF files from Hugging Face — including models that do not fit vLLM's formats. From $0.020/hr on an interruptible RTX 3060.
Template is free
You pay the GPU price only: Llama.cpp on a RTX 3060 is $0.041/hr on-demand, $0.020/hr interruptible, billed per second.
Volumes for state
Models, datasets and outputs live on a $0.08/GB/mo volume; the instance stays disposable.
Private by default
Dedicated GPU, encrypted disk, crypto payments, no KYC, no stored IPs — Jupyter and SSH behind your own credentials.
Best GPUs for Llama.cpp
Three price points that run this template well — Good, Better, Best. Every model in the catalogue works; these are the value picks.
| Tier | GPU | VRAM | On-demand | Interruptible | Why this card | Action |
|---|---|---|---|---|---|---|
| Good | RTX 3060 | 12 GB | $0.041 | $0.020 | 12 GB is enough for 7B–8B Q4 GGUF — the cheapest chat endpoint on the sheet. | Deploy |
| Better | RTX 4090 | 24 GB | $0.262 | $0.131 | 24 GB runs 14B–32B quantized models with full GPU offload. | Deploy |
| Best | RTX A6000 | 48 GB | $0.281 | $0.140 | 48 GB for 70B Q4 GGUF on a single card. | Deploy |
Need more VRAM? The full catalogue lists all 76 models with live availability; the VRAM guide sizes models to cards.
Deploy Llama.cpp from the console, CLI or API
Pick the template in the console deploy bar, or script it:
- Console — filter by GPU, choose Llama.cpp in the template picker, set disk and env, deploy.
- CLI — pip install powergpu, then the command on the right. CLI reference.
- API — POST /v1/instances with "template": "llama-cpp". REST reference.
- Own image — any OCI reference works too; we inject the NVIDIA runtime. Template docs.
$ powergpu launch --gpu rtx-3060 --template llama-cpp \
--disk 100 --volume models:/workspace/models
✓ instance i-7a41c0e2 running (27.9s)
# Llama.cpp · RTX 3060 · $0.041/hr · per second
# https://i-7a41c0e2.powergpu.io:8888 (TLS)
$ powergpu stop i-7a41c0e2 # billing ends this secondWhat is inside the Llama.cpp template
| Image | powergpu/llama-cpp |
|---|---|
| CUDA | CUDA 12.9 |
| Access | also builds for ARM hosts · SSH shell · JupyterLab on a mapped port |
| Category | LLM serving & chat |
| Storage | Instance NVMe disk (sized at deploy) + optional network volumes |
| Billing | GPU price only, per second — no template fee, no setup fee |
Other llm serving & chat templates
vLLM on a cloud GPU — Production LLM serving with PagedAttention — an OpenAI-compatible endpoint from any HF model.- vLLM Omni on a cloud GPU — vLLM extended for multimodal models — vision and audio inputs on the same fast server.
SGLang on a cloud GPU — High-throughput serving with RadixAttention — excels at structured and agentic workloads.
Ollama on a cloud GPU — Pull and run quantized models in one command, with a clean REST API.
Open WebUI (Ollama) on a cloud GPU — A full chat UI over Ollama — conversations, RAG and model management in the browser.
Oobabooga Text Gen on a cloud GPU — The text-generation WebUI & API — load GPTQ, EXL2 or GGUF with extensions.- All 37 templates
Llama.cpp on a cloud GPU: FAQ
Environment variables, ports and custom images are covered in the template docs.
How much does it cost to run Llama.cpp on a cloud GPU?
Only the GPU price — the template is free. From $0.020/hr on an interruptible RTX 3060, $0.041/hr on-demand on a RTX 3060. Billing is per second, so an hour of tinkering costs an hour, not a day. Storage is $0.08/GB/month.
How long does Llama.cpp take to start?
About 30 seconds from the deploy click: the image is pre-cached on hosts, ports are mapped and TLS is terminated for you. Restarting a stopped instance is faster, and your disk is exactly as you left it.
Can I keep my models and outputs between sessions?
Yes — attach a volume at deploy. Everything on the volume survives instance destruction and mounts on the next instance in the region in seconds, at $0.08/GB/month. The instance disk itself survives stop/start but not destroy.
Can I split a model between GPU and CPU RAM?
Yes — set the number of offloaded layers (-ngl) at launch. Machines with large RAM let you run models bigger than VRAM at reduced speed, which is handy for occasional 70B use on a cheap card.