For teams shipping RAG, voice and agent apps
Know your GPU headroom before you buy.
Size the GPUs for your whole agentic RAG stack, not just the LLM: embedders, rerankers, document parsers, speech-to-text, text-to-speech and guardrails share the same cards. Memory, tokens per second, question-to-first-word time, GPU count and cost per million tokens, from real architecture math.
No account needed to plan. Sign up to save, share and compare.
Gemma 4 31B · INT8 + 3 RAG models
200 users · 8,192-token context · 8× L40S 48GB
Where the GPU memory goes
- Model weights30.4 GB
- KV cache203.1 GB
- Runtime & buffers17.2 GB
- Activations1.43 GB
- RAG & agent models5.66 GB
Same GPUs, whole stack
- EmbeddingsBGE-M31.77 GB
- RerankerBGE Reranker v2 M31.77 GB
- Document parsing / OCRGranite-Docling 258M2.12 GB
- LLMs
- 138
- RAG & agent models
- 175
- GPUs & accelerators
- 93
- weight formats
- 20
- inference runtimes
- 7
What it answers
Everything that decides the hardware bill
The whole RAG & agent stack
The LLM plus 175 embedding, reranking, OCR, speech-to-text, text-to-speech and guardrail models, on the same GPUs, on the CPU or as an API.
Memory, line by line
Weights, KV cache per user, runtime, CUDA graphs, activations and every model in the stack, against what each GPU can really hold.
Speed and first word
Tokens per second per user, and the time from question to first word: embedding, vector search, reranking, guardrails and the LLM's first token, under a burst.
How many GPUs
The smallest server that fits, or a cluster with tensor and pipeline parallelism over NVLink, InfiniBand or Ethernet.
Every GPU, side by side
The same workload on all 93 GPUs: count needed, speed, first word, cost per hour and per million tokens.
Fine-tuning
Full, LoRA and QLoRA with optimizer states, ZeRO / FSDP sharding, checkpointing, training time and cost.
Offloading
Move layers or the KV cache to RAM or NVMe, run them on the CPU or stream over PCIe, and see what it costs in speed.
Cost and energy
Cloud price or your own, electricity, power draw, cost per million tokens, and the CPU and RAM the server needs.
The whole stack
One GPU budget for every model
An assistant is never just an LLM. It transcribes, checks, embeds, searches, reranks, answers and speaks, and every one of those models takes memory and compute from the same GPUs. Headroom sizes them together.
- Speech-to-text
- Guardrail
- Embedder
- Vector search
- Reranker
- LLM
- Text-to-speech
Behind it: document parsing, OCR and indexing for your knowledge base.
Example: a voice agent for 25 users
Qwen3 8B with Whisper large-v3-turbo, Kokoro-82M, BGE-M3 on 2× L4 24GB
85% of usable VRAM: 34.6 of 40.8 GB, of which 5.98 GB for the RAG and voice models
- Speech-to-textWhisper large-v3-turbo3.01 GB
× 78 real time · 4 live streams use ≈ 5% of a GPU · 1 hour of audio in ≈ 46 s
- Text-to-speechKokoro-82M1.21 GB
× 60 real time (estimate) · 4 live voices use ≈ 7% of a GPU
- EmbeddingsBGE-M31.77 GB
Query in 9 ms · Indexing ≈ 70 chunks/s of 512 tokens
Question → first word: embed 8.88 ms + search 15.0 ms + LLM 1.46 s = 1.49 s
12.2 tok/s per user · $2.21 per 1M tokens · the stack uses 5.9% of the GPUs' compute
How it works
Three choices, one answer
- 1Pick the modelsAn LLM from the catalog or your own architecture, plus the embedders, rerankers, parsers, speech and guardrail models around it.
- 2Pick the hardwareA GPU, a server or a cluster. Auto mode finds the smallest GPU count that fits the whole stack.
- 3Describe the loadConcurrent users, context length, questions per minute, live audio streams, pages to parse. Every number updates as you type.
Example: Llama 3.3 70B, FP8, 32 users, 8K context
Smallest single server on each GPU, computed by the same engine as the planner.
| GPU | GPUs | Speed / user | First word | $ / 1M tokens |
|---|---|---|---|---|
| H100 SXM 80GB | 2× | 28.2 tok/s | 2.49 s | $4.18 |
| H200 SXM 141GB | 1× | 21.6 tok/s | 4.72 s | $3.60 |
| B200 SXM (HGX B200) | 1× | 35.1 tok/s | 2.09 s | $3.34 |
| Instinct MI300X 192GB | 1× | 23.7 tok/s | 3.58 s | $1.87 |
| L40S 48GB | 4× | 12.3 tok/s | 6.55 s | $7.73 |
| RTX PRO 6000 Blackwell Server Edition 96GB | 2× | 13.4 tok/s | 5.78 s | $5.53 |
With a free account
Plans your team can trust
- Saved plans with autosaveEvery change is kept. Rename, star, duplicate, export to JSON.
- Read-only share linksSend the exact setup to a colleague or a vendor. Turn the link off any time.
- Compare plans side by sideUp to four plans with the best value per row marked.
- Setup codesPaste a code to load a setup, including codes from the original planner.
Start from a template
Team assistant with RAG
Gemma 4 31B on L40S for 200 people, embeddings, reranker and document parsing.
8× L40S 48GB
9.1 tok/s per user · $3.34/1M
Use templateLlama 3.3 70B on H100
FP8 weights and KV cache, 100 concurrent users with 16K context.
8× H100 SXM 80GB
26.8 tok/s per user · $5.61/1M
Use templateDeepSeek V3 on a B200 cluster
2 nodes × 8 B200, TP 8 × PP 2, 256 users, 16K context.
16× B200 SXM (HGX B200)
21.5 tok/s per user · $10.91/1M
Use templateLocal coding assistant
Qwen3-Coder 30B-A3B in GGUF Q4_K_M on one RTX 4090.
1× GeForce RTX 4090 24GB
96.1 tok/s per user · $3.47/1M
Use templateMethodology
Open math, no black box
Every result in the planner has a “How it is calculated” tab that shows each formula with your numbers plugged in.
How accurate are the numbers?+
They are planning estimates built from each model's published architecture and vendor datasheets: memory is computed exactly from the layer layout, speed from bytes read per token over memory bandwidth (with an efficiency factor per runtime), compute and interconnect. Real systems vary with kernels, batching and drivers, so validate with a load test before you buy.
Which units do you use?+
Memory is in GB = GiB (2³⁰ bytes), the way nvidia-smi reports it. Bandwidth is in decimal GB/s and compute in dense TFLOPS, without sparsity.
Does it handle MoE, MLA, sliding windows and hybrid models?+
Yes. Mixture-of-experts counts the experts each step actually touches, MLA models store their latent KV, sliding-window layers keep only their window, linear-attention and Mamba layers keep a fixed state, and quantized KV caches are supported per runtime.
Which models besides the LLM can it size?+
175 RAG and agent models in 9 types: embedders, rerankers, document parsers and OCR, speech-to-text, text-to-speech, guardrails, vision, PII detection and routers. Each one gets its memory and its speed (query time, pages per hour, × real time for live audio), on the GPU next to the LLM, on the CPU, or as a hosted API with its price.
Do I need an account?+
No. The planner works without one and keeps your draft in the browser. An account lets you save plans with autosave, share read-only links and compare plans side by side.
Can I bring setups from the original single-file planner?+
Yes. Setup codes from the original LLM GPU Planner load as they are: open the planner, choose More → Load a setup, and paste the code.
Is my data private?+
Plans are private to your account unless you turn on a share link, which you can turn off at any time. Calculations run in your browser.
Plan your next deployment
Answer the GPU question in minutes, then share the plan with the people who sign the purchase order.
Headroom · planning estimates, not measurements