Headroom

For teams shipping RAG, voice and agent apps

Know your GPU headroom before you buy.

Size the GPUs for your whole agentic RAG stack, not just the LLM: embedders, rerankers, document parsers, speech-to-text, text-to-speech and guardrails share the same cards. Memory, tokens per second, question-to-first-word time, GPU count and cost per million tokens, from real architecture math.

No account needed to plan. Sign up to save, share and compare.

Gemma 4 31B · INT8 + 3 RAG models

200 users · 8,192-token context · 8× L40S 48GB

Moderateof usable VRAM
79%257.9 GBof 326.4 GB usable
Speed per user9.1 tok/s110 ms per token
Question → first word13.7 sretrieval included · 10 at once
Total output1,814 tok/s200 answers at once
Per 1M output tokens$3.34$13.11 per hour

Where the GPU memory goes

  • Model weights30.4 GB
  • KV cache203.1 GB
  • Runtime & buffers17.2 GB
  • Activations1.43 GB
  • RAG & agent models5.66 GB

Same GPUs, whole stack

  • EmbeddingsBGE-M31.77 GB
  • RerankerBGE Reranker v2 M31.77 GB
  • Document parsing / OCRGranite-Docling 258M2.12 GB
LLMs
138
RAG & agent models
175
GPUs & accelerators
93
weight formats
20
inference runtimes
7

What it answers

Everything that decides the hardware bill

The whole RAG & agent stack

The LLM plus 175 embedding, reranking, OCR, speech-to-text, text-to-speech and guardrail models, on the same GPUs, on the CPU or as an API.

Memory, line by line

Weights, KV cache per user, runtime, CUDA graphs, activations and every model in the stack, against what each GPU can really hold.

Speed and first word

Tokens per second per user, and the time from question to first word: embedding, vector search, reranking, guardrails and the LLM's first token, under a burst.

How many GPUs

The smallest server that fits, or a cluster with tensor and pipeline parallelism over NVLink, InfiniBand or Ethernet.

Every GPU, side by side

The same workload on all 93 GPUs: count needed, speed, first word, cost per hour and per million tokens.

Fine-tuning

Full, LoRA and QLoRA with optimizer states, ZeRO / FSDP sharding, checkpointing, training time and cost.

Offloading

Move layers or the KV cache to RAM or NVMe, run them on the CPU or stream over PCIe, and see what it costs in speed.

Cost and energy

Cloud price or your own, electricity, power draw, cost per million tokens, and the CPU and RAM the server needs.

The whole stack

One GPU budget for every model

An assistant is never just an LLM. It transcribes, checks, embeds, searches, reranks, answers and speaks, and every one of those models takes memory and compute from the same GPUs. Headroom sizes them together.

  1. Speech-to-text
  2. Guardrail
  3. Embedder
  4. Vector search
  5. Reranker
  6. LLM
  7. Text-to-speech

Behind it: document parsing, OCR and indexing for your knowledge base.

Example: a voice agent for 25 users

Qwen3 8B with Whisper large-v3-turbo, Kokoro-82M, BGE-M3 on 2× L4 24GB

Moderate

85% of usable VRAM: 34.6 of 40.8 GB, of which 5.98 GB for the RAG and voice models

  • Speech-to-textWhisper large-v3-turbo3.01 GB

    × 78 real time · 4 live streams use ≈ 5% of a GPU · 1 hour of audio in ≈ 46 s

  • Text-to-speechKokoro-82M1.21 GB

    × 60 real time (estimate) · 4 live voices use ≈ 7% of a GPU

  • EmbeddingsBGE-M31.77 GB

    Query in 9 ms · Indexing ≈ 70 chunks/s of 512 tokens

Question → first word: embed 8.88 ms + search 15.0 ms + LLM 1.46 s = 1.49 s
12.2 tok/s per user · $2.21 per 1M tokens · the stack uses 5.9% of the GPUs' compute

Open this in the planner

How it works

Three choices, one answer

  1. 1Pick the modelsAn LLM from the catalog or your own architecture, plus the embedders, rerankers, parsers, speech and guardrail models around it.
  2. 2Pick the hardwareA GPU, a server or a cluster. Auto mode finds the smallest GPU count that fits the whole stack.
  3. 3Describe the loadConcurrent users, context length, questions per minute, live audio streams, pages to parse. Every number updates as you type.

Example: Llama 3.3 70B, FP8, 32 users, 8K context

Smallest single server on each GPU, computed by the same engine as the planner.

GPUGPUsSpeed / userFirst word$ / 1M tokens
H100 SXM 80GB2×28.2 tok/s2.49 s$4.18
H200 SXM 141GB1×21.6 tok/s4.72 s$3.60
B200 SXM (HGX B200)1×35.1 tok/s2.09 s$3.34
Instinct MI300X 192GB1×23.7 tok/s3.58 s$1.87
L40S 48GB4×12.3 tok/s6.55 s$7.73
RTX PRO 6000 Blackwell Server Edition 96GB2×13.4 tok/s5.78 s$5.53
Open this in the planner

Methodology

Open math, no black box

Every result in the planner has a “How it is calculated” tab that shows each formula with your numbers plugged in.

How accurate are the numbers?+

They are planning estimates built from each model's published architecture and vendor datasheets: memory is computed exactly from the layer layout, speed from bytes read per token over memory bandwidth (with an efficiency factor per runtime), compute and interconnect. Real systems vary with kernels, batching and drivers, so validate with a load test before you buy.

Which units do you use?+

Memory is in GB = GiB (2³⁰ bytes), the way nvidia-smi reports it. Bandwidth is in decimal GB/s and compute in dense TFLOPS, without sparsity.

Does it handle MoE, MLA, sliding windows and hybrid models?+

Yes. Mixture-of-experts counts the experts each step actually touches, MLA models store their latent KV, sliding-window layers keep only their window, linear-attention and Mamba layers keep a fixed state, and quantized KV caches are supported per runtime.

Which models besides the LLM can it size?+

175 RAG and agent models in 9 types: embedders, rerankers, document parsers and OCR, speech-to-text, text-to-speech, guardrails, vision, PII detection and routers. Each one gets its memory and its speed (query time, pages per hour, × real time for live audio), on the GPU next to the LLM, on the CPU, or as a hosted API with its price.

Do I need an account?+

No. The planner works without one and keeps your draft in the browser. An account lets you save plans with autosave, share read-only links and compare plans side by side.

Can I bring setups from the original single-file planner?+

Yes. Setup codes from the original LLM GPU Planner load as they are: open the planner, choose More → Load a setup, and paste the code.

Is my data private?+

Plans are private to your account unless you turn on a share link, which you can turn off at any time. Calculations run in your browser.

Plan your next deployment

Answer the GPU question in minutes, then share the plan with the people who sign the purchase order.

Headroom · planning estimates, not measurements