Quick overview
How much VRAM do you need for local LLMs? Model size, Q4 vs Q8 quant, KV cache, and GPU tiers explained for Indian PC builders with a sizing table.
VRAM is the ceiling on which open-weight models load on a Windows or Linux desktop. Gaming benchmarks rarely mention it, yet it is the first number to check when you plan a local LLM PC. This guide ties model size, quantization, and context length to realistic GPU tiers without promising fixed token speeds, which change by engine and driver.
Quick answer
For Q4_K_M, plan roughly 5 GB for 7B, 8–10 GB for 13B, 16–19 GB for 27B–32B, and 40–48 GB for 70B including modest context headroom. Long contexts and higher quants move those numbers up. Cross-check with a VRAM requirements table before you purchase a GPU.
What consumes VRAM during inference
Four buckets matter on a desktop GPU:
1. Model weights (fixed once you pick model + quant) 2. KV cache (grows with context length and batch size) 3. Runtime buffers (engine overhead, often under 1 GB) 4. Display and desktop reserve (commonly 1–2 GB if the GPU drives monitors)
Weights dominate for short chats. KV cache dominates when you pin 32K+ context or run multiple sessions.
VRAM planning table (Q4_K_M, modest context)
| Model class | Approximate VRAM need | Typical GPU tier to research |
|---|---|---|
| 7B–8B | 5–9 GB | 8–12 GB cards |
| 13B–14B | 8–12 GB | 12 GB cards |
| 24B | ~14–16 GB | 16 GB cards |
| 27B–32B | 16–24 GB | 24 GB class |
| 70B | 40–48 GB | Dual high-VRAM or workstation GPUs |
Numbers are planning estimates, not guarantees. Always test your exact model file in Ollama or LM Studio before finalizing a build.
Quantization changes the footprint
Moving from Q4 toward Q8 or FP16 improves quality but multiplies memory. A 7B model that fits an 8 GB card at Q4 may not fit at Q8. Read our Q4 vs Q8 quantization guide before you assume "7B always equals 8 GB."
When system RAM enters the picture
If weights do not fit in VRAM, runtimes can offload layers to CPU RAM. That can work for experiments but is slower than a full GPU fit. RAM does not create VRAM: it only holds spillover. See system RAM vs GPU VRAM for LLMs.
Delhi-specific reminder
Sustained inference in warm rooms pushes GPU hotspots up. Thermal throttling lowers effective speed even when VRAM is sufficient. Pair VRAM planning with airflow and honest load tests, similar to workstation maintenance in heat.
How Roger N Reckon can help
If you are between GPU tiers, Roger N Reckon can benchmark your target model on a proposed build where possible and document thermals under load. Start from Workstation Builds and bring the exact model name and quant you want to run.
Practical checklist
1. Write down the model size, quant level, and target context length you need. 2. Measure current GPU VRAM, system RAM, and free NVMe space before buying parts. 3. Confirm PSU wattage against the GPU vendor's required system power. 4. Plan cooling for sustained inference, not just short benchmarks. 5. Back up data before OS or hardware changes. 6. Request a written component list, testing scope, and warranty terms for any build or upgrade. 7. Re-test with the same prompt and context after changes.
Frequently Asked Questions
Can I run a 70B model on 24 GB VRAM?
Usually not at Q4 with useful context on a single 24 GB card. You need aggressive quant, heavy offload, or 40 GB+ effective memory across GPUs or unified memory platforms.
Does context length affect VRAM?
Yes. KV cache memory grows with context. Budget extra headroom if you rely on long documents or large codebases in one session.
Is VRAM the same as GDDR6 capacity on the spec sheet?
For planning, yes: use the GPU's advertised memory size. Some tools also use shared system memory on APUs, which behaves differently from discrete VRAM.
Do MoE models use total or active parameters for sizing?
Weight files are sized by total parameters in the checkpoint, even when only a subset activates per token. Do not size MoE models using active-parameter marketing alone.
Where should I verify a specific model file size?
Check the download page for the GGUF or Ollama tag size, then add KV and overhead. Community tables such as llmhardware.io document methodology for many 2026 open models.
Final takeaway
Pick the model first, then the quant, then add context headroom, then buy the GPU. VRAM tables beat hype every time.