How Much VRAM Does an LLM Need? Weights and KV-Cache Math

Ask "how much VRAM does an 8B model need?" on a forum and you get answers from 12 GB to 40 GB. The honest answer has two terms: VRAM is roughly weights plus KV cache, with about 20% engine overhead on top.

The weights term is easy. It is parameters x bytes per parameter, so an 8B model in fp16 takes about 16 GB. The KV cache is the term people forget. The same model serving one 128K-token context accumulates roughly 16 GiB of KV cache, as much as the weights. Your comfortable 24 GB card just got tight.

This post runs both calculations with plain arithmetic. By the end you can size any model from its config.json instead of guessing from forum posts.

In short:

• Weights = parameters x bytes per parameter. In fp16 that is about 2 GB per 1B params: an 8B model is ~16 GB, a 70B model ~141 GB.
• KV cache = 2 x layers x KV heads x head_dim x bytes, per token. For Llama-3.1-8B that works out to 128 KiB per token.
• The cache scales linearly with context and with batch. At 128K context, one sequence's cache (~16 GiB) rivals the entire model.
• Budget ~20% overhead as a floor, not a ceiling; long contexts need the full KV math added on top.
• Quantize weights before you touch the cache dtype, and only quantize the cache when context x batch forces you to.

First term: how much memory the weights take

The weights term is simple multiplication. Every parameter is stored at some precision, and precision maps directly to bytes:

So an 8B model in fp16 is 8 x 2 bytes ~ 16.1 GB; the same model at int4 is ~4 GB. A 70B model in bf16 is ~141 GB, and quantized to 4 bits, ~35 GB. The handy shortcut, which Modal's VRAM guide states plainly: figure on roughly 2 GB per 1B parameters in half precision.

One caveat: real checkpoints land a few percent heavier than the raw P x Q/8 math, because embeddings and some layers often stay at higher precision. Hugging Face's estimate-memory tool shows this on actual repos - the 80B idefics checkpoint reports 148.56 GB in fp16 and 37.14 GB at int4, so the 4x drop matches the byte math, but the absolute numbers sit slightly above the naive product. Treat the formula as a very good first estimate, not a guarantee.

MoE models: VRAM follows total parameter count, not active params

Mixture-of-experts models complicate the speed story more than the memory story. MoE hosting guides note that VRAM follows total parameter count, since every expert stays resident even though only a few fire per token; the active-parameter count affects speed, not the memory bill. For sizing purposes, use total params in the same weights formula.

Second term: the KV cache grows with every token

Weights are a fixed cost. The KV cache is the variable one, and it is usually what actually breaks deployments.

During generation, attention needs the key and value vectors for every token processed so far, and it keeps them until the sequence ends. That store is the KV cache. Context here is counted in tokens - roughly three-quarters of an English word each; see our tokenization explainer if you want that mapping in detail - so a 128K context window means up to 128K tokens of cached state per sequence.

Per token, the cost is:

kv_bytes per token = 2 x layers x n_kv_heads x head_dim x bytes

The 2 is for the K and the V; bytes is the dtype (2 for fp16 or bf16). GQA models shrink the cache by grouping attention heads onto shared KV heads, so n_kv_heads is smaller than the attention-head count. A plain MHA model uses the full head count.

Llama-3.1-8B's config gives the concrete example: 32 layers, 8 KV heads, head dimension 128, bf16.

2 x 32 x 8 x 128 x 2 = 131,072 bytes = 128 KiB per token

Without GQA - if all 32 attention heads kept their own KV - the same model would need 512 KiB per token. Grouping down to 8 KV heads cuts the cache 4x, which is exactly why modern architectures adopt GQA. You can cross-check the formula against the PagedAttention paper: its OPT-13B example needs 800 KB per token (2 x 40 layers x 5120 hidden size x 2 bytes), so a single 2,048-token request can hold up to 1.6 GB of cache.

Now multiply by context length, and remember batch: every concurrent sequence carries its own cache.

At 128K, a single sequence's cache matches the model's entire weights. Long context, not model size, is what usually blows up a VRAM budget.

Put it together: sizing an 8B model end-to-end

Say the whole budget out loud: an 8B fp16 deployment needs 16.1 GB of weights, a context-dependent KV cache, and overhead. Modal compresses this into M = (P x Q/8) x 1.2, where the 20% covers a small KV allowance plus engine overhead. Treat that 1.2x as a floor for short contexts. It is not a ceiling, and it does not begin to cover 128K contexts.

Production data agrees. Measuring a 13B model served on an A100-40GB, the PagedAttention paper found the memory split roughly 65% weights, ~30% KV cache, and the remainder activations - a third of the card spent on cache even in a fairly normal setup. The same paper documents why the serving engine matters as much as the math: in pre-PagedAttention systems, only 20.4-38.2% of KV cache memory held actual token states, with the rest lost to fragmentation and over-reservation. Paged allocation, as implemented in vLLM, cut that waste to near zero and lifted throughput 2-4x at the same latency. The lesson: the formulas tell you the true footprint; the engine decides how much of the card you waste on top of it.

The recipe, end to end:

1. Open your model's config.json: num_hidden_layers, num_key_value_heads (missing means MHA), hidden_size, num_attention_heads, and dtype.
2. Weights: parameters x bytes per parameter.
3. KV: the per-token number x context length x concurrent sequences.
4. Add ~20% overhead, with extra headroom for long context or high batch.
5. Compare against the card, and remember activations take a slice too.

Cutting the bill: quantize weights and cache separately

Weights and cache respond to different dials, so cut them in order.

Weights first. Modal's guidance matches industry consensus: 8-bit generally performs on par with 16-bit at half the memory, while 4-bit saves more but can noticeably impact quality. For evidence that 4-bit is workable rather than broken, the QLoRA paper finetuned a 65B model on a single 48 GB GPU at 4-bit without degrading performance. Workable, not free - and only one facet of the broader efficiency curve we traced in how inference costs keep falling.

Then, only if context x batch forces it, the cache. llama.cpp stores K and V as F16 by default; --cache-type-k and --cache-type-v accept q8_0, q4_0 and friends. vLLM exposes --kv-cache-dtype with fp8 variants, defaulting to auto (the model's dtype).

Cache quantization is not free either. Quantized K and V values must be converted back during attention, which can slow decoding as contexts grow, so test your own model and context length before you rely on it.

So, one line: quantize weights before the cache, and quantize the cache only when context x batch forces your hand.

The knobs your serving engine already exposes

You rarely need to build anything - the engines already expose the memory controls this math points at:

The current vLLM serve docs put the default at 0.92, and it is a per-instance limit - two vLLM instances sharing one GPU can take 0.5 each. On the HF side, accelerate estimate-memory <model> prints checkpoint sizes per dtype on the meta device, so run it before you rent a GPU. These knobs are one facet of running models in production generally - our LLMOps guide covers the wider picture.

The napkin version

VRAM = weights (P x Q/8) + KV cache (2 x L x n_kv_heads x head_dim x bytes x tokens x sequences) + ~20% overhead. Read the numbers out of config.json, multiply, compare with the card. The config tells you the model, the spec sheet tells you the GPU, and everything in between is arithmetic you can do on a napkin.

If you are weighing engines, quantization formats, or serving tools for a specific deployment, the Toolbit directory lets you compare what is out there before you commit.