Two numbers, not one

A model's memory use is two things added together: the weights, fixed once you pick a model and a quantization, and the KV cache, which grows as the conversation gets longer. Sizing for the weights alone is how a model works in a quick test and then runs out of memory in real use.

Weights: the easy part

Weight memory is roughly the parameter count times the bytes per parameter. The name gives you the first number ("7B" is 7 billion parameters) and the quantization gives you the second: 4-bit is about 0.5 bytes per parameter, so a 7B model at 4-bit needs about 3.5 GB for its weights. Double the bits and you roughly double that. (LLM quantization explained covers why schemes at the same bit width aren't interchangeable.) Model cards give you this number, and it's where most "how much RAM" answers stop, which is the gap that causes surprises later.

KV cache: the part that grows

The key-value cache holds attention state for every token in the current context, so unlike the weights, it grows with the conversation. The formula is 2 × layers × KV heads × head dimension × tokens × bytes per element. Every term is in the model's config, and most inference tools report the result so you don't have to. What matters is the effect. In a short exchange the weights dominate and the KV cache hardly registers. In a long agent session with a big context window, the kind of multi-turn, tool-calling conversation a coding agent produces, the KV cache can match or exceed the weights. That's why a model can be fine when you start and slow or crashing later: the model didn't change, the context kept growing.

The other consumer: macOS

With unified memory, the model isn't the only thing drawing from the pool. macOS and the display need their share, which is why the model table keeps about 20 to 25% of total memory aside. That's not an arbitrary margin; it covers real system use that a simple "weights plus KV cache" sum leaves out.

Putting it together

For any model: weights (parameters times bytes per parameter), plus a KV cache sized for the sessions you expect (small for a short chat, much more for a long agent run), plus 20 to 25% headroom for macOS. If the weights alone take most of your memory, there's nowhere for the KV cache to grow. Choose a smaller model rather than fighting swap.

What it looks like when you get it wrong

Memory pressure on a Mac rarely shows up as a crash. It shows up as swapping: an engine that was fast a minute ago suddenly isn't, with no error pointing at the cause. If a model gets steadily slower through a session instead of staying consistent, check the context length and KV cache before blaming the model or the engine.

Related reading: the model table this post explains, the local LLM tools, and how much memory to give a Linux VM for the VM side of the budget. Or download Velo Workspaces and try it yourself.