What quantization is

A model's weights start as 16-bit (occasionally 32-bit) floating-point numbers, the precision they're trained and usually published in. Quantization replaces them with fewer bits per weight, grouped into blocks that share a little extra data (a scale factor, sometimes a minimum) so the smaller numbers still map back close to the original range. Fewer bits means less memory and less data moved through the chip for each token, which is why quantized models are faster as well as smaller.

The cost is precision, not correctness. A quantized model doesn't become wrong the way a training bug would make it wrong. It becomes a slightly blurrier version of the same model: individual answers can differ from the full-precision original, more so at lower bit depths, but there's no bit width where it suddenly produces garbage. The question is always how much quality you'll trade for how much size and speed.

Why "4-bit" isn't one thing

A real GGUF download list is more specific than "4-bit": Q4_K_M, Q5_K_M, Q4_0, Q8_0 and more. The number is the average bits per weight; the letters name the scheme, and schemes aren't interchangeable at the same bit count. The older Q4_0 and Q4_1 quantize every block the same way. The newer k-quants (Q4_K_M, Q5_K_M and friends) spend more precision on the weights that matter most to output quality and less on the rest. At the same bit width, a k-quant is almost always the better choice; llama.cpp keeps the old schemes mainly for compatibility.

The trailing letter on a k-quant (_S, _M, _L, for small, medium and large) is a smaller lever within the same bit width: a little more size for a little more fidelity. When two model cards both say "Q4", check the full tag.

Choosing a level on purpose

A sensible default, not a rule: Q4_K_M is where most of the ecosystem lands. It's a clear size and speed win over 8-bit, with a quality gap that's hard to notice in normal use for most tasks. Move away from it deliberately:

  • Q5_K_M or Q6_K when the task is precision-sensitive. Code generation, exact syntax and careful multi-step reasoning tend to show quantization loss sooner than open-ended writing.
  • Q8_0, close to full precision, when you have memory to spare and want the smallest gap from the original model.
  • Q3_K or lower only when memory is the hard limit and a smaller model at a higher bit level isn't an option. A heavily quantized large model isn't reliably better than a lightly quantized smaller one in the same memory, and is often worse. Test both.

MLX-format models name their bit depths differently from GGUF, but the trade-off is the same: bits per weight against quality. The tools and formats post covers where each format comes from.

What it changes in the memory budget

Moving up a tier doesn't only cost weight memory. It changes the whole budget in the RAM requirements post: weights scale with bits per parameter, but the KV cache doesn't depend on quantization at all; it comes from the model's architecture. A model that fits at Q4_K_M with room for a long agent session may have no KV cache headroom left at Q8_0, even though its weights still fit. Check both numbers before moving up.

Related reading: how much RAM local LLMs need, and the local LLM tools and formats for where to find quantized models. Or download Velo Workspaces and try it yourself.