What quantization actually is

A model's weights start out as 16-bit (or occasionally 32-bit) floating-point numbers — that's the precision they were trained and typically distributed in. Quantization replaces those with lower-precision representations: fewer bits per weight, grouped into blocks that share a small amount of extra data (a scale factor, sometimes a minimum value) so the reduced-precision numbers can still be mapped back to something close to the original range. Fewer bits per weight means less memory and less data to move through the chip per token generated — which is why quantized models are both smaller and faster, not just smaller.

The cost is precision, not correctness in any simple sense: a quantized model doesn't become "wrong" the way a training bug would make it wrong. It becomes a slightly blurrier version of the same model — individual outputs can differ from the full-precision original, more so at lower bit depths, but there's no bit width at which the model suddenly starts producing garbage across the board. The practical question is always "how much quality am I willing to trade for how much size and speed," not "is this broken."

Why "4-bit" isn't one thing

Open an actual GGUF download list and the names are more specific than "4-bit" or "8-bit" — Q4_K_M, Q5_K_M, Q4_0, Q8_0, and several more. The number is the average bits per weight; the letters describe the quantization scheme, and they're not interchangeable at the same bit count. The older, plain Q4_0/Q4_1 schemes quantize uniformly, block by block, with no per-layer nuance. The newer k-quants (Q4_K_M, Q5_K_M, and similar) are more deliberate about it — allocating more precision to weights that empirically matter more to output quality and less to weights that don't, rather than treating every block identically. At the same nominal bit width, a k-quant variant is close to universally the better choice over the legacy scheme it replaced; llama.cpp still ships both mainly for compatibility with older tooling.

The trailing letter on a k-quant (_S, _M, _L) is a small additional lever within that same bit width — small, medium, large — trading a bit more size for a bit more fidelity without changing the headline bit count. When two model cards both say "Q4," check the full tag before assuming they mean the same tradeoff.

Picking a level, not defaulting to one

A workable default rather than a rule: Q4_K_M is the level most of the ecosystem converges on as the practical sweet spot — a meaningful size and speed win over 8-bit, with a quality gap that's genuinely hard to notice in normal use for most tasks. Move up from there deliberately, not reflexively:

  • Q5_K_M or Q6_K when the task is precision-sensitive — code generation and anything involving exact syntax or careful multi-step reasoning tends to show quantization loss sooner than open-ended writing does.
  • Q8_0, close to full precision, when you have the memory headroom to spare and want the smallest possible gap from the unquantized model, accepting the larger download and footprint.
  • Q3_K or lower only when memory is the hard constraint and running a smaller parameter count at a higher bit level isn't an option — a heavily quantized larger model is not reliably better than a lightly quantized smaller one at the same memory budget, and often worse. Test both rather than assuming bigger-but-blurrier wins.

MLX-format models use their own bit-depth conventions rather than the GGUF/k-quant naming above, but the underlying tradeoff — bits per weight traded against quality — is the same idea wearing a different label. The tools and formats post covers where each format actually comes from.

What this changes about the RAM math

Going up one quantization tier doesn't just cost more memory for weights — it changes the whole budget covered in the RAM requirements post: weights scale directly with bits per parameter, and KV cache size is independent of quantization level entirely, computed from the model's architecture rather than its weight precision. A model that just barely fits at Q4_K_M with room for a long agentic session may not have any KV cache headroom left at all at Q8_0, even though the weights themselves are still well within memory. Check both numbers before moving up a tier, not just whether the weights alone fit.

Related reading: how much RAM local LLMs actually need, the math this post's bit-width numbers feed into, and the local LLM tools and formats landscape for where to actually find quantized models. Or download Velo Workspaces and try it yourself.