Quantization: INT8, INT4, GPTQ, AWQ, and GGUF

A 70-billion-parameter model in FP16 is 140 GB of weights. That's two H100s just to hold it, before you've loaded an activation or a KV cache or a single request. Quantization is the lever that takes those same weights and makes them fit somewhere smaller: one H100, one consumer GPU, a Mac, a phone. In the best case, you barely notice the difference in output quality. In the worst case, the model forgets how to add two numbers.

I want to build the intuition for why that range exists, and why the good methods (GPTQ, AWQ, the GGUF schemes) land so far on the "you barely notice" side of it. It turns out to be less about how many bits you use and more about where you choose to spend them.

The problem this layer solves

The serving pipeline is a memory hierarchy problem. Your KV cache wants HBM. Your activations want HBM. Your weights want HBM. When the model doesn't fit, everything cascades: you can't batch, you can't extend context, you page from CPU, latency falls off a cliff. So you want the weights to be smaller.

The cheap way: store each weight in fewer bits. A weight matrix that was 16 bits per entry becomes 8 or 4 or 2 bits per entry. The hardware loads less from memory per matmul, which means the matmul is faster too, because for LLM decoding the bottleneck is usually memory bandwidth, not arithmetic. You're winning in two places at once.

But there's no such thing as "these 16 bits are just 8 bits of information in a trenchcoat." You're throwing bits away. The question is which bits.

Quantization, as a word, gets thrown around a lot. For this post it means one specific thing: take the continuous FP16 weight values, map each of them to the nearest point on a coarse grid, store the grid index, and remember the spacing. Everything interesting is in how the grid is chosen.

Snapping weights to a grid

Here's the basic move. Suppose your weights are a vector $\mathbf{w} \in \mathbb{R}^n$, and you want to store them with $b$ bits each, signed. You pick a scale factor $s$ and write:

The integer part $q_i = \text{round}(w_i / s)$ is what actually lives on disk, packed into $b$-bit slots. The scale $s$ is a single FP16 number stored off to the side. At inference time you dequantize: $\hat{w}_i = s \cdot q_i$.

How do you pick $s$? The simplest answer: look at the largest absolute weight in the tensor and divide by $2^{b-1} - 1$. That way the grid just barely reaches the most extreme value and every other weight sits safely inside. For INT4, that gives you 16 bins spread between $-|w|_{\max}$ and $+|w|_{\max}$.

I want to emphasise one thing before we move on. This is a reconstruction problem. We're approximating each weight by a grid point. The error is $w_i - \hat{w}_i$, and it's bounded by $s/2$. Make the grid finer (more bits) and the error shrinks. Make the grid coarser (fewer bits) and the error grows. Nothing magical.

The viz above is the meat of this whole post. Try the combinations. INT4 per-tensor is noticeably lossy, because one global scale has to span the entire weight range. INT4 per-row is within a whisker of INT8 per-tensor. INT4 per-group (small groups of 32 weights sharing one scale) is visually identical to the original on most of the mass.

That's the core trick: if you give each chunk of weights its own scale, you can get away with a much coarser grid per chunk. The total number of stored bits barely changes, because the scales are cheap (one FP16 per chunk), but the effective dynamic range is enormous.

Why the scale matters so much: outliers

The reason per-channel or per-group scales work is that the weight matrix isn't uniform. Most weights are small, clustered near zero, with a tight distribution. But there are always a few outliers, and a single outlier can ruin uniform quantization for the entire tensor it belongs to.

Think about it. If you have a tensor where 99% of weights live in $[-1, 1]$ and one weight sits at $+12$, the per-tensor scale has to be $12 / 7 \approx 1.71$ (for INT4 signed). So every weight inside that $[-1, 1]$ range can only take values in $\{-1.71, 0, +1.71\}$ (the three INT4 bins that fit). You've just thrown away almost all the signal.

Toggle between the three schemes and read the cell values. Under naive INT4, the small weights in row 0 all snap to either $-1.71$, $0$, or $+1.71$. Row 1 is fine only because it lives in a separate row; if we'd flattened and used one scale, it would have been destroyed too.

Dettmers et al. (LLM.int8) noticed that in large transformers the outliers concentrate in specific feature dimensions. Specific hidden dimensions of the activations get consistently large values, and naive INT8 quantization loses accuracy catastrophically because of them. Their fix was a mixed-precision decomposition: identify the outlier dimensions, keep them in FP16, quantize the rest to INT8, and add the two matmuls back together. Same basic idea as AWQ, different knob.

So: the enemy is outliers, and the remedy is some flavor of "don't use one scale for everything."

GPTQ and AWQ: two ways to spend your bits well

Per-channel or per-group scales get you most of the way. INT4 with group size 128 is a strong baseline. But the top-tier methods go one step further.

GPTQ (Frantar et al., 2022) reframes the problem. Instead of "find $s$ and $q$ that minimize $\|W - \hat{W}\|^2$" (which treats every weight as equally important), it asks "find $s$ and $q$ that minimize $\|WX - \hat{W}X\|^2$ on a calibration dataset $X$." That is, minimize the error in the output of the layer, not the error in the weights themselves. Weights that barely affect the output can be quantized aggressively; weights that matter a lot get extra attention.

The computational trick is that you can do this layer by layer, column by column, using the Hessian of the per-layer reconstruction loss. For each column you quantize, you update the remaining (not-yet-quantized) columns to absorb some of the rounding error. It ends up being efficient enough to quantize a 175B model on one GPU in a few hours.

AWQ (Lin et al., 2023) takes a different angle. The authors observed that the activations have outlier channels too, and those channels correspond to particular weight columns that receive disproportionately large inputs. Their fix is a pre-quantization rescaling: for each outlier channel, scale down the weights (which you can quantize to INT4 safely), and scale up the corresponding activation direction at runtime. Arithmetically it's a no-op: $W X = (W / s)(s X)$. The payoff is that the weights the INT4 grid has to handle are now tame.

Both methods hit roughly the same quality at INT4 that naive quantization gets at INT8. Different routes, same destination: spend more bits on the weights that matter, fewer on the ones that don't.

What happens at runtime

The things above are storage tricks. The inference-time picture is simpler.

On a matmul $Y = W X$, the engine packs the quantized integer weights $Q$ tightly in memory, reads them with one memory fetch per tile, and at some point combines them with the scales $S$ and the input $X$. There are two common implementations:

1. Dequantize-then-matmul: read the INT4 weights, reconstruct $\hat{W} = S \odot Q$ in a register or shared memory tile, run the matmul in FP16 or BF16. Simple, and usually the bottleneck is the weight read anyway, so you don't lose much.
2. Fused INT matmul: do the matmul in integer arithmetic on Tensor Cores that support low-bit inputs, and apply the scale once per output tile. This needs hardware support. NVIDIA's Hopper and newer cards have INT8 Tensor Cores and FP8 Tensor Cores; Blackwell adds FP4 and INT4 paths. Faster but more finicky to implement.

For LLM decoding, where you're loading hundreds of gigabytes of weights per token generated, the memory reduction itself is the win. A 4× smaller weight matrix means 4× fewer bytes to shuttle from HBM to SRAM, which translates almost directly to throughput if you're bandwidth-bound (and for decode you almost always are). The quantized arithmetic on top is a bonus.

One worked example

Let me walk through a tiny tensor by hand, because the arithmetic is straightforward and it pins the intuition. Take a vector $\mathbf{w} = [0.1, -0.4, 0.2, 1.5, -0.1, 0.3, -0.05, 0.8]$ and quantize it to INT4 per-tensor.

First, the scale. $|w|_{\max} = 1.5$. With 4 bits signed, the positive grid goes up to $q_{\max} = 2^3 - 1 = 7$. So $s = 1.5 / 7 \approx 0.2143$.

Now divide each weight by $s$, round, and clip to $[-8, 7]$:

A bunch of small weights got snapped to zero. That's the cost of using a coarse grid stretched to fit the $1.5$ outlier. RMSE works out to about $0.065$, not terrible but also not great for values that should have been $\pm 0.1$.

Swap to per-row quantization on a 2-row matrix where row 1 has only small values, and the row-1 scale drops to $\sim 0.05$, giving plenty of resolution to distinguish $0.09$ from $0.11$. The code widget below shows it end-to-end.

import torch
 
# A small weight vector (8 weights, dynamic range up to 1.5)
W = torch.tensor([0.1, -0.4, 0.2, 1.5, -0.1, 0.3, -0.05, 0.8])
 
def quantize_int4_per_tensor(w, n_bits=4):
    qmax = 2**(n_bits - 1) - 1         # 7 for INT4
    scale = w.abs().max() / qmax
    q = (w / scale).round().clamp(-qmax - 1, qmax).to(torch.int8)
    w_hat = q.to(torch.float32) * scale
    return q, scale, w_hat
 
q, s, W_hat = quantize_int4_per_tensor(W)
print("original :", W.tolist())
print("quantized:", q.tolist(), "scale =", round(s.item(), 4))
print("reconstr.:", [round(x, 3) for x in W_hat.tolist()])
print("rmse     :", round(((W - W_hat)**2).mean().sqrt().item(), 4))
 
# Per-row quantization for a 2x4 matrix — each row gets its own scale.
W2 = torch.tensor([[0.1, -0.4, 0.2, 1.5],     # row 0: outlier 1.5
                   [-0.1, 0.3, -0.05, 0.08]]) # row 1: small range
 
def quantize_int4_per_row(W, n_bits=4):
    qmax = 2**(n_bits - 1) - 1
    scale = W.abs().max(dim=1, keepdim=True).values / qmax
    Q = (W / scale).round().clamp(-qmax - 1, qmax).to(torch.int8)
    return Q, scale, Q.to(torch.float32) * scale
 
Q, S, W2_hat = quantize_int4_per_row(W2)
print("\nper-row scales:", S.flatten().tolist())
print("per-row rmse  :", round(((W2 - W2_hat)**2).mean().sqrt().item(), 4))
# Per-row keeps row 1 (small values) sharp; per-tensor would have crushed it.

This is the actual inner loop of every PTQ method, stripped to its essentials. GPTQ adds the Hessian-based error correction; AWQ adds the activation-aware rescaling first. But the core (w / scale).round().clamp() is the same.

GGUF is a file format, not a method

This is a common source of confusion. GGUF (the format used by llama.cpp) is not a quantization algorithm. It's a container that specifies how quantized tensors, metadata, and tokenizer state are laid out on disk, with streaming-friendly alignment for mmap.

Inside a GGUF file, the actual quantization is one of a dozen schemes: Q4_0, Q4_1, Q4_K_M, Q5_K_M, Q6_K, Q8_0, and so on. Most of these are variations on "take 32 or 64 weights, find a scale (and sometimes a min), store everything in a tightly-packed block." The "K" variants use mixed block sizes and mixed precisions inside a single tensor; they're roughly equivalent to the "per-group + protect important weights" idea from AWQ, just implemented in C++ with handwritten SIMD kernels targeting CPU.

The reason GGUF is everywhere on the edge is entirely about llama.cpp's deployment story: the format is self-contained, memory-maps cleanly, runs on anything from Mac unified memory to Raspberry Pis. The quantization quality varies by scheme (Q4_K_M is the common sweet spot; Q8_0 is near-lossless), but those are orthogonal choices inside the format.

In 2026, if someone hands you a .gguf file, the file format is GGUF and the quantization scheme is whichever one they chose, usually stamped into the filename like llama-3.1-70b-instruct.Q4_K_M.gguf.

Misconceptions

"INT4 is half as accurate as INT8." Counting bins, yes: INT4 has 16, INT8 has 256. But the question is whether your weight distribution actually needed 256 bins, and for LLM weights with per-channel or per-group scales, the answer is mostly no. Calibrated INT4 (GPTQ, AWQ) lands within a fraction of a perplexity point of FP16 on LLaMA-scale models. The bins went 256 → 16, but the useful dynamic range was preserved. You only pay the cost when a tensor's distribution genuinely has 5+ decades of dynamic range to capture, which is rare for trained weights (and common for activations).

"GGUF is a quantization method." GGUF is the file format llama.cpp uses. The quantization happens inside, using schemes like Q4_0, Q4_K_M, Q8_0, each of which is a separate algorithm for packing integer weights with scales and zero-points. A single GGUF file could in principle hold any of those schemes (and picking between them is the first thing you do when you download a quantized model).

"Activations don't need quantization." They often do, and they're harder than weights. Weights are static; you can calibrate once. Activations are dynamic, so you need either per-token rescaling (cheap, done on the fly) or per-channel stats collected from a calibration pass. SmoothQuant's trick is that it shifts the hard part of activation quantization onto the weights (which can absorb it), keeping both sides in INT8 range. For serving at scale, INT8 KV cache is a real win and increasingly standard.

"Quantized models are always fine." For most chat and coding workloads, INT4 is genuinely fine. For multi-step reasoning, math, or long-context benchmarks, the degradation shows up. The numbers in the chart earlier are perplexity on a generic language corpus; they under-represent task-specific regression. Always evaluate on your workload before shipping quantized weights to users. A 0.1 perplexity bump can hide a 5-point drop on your specific eval.

What's next

The next post takes a different lever for fitting and serving large models: instead of shrinking each weight, split the model across multiple GPUs. Start with Tensor, Pipeline, and Expert Parallelism.

Additional reading (and watching)

• Dettmers, T., et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. NeurIPS 2022. Documents the emergent outlier phenomenon in LLM weights and activations and introduces the mixed-precision decomposition that keeps outliers in FP16.

• Xiao, G., et al. (2022). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. ICML 2023. Shifts the activation-outlier problem into the weight matrix via an equivalent transformation, enabling INT8 W8A8 without catastrophic quality loss.

• Frantar, E., et al. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR 2023. Introduces layer-wise Hessian-based quantization that minimizes output error rather than weight error; the method of choice for calibrated INT3/INT4 GPU serving.

• Lin, J., et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. MLSys 2024. Uses activation statistics to decide which weight channels to protect, then applies an equivalent rescaling so INT4 quantization preserves the protected channels' precision.

• NVIDIA (2024). NVIDIA Blackwell Architecture Technical Brief. Introduces FP4 and INT4 Tensor Core paths alongside Hopper's FP8/INT8. The hardware that makes low-bit fused matmuls fast.

• Gerganov, G., et al. (2023+). GGUF specification — ggml-org/ggml. The on-disk format used by llama.cpp and ecosystem tools. Defines how tensors, metadata, and tokenizer state are laid out for memory-mapped loading.

• Dettmers, T., et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023. Introduces the NF4 data type and double quantization, enabling fine-tuning of 70B-class models on a single 48 GB GPU.

• Hooper, C., et al. (2024). KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization. NeurIPS 2024. Covers why activation/KV quantization is harder than weight quantization and offers the current state of the art for long-context serving.

• Kwon, W., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. The vLLM paper; the same engine that ships first-class AWQ and GPTQ support for INT4 GPU serving.

• Micikevicius, P., et al. (2022). FP8 Formats for Deep Learning. arXiv:2209.05433. Defines the E4M3 and E5M2 FP8 formats used by Hopper and later NVIDIA hardware for near-lossless mixed-precision training and inference.

• Dettmers, T., & Zettlemoyer, L. (2022). The case for 4-bit precision: k-bit Inference Scaling Laws. ICML 2023. Empirically argues that 4 bits per weight is close to the Pareto frontier for inference under real hardware constraints.