Quantization: INT8, INT4, GPTQ, AWQ, and GGUF

A 70-billion-parameter model needs roughly 140 GB just for raw 16-bit weights. Four-bit storage reduces that payload to 35 GB. The practical questions are what else the format stores, how much the approximation changes the model, and whether the runtime can use it efficiently.

Quantization replaces values with a smaller set of representable values. Uniform integer grids are a useful starting point, though floating-point formats and nonuniform codebooks make different choices.

A scale turns integers back into weights

For a symmetric $b$-bit example, choose $q_{\max}=2^{b-1}-1$ and

At four bits, this symmetric construction uses codes −7 through 7: fifteen levels, with one of the sixteen possible bit patterns unused. Other schemes use an asymmetric range and a zero point. An all-zero group needs special handling so that its scale is not used as a zero divisor.

Without clipping, rounding to a uniform grid introduces at most half a grid step of absolute error. That bound says little about the downstream model: different weights have different effects, and errors accumulate through many layers.

FP16 itself also has finitely many representable values. Its spacing changes with magnitude, unlike the uniform integer grid shown here.

Outliers can spend the available range

If most values are near zero but a few are much larger, a single scale must cover those extremes. Small values then share relatively coarse grid steps. Using separate scales for groups can reduce this problem, at the cost of storing more scales and handling more groups.

Outliers in activations and outliers in weights are related but distinct problems. LLM.int8() uses a mixed-precision decomposition for activation outlier dimensions. Keeping selected values at high precision can help, but it also complicates execution. A format needs a kernel that supports its layout efficiently.

GPTQ protects layer outputs

Rounding weights independently minimizes a local weight error. GPTQ instead uses calibration activations to approximate how a layer's output changes. For a weight row and calibration matrix $X$, the reconstruction objective has the form

Correlations in $XX^T$ matter: changing another unquantized weight can compensate for some output error from a weight just rounded. GPTQ uses damped second-order information to choose these updates efficiently.

Minimizing this layer-output objective can increase ordinary weight RMSE: the calibration activations determine which errors matter most.

AWQ rescales important channels

AWQ uses activation statistics to identify weight channels worth protecting. It scales selected weight channels up before quantization and compensates by scaling the corresponding input activations down. For a diagonal scale matrix $S$,

Before quantization, these are the same product. Quantizing the rescaled weights changes which errors the integer grid introduces. AWQ searches for useful scales while retaining a low-bit weight representation; it does not simply store its important weights in FP16.

SmoothQuant addresses activation outliers by moving difficulty from activations into weights through an equivalent rescaling, with a different objective.

Bit width is not a quality score

Four bits means sixteen possible bit patterns, not half the accuracy of eight bits. To evaluate a quantized checkpoint, compare the same base model, tokenizer, prompts, context lengths, and evaluation protocol. Measure the tasks that matter to the application, including long-context or structured-output behavior when relevant.

Perplexities from different datasets or tokenizers cannot establish a universal ranking of quantization methods. A small average change can also hide a larger change on a particular task.

Storage savings and execution speed

A W4A16 kernel stores weights at four bits while consuming 16-bit activations. It may unpack and dequantize weights into a floating-point computation; it is not necessarily an integer matrix multiply. W8A8 and floating-point low-precision kernels have different hardware paths.

Reducing weight traffic can help a bandwidth-limited decode step. It does not imply a fourfold end-to-end speedup: dequantization, kernel efficiency, batching, attention, KV traffic, and host overhead still contribute. A representation that saves memory can even run slower on an unsupported or poorly matched backend.

The KV cache has its own dtype and scaling scheme. Quantizing weights alone does not reduce KV payload, and a runtime's dtype setting does not automatically produce a valid quantized checkpoint.

GGUF describes more than a bit count

GGUF is a container used by llama.cpp and related tools. The tensors inside can use different quantization layouts. Block metadata means a nominal four-bit scheme occupies more than four bits per weight.

For example, Q4_0 stores 32 weights in 18 bytes, including its scale: 4.5 bits per weight. Q4_K also averages 4.5 bits per weight within its 256-weight superblock, using a different arrangement of scale information. Equal storage density does not make the two quantizers equivalent.

Try the rounding rule

import numpy as np
 
weights = np.array([-0.53, -0.21, -0.08, 0.0, 0.06, 0.17, 0.31, 0.55])
 
def quantize_symmetric(values, bits):
    if not isinstance(bits, int) or not 2 <= bits <= 8:
        raise ValueError("This int8-backed example supports 2 through 8 bits")
    qmax = 2 ** (bits - 1) - 1
    maximum = np.max(np.abs(values))
    scale = maximum / qmax if maximum else 1.0
    codes = np.clip(np.rint(values / scale), -qmax, qmax).astype(np.int8)
    return codes, scale, codes.astype(float) * scale
 
codes, scale, restored = quantize_symmetric(weights, 4)
print("codes:", codes)
print("scale:", scale)
print("RMSE:", np.sqrt(np.mean((weights - restored) ** 2)))
assert np.max(np.abs(weights - restored)) <= scale / 2 + 1e-12
assert np.all(quantize_symmetric(np.zeros(8), 4)[2] == 0)

The NumPy codes above are stored as eight-bit integers for convenience; this teaching example does not pack two four-bit codes into one byte. Real formats must specify that packing and their metadata.

When one device still cannot meet the memory or latency requirement, the next option is parallelism across devices.

Additional reading

• GPTQ, Frantar et al., and its reference implementation.
• AWQ, Lin et al. Activation-aware weight rescaling.
• LLM.int8(), Dettmers et al. Mixed-precision treatment of activation outliers.
• SmoothQuant, Xiao et al. Rescaling for weight and activation quantization.
• ggml quantization block definitions. Storage layouts behind the GGUF examples.