GPUs, Floating Point, and Why Precision Matters

A 70-billion-parameter model needs 280 GB for fp32 weights alone. Halving the bits per weight halves that storage, but it also changes which numbers the machine can represent. The useful question is where that loss of range or precision affects the computation.

We will inspect the bit patterns, compare numerical ranges, and use mixed precision in a small training loop.

This post assumes you've read Vectors, Matrices, and the Spaces They Live In and Gradients and Learning. You'll need to know what a matrix multiply is and why we compute gradients during training.

Billions of numbers

The learned parameters occupy a substantial part of a model's storage. GPT-3 had 175 billion parameters; Llama 3 405B has about 405 billion. The chosen numerical format determines how many bytes each parameter needs.

If you store each parameter in the standard 32-bit floating-point format (fp32), that's 4 bytes per parameter. For a 70-billion-parameter model, that's 280 GB just for the weights. An A100 80 GB GPU has 80 GB of memory. You can't even load the model onto one GPU, let alone train it.

If you could store each number in 16 bits instead of 32, you'd cut that to 140 GB. In 8 bits, 70 GB. And it's not just about storage. The GPU has to move all those numbers through its memory bus to actually compute with them. Smaller numbers means faster transfers. And modern GPU hardware has specialized circuits that can do math on smaller numbers much, much faster.

How computers store decimal numbers

IEEE 754, first standardized in 1985, defines the binary floating-point formats used by CPUs and GPUs. Hardware also supports formats such as bfloat16 and modes that use different precision or rounding behavior.

A floating-point number is stored as three fields packed into a fixed number of bits. For normalized finite nonzero numbers, which is the common case, the value is:

Special exponent patterns (all zeros, all ones) carve out the edge cases: signed zeros, subnormals (very small numbers below the normal range), infinities, and NaNs. Subnormals are going to matter a lot when we get to fp16 underflow.

• Sign (1 bit). 0 for positive, 1 for negative. Simple.
• Exponent (variable bits). Determines the range of the number. A wider exponent field means you can represent very large and very small numbers. The bias is subtracted so you can represent negative exponents without a separate sign bit.
• Mantissa (variable bits). Also called the significand or fraction. Determines the precision. More mantissa bits means finer distinctions between nearby values.

Think of it like scientific notation. In $6.022 \times 10^{23}$, the "6.022" part is your mantissa (precision) and the "23" is your exponent (range). If I only let you write two decimal places in the mantissa, you'd get $6.02 \times 10^{23}$. You can still represent the same enormous magnitudes, but you lose the ability to distinguish $6.022$ from $6.025$.

The formats

fp32: 32-bit floating point

The standard 32-bit float uses 1 sign bit, 8 exponent bits, and 23 mantissa bits. This gives you:

• Range: roughly $\pm 3.4 \times 10^{38}$. Wider than fp16, but exponentials and large intermediate products can still overflow.
• Precision: about 7 decimal digits. More than enough for most purposes.

fp32 has long been common in machine learning; scientific computing also relies heavily on fp64. It's the forgiving option for baseline ML training. fp32 still has overflow, underflow, NaNs, non-associative addition, catastrophic cancellation, and softmax/exp blow-ups if you're not careful.

The problem is that it's also big. 4 bytes per number. And GPUs can only do so many fp32 operations per second.

fp16: a narrower exponent range

IEEE 754 also defines a 16-bit format. It uses 1 sign bit, 5 exponent bits, and 10 mantissa bits. Half the memory of fp32. But look at what we gave up:

• Range: roughly $\pm 6.5 \times 10^{4}$. The maximum finite value is 65,504. Activations or gradients can exceed the format’s range, producing infinities that may lead to NaNs in later operations.
• Precision: about 3 decimal digits.

An activation or gradient can exceed fp16's range. Under round-to-nearest, 65,505 rounds to 65,504, but 70,000 overflows to infinity. Subsequent operations involving that infinity can produce NaNs.

bfloat16: range with fewer fraction bits

Around 2018, Google Brain introduced bfloat16 ("Brain Float 16") for their TPUs. The design is pragmatic: 1 sign bit, 8 exponent bits, and 7 mantissa bits.

Those 8 exponent bits give bfloat16 essentially the same dynamic range as fp32. Its largest finite value is about $3.39 \times 10^{38}$, just slightly under fp32's $3.40 \times 10^{38}$, because the smaller mantissa caps the representable magnitude. The wider range avoids many fp16 overflows.

The cost is precision. With only 7 mantissa bits instead of fp32's 23, you get roughly 2-3 decimal digits of precision. Under round-to-nearest-even, 3.14159 is stored as 3.140625. You lose the fine distinctions.

Kalamkar et al. found that bf16 could match fp32 training quality on the workloads they studied when combined with suitable higher-precision computation. The wider exponent helps avoid range failures.

bfloat16 is not an IEEE 754 interchange format. Its layout matches the upper 16 bits of fp32, preserving the sign and exponent fields. Conversions commonly use round-to-nearest-even rather than truncating the lower bits; the worked example below shows the difference.

fp8: scaling becomes essential

NVIDIA’s H100 introduced Tensor Core support for fp8; Blackwell GPUs also support it. Two common encodings are:

• E4M3. 1 sign, 4 exponent, 3 mantissa bits. Better precision, less range.
• E5M2. 1 sign, 5 exponent, 2 mantissa bits. More range, barely any precision.

E4M3 uses one byte per number and has eight representable values per power-of-two interval in its normal range.

But it works for inference, and increasingly for parts of training too. The Micikevicius et al. 2022 paper showed that you can do the forward pass in E4M3 and the backward pass in E5M2 (where you need the range more than the precision, since gradients can be very small) and still converge.

TF32: reduced-precision multiplication of fp32 inputs

There's one more format to know about, even though you rarely see it in user code. On supported Ampere and later NVIDIA GPUs, Tensor Cores expose a mode called TF32. Despite the name, it's not a 32-bit storage format. Tensors stay stored as fp32. What changes is that when a matmul runs on the Tensor Cores, the inputs get internally rounded to a 19-bit format (1 sign bit, 8 exponent bits, 10 mantissa bits) before the multiply-accumulate, while the accumulator stays in fp32.

TF32 keeps fp32's exponent range but rounds inputs to ten fraction bits for Tensor Core multiplication. It therefore trades precision for throughput. PyTorch's CUDA matrix multiplications have defaulted to IEEE fp32 precision since version 1.12; TF32 is not a universal default. In PyTorch 2.9 and later, torch.backends.cuda.matmul.fp32_precision = "tf32" opts into it. Convolution has a separate setting.

Range and precision side by side

It helps to see the range of each format laid out on a single axis. The marker below sweeps through the displayed orders of magnitude. The bars light up when a format can represent the current value, and the readouts show that format's step size (ULP) at that magnitude. Watch fp16 fall off near $10^5$ while bf16 keeps going up to $10^{38}$ alongside fp32.

See it yourself

You can inspect the bit patterns directly. Type a number below and watch how each format encodes it. Pay attention to what happens with 0.0001 (very small), 65504 (fp16's limit), and 100000 (beyond fp16's range).

Try typing 100000. Watch fp16 go to infinity while bfloat16 handles it fine. Same exponent range as fp32, so it can represent the same magnitudes. The mantissa is rougher, so the stored value won't be exactly 100000, but it won't explode.

Now compare bfloat16 against fp8.

With three fraction bits, E4M3 rounds 3.14 to 3.25. Whether that precision is sufficient depends on the tensor and computation. Practical fp8 training and inference use scale factors and higher-precision operations where needed; the scaling depends on the tensor’s range.

The hardware: why GPUs are good at this

Neural-network training repeatedly multiplies large matrices. GPUs provide many arithmetic lanes and specialized matrix units for computing different parts of those products in parallel.

The architecture in brief

A modern GPU like the A100 is organized into Streaming Multiprocessors (SMs). The A100 has 108 of them. Each SM contains a collection of CUDA cores (for general-purpose floating-point math) and, critically, Tensor Cores (specialized hardware for matrix multiply-accumulate operations).

All of these SMs share a pool of HBM (High Bandwidth Memory). The A100 80 GB SXM model has HBM2e with about 2 TB/s of bandwidth; other A100 configurations differ. The SMs pull data from HBM, crunch numbers, and write results back.

SIMT: where the parallelism comes from

NVIDIA GPUs group 32 threads into a warp. Their SIMT (Single Instruction, Multiple Threads) execution issues a common instruction across active lanes, each operating on its own data. Matrix multiplication supplies many repetitions of the same arithmetic, which fits this execution model.

From Volta (compute capability 7.0) onward, NVIDIA GPUs also support independent thread scheduling, with per-thread program counters and divergence and reconvergence at sub-warp granularity. Code that requires synchronization within a warp must use __syncwarp() or the appropriate warp-level intrinsics; sharing a warp does not itself guarantee synchronization.

Memory bandwidth and compute limits

A kernel is memory-bound when moving its inputs and outputs limits throughput more than arithmetic does. It is compute-bound when arithmetic throughput is the limit. The roofline model compares these limits using the kernel's work per byte transferred.

Decode at small batch sizes and optimizer-state movement often depend on memory bandwidth, while large training and prefill matrix multiplications can be compute-bound. Lower precision reduces bytes transferred and can increase Tensor Core throughput. The end-to-end gain depends on which operations limit the workload.

Tensor Cores and the precision speedup

Tensor Cores accelerate matrix multiply-accumulate operations on tiles. Supported tile shapes, input formats, accumulation formats, and throughput depend on the GPU architecture. Their peak rates require kernels with suitable shapes and enough work to keep the hardware busy.

The table compares advertised dense peaks for two specific SXM GPUs. Ordinary fp32 uses CUDA cores here, while bf16 and fp8 use Tensor Cores. These are hardware ceilings, not measured application speedups.

The A100's roughly 16-fold advertised peak-throughput ratio compares plain fp32 on CUDA cores with bf16 matrix arithmetic on Tensor Cores. It reflects different hardware paths as well as precision. Storing operands in bf16 also halves their byte count relative to fp32.

The H100 SXM advertises 1,979 dense fp8 TFLOPS. That number does not imply a corresponding end-to-end speedup: data movement, non-matmul operations, and kernel utilization remain part of the workload.

Worked example

To encode $3.14$, start with its exponent, then round its binary fraction.

In fp32, the bias is 127. We need to find an exponent $e$ such that $2^e \le 3.14 < 2^{e+1}$. That's $e = 1$ (since $2 \le 3.14 < 4$). The biased exponent stored in the bits is $1 + 127 = 128$, which in 8-bit binary is 10000000.

Now the mantissa. The value is $3.14 = 2^1 \times (1 + m)$, so $m = 3.14/2 - 1 = 0.57$. We need to write $0.57$ as a sum of negative powers of two. Doubling gives $1.14$, so the first bit is 1 and we keep $0.14$. Doubling gives $0.28$, so the next bit is 0. Keeping 23 fraction bits and rounding gives the fp32 mantissa 10010001111010111000011. Sign bit is 0. Stitch them together and that's the fp32 bit pattern for $3.14$.

In bf16, we keep the top 7 mantissa bits (rounded, not just truncated). Same sign, same exponent, mantissa becomes 1001001 after round-to-nearest-even. The reconstructed value is $2 \times (1 + 1/2 + 1/16 + 1/128) \approx 3.1406$. The bits beyond that are gone.

In fp16, the bias is 15 instead of 127, so the biased exponent is $1 + 15 = 16$, which is 10000. The 10-bit mantissa, after round-to-nearest-even, is 1001001000. Reconstructed: $2 \times (1 + 1/2 + 1/16 + 1/128) = 2 \times 1.5703125 = 3.140625$. Same answer as bf16 to four decimals, because both formats happen to land on the same representable value here. fp16 has more mantissa bits in general, but at this magnitude the extra resolution doesn't kick in until further down the fraction.

You can poke at this directly above by typing values into the bit viewer. Try 65504 (the largest finite fp16 number), 65505 (rounds back to 65504), and 100000 (overflows to inf).

Mixed-precision training: having it both ways

Reduced-precision matrix operations can save time and memory, while higher-precision updates and selected operations limit accumulated rounding error.

Mixed-precision training, described by Micikevicius et al. in a 2017 preprint and published at ICLR 2018, combines formats in one training run. Their fp16 recipe uses:

1. Keep a master copy of the weights in fp32. Apply weight updates to this higher-precision copy.
2. Cast to fp16/bf16 for the forward and backward pass. The actual matrix multiplies, which are the expensive part, happen in reduced precision. Hardware support determines the throughput gain.
3. Loss scaling (for fp16 specifically). Small gradient values can underflow to zero in fp16. The trick is to multiply the loss by a large scalar before backpropagation, which scales all the gradients up into fp16's representable range. After computing the gradients, you divide by the same scalar before updating the master weights. In exact arithmetic the scale cancels. In floating point it changes which gradients survive rounding, and an excessive scale can overflow.

With bfloat16, you often don't need loss scaling because the exponent range matches fp32. Gradients that are tiny (like $10^{-8}$) are still representable.

Here's what the PyTorch implementation looks like. This runnable example requires a CUDA GPU.

import torch
 
torch.manual_seed(0)
model = torch.nn.Linear(16, 8).cuda()
dataloader = [(torch.randn(32, 16, device="cuda"), torch.randn(32, 8, device="cuda"))]
criterion = torch.nn.MSELoss()
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)
 
# Pick one. bf16 doesn't need a scaler; fp16 does.
dtype = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16
use_scaler = dtype == torch.float16
scaler = torch.amp.GradScaler("cuda", enabled=use_scaler)
 
for inputs, targets in dataloader:
    optimizer.zero_grad(set_to_none=True)
 
    # Forward pass in reduced precision
    with torch.amp.autocast("cuda", dtype=dtype):
        output = model(inputs)
        loss = criterion(output, targets)
 
    # Backward pass. With bf16, scaler is a no-op; with fp16 it
    # handles the loss-scaling dance.
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()
 
# Quick demo: see what fp16 does to a value past the cliff
x = torch.tensor([65504.0, 65505.0, 100000.0], dtype=torch.float16)
print(x)   # tensor([65504., 65504.,    inf], dtype=torch.float16)
y = torch.tensor([65505.0, 100000.0], dtype=torch.bfloat16)
print(y)   # tensor([65536., 99840.], dtype=torch.bfloat16)

Autocast chooses computation dtypes by operation; model parameters in this example remain fp32. With fp16, GradScaler scales the loss, unscales gradients, and skips an optimizer update when it detects nonfinite gradients. With bf16, scaling is usually unnecessary and the disabled scaler passes calls through.

In this implementation, reduced-precision operations can save activation memory, but weights, gradients, and optimizer states still occupy fp32 storage.

What happens when precision goes wrong

Suppose an attention score rounds to 70,000 in fp16. It overflows to infinity; a softmax calculation involving it can then produce NaNs. If the training loop applies nonfinite gradients without detecting them, parameter updates can become nonfinite too.

When the loss becomes NaN, inspecting earlier intermediate values can help locate the first overflow.

A gradient of $10^{-8}$ rounds to zero in fp16 under round-to-nearest-even. The smallest positive normal value is about $6.10 \times 10^{-5}$, but subnormals extend down to $5.96 \times 10^{-8}$. Crossing the normal boundary reduces relative precision; values below half the smallest subnormal round to zero. Loss scaling can preserve these small gradients during backpropagation.

Loss scaling addresses gradient underflow; it does not prevent an activation from overflowing during the forward pass. If you multiply the loss by something like $2^{15}$ before backprop, that $1 \times 10^{-8}$ gradient becomes about $3 \times 10^{-4}$, comfortably inside fp16's normal range. After the backward pass, you divide by the same factor and you're back to the correct value.

Misconceptions

"fp16 is always fine for training." Its limited range can overflow activations or underflow gradients. The mixed-precision recipe combines fp32 weights, selected higher-precision computation, and loss scaling; even then, stability depends on the model and operations.

"bfloat16 is worse because it's less precise." bf16 trades mantissa precision for exponent range: 7 mantissa bits and 8 exponent bits, versus fp16's 10 and 5. For example, bf16 can represent a gradient near $10^{-8}$ while fp16 rounds it to zero. Which error matters depends on the computation.

"Quantization is the same as lower-precision training." Mixed-precision training combines numerical formats while optimizing the model. Quantization selects compact representations, often for inference. Post-training quantization does this after training; quantization-aware training accounts for its effects during training. The later quantization post focuses on inference representations.

"GPUs are fast because each core is fast." The GPU's advantage comes from parallel work across many lanes. The speed comes from having tens of thousands of them executing the same instruction across different data lanes, plus Tensor Cores that do whole tile-sized matmuls per clock. High throughput requires enough parallel work to keep those units busy.

What's next

Next up: A Short Prehistory of Statistical NLP. We'll look at how language modeling developed before transformers, from counting word sequences to learning embeddings.

Additional reading (and watching)

• IEEE Computer Society. (2019). IEEE 754-2019, Standard for Floating-Point Arithmetic. The canonical specification for sign/exponent/mantissa formats and rounding modes.
• Kalamkar, D., et al. (2019). A Study of BFLOAT16 for Deep Learning Training. Experiments combining bf16 with higher-precision computation on several training workloads.
• Micikevicius, P., et al. (2022). FP8 Formats for Deep Learning. Proposes the E4M3 / E5M2 split and shows it converges on language and vision benchmarks.
• NVIDIA. (2020). NVIDIA A100 Tensor Core GPU Architecture. Whitepaper detailing SMs, warps, Tensor Cores, and HBM bandwidth on Ampere.
• NVIDIA. H100 Tensor Core GPU specifications. The SXM dense rates used in the comparison; starred headline figures include sparsity.
• PyTorch. CUDA semantics. TF32 settings, defaults, and precision tradeoffs.
• PyTorch. Automatic Mixed Precision recipe. Autocast and gradient scaling in a training loop.
• Micikevicius, P., et al. (2018). Mixed Precision Training. ICLR 2018. The original recipe for fp32 master weights + fp16 compute + loss scaling.
• Dettmers, T., et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. NeurIPS 2022. Foundational work showing that even 175B-parameter models can be quantized below fp16 with care around outlier features.
• Williams, S., Waterman, A., & Patterson, D. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures. CACM. Origin of the "memory-bound vs compute-bound" framing used throughout this post.