GPUs, Floating Point, and Why Precision Matters

So we keep hearing about fp16 and bfloat16 and quantization. What does any of that actually mean? Like, why can't we just use normal numbers?

I spent a long time vaguely nodding along whenever someone mentioned "mixed-precision training" or "bfloat16," assuming I'd figure it out eventually. Then I actually sat down and looked at how floating-point numbers work at the bit level, and a bunch of things clicked. Why training sometimes explodes. Why Google invented a weird new number format. Why an H100 is so much faster than an A100 at certain workloads. It all traces back to the same handful of ideas.

This post is about those ideas. We'll start with how computers represent decimal numbers at all, work through the different precision formats, look at the actual hardware that processes them, and end up at the mixed-precision training recipe that basically every large model uses today.

This post assumes you've read Vectors, Matrices, and the Spaces They Live In and Gradients and Learning. You'll need to know what a matrix multiply is and why we compute gradients during training.

Billions of numbers

A large language model is, at its core, a massive collection of numbers. GPT-3 had 175 billion parameters. Llama 3 405B has, well, 405 billion. Each parameter is a floating-point number sitting in GPU memory.

If you store each parameter in the standard 32-bit floating-point format (fp32), that's 4 bytes per parameter. For a 70-billion-parameter model, that's 280 GB just for the weights. A single A100 GPU has 80 GB of memory. You can't even load the model onto one GPU, let alone train it.

If you could store each number in 16 bits instead of 32, you'd cut that to 140 GB. In 8 bits, 70 GB. And it's not just about storage. The GPU has to move all those numbers through its memory bus to actually compute with them. Smaller numbers means faster transfers. And modern GPU hardware has specialized circuits that can do math on smaller numbers much, much faster.

So there's an enormous practical incentive to use the smallest possible number format. The catch is that smaller formats represent fewer distinct values. At some point you lose so much precision that your model can't train correctly, or your trained model gives garbage answers.

The entire field of mixed-precision training, quantization, and the alphabet soup of fp32/fp16/bf16/fp8 is fundamentally about navigating this tradeoff. How small can we make our numbers before things break?

How computers store decimal numbers

Okay, let's actually look at how this works. The dominant standard is IEEE 754, which has been around since 1985 and is the basis for binary floating-point arithmetic on virtually every CPU and GPU you've ever touched. Hardware and libraries do sometimes use non-IEEE formats (bfloat16 is a famous one) or fast modes that relax some of the rounding rules, but IEEE is the gravity well everything else orbits.

A floating-point number is stored as three fields packed into a fixed number of bits. For normalized finite nonzero numbers, which is the common case, the value is:

Special exponent patterns (all zeros, all ones) carve out the edge cases: signed zeros, subnormals (very small numbers below the normal range), infinities, and NaNs. Subnormals are going to matter a lot when we get to fp16 underflow.

• Sign (1 bit). 0 for positive, 1 for negative. Simple.
• Exponent (variable bits). Determines the range of the number. A wider exponent field means you can represent very large and very small numbers. The bias is subtracted so you can represent negative exponents without a separate sign bit.
• Mantissa (variable bits). Also called the significand or fraction. Determines the precision. More mantissa bits means finer distinctions between nearby values.

Think of it like scientific notation. In $6.022 \times 10^{23}$, the "6.022" part is your mantissa (precision) and the "23" is your exponent (range). If I only let you write two decimal places in the mantissa, you'd get $6.02 \times 10^{23}$. You can still represent the same enormous magnitudes, but you lose the ability to distinguish $6.022$ from $6.025$.

Exponent bits buy you range. Mantissa bits buy you precision. And different floating-point formats make different tradeoffs between the two.

The formats

fp32: the safe default

The standard 32-bit float uses 1 sign bit, 8 exponent bits, and 23 mantissa bits. This gives you:

• Range: roughly $\pm 3.4 \times 10^{38}$. Enormous. You're not going to overflow in any normal computation.
• Precision: about 7 decimal digits. More than enough for most purposes.

For decades, fp32 was the default for everything in scientific computing and machine learning. It's the forgiving option for baseline ML training. It's not numerically magic, though. fp32 still has overflow, underflow, NaNs, non-associative addition, catastrophic cancellation, and softmax/exp blow-ups if you're not careful. The headroom is just wide enough that most code stops worrying about those things.

The problem is that it's also big. 4 bytes per number. And GPUs can only do so many fp32 operations per second.

fp16: half the bits, half the fun

IEEE 754 also defines a 16-bit format. It uses 1 sign bit, 5 exponent bits, and 10 mantissa bits. Half the memory of fp32. But look at what we gave up:

• Range: roughly $\pm 6.5 \times 10^{4}$. The maximum finite value is 65,504. That might sound like a lot, but during neural network training, intermediate activations and gradient values can easily exceed this. When they do, you overflow to infinity. Your loss becomes NaN. Training crashes.
• Precision: about 3 decimal digits.

That range limitation is a killer. During the forward pass of a deep network, activation values get multiplied together across many layers. Gradient values during backpropagation can spike. Values just past the cliff (like 65,505) round back down to 65,504 under round-to-nearest, but anything sufficiently larger, say 70,000 or 100,000, overflows to infinity. Once one element of your tensor is inf, the rest of the math is downhill from there.

bfloat16: Google's insight

This is where things get interesting. Around 2018, Google Brain introduced bfloat16 ("Brain Float 16") for their TPUs. The design is pragmatic: 1 sign bit, 8 exponent bits, and 7 mantissa bits.

Wait, 8 exponent bits? That's the same as fp32. Which means bfloat16 keeps fp32's exponent range and so has essentially the same dynamic range. Its largest finite value is about $3.39 \times 10^{38}$, just slightly under fp32's $3.40 \times 10^{38}$, because the smaller mantissa caps the representable magnitude. For practical purposes the overflow problem is gone.

The cost is precision. With only 7 mantissa bits instead of fp32's 23, you get roughly 2-3 decimal digits of precision. The number 3.14159 might get stored as 3.14 or 3.15625. You lose the fine distinctions.

But here's Google's key insight, confirmed by Kalamkar et al. in their 2019 study: for neural network training, range matters more than precision. The small errors introduced by reduced mantissa precision mostly wash out during stochastic gradient descent. The network is already dealing with noisy mini-batch gradients, random initialization, dropout. A little extra rounding noise in the mantissa doesn't meaningfully hurt convergence. But overflow from a truncated exponent? That kills your training run dead.

bfloat16 is also a pragmatic hack. It's not a proper IEEE 754 format. The layout is the upper 16 bits of fp32, with the same sign and exponent. Some explanations describe converting fp32 to bf16 as "chopping off the lower bits," and that's a useful intuition, but real conversions usually round to nearest (commonly round-to-nearest-even) rather than just truncating. The hardware path is still cheap, but the precise number you get out is the rounded one, not the truncated one. (We'll see this in the worked example below.)

fp8: the new frontier

With the H100 and now the B200 (Blackwell), NVIDIA introduced hardware support for 8-bit floating point. There are two variants:

• E4M3. 1 sign, 4 exponent, 3 mantissa bits. Better precision, less range.
• E5M2. 1 sign, 5 exponent, 2 mantissa bits. More range, barely any precision.

Eight bits total. One byte per number. This is getting aggressive. With 3 mantissa bits, you can distinguish between maybe 8 values in any given power-of-two interval. That's coarse.

But it works for inference, and increasingly for parts of training too. The Micikevicius et al. 2022 paper showed that you can do the forward pass in E4M3 and the backward pass in E5M2 (where you need the range more than the precision, since gradients can be very small) and still converge.

TF32: the hidden default on NVIDIA Ampere and later

There's one more format to know about, even though you rarely see it in user code. On Ampere (A100) and every NVIDIA GPU since, Tensor Cores expose a mode called TF32. Despite the name, it's not a 32-bit storage format. Tensors stay stored as fp32. What changes is that when a matmul runs on the Tensor Cores, the inputs get internally rounded to a 19-bit format (1 sign bit, 8 exponent bits, 10 mantissa bits) before the multiply-accumulate, while the accumulator stays in fp32.

The 8-bit exponent is the key part: TF32 keeps fp32's range, so overflow behavior is fp32-like. It just trades 13 mantissa bits for a much higher Tensor Core throughput. PyTorch enables TF32 by default for matmuls on Ampere and later (you can toggle it with torch.backends.cuda.matmul.allow_tf32). For a lot of fp32 training workloads, TF32 is the speedup you've been getting without realizing it.

Range and precision side by side

It helps to see the range of each format laid out on a single axis. The marker below sweeps through ten orders of magnitude. The bars light up when a format can represent the current value, and the readouts show that format's step size (ULP) at that magnitude. Watch fp16 fall off near $10^5$ while bf16 keeps going up to $10^{38}$ alongside fp32.

That picture is the entire bf16-vs-fp16 argument. Same memory footprint, dramatically different exponent reach.

See it yourself

Theory is one thing. Let me show you the actual bit patterns. Type a number below and watch how each format encodes it. Pay attention to what happens with 0.0001 (very small), 65504 (fp16's limit), and 100000 (beyond fp16's range).

Try typing 100000. Watch fp16 go to infinity while bfloat16 handles it fine. Same exponent range as fp32, so it can represent the same magnitudes. The mantissa is rougher, so the stored value won't be exactly 100000, but it won't explode.

Now compare bfloat16 against fp8.

With only 3 mantissa bits, fp8 E4M3 rounds pretty aggressively. The number 3.14 becomes 3.0 or 3.25 depending on the nearest representable value. For a single number that's rough. But across billions of parameters, the errors average out a lot.

The hardware: why GPUs are good at this

I should say a little about why GPUs are involved at all. The core operation in neural networks is matrix multiplication. Training a transformer involves multiplying enormous matrices together millions of times. CPUs can do matrix multiplies, but GPUs can do thousands of them in parallel.

The architecture in brief

A modern GPU like the A100 is organized into Streaming Multiprocessors (SMs). The A100 has 108 of them. Each SM contains a collection of CUDA cores (for general-purpose floating-point math) and, critically, Tensor Cores (specialized hardware for matrix multiply-accumulate operations).

All of these SMs share a pool of HBM (High Bandwidth Memory). On the A100, that's 80 GB of HBM2e with about 2 TB/s of bandwidth. The SMs pull data from HBM, crunch numbers, and write results back.

SIMT: where the parallelism comes from

GPU threads aren't independent the way CPU threads are. They run in warps, groups of 32 threads that execute the same instruction on different data lanes. This is called SIMT (Single Instruction, Multiple Threads). For performance intuition, think "same instruction over many lanes": if thread 4 needs to multiply two numbers, threads 0 through 31 all multiply two numbers in roughly the same cycle.

That intuition is the right shape, but it's slightly out of date as a hardware claim. From Volta (compute capability 7.0) onward, NVIDIA GPUs have independent thread scheduling, which keeps per-thread program counters and lets threads in a warp diverge and reconverge at sub-warp granularity. Code that wants warp-synchronous behavior has to ask for it explicitly, with __syncwarp() or warp-level intrinsics. The lockstep model is still the right one for thinking about throughput; it's just no longer the right one for thinking about correctness.

This is why matrix multiplies map so well onto GPUs. Every output element of a matmul is the same computation (a dot product) on different rows and columns. A GPU can hand each warp a different chunk of output cells and let lockstep execution rip through all of them in parallel.

Memory bandwidth is often the real bottleneck

For many transformer workloads, memory bandwidth is the bottleneck, not compute. The GPU can multiply numbers faster than it can fetch them from memory. This is called being "memory-bound," and the roofline model is the standard way to reason about where a kernel sits on the memory-bound vs compute-bound spectrum.

Which workloads, though, matters a lot. Inference decode (one new token at a time, hauling the whole KV cache through the chip every step), optimizer-state movement, and small-batch kernels are usually memory-bound. Large GEMMs in training and prefill are typically compute-bound on the Tensor Cores. Lower precision helps both, but for different reasons: memory-bound kernels benefit because fewer bytes move through the same bandwidth budget, and compute-bound GEMMs benefit because Tensor Cores deliver dramatically higher FLOP/s at lower precision. End-to-end speedups depend on which side of that line your workload actually sits on.

That's the cheap win. Now look at the expensive one.

Tensor Cores and the precision speedup

Tensor Cores are special-purpose hardware units that perform matrix multiply-accumulate (MMA) operations on small tiles. CUDA exposes them through warp-level MMA instructions, and the exact tile shapes, supported dtypes, and throughput depend on the architecture. The cartoon version is "a 4x4 or 8x8 multiply-accumulate per cycle," and that's still useful as a mental model, but real Tensor Core kernels in 2026 work over architecture-specific shapes (16x8x16 on Ampere, larger on Hopper and Blackwell), often with structured-sparsity modes and dtype-dependent accumulator widths. The thing that hasn't changed is that they're designed to operate on reduced-precision inputs.

The performance difference is large enough to be worth looking at concretely. Here are the advertised dense (non-sparse) Tensor Core peaks across the last three NVIDIA generations.

Going from fp32 to bf16 on an A100 is a ~16x speedup (comparing plain fp32 CUDA cores against bf16 Tensor Cores — it's a hardware-path comparison, not a pure precision comparison). Not because fp16 math is inherently simpler (it is, slightly), but because the Tensor Cores are massively parallel hardware dedicated to reduced-precision matrix math, and the memory bandwidth pressure is halved at the same time.

On the H100, fp8 Tensor Core performance hits ~1,979 TFLOPS. On the B200 (Blackwell), fp8 is up to ~4,500 TFLOPS, with fp4 reaching ~9,000 TFLOPS. The trend is clear. Every hardware generation doubles down on lower-precision compute.

So the incentive structure is to use the lowest precision that doesn't break your training. You get a memory savings and a compute speedup at the same time.

Worked example

Let me actually walk through one number so the formula stops being abstract. Take $3.14$.

In fp32, the bias is 127. We need to find an exponent $e$ such that $2^e \le 3.14 < 2^{e+1}$. That's $e = 1$ (since $2 \le 3.14 < 4$). The biased exponent stored in the bits is $1 + 127 = 128$, which in 8-bit binary is 10000000.

Now the mantissa. The value is $3.14 = 2^1 \times (1 + m)$, so $m = 3.14/2 - 1 = 0.57$. We need to write $0.57$ as a sum of negative powers of two. Doubling gives $1.14$, so the first bit is 1 and we keep $0.14$. Doubling gives $0.28$, so the next bit is 0. Doing this 23 times gives the fp32 mantissa 10010001111010111000011. Sign bit is 0. Stitch them together and that's the fp32 bit pattern for $3.14$.

In bf16, we keep the top 7 mantissa bits (rounded, not just truncated). Same sign, same exponent, mantissa becomes 1001001 after round-to-nearest-even. The reconstructed value is $2 \times (1 + 1/2 + 1/16 + 1/128) \approx 3.1406$. The bits beyond that are gone.

In fp16, the bias is 15 instead of 127, so the biased exponent is $1 + 15 = 16$, which is 10000. The 10-bit mantissa, after round-to-nearest-even, is 1001001000. Reconstructed: $2 \times (1 + 1/2 + 1/16 + 1/128) = 2 \times 1.5703125 = 3.140625$. Same answer as bf16 to four decimals, because both formats happen to land on the same representable value here. fp16 has more mantissa bits in general, but at this magnitude the extra resolution doesn't kick in until further down the fraction.

This is why bf16's range is identical to fp32's (same exponent field, same bias) and fp16's isn't. The exponent field is doing all the work.

You can poke at this directly above by typing values into the bit viewer. Try 65504 (the largest finite fp16 number), 65505 (rounds back to 65504), and 100000 (overflows to inf).

Mixed-precision training: having it both ways

Okay, so lower precision is faster and uses less memory, but fp16 overflows and even bf16 introduces rounding errors. How do we actually train a model with reduced precision without things going wrong?

The answer is mixed-precision training, formalized by Micikevicius et al. in a 2017 preprint (published at ICLR 2018). The recipe has three parts:

1. Keep a master copy of the weights in fp32. This is your source of truth. It lives in memory and gets updated with the full-precision gradients.
2. Cast to fp16/bf16 for the forward and backward pass. The actual matrix multiplies, which are the expensive part, happen in reduced precision. This is where you get the speedup.
3. Loss scaling (for fp16 specifically). Small gradient values can underflow to zero in fp16. The trick is to multiply the loss by a large scalar before backpropagation, which scales all the gradients up into fp16's representable range. After computing the gradients, you divide by the same scalar before updating the master weights. The math is identical; you're just shifting the numbers into a range that fp16 can handle.

With bfloat16, you often don't need loss scaling because the exponent range matches fp32. Gradients that are tiny (like $10^{-8}$) are still representable. This is another reason bfloat16 became so popular.

Here's what the PyTorch implementation looks like. It's almost disappointingly simple.

import torch
 
model = MyModel().cuda()
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)
 
# Pick one. bf16 doesn't need a scaler; fp16 does.
dtype = torch.bfloat16              # or torch.float16
use_scaler = dtype == torch.float16
scaler = torch.amp.GradScaler("cuda", enabled=use_scaler)
 
for batch in dataloader:
    optimizer.zero_grad(set_to_none=True)
 
    # Forward pass in reduced precision
    with torch.amp.autocast("cuda", dtype=dtype):
        output = model(batch["input_ids"])
        loss = criterion(output, batch["labels"])
 
    # Backward pass. With bf16, scaler is a no-op; with fp16 it
    # handles the loss-scaling dance.
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()
 
# Quick demo: see what fp16 does to a value past the cliff
x = torch.tensor([65504.0, 65505.0, 100000.0], dtype=torch.float16)
print(x)   # tensor([65504., 65504.,    inf], dtype=torch.float16)
y = torch.tensor([65505.0, 100000.0], dtype=torch.bfloat16)
print(y)   # tensor([65536., 99840.], dtype=torch.bfloat16)

The autocast context manager automatically casts operations to bf16 (or fp16) where it's safe and keeps certain numerically sensitive operations (like layer norm, softmax) in fp32. The GradScaler handles the loss-scaling dance, but only when you actually need it. With bf16 the exponent range matches fp32, so loss scaling isn't necessary; the enabled=False path makes the scaler a passthrough. With fp16, the scaler is doing real work.

A wrapper context manager and a scaler. That gives you a ~2x memory reduction and a massive speedup on any GPU with Tensor Cores. Every major training framework (PyTorch, JAX, DeepSpeed, Megatron) supports this out of the box.

What happens when precision goes wrong

Let me give a concrete scenario for why this stuff matters in practice.

Say you're training a model in pure fp16, no loss scaling, no master weights. During the forward pass, an attention score computation produces a value of 70,000. In fp32, that's fine. In fp16, the maximum is 65,504. Your value becomes infinity. That infinity propagates through the softmax, corrupts the loss, and the gradients become NaN. By the time you notice, every parameter in your model has been updated with garbage.

The classic failure mode: training loss suddenly jumps to NaN with no warning, and now you have to figure out which intermediate value overflowed.

Or consider the underflow case. A gradient value of $1 \times 10^{-8}$ is perfectly representable in fp32. In fp16, the smallest positive normal number is about $6 \times 10^{-5}$. Anything below that lands in the subnormal range (the smallest positive subnormal in fp16 is about $6 \times 10^{-8}$), where you keep representing the value but lose precision. And many GPU paths flush fp16 subnormals to zero outright for performance. So a gradient that small either underflows hard to zero, or limps along as a low-precision subnormal that's effectively noise. That parameter just stops learning. If enough parameters stop learning, your model stalls and you don't even get a clear error message. It just converges to something bad.

This is exactly what loss scaling fixes. If you multiply the loss by something like $2^{15}$ before backprop, that $1 \times 10^{-8}$ gradient becomes about $3 \times 10^{-4}$, comfortably inside fp16's normal range. After the backward pass, you divide by the same factor and you're back to the correct value.

Misconceptions

"fp16 is always fine for training." Without loss scaling and fp32 master weights, fp16 will blow up on many architectures. The exponent range is too small. bfloat16 is much more forgiving because it keeps fp32's exponent range, which is why it's become the default for training. fp16 is still useful, but you need the full mixed-precision recipe with loss scaling.

"bfloat16 is worse because it's less precise." It has less mantissa precision than fp16, yes (7 bits vs 10 bits). But it has dramatically more range (8 exponent bits vs 5). For training neural networks, the range matters more. You'd rather represent a gradient of $10^{-8}$ imprecisely than not represent it at all.

"Quantization is the same as lower-precision training." These are different things. Mixed-precision training uses fp16/bf16 during the training process itself. Quantization typically happens after training: you take a model trained in fp32/bf16 and compress its weights down further for cheap inference. We have a whole later post about that. The techniques, goals, and failure modes are different. Quantization is about deploying a trained model cheaply; mixed-precision is about training efficiently in the first place.

"GPUs are fast because each core is fast." Each CUDA core is honestly pretty modest by CPU standards. The speed comes from having tens of thousands of them executing the same instruction across different data lanes, plus Tensor Cores that do whole tile-sized matmuls per clock. The GPU's architectural bet is that you have enough work that's the same shape to keep the wide SIMT machinery busy.

What's next

This post covered why the bit width of your numbers matters as much as the math you do with them. Floating point as sign + exponent + mantissa, the fp32/fp16/bf16/fp8 tradeoffs, why GPUs are wide SIMT machines that often spend their time waiting on memory, and how mixed-precision training threads the needle between speed and stability.

Next up: A Short Prehistory of Statistical NLP. We've spent Arc 1 building the math and hardware substrate. Now we'll back up and look at what people were doing with text before transformers showed up — n-grams, sparsity, the long road from "count the words" to learned embeddings.

Additional reading (and watching)

• IEEE Computer Society. (2019). IEEE 754-2019, Standard for Floating-Point Arithmetic. The canonical specification for sign/exponent/mantissa formats and rounding modes.
• Kalamkar, D., et al. (2019). A Study of BFLOAT16 for Deep Learning Training. Empirical evidence that bfloat16's wider exponent makes it a drop-in replacement for fp32 in most training workloads.
• Micikevicius, P., et al. (2022). FP8 Formats for Deep Learning. Proposes the E4M3 / E5M2 split and shows it converges on language and vision benchmarks.
• NVIDIA. (2020). NVIDIA A100 Tensor Core GPU Architecture. Whitepaper detailing SMs, warps, Tensor Cores, and HBM bandwidth on Ampere.
• NVIDIA. (2024). NVIDIA Blackwell Architecture Technical Brief. Documents the B200's fp8 and fp4 Tensor Core peaks plus the dynamic precision Transformer Engine.
• Micikevicius, P., et al. (2018). Mixed Precision Training. ICLR 2018. The original recipe for fp32 master weights + fp16 compute + loss scaling.
• Dettmers, T., et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. NeurIPS 2022. Foundational work showing that even 175B-parameter models can be quantized below fp16 with care around outlier features.
• Williams, S., Waterman, A., & Patterson, D. (2009). Roofline: An Insightful Visual Performance Model for Multicore Architectures. CACM. Origin of the "memory-bound vs compute-bound" framing used throughout this post.