Positional Encodings: Sinusoidal, Learned, RoPE, and ALiBi

Each token's position is encoded as a set of rotations at different frequencies. Fast-rotating pairs (blue) encode fine position, slow pairs (amber) encode coarse position.

Something confused me for a long time about attention. Take the sentence "the cat sat on the mat" and scramble it into "mat the on sat cat the." Feed both through a self-attention layer. You get the same set of attention weights. Same values, just permuted.

That sounds wrong. But it's true. Self-attention computes pairwise dot products between queries and keys, and those dot products don't depend on where a token sits in the sequence. Swap two tokens and their Q, K, V vectors swap too, but the set of all pairwise scores is identical. Attention treats its input as a set, not a sequence.

The technical name for this is permutation equivariance. Permute the input and the output gets permuted the same way. That's a problem, because language is not a set. "Dog bites man" and "man bites dog" have very different meanings, and an order-blind model can't distinguish them.

So we need to inject position somehow. The question is how, and it turns out the choice has major implications for whether a model can handle long contexts, extend beyond its training length, and play nicely with the KV cache during inference.

Prerequisites

You should be comfortable with single-head attention and multi-head attention: queries, keys, values, and the dot-product score matrix. The residual stream picture from the self-attention post also matters here, because where position information gets injected (into the embedding once vs. into Q and K at every layer) is partly a story about how cleanly position survives the residual path. For the RoPE math, a little familiarity with 2D rotation matrices and complex number multiplication helps, but I'll walk through the key intuitions.

Sinusoidal encodings (Vaswani 2017)

The original transformer paper proposed a simple solution: add a position-dependent vector to each token embedding before it enters the first layer. No trainable parameters. Just a fixed pattern of sines and cosines.

The formula looks like this. For a token at position $pos$ and embedding dimension $i$:

Each dimension gets a sine or cosine wave, and the frequency decreases as you move to higher dimensions. The low-order dimensions oscillate fast, changing rapidly from position to position. The high-order dimensions oscillate slowly, barely changing across dozens of positions.

The intuition I find most helpful is the clock analogy. A clock has a second hand (high frequency, fine resolution), a minute hand (medium frequency), and an hour hand (low frequency). Together the three hands uniquely identify any time within a 12-hour cycle. Sinusoidal encodings work the same way. The combination of different-frequency waves produces a unique fingerprint for each position. That clock is exactly what the hero loop above is showing: three dial hands at different speeds, sweeping out the "fingerprint" for each token's position.

Hover over the heatmap. You can see the wave patterns directly. The leftmost columns oscillate rapidly between red and blue (high frequency), while the rightmost columns barely change (low frequency). Each row has a unique pattern.

There's a nice mathematical property here. The dot product $PE(pos+k) \cdot PE(pos)$ depends only on the offset $k$, not on the absolute position $pos$. Relative position information is implicitly baked in, even though we're adding absolute position vectors. Vaswani et al. pointed this out in the original paper but didn't fully exploit it. RoPE will.

Ten lines of code. You precompute the table once and add it to the embeddings. No gradients, no training, no parameters.

Learned absolute encodings (GPT-2)

GPT-2 took an even simpler approach: just make the position embeddings a trainable parameter. Create an embedding table of shape (max_len, d_model), initialize it randomly, and let gradient descent figure out the rest.

This works well in practice. The model learns whatever position patterns are useful for its training distribution. And it's dead simple to implement.

But there's a hard ceiling. If you trained with max_len=1024, position 1025 simply does not exist in the embedding table. You can't generalize beyond your training length. That might seem fine when your context window is large enough, but in 2024 and beyond models started needing 32K, 128K, even 1M token contexts. Learned absolute encodings can't get there without retraining.

RoPE: rotary position embedding

This is the one that won. Su et al. introduced RoPE in 2021, and it has since become the default position encoding in essentially every major open-weight model: Llama, Mistral, Qwen, Gemma, and more.

The core idea is different from both sinusoidal and learned encodings. Instead of adding position information to the token embedding, RoPE rotates the query and key vectors by a position-dependent angle. The embedding itself is left alone. Position lives in Q and K.

The mechanism is straightforward. Pair up the dimensions of your query vector: $(q_0, q_1)$, $(q_2, q_3)$, $(q_4, q_5)$, and so on. Each pair lives in its own 2D plane. Now rotate each pair by an angle that depends on both the position and the dimension pair:

where $m$ is the position and $\theta_i = 10000^{-2i/d}$ controls the rotation speed per dimension pair. Low-index pairs rotate fast. High-index pairs rotate slowly. The same frequencies as sinusoidal encodings, but applied as rotation instead of addition.

The magic is in the dot product. When you compute $q(pos_m) \cdot k(pos_n)$, the two rotation matrices partially cancel. What survives depends only on the difference $m - n$. So RoPE naturally encodes relative position, which is exactly what attention cares about. Position 47 attending to position 44 should look the same as position 100 attending to position 97.

If you want to feel that yourself, switch to the RoPE tab on the heatmap viz earlier and drag the position slider. The red query vector rotates against a fixed blue key, and the readout under the circle shows the dot product moving smoothly as the gap widens.

There are a few reasons RoPE beat the alternatives. It works naturally with the KV cache, because position information is baked into Q and K at computation time. You don't need to store separate position embeddings or recalculate anything when extending the cache. And it supports graceful extension to longer sequences through frequency scaling, which we'll get to next.

One thing about that code. Instead of manually applying a 2×2 rotation matrix to each pair of dimensions, you reinterpret each pair as a complex number and multiply by $e^{im\theta}$. Complex multiplication is 2D rotation. Same result, cleaner code, and the GPU handles complex arithmetic efficiently.

NTK-aware scaling and YaRN

RoPE encodes relative position elegantly. But what happens when you try to use a model at positions it never saw during training? If a model was trained with a 4K context and you feed it 32K tokens, the rotation angles at positions 4001 through 32000 are completely out of distribution. The high-frequency dimension pairs wrap around multiple times at angles the model never learned to interpret. Performance degrades sharply.

The naive fix is linear interpolation. Divide all positions by a scaling factor so position 32K maps to where position 4K used to be. This compresses the rotation schedule and keeps all angles in the trained range. But it crushes the high-frequency components, smashing nearby positions together and making the model struggle to distinguish adjacent tokens.

In 2023, a Reddit user named bloc97 proposed NTK-aware scaling. The insight was that different frequency bands need different treatment. High-frequency dimensions, which encode fine-grained local position, should not be compressed much. Low-frequency dimensions, which encode coarse global position, can tolerate more compression because they were already changing slowly.

NTK-aware scaling achieves this by increasing the base frequency $\theta$ rather than scaling positions directly. Instead of $\theta = 10000$, you use:

where $\alpha$ is the extension ratio. This naturally spreads the compression across frequency bands, preserving local resolution while extending the usable range.

YaRN (Peng et al., 2023) took this further by combining NTK-aware frequency scaling with a temperature correction on the attention logits. The temperature adjustment accounts for the fact that longer sequences produce larger attention score magnitudes, which pushes softmax toward sharper distributions than the model expects. YaRN showed that with a few hundred steps of fine-tuning on longer data (around 400 to extend 16x, plus another couple hundred to push to 32x) you could extend a 4K model to 128K with minimal quality loss.

This is how the context window arms race actually works in practice. You don't retrain the model from scratch at 128K context. You train at 4K or 8K, then apply RoPE scaling and a short fine-tuning phase to unlock far longer sequences. The compute cost is a fraction of the original pretraining run.

ALiBi: attention with linear biases

ALiBi takes a completely different approach. There are no position embeddings and no modifications to Q or K. Instead, ALiBi adds a bias directly to the attention scores that penalizes long-range attention.

For each attention head, before softmax, subtract $m \cdot |i - j|$ from the attention score between query position $i$ and key position $j$. The slope $m$ varies by head in a geometric sequence. Some heads get steep slopes (strong locality bias, attending mostly nearby) and others get gentle slopes (weaker bias, attending more broadly).

Switch to the ALiBi tab in the visualization and toggle between the three heads. Head 1 with $m=1.0$ applies a steep penalty, basically forcing local attention. Head 3 with $m=0.125$ is much more permissive, allowing the model to attend across long ranges with only a gentle discount.

The "train short, test long" property is ALiBi's main selling point. Because the bias is a linear function of distance, it naturally extrapolates to positions longer than training. If position 512 had a bias of $-512m$, position 2048 gets a bias of $-2048m$. The penalty increases smoothly and the model handles it gracefully.

ALiBi is elegant and it genuinely extrapolates well. But it imposes a structural bias. Distant tokens always receive a penalty. The model can still attend to them if the raw attention score is high enough, but it has to overcome the bias. For tasks that require precise long-range retrieval (like finding a needle in a haystack), that built-in penalty can hurt. This is probably why RoPE, which has no such bias, ended up winning the long-context race.

Runtime behavior and practical tradeoffs

In production, the choice of positional encoding mostly affects two things: context extension and KV cache compatibility.

Sinusoidal and learned absolute encodings are the cheapest at inference time. They add a vector to the embedding once and forget about it. But they can't extend beyond their training length, and any change to the position scheme requires retraining from scratch.

RoPE adds a small cost per layer, because it rotates Q and K at every attention computation. In practice this is negligible compared to the matmuls. The real benefit is that position information stays fresh at every layer instead of getting diluted through residual connections. And because the rotation is applied directly to Q and K, the KV cache "just works." Each cached key already has its position baked in, so there's nothing to recompute when the context grows.

ALiBi has a different cost profile. The bias matrix is precomputed and stored, then added to attention scores before softmax. For very long sequences the storage for the bias matrix matters, but in practice you only need one triangle of shape (seq_len, seq_len) shared across the batch.

The real differentiator in 2024-2026 has been context extension. RoPE with NTK-aware or YaRN scaling lets you train at 4K-8K and deploy at 128K+, which is dramatically cheaper than training at the target length. ALiBi extrapolates naturally but hits accuracy issues on needle-in-a-haystack tasks. Learned encodings simply can't play this game at all.

Worked example

Let me walk through a small concrete example. Take 8 positions and $d_{model}=4$, so we have two dimension pairs.

For sinusoidal encodings, position 3:

• Dimension 0: $\sin(3 / 10000^{0/4}) = \sin(3) = 0.141$
• Dimension 1: $\cos(3 / 10000^{0/4}) = \cos(3) = -0.990$
• Dimension 2: $\sin(3 / 10000^{2/4}) = \sin(3/100) = \sin(0.03) = 0.030$
• Dimension 3: $\cos(3 / 10000^{2/4}) = \cos(0.03) = 1.000$

See the pattern? Dimensions 0 and 1 oscillate at full speed. Dimensions 2 and 3 barely move because the denominator $10000^{0.5} = 100$ makes the angle tiny.

For RoPE at the same position, say we have a query vector $q = [1.0, 0.5, 0.8, 0.3]$. We pair up: $(1.0, 0.5)$ and $(0.8, 0.3)$.

For the first pair, $\theta_0 = 1.0$, so the rotation angle is $3 \times 1.0 = 3.0$ radians:

For the second pair, $\theta_1 = 10000^{-0.5} = 0.01$, so the angle is $3 \times 0.01 = 0.03$ radians. Barely any rotation:

Same multi-resolution structure as sinusoidal encodings. The first pair gets aggressively rotated (fine position info), the second pair barely budges (coarse position info). But the rotation lives in Q and K, not in the embeddings.

Misconceptions

"Sinusoidal encodings are the best because they're analytically derived." They were a good first attempt and the math is elegant. Nobody uses them in production anymore. Every major model from 2023 onward uses RoPE or a variant. Sinusoidal encodings get added to embeddings, so position information gets diluted as it propagates through layers. RoPE re-injects position at every attention computation.

"RoPE can't handle longer contexts than training." It can, with scaling. NTK-aware scaling and YaRN let you extend a 4K-trained model to 128K+ with minimal fine-tuning. The key is adjusting the frequency schedule so that rotation angles at extended positions stay in a range the model has seen before.

"To get longer context, just bump the RoPE base frequency." Llama 3 training at base 500000 instead of 10000 might suggest the base is a knob you turn freely. The relationship is more delicate. Raising the base flattens the per-pair frequency schedule, which spreads the rotations out over longer wavelengths, but it also changes what every dim pair encodes. Bumping the base without retraining tends to hurt short-context quality. NTK-aware scaling, YaRN, and Llama 3.1's 128k extension all do something subtler: scale frequencies non-uniformly, then fine-tune on long data so the model relearns the distribution.

"RoPE adds position to the embedding, like sinusoidal does." They share the same sin/cos frequencies but do different things. Sinusoidal adds a position vector to the token embedding before the first layer, so position info gets diluted by every subsequent transformation. RoPE multiplies Q and K by a rotation at every attention computation. Position is re-injected fresh at each layer, which is part of why RoPE survived deep stacks better.

What's next

In Post 4.5 we'll step away from attention entirely and look at the other half of the transformer block: the feed-forward layer, and the key-value memory interpretation that makes it the model's actual long-term storage.

Additional reading (and watching)

• Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS 2017. Introduces the sinusoidal position encoding and notes the offset-only dot-product property that RoPE eventually exploited.
• Su, J., et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. The paper that introduced RoPE, now the default position encoding across open-weight LLMs.
• Press, O., Smith, N., & Lewis, M. (2022). Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. ICLR 2022. The ALiBi paper, and the clearest statement of the "train-short, test-long" framing.
• bloc97 (2023). NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning. Reddit. The original post on frequency-wise scaling that became a standard technique.
• Peng, B., et al. (2023). YaRN: Efficient Context Window Extension of Large Language Models. Formalizes NTK-aware scaling and adds the attention temperature correction; the production recipe behind many long-context models.
• Meta AI (2024). The Llama 3 Herd of Models. The 3.1 release notes and tech report describe extending context from 8K to 128K via RoPE base-frequency scaling plus long-context fine-tuning, and the resulting rope_scaling config that inference engines now consume.