Positional Encodings: Sinusoidal, Learned, RoPE, and ALiBi

Each token's position is encoded as a set of rotations at different frequencies. Fast-rotating pairs (blue) encode fine position, slow pairs (amber) encode coarse position.

Take the sentence "the cat sat on the mat" and scramble it into "mat the on sat cat the." Feed both through an unmasked self-attention layer with no positional encoding. You get the same set of attention weights. Same values, just permuted.

Self-attention computes pairwise dot products between queries and keys, and those dot products don't depend on where a token sits in the sequence. Swap two tokens and their Q, K, V vectors swap too, but the set of all pairwise scores is identical. Attention treats its input as a set, not a sequence.

The technical name for this is permutation equivariance. Permute the input and the output gets permuted the same way. That's a problem, because language is not a set. "Dog bites man" and "man bites dog" have very different meanings, and an order-blind model can't distinguish them.

Position information must enter through another operation. We will compare adding position vectors to embeddings, rotating queries and keys, and biasing attention scores by distance.

Prerequisites

You should be comfortable with single-head attention and multi-head attention: queries, keys, values, and the dot-product score matrix. The residual stream picture from the self-attention post also matters here, because where position information gets injected (into the embedding once vs. into Q and K at every layer) is partly a story about how cleanly position survives the residual path. The RoPE example uses 2D rotation matrices and complex multiplication.

Sinusoidal encodings (Vaswani 2017)

The original transformer paper proposed a simple solution: add a position-dependent vector to each token embedding before it enters the first layer. No trainable parameters. Just a fixed pattern of sines and cosines.

The formula looks like this. For a token at position $pos$ and embedding dimension $i$:

Each dimension gets a sine or cosine wave, and the frequency decreases as you move to higher dimensions. The low-order dimensions oscillate fast, changing rapidly from position to position. The high-order dimensions oscillate slowly, barely changing across dozens of positions.

A clock has a second hand (high frequency, fine resolution), a minute hand (medium frequency), and an hour hand (low frequency). Together the three hands uniquely identify any time within a 12-hour cycle. Sinusoidal encodings work the same way. The combination of different-frequency waves produces a unique fingerprint for each position. That clock is exactly what the hero loop above is showing: three dial hands at different speeds, sweeping out the "fingerprint" for each token's position.

Tap or hover over the heatmap. You can see the wave patterns directly. The leftmost columns oscillate rapidly between red and blue (high frequency), while the rightmost columns barely change (low frequency). Each row has a unique pattern.

For complete sine/cosine pairs, the dot product $PE(pos+k) \cdot PE(pos)$ depends only on the offset $k$, not on the absolute position $pos$. Relative position information is implicitly baked in, even though we're adding absolute position vectors. This follows from the cosine difference identity. The paper motivates the schedule by a related fact: a fixed positional offset can be represented as a linear transformation of the encoding.

The table can be precomputed and added to token embeddings; its entries are fixed during training.

Learned absolute encodings (GPT-2)

GPT-2 took an even simpler approach: just make the position embeddings a trainable parameter. Create an embedding table of shape (max_len, d_model), initialize it randomly, and let gradient descent figure out the rest.

A table with max_len=1024 has indices 0 through 1023; requesting index 1024 is out of bounds. Extending it requires a scheme for the added positions and usually model adaptation.

RoPE: rotary position embedding

Su et al. introduced RoPE in 2021. It is now used in several widely deployed open model families, including Llama and Mistral.

Instead of adding position information to the token embedding, RoPE rotates the query and key vectors by a position-dependent angle. The embedding itself is left alone.

Pair up the dimensions of your query vector: $(q_0, q_1)$, $(q_2, q_3)$, $(q_4, q_5)$, and so on. Each pair lives in its own 2D plane. Now rotate each pair by an angle that depends on both the position and the dimension pair:

where $m$ is the position and $\theta_i = 10000^{-2i/d}$ controls the rotation speed per dimension pair. Low-index pairs rotate fast. High-index pairs rotate slowly. The same frequencies as sinusoidal encodings, but applied as rotation instead of addition.

When you compute $q(pos_m) \cdot k(pos_n)$, the two rotation matrices partially cancel. For fixed unrotated query and key vectors, the positional part depends only on the difference $m - n$. The vectors themselves still depend on token content and context. So RoPE naturally encodes relative position. Shifting both positions by the same amount leaves that positional relationship unchanged when the unrotated vectors are held fixed.

Switch to the RoPE tab on the heatmap viz earlier and drag the position slider. The red query vector rotates against a fixed blue key, and the readout under the circle shows the dot product moving smoothly as the gap widens.

Instead of manually applying a 2×2 rotation matrix to each pair of dimensions, you reinterpret each pair as a complex number and multiply by $e^{im\theta}$. Complex multiplication is 2D rotation. The two forms compute the same rotation. Production kernels often use real-valued sine and cosine operations; the complex form makes the algebra concise.

NTK-aware scaling and YaRN

What happens when you try to use a model at positions it never saw during training? A model trained on 4K-token sequences has not learned from the same long-range offsets and attention competition as a 32K-token sequence. Fast pairs already complete many rotations during training; slow pairs may cover only part of a turn. Extending the sequence changes the combination of phases across pairs. RoPE defines those rotations mathematically, but does not guarantee useful extrapolation.

The naive fix is linear interpolation. Divide all positions by a scaling factor so position 32K maps to where position 4K used to be. This maps extended positions into the original position range. It also reduces the angular separation between adjacent positions, including in fast pairs; adaptation and evaluation determine the effect on quality.

In 2023, a Reddit user named bloc97 proposed NTK-aware scaling. The insight was that different frequency bands need different treatment. High-frequency dimensions, which encode fine-grained local position, should not be compressed much. Low-frequency dimensions, which encode coarse global position, can tolerate more compression because they were already changing slowly.

NTK-aware scaling achieves this by increasing the base frequency $\theta$ rather than scaling positions directly. Instead of $\theta = 10000$, you use:

where $\alpha$ is the extension ratio. This naturally spreads the compression across frequency bands, preserving local resolution while extending the usable range.

YaRN combines frequency-dependent interpolation with an empirically chosen attention scale. Its recommended factor multiplies both Q and K magnitudes by $1+0.1\ln(s)$, sharpening softmax for $s>1$. The paper tested this combination with long-context fine-tuning to extend Llama models to 128K.

ALiBi: attention with linear biases

ALiBi takes a completely different approach. There are no position embeddings and no modifications to Q or K. Instead, ALiBi adds a bias directly to the attention scores that penalizes long-range attention.

For each attention head, before softmax, subtract $m \cdot |i - j|$ from the attention score between query position $i$ and key position $j$. The slope $m$ varies by head in a geometric sequence. Some heads get steep slopes (strong locality bias, attending mostly nearby) and others get gentle slopes (weaker bias, attending more broadly).

Switch to the ALiBi tab in the visualization and toggle between the three heads. Head 1 with $m=1.0$ applies a steep penalty, favoring local attention. Head 3 with $m=0.125$ is much more permissive, allowing the model to attend across long ranges with only a gentle discount.

ALiBi’s linear distance penalty remains defined beyond the training window: if distance 512 has a bias of $-512m$, distance 2048 has a bias of $-2048m$. The paper tested whether models trained with that penalty could use longer inputs.

ALiBi demonstrated length extrapolation in those experiments. Its distance penalty also makes a distant key require a higher raw attention score than a nearby key to receive the same weight. The effect on retrieval quality must be measured for the model and task.

Runtime behavior and practical tradeoffs

In production, the choice of positional encoding mostly affects two things: context extension and KV cache compatibility.

Sinusoidal and learned absolute encodings are the cheapest at inference time. They add a vector to the embedding once and forget about it. A learned table has a finite index range. Sinusoidal encodings are defined beyond training length, but neither scheme guarantees that the trained model will generalize there. Extending a scheme may require adaptation, not necessarily training from scratch.

RoPE rotates Q and K at every attention layer. The rotations usually cost little compared with the projections and attention matrix products. Each cached key already includes its position, so it can be reused while the rotation schedule stays fixed. Changing a dynamic scaling factor may require updating previously rotated keys.

The ALiBi implementation above materializes a per-head bias matrix. A fused attention kernel can compute each distance bias as needed, avoiding that quadratic allocation.

Worked example

Take 8 positions and $d_{model}=4$, so we have two dimension pairs.

For sinusoidal encodings, position 3:

• Dimension 0: $\sin(3 / 10000^{0/4}) = \sin(3) = 0.141$
• Dimension 1: $\cos(3 / 10000^{0/4}) = \cos(3) = -0.990$
• Dimension 2: $\sin(3 / 10000^{2/4}) = \sin(3/100) = \sin(0.03) = 0.030$
• Dimension 3: $\cos(3 / 10000^{2/4}) = \cos(0.03) = 1.000$

Dimensions 0 and 1 oscillate at full speed. Dimensions 2 and 3 barely move because the denominator $10000^{0.5} = 100$ makes the angle tiny.

For RoPE at the same position, say we have a query vector $q = [1.0, 0.5, 0.8, 0.3]$. We pair up: $(1.0, 0.5)$ and $(0.8, 0.3)$.

For the first pair, $\theta_0 = 1.0$, so the rotation angle is $3 \times 1.0 = 3.0$ radians:

For the second pair, $\theta_1 = 10000^{-0.5} = 0.01$, so the angle is $3 \times 0.01 = 0.03$ radians. Barely any rotation:

Same multi-resolution structure as sinusoidal encodings. The first pair gets aggressively rotated (fine position info), the second pair barely budges (coarse position info). But the rotation lives in Q and K, not in the embeddings.

Misconceptions

"A formula defined at every position guarantees length generalization." Sinusoidal encodings and RoPE both exist beyond the training window. The learned network may still behave poorly there. Mathematical coverage and empirical generalization are separate questions.

"RoPE cannot extend beyond its training window." Its formula is defined there, and methods such as position interpolation and YaRN have demonstrated useful extensions with adaptation. The achievable length and training cost depend on the checkpoint, extension factor, and evaluation.

"To get longer context, just bump the RoPE base frequency." Llama 3 training at base 500000 instead of 10000 might suggest the base is a knob you turn freely. Raising the base lowers the frequencies of pairs beyond the first, which spreads the rotations out over longer wavelengths, but it also changes what every dim pair encodes. Bumping the base without retraining tends to hurt short-context quality. NTK-aware scaling, YaRN, and Llama 3.1's 128k extension all do something subtler: adjust the frequency schedule. Whether further training is needed, and how much, depends on the method and extension factor.

"RoPE adds position to the embedding, like sinusoidal does." Sinusoidal encoding adds a position vector before the first layer. RoPE rotates Q and K within attention layers.

What's next

In Post 4.5 we'll step away from attention entirely and look at the other half of the transformer block: the feed-forward layer, and the key-value memory interpretation that makes it a useful view of its learned parameters.

Additional reading (and watching)

• Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS 2017. Introduces the sinusoidal position encoding and motivates its frequencies through the linear relationship between encodings at fixed offsets.
• Su, J., et al. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. The paper that introduced RoPE, now the default position encoding across open-weight LLMs.
• Press, O., Smith, N., & Lewis, M. (2022). Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. ICLR 2022. The ALiBi paper, and the clearest statement of the "train-short, test-long" framing.
• bloc97 (2023). NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning. Reddit. The original post on frequency-wise scaling that became a standard technique.
• Peng, B., et al. (2023). YaRN: Efficient Context Window Extension of Large Language Models. Describes frequency-dependent interpolation and an attention-scale correction, with Llama context-extension experiments.
• Meta AI (2024). The Llama 3 Herd of Models. The 3.1 release notes and tech report describe extending context from 8K to 128K via RoPE base-frequency scaling plus long-context fine-tuning, and the resulting rope_scaling config that inference engines now consume.