The Link Between Positional Encodings and Group Theory

I’ve recently been trying to understand the inner workings of LLMs. Some days ago, I wrote a detailed walkthrough of how Llama 3.1 generates the next token, that follows the computation from tokenization all the way through to sampling. Writing it helped me make sense of the mechanical process, but many parts of the pipeline still felt mysterious to me, one of them being the positional encoding through which the transformer blocks get information about where tokens are relative to each other.

In Llama 3.1, rotary positional embeddings (RoPE) takes pairs of components from the query and key vectors inside each transformer block and rotates each pair by an angle that depends on the token’s position. Somehow, this lets the attention block know the relative position between tokens. But why pairs? How do those rotations make relative position show up in the attention score? And what is the mathematical structure underneath it all?

After seeing all kinds of positional encodings in papers, I often find the choices feel ad hoc, almost pulled out of the ass. Why this particular formula? Why sines and cosines? Why rotations? Even when the results look good, I still want to know: what is the first principle behind putting position information into attention? Can we start with the properties we want and arrive at different positional encodings in a principled way?

While trying to understand this and reading some of the surrounding material, I came across Alok Puranik’s Using group theory to explore the space of positional encodings for attention on the Jane Street blog. This article is my attempt to work through some of those ideas and explain them as I understand them, starting with the assumptions and then coming back to RoPE.

Where Does Position Enter the Computation?

Let’s zoom into a single transformer block in Llama. The residual stream carries one vector for each token down through the block. Two branches read from that stream and add their results back into it: the attention block, then an MLP block.

The attention branch starts by normalizing the residual stream, then applying three learned projections to produce query, key, and value vectors for each token position. RoPE rotates the query and key vectors within each head, using each token’s position to set the rotation angle. The rotated queries and keys are then compared through dot products to produce the attention scores.

After scaling, a causal mask prevents each token from attending to future tokens, and softmax is applied to turn the scores into weights that multiply into the value vectors to produce a weighted sum for each query. The head outputs are concatenated and projected back to the residual stream’s width before being added to it. This follows Meta’s reference implementation.

So the place I want to focus on is those two RoPE operations. How do they actually inject information about position into the computation?

Let’s focus on just one attention head. Let $q_s$ be the query at position $s$ and $k_t$ the key at position $t$. The diagram places RoPE after these vectors have been calculated but before their dot product. Writing $R(s)$ for the rotation at position $s$, the score before masking and softmax is

$$e_{st} = \frac{\bigl(R(s)q_s\bigr)^\top\bigl(R(t)k_t\bigr)}{\sqrt{d_h}},$$

where $d_h$ is the dimension of a head. Each vector gets its own position-dependent rotation.

What Should Attention Know About Position?

Before choosing a rotation, or any other encoding, it helps to ask what we want that comparison to tell us. Does attention need to know that the query is at position 12 and the key is at position 9? Or would it be enough to know that the key is three tokens behind the query?

For a relative encoding, we want our model to know the second kind of information. If we held the query and key vectors fixed and moved their positions from $(12, 9)$ to $(112, 109)$, the positional effect on their score should stay the same. The signed offset $t-s$ is still $-3$. Holding the vectors fixed matters here: we’re testing the encoding, rather than claiming that moving words around in a sentence leaves the model’s representations unchanged.

The Assumptions Behind the Jane Street Analysis

The Jane Street article by Alok Puranik asks the more general question that I had in my mind: can we make some assumptions about the properties of the kind of transformations queries and keys need to undergo to inject position information? And can we use these assumptions to derive different positional encodings from first principles? Puranik’s idea was to assume four properties:

1. Linearity in the vectors. At fixed positions, the transformations are matrices: $\tilde q_s = F(s)q_s$ and $\tilde k_t = G(t)k_t$. Their positional contribution to the score is therefore $F(s)^\top G(t)$.
2. Translation invariance. Shifting both positions equally leaves that contribution unchanged. Equivalently, $F(s)^\top G(t) = H(t-s)$ for some matrix-valued function $H$.
3. Identity at zero offset. Require $H(0)=I$: comparing vectors at the same position preserves their original dot product.
4. Continuity. Extend positions to real numbers and require $H(\delta)$ to vary continuously with the offset $\delta$. This is an additional assumption beyond integer token indices.

His next step is to show that these assumptions force the relative-offset matrices $H(\delta)$ to form a mathematical object called a one-parameter group. In particular, they obey

$$H(a+b) = H(a)H(b).$$

The intuition is that moving through an offset of two tokens and then another three should have the same effect as moving through five in one go. The matrices compose in the same way that offsets add. An offset of zero does nothing, and a negative offset undoes the corresponding positive one: $H(-a)=H(a)^{-1}$. The “one parameter” is the single real number specifying the offset, and continuity means the transformation changes continuously as that number varies. Rotating a plane at a fixed angular speed is one example: two successive rotations add their angles, and reversing the angle undoes the rotation. Puranik’s result lets him use the established mathematics of these groups to work out which positional encodings satisfy his assumptions.

Digging into Group Theory

Before coming back to RoPE, I want to unpack that group theory bit. What is a group here, and what does it mean for a matrix to generate one?

Take all the rotations of a two-dimensional plane. Each rotation is an element of a group called $SO(2)$, and the operation that combines two elements is matrix multiplication. Rotating through an angle $a$ and then an angle $b$ gives another rotation, through $a+b$. There’s an identity element, $R(0)=I$, which does nothing, and every rotation has an inverse: $R(-a)$ undoes $R(a)$. Matrix multiplication is associative, so it doesn’t matter how we bracket a sequence of these operations. These are the properties that make this collection a group.

Now imagine turning the angle up continuously from zero. At zero we’re at the identity matrix. The generator tells us how the rotation matrix starts changing as we increase the angle:

$$J = \left.\frac{dR(\phi)}{d\phi}\right|_{\phi=0} = \begin{bmatrix}0&-1\\1&0\end{bmatrix}.$$

For a tiny angle $\varepsilon$, we have $R(\varepsilon)\approx I+\varepsilon J$. Applied to a vector $v$, the term $Jv$ gives its initial direction of motion as we begin to rotate it. The generator describes a rate of change; it isn’t itself the small rotation. In this particular example, $J$ also happens to be the matrix for a quarter-turn, but its role here is as the derivative at zero.

The next question is how to get from that initial change to a rotation through a finite angle. That’s what the exponential map does: $\phi J\mapsto e^{\phi J}=R(\phi)$. This is a matrix exponential, defined by the same power series as the ordinary exponential, with matrix powers and the identity matrix in place of the number one:

$$e^{\phi J}=\sum_{k=0}^{\infty}\frac{(\phi J)^k}{k!} =I+\phi J+\frac{\phi^2J^2}{2!}+\frac{\phi^3J^3}{3!}+\cdots.$$

Since $J^2=-I$, the powers cycle through $I,J,-I,-J$. Collecting the even and odd powers gives $\cos\phi\,I+\sin\phi\,J$, which is exactly the usual rotation matrix. We can also think of the exponential as the limit of many tiny steps: $e^{\phi J}=\lim_{N\to\infty}(I+\phi J/N)^N$.

The matrices $\phi J$ form a line through zero called the Lie algebra of $SO(2)$: the space of possible initial changes at the identity. The exponential map takes us from this space into the group. For a continuous one-parameter matrix group, the corresponding result is that $H(\delta)=e^{\delta X}$ for a fixed generator $X$.

The earlier assumptions don’t require $X$ to generate rotations. A general real matrix $X$ can generate transformations involving scaling, rotation, or shear. If we additionally require $X$ to be skew-symmetric, meaning $X^\top=-X$, then $e^{\delta X}$ is an orthogonal rotation for every $\delta$. For a RoPE pair rotating at frequency $\omega$, the angle is $\delta\omega$, so its generator with respect to position is $X=\omega J$. This is a skew-symmetric generator, one particular choice within the more general matrix-group framework.

Digging into RoPE

With that structure in mind, let’s come back to how the rotations in Llama make the positional effect depend on the relative offset.

RoPE achieves this by splitting each query and key vector into pairs of components and rotating each pair in its own two-dimensional plane. At position $p$, pair $i$ is rotated through an angle $p\,\omega_i$, where $\omega_i$ is a fixed frequency for that pair. Queries and keys use the same frequencies, but each token’s position sets its rotation angles. These rotations make the positional effect on their dot product depend on the relative offset, as described in the RoFormer paper.

But why does rotating pairs of components work? The query and key are vectors in a much higher-dimensional space. Why split them into two-dimensional pieces instead of applying one big rotation to the whole vector? What does an $N$-dimensional rotation matrix even look like?

An important property of a real skew-symmetric generator is that an orthogonal change of basis puts it into independent $2\times2$ blocks of the form $\omega_i J$, with zeros for any remaining directions. Taking the matrix exponential turns those blocks into two-dimensional rotations and leaves the remaining directions fixed. So a continuous one-parameter family of rotations in $N$ dimensions uses the same independent planes at every position, with a fixed frequency in each plane. This decomposition applies to rotation groups, rather than to every matrix group allowed by the earlier assumptions. RoPE (Rotary Positional Embedding) chooses its generator in this block-diagonal form, applying a rotation to each pair of coordinates in the query and key vectors. Each pair rotates at a different frequency, and the angle of rotation depends on the token’s position $p$:

$$R(\phi)= \begin{bmatrix} \cos\phi&-\sin\phi\\ \sin\phi&\cos\phi \end{bmatrix}, \qquad \boxed{\phi_{p,i}=p\,\omega_i=\frac{p}{\theta^{2i/d}}}$$

where $d$ is the even vector dimension, $i$ indexes the coordinate pairs, and $\theta$ is a fixed frequency-base hyperparameter. Writing $c_{p,i}=\cos\phi_{p,i}$ and $s_{p,i}=\sin\phi_{p,i}$, the full rotation matrix is

$$\begin{aligned} R_d(p)&= \underbrace{ \begin{bmatrix} c_{p,0} & -s_{p,0} & \cdots & 0 & 0 \\ s_{p,0} & c_{p,0} & \cdots & 0 & 0 \\ \vdots & \vdots & \ddots & \vdots & \vdots \\ 0 & 0 & \cdots & c_{p,d/2-1} & -s_{p,d/2-1} \\ 0 & 0 & \cdots & s_{p,d/2-1} & c_{p,d/2-1} \end{bmatrix} }_{d\times d} \\[6pt] &=\operatorname{diag}\!\left( \underbrace{R(\phi_{p,0})}_{2\times2},\ldots, \underbrace{R(\phi_{p,d/2-1})}_{2\times2} \right). \end{aligned}$$

There are $d/2$ rotation blocks along the diagonal, one for each coordinate pair; all entries outside those blocks are zero. As we move along the vector dimensions, the rotation frequencies decrease geometrically; as we move through token positions, each pair’s rotation angle increases linearly at its own frequency. We can think of this as a multi-frequency positional fingerprint, somewhat analogous to Gray code: each position corresponds to a particular configuration of rotations, with fast-changing and slow-changing components jointly representing position. Unlike Gray code, however, these are continuous, periodic signals rather than discrete bits.

The particularly useful property of these rotations comes from their group structure: composing two rotations adds their angles, so $R_d(s)^\top R_d(t)=R_d(t-s)$. Consequently, when RoPE rotates a query at position $s$ and a key at position $t$, their attention score becomes:

$$\boxed{(R_d(s)q_s)^\top(R_d(t)k_t)=q_s^\top R_d(t-s)k_t.}$$

Although RoPE rotates the query and key according to their absolute positions, the positional contribution to their dot product depends only on the relative displacement. This gives the transformer access to relative positional information at multiple frequencies without explicitly computing or storing pairwise positional embeddings. Mathematically, these rotations form a one-parameter group generated by a block-diagonal skew-symmetric matrix $X$, with $R_d(p)=e^{pX}$. RoPE is thus one example of the one-parameter groups in Puranik’s analysis: here, the generator produces rotations in independent two-dimensional planes.

Adding decay through the generator

Puranik’s analysis also considers generators that combine rotation with exponential decay. For one coordinate pair, choose

$$X=\lambda I+\omega J,\qquad \lambda>0.$$

Because $I$ and $J$ commute, the relative-offset matrix becomes

$$H(\delta)=e^{\delta X}=e^{\lambda\delta}R(\omega\delta).$$

Our convention is $\delta=t-s$, key position minus query position. In causal attention, the key is at or before the query, so the lag $\ell=s-t$ is nonnegative. The pair’s contribution to the unscaled dot product is then

$$q_s^\top H(-\ell)k_t =e^{-\lambda\ell}q_s^\top R(-\omega\ell)k_t.$$

The rotation encodes the offset, while the exponential gives the contribution a decaying envelope for fixed content vectors. This doesn’t guarantee that the softmax attention weight decreases, since the score also depends on content and competes with other scores.

Unlike RoPE, this construction uses different query and key transformations. Taking $F(p)=e^{-pX^\top}$ and $G(p)=e^{pX}$ gives $F(s)^\top G(t)=H(t-s)$ and preserves the dot product at zero offset. Damping both vectors identically would instead introduce dependence on the sum of their positions. The decay is directional, too. Reversing the offset produces growth, so causal masking matters here.

The diagram compares three choices of generator. Each row separates the position embeddings applied to individual queries and keys, $F(p)$ and $G(p)$, from the relative encoding that appears in their dot product, $H(t-s)$. The paths show what happens to a fixed vector as we increase the causal lag $\ell$, rather than the absolute position $p$.

The damped-rotation construction appears as xPos in Sun et al.’s paper. In our notation, its per-pair scale is $\zeta_i=e^{-\lambda_i}$, so the query transform is $F_i(p)=\zeta_i^pR(p\omega_i)$ and the key transform is $G_i(p)=\zeta_i^{-p}R(p\omega_i)$. Their product gives $H_i(-\ell)=\zeta_i^\ell R(-\ell\omega_i)$. Setting the rotation frequency to zero gives the pure-decay ablation. These formulas express the paper’s construction using our column-vector and offset conventions. The generator description is our mathematical interpretation of those transforms. Sun et al., §3.2 and §4.4.1.

Putting it together

So, putting it together, and trying to answer the questions I had about positional embeddings:

• Where does position enter the computation? RoPE rotates queries and keys before their dot product, making position affect the attention scores that determine how values are combined.
• What should attention know about position? For a relative encoding, the positional effect on the score should depend only on the signed offset between tokens, with the content vectors held fixed.
• Can we derive positional encodings from first principles? Under Puranik’s assumptions of linearity, translation invariance, identity at zero offset, and continuity, the relative transformations form a one-parameter group, $H(\delta)=e^{\delta X}$.
• What is a group here, and what generates it? The relative transformations compose as offsets add, and a fixed matrix $X$ describes their initial rate of change and generates every finite transformation through the matrix exponential.
• Why rotations, sines, and cosines? Choosing a skew-symmetric generator gives rotations, whose two-dimensional blocks have sine and cosine entries because $J^2=-I$.
• Why rotate pairs instead of the whole vector at once? In a suitable basis, any continuous one-parameter rotation in higher dimensions decomposes into independent two-dimensional rotation planes, with any remaining directions fixed.
• How do absolute rotations reveal relative position? The identity $R(s)^\top R(t)=R(t-s)$ makes the positional contribution to the query–key dot product depend only on their offset.
• Does this explain every choice in RoPE’s formula? The group structure explains the independent rotation planes and angles linear in position, but leaves choices such as the frequency schedule open.
• Can the same framework produce other encodings? Adding a positive scalar term to the rotation generator gives an exponential decay envelope over causal lag when queries and keys use the paired transforms derived above.