complextorch.nn.modules.position#

Positional information for complex-valued sequence models (e.g. complextorch.nn.MultiheadAttention, complextorch.nn.TransformerEncoderLayer). The native complextorch.nn.Transformer applies no positional encoding on its own, so one of these modules must be used to make attention position-aware.

Three flavours are provided:

RotaryEmbedding – relative position via Rotary Position Embedding (RoPE). Each complex feature channel \(k\) of a token at position \(n\) is multiplied by a unit phasor \(e^{j\omega_k n}\). Under the Hermitian attention inner product \(q\,k^H\) the per-channel contribution becomes \(q_k \overline{k_k}\, e^{j\omega_k (m-n)}\), which depends only on the relative offset \(m-n\). Apply it to the query and key tensors (see RotaryEmbedding.rotate_q_k()).
SinusoidalPositionalEncoding – fixed absolute encoding; adds a complex sinusoidal phasor bank \(e^{j\omega_k n}\) to the embeddings.
CoPE – a lightweight learnable complex positional encoding with per-channel learnable frequency and phase offset (2 * dim parameters).

RoPE is described in:

J. Su, et al. RoFormer: Enhanced Transformer with Rotary Position Embedding.

https://arxiv.org/abs/2104.09864

The complex / phase-modulation reading of RoPE (each rotation is literally a multiplication by \(e^{j\theta}\) on a bank of complex oscillators) makes it a natural fit for a complex-valued library.

Classes#

`CoPE`	Lightweight Learnable Complex Positional Encoding
`RotaryEmbedding`	Complex Rotary Position Embedding (RoPE)
`SinusoidalPositionalEncoding`	Complex Sinusoidal Positional Encoding

Module Contents#

class complextorch.nn.modules.position.CoPE(dim: int, base: float = 10000.0)[source]#

Bases: torch.nn.Module

Lightweight Learnable Complex Positional Encoding#

Multiplies the complex feature at position \(n\) by a learnable rotor

\[\tilde{x}_{n,k} = x_{n,k}\, e^{j (\omega_k n + \phi_k)},\]

where both the per-channel frequency \(\omega_k\) and phase offset \(\phi_k\) are learnable (2 * dim parameters total). Unlike RotaryEmbedding – which uses a fixed frequency schedule and is designed to be applied to the query/key tensors for relative encoding – CoPE is a standalone learnable absolute encoding applied to the sequence embeddings.

The frequencies are initialised to the RoPE schedule and the phase offsets to zero, so an untrained CoPE behaves like RotaryEmbedding.

param dim:: number of complex feature channels (size of the last dim).
param base:: geometric base used to initialise the frequencies.

Initialize internal Module state, shared by both nn.Module and ScriptModule.

extra_repr() → str[source]#

Return the extra representation of the module.

To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable.

forward(input: torch.Tensor, offset: int = 0) → torch.Tensor[source]#

Rotate input by the learnable positional rotor.

Parameters:

input (torch.Tensor) – (..., L, dim) complex tensor.
offset (int) – position of the first token.

Returns:

torch.Tensor – rotated complex tensor of the same shape.

base = 10000.0#

dim#

omega#

phi#

class complextorch.nn.modules.position.RotaryEmbedding(dim: int, base: float = 10000.0, learnable: bool = False)[source]#

Bases: torch.nn.Module

Complex Rotary Position Embedding (RoPE)#

Multiplies the complex feature at sequence position \(n\) by the position-dependent unit phasor \(e^{j\omega_k n}\):

\[\tilde{x}_{n,k} = x_{n,k}\, e^{j \omega_k n}, \qquad \omega_k = \text{base}^{-k/d}.\]

Apply it to the query and key tensors before the attention dot product. Because the attention score uses the Hermitian inner product \(q\,k^H\), the rotation injected at positions \(m\) (query) and \(n\) (key) leaves a residual phase \(e^{j\omega_k(m-n)}\) that encodes the relative position.

The last tensor dimension is treated as the (complex) feature dimension and must equal dim; the second-to-last dimension is the sequence dimension. This matches the per-head attention layout (B, n_heads, L, d_k) as well as a plain (B, L, d) embedding layout.

param dim:: number of complex feature channels (size of the last dim).
param base:: geometric base for the frequency schedule (RoPE default 10000).
param learnable:: if True the per-channel frequencies are learnable.

Initialize internal Module state, shared by both nn.Module and ScriptModule.

extra_repr() → str[source]#

Return the extra representation of the module.

To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable.

forward(input: torch.Tensor, offset: int = 0) → torch.Tensor[source]#

Rotate input by the RoPE phasors.

Parameters:

input (torch.Tensor) – (..., L, dim) tensor; the last dim is the complex feature dim and the second-to-last is the sequence dim.
offset (int) – position of the first token (useful for KV caching).

Returns:

torch.Tensor – rotated complex tensor of the same shape.

rotate_q_k(q: torch.Tensor, k: torch.Tensor, offset: int = 0) → tuple[torch.Tensor, torch.Tensor][source]#: Apply RoPE to both the query and key tensors.

base = 10000.0#

dim#

learnable = False#

class complextorch.nn.modules.position.SinusoidalPositionalEncoding(dim: int, base: float = 10000.0)[source]#

Bases: torch.nn.Module

Complex Sinusoidal Positional Encoding#

Fixed absolute positional encoding that adds a bank of complex sinusoids to the input embeddings:

\[\tilde{x}_{n,k} = x_{n,k} + e^{j \omega_k n}, \qquad \omega_k = \text{base}^{-k/d}.\]

The last tensor dimension is the (complex) feature dimension and must equal dim; the second-to-last dimension is the sequence dimension.

param dim:: number of complex feature channels (size of the last dim).
param base:: geometric base for the frequency schedule.

Initialize internal Module state, shared by both nn.Module and ScriptModule.

extra_repr() → str[source]#

Return the extra representation of the module.

To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable.

forward(input: torch.Tensor, offset: int = 0) → torch.Tensor[source]#

Add the complex sinusoidal encoding to input.

Parameters:

input (torch.Tensor) – (..., L, dim) complex tensor.
offset (int) – position of the first token.

Returns:

torch.Tensor – input plus the positional phasor bank.

base = 10000.0#

dim#