Complex-valued activations#
Complex-valued activation functions must take into account the two degrees-of-freedom inherent to complex-valued data, typically represented as real / imaginary parts or magnitude / phase. Two generalised classes of activation operate on those respective representations and are defined as Type-A and Type-B functions.
Type-A — split on real / imaginary#
Type-A activations consist of two real-valued functions, \(G_\mathbb{R}(\cdot)\) and \(G_\mathbb{I}(\cdot)\), applied to the real and imaginary parts of the input tensor independently:
where \(\mathbf{z} = \mathbf{x} + j\mathbf{y}\).
Under the hood, Type-A activations call
complextorch.nn.functional.apply_complex_split(). Examples in the
package: CVSplitReLU, CVSplitTanh, CVSplitSigmoid, CELU, CCELU,
CGELU. See the activation reference
for the full list.
Type-B — split on magnitude / phase#
Type-B activations consist of two real-valued functions, \(G_{||}(\cdot)\) and \(G_\angle(\cdot)\), applied to the magnitude (modulus) and phase (argument) of the input tensor:
Type-B activations call
complextorch.nn.functional.apply_complex_polar(). Passing phase_fun=None
is an optimisation that skips the polar round-trip when the activation only
modifies magnitude. Examples: modReLU, AdaptiveModReLU, CVPolarTanh.
Fully complex#
Fully-complex activations fit neither the Type-A nor the Type-B designation — they operate on the complex tensor directly. Use them when an activation has a natural complex form (e.g., a learnable phase rotation).
ReLU variants#
A separate family generalises the rectified linear unit to the complex plane:
complextorch.nn.CReLU/complextorch.nn.CVSplitReLU(alias) — split Type-A: ReLU on real and imaginary parts independently.complextorch.nn.zReLU— first-quadrant gating: passes \(z\) unchanged only when both \(\Re z > 0\) and \(\Im z > 0\).complextorch.nn.zAbsReLU— magnitude threshold with a learnable cutoff; phase preserved. The forward pass is the exact hard gate; because the indicator has zero gradient almost everywhere, the cutoff learns through a straight-through sigmoid surrogate of widthtau.complextorch.nn.zLeakyReLU— softzReLUwith a leaky slope outside the first quadrant.
Phase-aware / manifold ReLUs#
A third family operates on the magnitude / phase representation in a phase-aware way (the operation depends on the phase, not just on each component independently). These come from two paper lineages — CDS (Singhal et al., CVPR 2022) and SurReal (Chakraborty, Xing, Yu, arxiv:1910.11334) — and pair naturally with the symmetry-aware modules described in Co-domain symmetry.
complextorch.nn.GTReLU— learnable complex scaling \((\alpha + j\beta)\,z\) followed by an upper-half-plane phase mask \(\theta \mapsto \theta \cdot \mathbf{1}[\theta \bmod 2\pi \in [0, \pi]]\) with a custom autograd whose backward gradient is the mask itself. Optional learnable phase-rescaling.complextorch.nn.EquivariantPhaseReLU— thresholds phase relative to the channel-mean direction so the operator commutes with any global phase rotation (strictly U(1)-equivariant).complextorch.nn.tReLU— the tangent-space ReLU from SurReal Eq. 21-22: \(r \mapsto \max(r, 1)\), \(\arg z \mapsto \max(\arg z, 0)\). Parameter-free; the principled lift of standard ReLU onto the rotation+scaling manifold. Not equivariant by design (analogous to ReLU’s lack of translation-equivariance on \(\mathbb{R}\)).complextorch.nn.wFMReLU— learned affine on \(\log|z|\) and \(\arg z\) on the manifold; the port of RotLieNet’smanifoldReLUv2angle, distinct fromtReLU.
When to use which#
Need |
Reach for |
|---|---|
Drop-in replacement for |
|
Preserve phase, modulate magnitude only |
|
Phase-aware operation |
Type-B with both |
Strictly U(1)-equivariant ReLU |
|
U(1)-invariant ReLU after a U(1)-invariant block |
|
Tangent-space ReLU on the manifold |
|
Manifold-aware affine (paired with |
|
Learnable scalar phase shift |
|
Learnable complex scaling |
|