mlx.nn.SinusoidalPositionalEncoding

mlx.nn.SinusoidalPositionalEncoding#

class SinusoidalPositionalEncoding(dims: int, min_freq: float = 0.0001, max_freq: float = 1, scale: float | None = None, cos_first: bool = False, full_turns: bool = False)#

Implements sinusoidal positional encoding.

For more details see the paper Attention Is All You Need.

参数:

  • dims (int) -- The dimensionality of the resulting positional embeddings.

  • min_freq (float, optional) -- The minimum frequency expected. Default: 0.0001.

  • max_freq (float, optional) -- The maximum frequency expected. Default: 1.

  • scale (float, optional) -- A multiplicative scale for the embeddings. Default: sqrt(2/dims).

  • cos_first (bool, optional) -- If True embed using [cos(x); sin(x)] instead of the reverse. Default: False.

  • full_turns (bool, optional) -- If True multiply the frequencies with \(2\pi\). Default: False.

Methods

目录