RotaryEmbedding (Attention)

Rotary Position Embedding for RoFormer (opens in a new tab).

Attention(
    dim=768,
    num_heads=12,
    head_dim= 64,
    plugins=[
        RotaryEmbedding(
            head_dim=64,
        )
    ],
)

Parameters

• head_dim: The dimension size for each attention head.
• seq_len: The length of given sequence.