RelativePositionBias (Attention)

Relative Position Bias for T5 (opens in a new tab).

Attention(
    dim=768,
    num_heads=12,
    plugins=[
        RelativePositionBias(
            num_buckets=256,
            num_heads=12,
        )
    ],
)

DropProjection RelativePositionBiasViT