Purpose of applying transform to V and out

[smileyenot983]

Hello!
I am wondering about the purpose of applying the P^{-1} to values vector and also about applying P to outputs. Can anyone give intuition? Has anyone tested without applying this transforms

[brentyi]

The GTA paper discusses it: https://arxiv.org/pdf/2310.10375

The contrast is to standard RoPE where we only transform the Q and K vectors, right? My intuition for why this is limiting is:

• Each transformed q and k vector is immediately collapsed to a scalar via an inner product.
• Hard to get much information to flow through a single scalar value. This may be ok for 1D position encoding where adjusting the softmax/attention weights is enough, but cameras are higher-dimensional.
• In GTA/PRoPE, transforming the values vector lets you condition on the relative camera info without the scalar bottleneck.

It would be interesting actually to compare "only Q/K" multiplication vs "only V/output" multiplication. It's possible the latter is more important!