Symmetric Power Transformers

The article discusses the development of Symmetric Power Transformers, a new variant of linear transformers designed to improve performance while maintaining efficient computational costs. Traditional linear transformers, while theoretically advantageous for long contexts, often suffer from degraded performance due to small state sizes relative to their weights. The authors propose a solution by embedding keys and queries in a higher-dimensional space, which allows for larger state sizes and better performance. Symmetric Power Transformers introduce a hyperparameter \(p\) that controls the state size, enabling configurations that outperform standard transformers when \(p\) is set to 4 or higher. The article also highlights the compatibility of these transformers with rotary embeddings, enhancing their utility. The authors conducted experiments using an attention formulation of linear transformers to validate the architecture's learning capabilities, with plans to release an efficient implementation in future work. The findings suggest that by adjusting the embedding function and state size, it is possible to achieve competitive performance with a manageable computational footprint.

- Symmetric Power Transformers improve performance of linear transformers by embedding keys and queries in higher-dimensional spaces.

- A hyperparameter \(p\) controls the state size, allowing configurations that outperform standard transformers.

- The architecture is compatible with rotary embeddings, enhancing its effectiveness.

- Experiments validate the learning capabilities of the proposed architecture.

- An efficient implementation of the chunked algorithm for these transformers is planned for future release.