SepLLM: Accelerate LLMs by Compressing One Segment into One Separator

SepLLM is a new framework designed to enhance the efficiency of Large Language Models (LLMs) by compressing segments of text into special separator tokens, thereby reducing computational demands and improving inference speed. The research identifies that certain meaningless special tokens significantly impact attention scores, leading to the hypothesis that the information between these tokens can be effectively condensed. SepLLM operates as a plug-and-play solution, allowing for inference acceleration by eliminating redundant tokens and implementing efficient training kernels. Experimental results demonstrate that SepLLM can achieve over a 50% reduction in key-value (KV) cache usage on the GSM8K-CoT benchmark while maintaining performance levels. Additionally, it shows promise in streaming applications, effectively managing language modeling tasks across millions of tokens. The framework has been validated through various settings, including training-free, training-from-scratch, and post-training scenarios, showcasing its versatility and effectiveness in optimizing LLM performance.

- SepLLM compresses segments of text into special tokens to enhance LLM efficiency.

- The framework reduces computational demands and improves inference speed.

- It achieves over 50% reduction in KV cache usage with minimal performance loss.

- SepLLM is effective in both training and streaming applications.

- The research highlights the potential of optimizing attention mechanisms in LLMs.