Optimizing AI Inference at Character.ai

Character.AI is focused on optimizing AI inference to enhance daily life with large language models (LLMs). They aim to achieve highly efficient inference to serve a global audience, currently handling over 20,000 queries per second. By implementing memory-efficient architecture design techniques like Multi-Query Attention and Hybrid Attention Horizons, they have significantly reduced KV cache size without compromising quality. Stateful caching innovation allows for efficient caching of attention KV on host memory between chat turns, achieving a 95% cache rate. Additionally, they utilize int8 quantization for training and serving, improving efficiency and reducing costs. These innovations have led to a 33x reduction in serving costs since late 2022. Character.AI envisions a future where LLMs drive innovation and improve experiences globally, inviting others to join them in advancing the capabilities of AI systems.