How to evaluate performance of LLM inference frameworks

LLM inference frameworks are currently facing a "memory wall," a hardware-imposed limit on performance due to memory constraints. Developers should focus on understanding their system's memory wall and select an inference framework that approaches this limit, rather than getting bogged down in the nuances of different frameworks. Performance metrics like requests per second can be misleading, especially since server and offline scenarios yield higher throughput than single-stream scenarios. While optimizations such as quantization and sparsity can enhance performance, they must be applied judiciously, as aggressive pruning can lead to significant accuracy loss. It is generally advisable to use well-validated models in their published formats. The Lamini inference engine is designed to optimize performance on various GPUs, focusing on the MLPerf server scenario for maximum throughput. The memory wall affects transformer models during inference, as they require loading extensive weights for each token generated. Although higher bandwidth memory could theoretically improve performance, practical limitations exist due to hardware constraints. The MLPerf benchmark defines various scenarios for measuring LLM performance, with server and offline scenarios allowing for better performance through batching requests. Developers are encouraged to structure applications to leverage these scenarios for improved efficiency.

- LLM inference frameworks are limited by a hardware "memory wall."

- Performance metrics can vary significantly between single-stream and server scenarios.

- Caution is advised when applying optimizations like quantization and sparsity.

- The Lamini engine is optimized for high throughput on multiple GPU types.

- Structuring applications for server or offline scenarios can enhance performance.