The Influence of LLM Inference Optimization on LLM Quality

The exponential growth of large language models (LLMs) in recent years has driven signif- icant advancements in natural language processing (NLP). These models, with their unprece- dented capacity to understand and generate human-like text, have demonstrated remarkable performance across a wide range of applications. However, their increasing size and complex- ity pose substantial challenges in terms of computational requirements, memory consumption, and inference speed. This diploma explores three inference acceleration methods: quantiza- tion, flash-attention, and deepspeed. Quantization reduces the precision of data representations within neural networks, thereby decreasing computational demands and minimizing memory us- age. Flash-attention optimizes memory access patterns and computational efficiency, while deep- speed inference employs kernel-fusion and memory optimization techniques to enhance model performance. The study investigates these optimization strategies’ impact on model accuracy and performance using popular open-source models Llama2-7b, Mistral-7b and Mixtral-8x7b. The results provide insights into the trade-offs between efficiency and accuracy, offering a com- prehensive evaluation of these methods on MMLU and RUMMLU benchmarks. This work aims to contribute to making large language models more accessible and efficient, extending their applicability to a broader range of tasks and environments.