Accelerating Transformer Inference through Contextual Sparsity Exploitation

Contextual sparsity is the phenomenon where only a subset of the model’s parameters are activated in response to specific inputs. It offers a promising avenue to reduce computational demands while maintaining high levels of performance. This research explores contextual sparsity in Chinese Large Language Models (LLMs), which has the potential to improve inference efficiency without compromising performance degradation. As activation sparsity is recently discovered approach for inference acceleration, there was performed lack of investigation into the coverage of this method for different LLMs. The aim is to uncover how sparsity can be strategically exploited to enhance inference efficiency in existing pre-trained LLMs. Our findings show advantages of this method and its applicability in Chinese LLMs and further investigate prospective benefits for inference speedup as well as implementing these models for the sake of deploying them on edge devices.