Managing Block Allocation to Servers for Decentralized Inference of LLM Model in Petals System

In the modern world, large language models (LLMs) with often more than 100 billion parameters have gained immense popularity. Today, anyone can freely download pre-trained models of such scale. However, using these models requires high-tech and expensive equipment, which is inaccessible to many researchers. One solution to this problem is PETALS, a system for collaborative work with large NLP models that combines the resources of multiple participants into a single pool. One of the main use cases for this system is the inference of a trained model to provide predictions for users. In the current implementation of PETALS, the strategy for allocating blocks that the server will service optimizes computational throughput at each layer of the model. This approach works well for training but not for inference. This study investigated the problem of block allocation that optimizes the response time for client inference requests of a trained model. The algorithms considered in this work managed to speed up the inference of a large language model by up to 1.8 times compared to the baseline strategy in a test simulator.