Software Suite for Streaming Data Backfill Using YT Distributed Computing System

This work focuses on developing a solution for data recalculation within the context of stream processing, aimed at facilitating the scalability of the process and minimizing the workload on data engineers during its preparation. In today’s digital world, stream processing is actively used for analytics and real-time decision-making. Apache Flink, a well-known data streaming engine and framework, is widely employed by data engineers to create and execute data transformations. However, during the operation of these transformations, the need often arises for their recalculation, owing to errors or new requirements. Such recalculation process, known as backfill, requires significantly more computing capabilities compared to regular transformations due to the increased volume of data. In this work, a new approach to streaming data backfill is introduced that uses batch processing, natively executing the user’s Flink transformation, while also partly utilizing the MapReduce model on the distributed computing system YT. The approach is scalable and requires minimal additional coding, reducing the workload for data engineers and minimizing the potential for errors. The work contains: 45 pages, 3 chapters, 10 figures, 5 tables, 24 sources and 3 applications. Keywords: distributed computing systems, data processing, data streaming, MapReduce, real-time data processing