Switching to a Non-Transactional Model in the Yandex Market Unified Offer Storage Service

The amount of data that needs to be processed is growing every day. Software developers are creating new tools that simplify the processing of the required data, but optimization options are limited. If your goal is to increase the amount of data that your service is capable of processing, there are two common solutions: vertical and horizontal scaling, but they require increasing the cost of maintaining your service. That is why sometimes it is better to look at the problem from the other side and change the idea underlying the solution used. Rethinking the components of a functioning service always requires a lot of effort so as not to disrupt their performance. This is the essence of this project: to implement as safely as possible an approach to processing streaming data that differs from the solution that our service has been using on an ongoing basis for more than 5 years. By adding a microservice sharding incoming data to the pipeline used, it turned out to reduce the number of transaction conflicts by 5 times, which in turn reduced the load on data processors by 10%.