Orchestration System for Machine Learning Tasks

The rapid progagation of machine learning(ML) tasks, particularly within scientific domains, underscores the growing reliance on computational methods for data analysis and prediction. However, as the number of ML models continues to expand, scientists often encounter challenges in effectively utilizing these models due to their intricate nature and the specialized expertise required. In many cases, scientists seek to apply pre-configured neural network models to their specific datasets without extensive knowledge of ML intricacies, other times they want to build training pipelines but again using configured logic blocks, not writing complicated data transformations from scratch. This demand is enhanced by the escalating need for computational resources to accommodate the increasing complexity and volume of ML tasks. Dedicated servers are often required to support the execution of resource-intensive ML computations, posing logistical and accessibility challenges for scientists. To address these pressing issues, this paper proposes the development of an orchestration system tailored to the needs of scientists working in interdisciplinary fields. By abstracting away infrastructure complexities and providing intuitive interfaces for ML task execution, the proposed system aims to empower scientists to effortlessly utilize pre-configured ML models and logic blocks on their datasets. Key benefits of the proposed orchestration system include streamlined access to computational resources, efficient utilization of computing resources through task queuing and load distribution mechanisms, seamless scalability to accommodate growing computational demands, well defined business process for scientists to work with ML algorithms.