Fine-Tuning of Universal Audio Source Separation Model Using Low-Rank Adaptation Technique

This study investigates the effectiveness of fine-tuning query-based universal audio source separation (USS) models to improve the quality of extracting a limited number of target classes. The focus is on USS models built on the U-net architecture, particularly the AudioSep model. The main attention is devoted to the application and comparison of Low Rank Adaptation (LoRa) and Prompt Tuning techniques for fine-tuning this model. The results reveal that despite a relatively small number of trainable parameters (1% of the number of parameters in the separation net of the original model for LoRa and 0.015% for Prompt Tuning), the quality of separation significantly improved (by an average of 4 to 10 dB) across three different datasets. Additionally, a pipeline for using the model was proposed, within which optimal weights are dynamically selected for the model. To ensure reproducibility, the source code of the modified models and the fine-tuned checkpoints were published.