Tensor Algorithms for Effective Training of Quantum and Optical Neural Networks

This work explores the development and optimization of quantum and optical computing in application to transformer models. By employing surrogate modeling on transformer architectures, we test various quantum architectures alongside an optical case. Our research demonstrates a possibility of significant reduction in the number of parameters required in large linear layers in transformers, while leaving most of the expressivity of the model. To address the problem of training directly on a quantum or optical device we propose a training technique incorporating black-box optimization. Our mixed classical and zero-order training techniques show potential but require further refinement to improve their practical applicability. We also introduce a new zero-order optimization method, designed to handle high-dimensional problems and tested it against strong baselines. In summary, this thesis contributes to the fields of zero-order optimization and optical computing by presenting an approach aiming at approximation of classical layers, rather then their transfer. While our findings lay the groundwork for more efficient AI systems, addressing the practical challenges is essential for realizing the full potential of these innovations.