The quantum-inspired adaptive superposition optimization for neural network training

Training deep neural networks is often hindered by the fragility of gradient-based methods, which suffer from vanishing or exploding gradients, sensitivity to initialization, and entrapment in poor local minima. In response to these shortcomings, we introduce a new gradient-free algorithm called Quantum-Inspired Adaptive Superposition Optimization (QIASO), which views weight learning as a probabilistic superposition of candidate solutions, a fundamentally new optimization approach. In contrast to being dedicated to a single weight ensemble, QIASO maintains a distribution over several candidates, which are amplified and suppressed according to dynamically changing weights assigned to them. The variational formulation of the amplitude evolution leads to a KL-regularized formulation of their evolution, which generalizes statistical physics, information geometry, and online optimization viewpoints. To prevent invalid convergence, QIASO incorporates a stochastic perturbation operator based on quantum tunnelling into the optimizer, enabling the optimization process to overcome local minima on the loss surface and converge to the optimal solution. We provide theoretical bounds, monotone convergence of loss reduction, and almost-sure convergence to local optima with mild assumptions. Complexity analysis via empirical techniques suggests that QIASO scales more efficiently than Grover-based quantum-inspired algorithms and incurs no overhead in gradient computation compared to ADAM. The overall findings indicated that QIASO is a viable option for neural training, particularly when combined with other paradigms that utilise either large-scale or gradient-free approaches.