Reinforcement-learning-based Algorithms for Optimization Problems and Applications to Inverse Problems

Abstract

We design a new iterative algorithm, called REINFORCE-OPT, for solving a general type of optimization problems. This algorithm parameterizes the solution search rule and iteratively updates the parameter using a reinforcement learning (RL) algorithm resembling REINFORCE. To gain a deeper understanding of the RL-based methods, we show that REINFORCE-OPT essentially solves a stochastic version of the given optimization problem, and that under standard assumptions, the searching rule parameter almost surely converges to a locally optimal value. Experiments show that REINFORCE-OPT outperforms other optimization methods such as gradient descent, the genetic algorithm, and particle swarm optimization, via its ability to escape from locally optimal solutions and its robustness to the choice of initial values. With rigorous derivations, we formally introduce the use of reinforcement learning to deal with inverse problems. By choosing specific probability models for the action-selection rule, we can also connect our approach to the conventional methods of Tikhonov regularization and iterative regularization. We take non-linear integral equations and parameter-identification problems in partial differential equations as examples to show how reinforcement learning can be applied in solving non-linear inverse problems. The numerical experiments highlight the strong performance of REINFORCE-OPT, as well as its ability to quantify uncertainty in error estimates and identify multiple solutions for ill-posed inverse problems that lack solution stability and uniqueness.