Hybrid Random Concentrated Optimization Without Convexity Assumption

Abstract

We propose a new random method to minimize deterministic continuous functions over subsets $\mathcal{S}$ of high-dimensional space $\mathbb{R}^K$ without assuming convexity. Our procedure alternates between a Global Search (GS) regime to identify candidates and a Concentrated Search (CS) regime to improve an eligible candidate in the constraint set $\mathcal{S}$. Beyond the alternation between those completely different regimes, the originality of our approach lies in leveraging high dimensionality. We demonstrate rigorous concentration properties under the $CS$ regime. In parallel, we also show that $GS$ reaches any point in $\mathcal{S}$ in finite time. Finally, we demonstrate the relevance of our new method by giving two concrete applications. The first deals with the reduction of the $\ell_{1}-$norm of a LASSO solution. Secondly, we compress a neural network by pruning weights while maintaining performance; our approach achieves significant weight reduction with minimal performance loss, offering an effective solution for network optimization.