arXiv subjects

Estelle Massart

Publications and source records attributed to Estelle Massart.

15 recordsLinked to original sources

arXiv2026

Expressivity of congruence-based architectures for DNNs on positive-definite matrices

This work studies neural architectures for classifying symmetric positive-definite matrices, focusing on congruence-like layers, in which the input matrix is multiplied on the left and right by a (possibly rectangular) weight matrix $W$ and its transpose. Such layers lie at the core of the celebrated SPDNet and have also been employed independently for dimensionality reduction on positive-definite data. We show that the (semi)-orthogonality constraint commonly imposed on $W$ limits the expressivity of these layers: for certain activation functions, the resulting architecture collapses to a one-hidden-layer equivalent. This lack of expressivity follows from a loss of spectral diversity in congruence-like layers for semi-orthogonal $W$ and is a direct consequence of Poincaré's separation theorem. We then examine the choice of the final classifier, comparing several Riemannian classifiers and discussing their compatibility with the feature maps produced by congruence-like layers.

cs.LG↗

arXiv2026

The role of class encoding in neural collapse

Neural collapse is a structural property of the last-hidden-layer activations in neural network classification models, when trained beyond a zero classification error. In this work, we explore the role of label encoding in neural collapse by relying on the unrestricted feature model with mean squared error training loss. We demonstrate that, for one-hot encoded labels and balanced data, the uncentered mean features associated with each class transition from a simplex equiangular tight frame to an orthogonal frame when increasing the bias regularization coefficient associated with the final classifier. These structures are reminiscent of the orthogonal frame structure of one-hot encoded labels. For any arbitrary encoding, we also show that the final classifier's bias aims at centering the labels, compensating for the discrepancy between the global mean of the labels and the origin. We further discuss the role of the encoding in other neural collapse properties.

cs.LG↗

arXiv2026

Efficient Techniques for Data Reconstruction, with Finite-Width Recovery Guarantees

Data reconstruction attacks on trained neural networks aim to recover the data on which the network has been trained and pose a significant threat to privacy, especially if the training dataset contains sensitive information. Here, we propose a unified optimization formulation of the data reconstruction problem based on initial and trained parameter values, incorporating state-of-the-art proposals. We show that in the random feature model, this formulation provably leads to training data reconstruction with high probability, provided the network width is sufficiently large; this unprecedented finite-width result uses PAC-style bounds. Furthermore, when the data lies in a low-dimensional subspace, we show that the network width requirement for successful reconstruction can be relaxed, with bounds depending on the subspace dimension rather than the ambient dimension. For general neural network models and unknown data orientations, we propose an efficient reconstruction algorithm that approximates the low-dimensional data subspace through the change in the first-layer weights during training and uses only the last-layer weights for reconstruction, thus reducing the search space dimension and the required network width for high-quality reconstructions. Our numerical experiments on synthetic datasets and CIFAR-10 confirm that our subspace-aware reconstruction approach outperforms standard full-space techniques.

cs.LG↗

arXiv2026

Riemannian optimization with finite-difference gradient approximations

Derivative-free Riemannian optimization (DFRO) aims to minimize an objective function using only function evaluations, under the constraint that the decision variables lie on a Riemannian manifold. The rapid increase in problem dimensions over the years calls for computationally cheap DFRO algorithms, that is, algorithms requiring as few function evaluations and retractions as possible. We propose a novel DFRO method based on finite-difference gradient approximations that relies on an adaptive selection of the finite-difference accuracy and stepsize that is novel even in the Euclidean setting. When endowed with an intrinsic finite-difference scheme, that measures variations of the objective in tangent directions using retractions, our proposed method requires $O(dε^{-2})$ function evaluations and retractions to find an $ε$-critical point, where $d$ is the manifold dimension. We then propose a variant of our method when the search space is a Riemannian submanifold of an $n$-dimensional Euclidean space. This variant relies on an extrinsic finite-difference scheme, approximating the Riemannian gradient directly in the embedding space, assuming that the objective function can be evaluated outside of the manifold. This approach leads to worst-case complexity bounds of $O(dε^{-2})$ function evaluations and $O(ε^{-2})$ retractions. We also present numerical results showing that the proposed methods achieve superior performance over existing derivative-free methods on various problems in both Euclidean and Riemannian settings.

math.OC↗

arXiv2026

A Langevin sampler for quantum tomography

Quantum tomography involves obtaining a full classical description of a prepared quantum state from experimental results. We propose a Langevin sampler for quantum tomography, that relies on a new formulation of Bayesian quantum tomography exploiting the Burer-Monteiro factorization of Hermitian positive-semidefinite matrices. If the rank of the target density matrix is known, this formulation allows us to define a posterior distribution that is only supported on matrices whose rank is upper-bounded by the rank of the target density matrix. Conversely, if the target rank is unknown, any upper bound on the rank can be used by our algorithm, and the rank of the resulting posterior mean estimator is further reduced by the use of a low-rank promoting prior density. This prior density is a complex extension of the one proposed in (Annales de l'Institut Henri Poincare Probability and Statistics, 56(2):1465-1483, 2020). We derive a PAC-Bayesian bound on our proposed estimator that matches the best bounds available in the literature, and we show numerically that it leads to strong scalability improvements compared to existing techniques when the rank of the density matrix is known to be small.

math.ST↗

arXiv2025

On the Neural Feature Ansatz for Deep Neural Networks

Understanding feature learning is an important open question in establishing a mathematical foundation for deep neural networks. The Neural Feature Ansatz (NFA) states that after training, the Gram matrix of the first-layer weights of a deep neural network is proportional to some power $α>0$ of the average gradient outer product (AGOP) of this network with respect to its inputs. Assuming gradient flow dynamics with balanced weight initialization, the NFA was proven to hold throughout training for two-layer linear networks with exponent $α= 1/2$ (Radhakrishnan et al., 2024). We extend this result to networks with $L \geq 2$ layers, showing that the NFA holds with exponent $α= 1/L$, thus demonstrating a depth dependency of the NFA. Furthermore, we prove that for unbalanced initialization, the NFA holds asymptotically through training if weight decay is applied. We also provide counterexamples showing that the NFA does not hold for some network architectures with nonlinear activations, even when these networks fit arbitrarily well the training data. We thoroughly validate our theoretical results through numerical experiments across a variety of optimization algorithms, weight decay rates and initialization schemes.

cs.LG↗

arXiv2025

Enhancing finite-difference based derivative-free optimization methods with machine learning

Derivative-Free Optimization (DFO) involves methods that rely solely on evaluations of the objective function. One of the earliest strategies for designing DFO methods is to adapt first-order methods by replacing gradients with finite-difference approximations. The execution of such methods generates a rich dataset about the objective function, including iterate points, function values, approximate gradients, and successful step sizes. In this work, we propose a simple auxiliary procedure to leverage this dataset and enhance the performance of finite-difference-based DFO methods. Specifically, our procedure trains a surrogate model using the available data and applies the gradient method with Armijo line search to the surrogate until it fails to ensure sufficient decrease in the true objective function, in which case we revert to the original algorithm and improve our surrogate based on the new available information. As a proof of concept, we integrate this procedure with the derivative-free method proposed in (Optim. Lett. 18: 195--213, 2024). Numerical results demonstrate significant performance improvements, particularly when the approximate gradients are also used to train the surrogates.

math.OC↗

arXiv2024

Learning the subspace of variation for global optimization of functions with low effective dimension

We propose an algorithmic framework, that employs active subspace techniques, for scalable global optimization of functions with low effective dimension (also referred to as low-rank functions). This proposal replaces the original high-dimensional problem by one or several lower-dimensional reduced subproblem(s), capturing the main directions of variation of the objective which are estimated here as the principal components of a collection of sampled gradients. We quantify the sampling complexity of estimating the subspace of variation of the objective in terms of its effective dimension and hence, bound the probability that the reduced problem will provide a solution to the original problem. To account for the practical case when the effective dimension is not known a priori, our framework adaptively solves a succession of reduced problems, increasing the number of sampled gradients until the estimated subspace of variation remains unchanged. We prove global convergence under mild assumptions on the objective, the sampling distribution and the subproblem solver, and illustrate numerically the benefits of our proposed algorithms over those using random embeddings.

math.OC↗

arXiv2021

Coordinate descent on the orthogonal group for recurrent neural network training

We propose to use stochastic Riemannian coordinate descent on the orthogonal group for recurrent neural network training. The algorithm rotates successively two columns of the recurrent matrix, an operation that can be efficiently implemented as a multiplication by a Givens matrix. In the case when the coordinate is selected uniformly at random at each iteration, we prove the convergence of the proposed algorithm under standard assumptions on the loss function, stepsize and minibatch noise. In addition, we numerically demonstrate that the Riemannian gradient in recurrent neural network training has an approximately sparse structure. Leveraging this observation, we propose a faster variant of the proposed algorithm that relies on the Gauss-Southwell rule. Experiments on a benchmark recurrent neural network training problem are presented to demonstrate the effectiveness of the proposed algorithm.

cs.LG↗

arXiv2021

Global optimization using random embeddings

We propose a random-subspace algorithmic framework for global optimization of Lipschitz-continuous objectives, and analyse its convergence using novel tools from conic integral geometry. X-REGO randomly projects, in a sequential or simultaneous manner, the high-dimensional original problem into low-dimensional subproblems that can then be solved with any global, or even local, optimization solver. We estimate the probability that the randomly-embedded subproblem shares (approximately) the same global optimum as the original problem. This success probability is then used to show convergence of X-REGO to an approximate global solution of the original problem, under weak assumptions on the problem (having a strictly feasible global solution) and on the solver (guaranteed to find an approximate global solution of the reduced problem with sufficiently high probability). In the particular case of unconstrained objectives with low effective dimension, that only vary over a low-dimensional subspace, we propose an X-REGO variant that explores random subspaces of increasing dimension until finding the effective dimension of the problem, leading to X-REGO globally converging after a finite number of embeddings, proportional to the effective dimension. We show numerically that this variant efficiently finds both the effective dimension and an approximate global minimizer of the original problem.

math.OC↗

arXiv2020

Constrained global optimization of functions with low effective dimensionality using multiple random embeddings

We consider the bound-constrained global optimization of functions with low effective dimensionality, that are constant along an (unknown) linear subspace and only vary over the effective (complement) subspace. We aim to implicitly explore the intrinsic low dimensionality of the constrained landscape using feasible random embeddings, in order to understand and improve the scalability of algorithms for the global optimization of these special-structure problems. A reduced subproblem formulation is investigated that solves the original problem over a random low-dimensional subspace subject to affine constraints, so as to preserve feasibility with respect to the given domain. Under reasonable assumptions, we show that the probability that the reduced problem is successful in solving the original, full-dimensional problem is positive. Furthermore, in the case when the objective's effective subspace is aligned with the coordinate axes, we provide an asymptotic bound on this success probability that captures its algebraic dependence on the effective and, surprisingly, ambient dimensions. We then propose X-REGO, a generic algorithmic framework that uses multiple random embeddings, solving the above reduced problem repeatedly, approximately and possibly, adaptively. Using the success probability of the reduced subproblems, we prove that X-REGO converges globally, with probability one, and linearly in the number of embeddings, to an $ε$-neighbourhood of a constrained global minimizer. Our numerical experiments on special structure functions illustrate our theoretical findings and the improved scalability of X-REGO variants when coupled with state-of-the-art global - and even local - optimization solvers for the subproblems.

math.OC↗

arXiv2020

Low-rank multi-parametric covariance identification

We propose a differential geometric construction for families of low-rank covariance matrices, via interpolation on low-rank matrix manifolds. In contrast with standard parametric covariance classes, these families offer significant flexibility for problem-specific tailoring via the choice of "anchor" matrices for the interpolation. Moreover, their low-rank facilitates computational tractability in high dimensions and with limited data. We employ these covariance families for both interpolation and identification, where the latter problem comprises selecting the most representative member of the covariance family given a data set. In this setting, standard procedures such as maximum likelihood estimation are nontrivial because the covariance family is rank-deficient; we resolve this issue by casting the identification problem as distance minimization. We demonstrate the power of these differential geometric families for interpolation and identification in a practical application: wind field covariance approximation for unmanned aerial vehicle navigation.

stat.CO↗

arXiv2020

Balanced truncation for parametric linear systems using interpolation of Gramians: a comparison of algebraic and geometric approaches

When balanced truncation is used for model order reduction, one has to solve a pair of Lyapunov equations for two Gramians and uses them to construct a reduced-order model. Although advances in solving such equations have been made, it is still the most expensive step of this reduction method. Parametric model order reduction aims to determine reduced-order models for parameter-dependent systems. Popular techniques for parametric model order reduction rely on interpolation. Nevertheless, the interpolation of Gramians is rarely mentioned, most probably due to the fact that Gramians are symmetric positive semidefinite matrices, a property that should be preserved by the interpolation method. In this contribution, we propose and compare two approaches for Gramian interpolation. In the first approach, the interpolated Gramian is computed as a linear combination of the data Gramians with positive coefficients. Even though positive semidefiniteness is guaranteed in this method, the rank of the interpolated Gramian can be significantly larger than that of the data Gramians. The second approach aims to tackle this issue by performing the interpolation on the manifold of fixed-rank positive semidefinite matrices. The results of the interpolation step are then used to construct parametric reduced-order models, which are compared numerically on two benchmark problems.

math.NA↗

arXiv2019

Fitting, Comparison, and Alignment of Trajectories on Positive Semi-Definite Matrices with Application to Action Recognition

In this paper, we tackle the problem of action recognition using body skeletons extracted from video sequences. Our approach lies in the continuity of recent works representing video frames by Gramian matrices that describe a trajectory on the Riemannian manifold of positive-semidefinite matrices of fixed rank. In comparison with previous works, the manifold of fixed-rank positive-semidefinite matrices is here endowed with a different metric, and we resort to different algorithms for the curve fitting and temporal alignment steps. We evaluated our approach on three publicly available datasets (UTKinect-Action3D, KTH-Action and UAV-Gesture). The results of the proposed approach are competitive with respect to state-of-the-art methods, while only involving body skeletons.

cs.CV↗

arXiv2018

Blended smoothing splines on Riemannian manifolds

We present a method to compute a fitting curve B to a set of data points d0,...,dm lying on a manifold M. That curve is obtained by blending together Euclidean Bézier curves obtained on different tangent spaces. The method guarantees several properties among which B is C1 and is the natural cubic smoothing spline when M is the Euclidean space. We show examples on the sphere S2 as a proof of concept.

cs.IT↗