arXiv subjects

Michael Unser

Publications and source records attributed to Michael Unser.

At least 73 records · Page 4Linked to original sources

arXiv2022

Asymptotic Stability in Reservoir Computing

Reservoir Computing is a class of Recurrent Neural Networks with internal weights fixed at random. Stability relates to the sensitivity of the network state to perturbations. It is an important property in Reservoir Computing as it directly impacts performance. In practice, it is desirable to stay in a stable regime, where the effect of perturbations does not explode exponentially, but also close to the chaotic frontier where reservoir dynamics are rich. Open questions remain today regarding input regularization and discontinuous activation functions. In this work, we use the recurrent kernel limit to draw new insights on stability in reservoir computing. This limit corresponds to large reservoir sizes, and it already becomes relevant for reservoirs with a few hundred neurons. We obtain a quantitative characterization of the frontier between stability and chaos, which can greatly benefit hyperparameter tuning. In a broader sense, our results contribute to understanding the complex dynamics of Recurrent Neural Networks.

cs.NE↗

arXiv2022

Bona fide Riesz projections for density estimation

The projection of sample measurements onto a reconstruction space represented by a basis on a regular grid is a powerful and simple approach to estimate a probability density function. In this paper, we focus on Riesz bases and propose a projection operator that, in contrast to previous works, guarantees the bona fide properties for the estimate, namely, non-negativity and total probability mass $1$. Our bona fide projection is defined as a convex problem. We propose solution techniques and evaluate them. Results suggest an improved performance, specifically in circumstances prone to rippling effects.

eess.SP↗

arXiv2022

Approximation of Lipschitz Functions using Deep Spline Neural Networks

Lipschitz-constrained neural networks have many applications in machine learning. Since designing and training expressive Lipschitz-constrained networks is very challenging, there is a need for improved methods and a better theoretical understanding. Unfortunately, it turns out that ReLU networks have provable disadvantages in this setting. Hence, we propose to use learnable spline activation functions with at least 3 linear regions instead. We prove that this choice is optimal among all component-wise $1$-Lipschitz activation functions in the sense that no other weight constrained architecture can approximate a larger class of functions. Additionally, this choice is at least as expressive as the recently introduced non component-wise Groupsort activation function for spectral-norm-constrained weights. Previously published numerical results support our theoretical findings.

cs.LG↗

arXiv2022

Stable Parametrization of Continuous and Piecewise-Linear Functions

Rectified-linear-unit (ReLU) neural networks, which play a prominent role in deep learning, generate continuous and piecewise-linear (CPWL) functions. While they provide a powerful parametric representation, the mapping between the parameter and function spaces lacks stability. In this paper, we investigate an alternative representation of CPWL functions that relies on local hat basis functions. It is predicated on the fact that any CPWL function can be specified by a triangulation and its values at the grid points. We give the necessary and sufficient condition on the triangulation (in any number of dimensions) for the hat functions to form a Riesz basis, which ensures that the link between the parameters and the corresponding CPWL function is stable and unique. In addition, we provide an estimate of the $\ell_2\rightarrow L_2$ condition number of this local representation. Finally, as a special case of our framework, we focus on a systematic parametrization of $\mathbb{R}^d$ with control points placed on a uniform grid. In particular, we choose hat basis functions that are shifted replicas of a single linear box spline. In this setting, we prove that our general estimate of the condition number is optimal. We also relate our local representation to a nonlocal one based on shifts of a causal ReLU-like function.

math.NA↗

arXiv2022

Ridges, Neural Networks, and the Radon Transform

A ridge is a function that is characterized by a one-dimensional profile (activation) and a multidimensional direction vector. Ridges appear in the theory of neural networks as functional descriptors of the effect of a neuron, with the direction vector being encoded in the linear weights. In this paper, we investigate properties of the Radon transform in relation to ridges and to the characterization of neural networks. We introduce a broad category of hyper-spherical Banach subspaces (including the relevant subspace of measures) over which the back-projection operator is invertible. We also give conditions under which the back-projection operator is extendable to the full parent space with its null space being identifiable as a Banach complement. Starting from first principles, we then characterize the sampling functionals that are in the range of the filtered Radon transform. Next, we extend the definition of ridges for any distributional profile and determine their (filtered) Radon transform in full generality. Finally, we apply our formalism to clarify and simplify some of the results and proofs on the optimality of ReLU networks that have appeared in the literature.

math.FA↗

arXiv2022

Measuring Complexity of Learning Schemes Using Hessian-Schatten Total Variation

In this paper, we introduce the Hessian-Schatten total variation (HTV) -- a novel seminorm that quantifies the total "rugosity" of multivariate functions. Our motivation for defining HTV is to assess the complexity of supervised-learning schemes. We start by specifying the adequate matrix-valued Banach spaces that are equipped with suitable classes of mixed norms. We then show that the HTV is invariant to rotations, scalings, and translations. Additionally, its minimum value is achieved for linear mappings, which supports the common intuition that linear regression is the least complex learning model. Next, we present closed-form expressions of the HTV for two general classes of functions. The first one is the class of Sobolev functions with a certain degree of regularity, for which we show that the HTV coincides with the Hessian-Schatten seminorm that is sometimes used as a regularizer for image reconstruction. The second one is the class of continuous and piecewise-linear (CPWL) functions. In this case, we show that the HTV reflects the total change in slopes between linear regions that have a common facet. Hence, it can be viewed as a convex relaxation (l1-type) of the number of linear regions (l0-type) of CPWL mappings. Finally, we illustrate the use of our proposed seminorm.

cs.LG↗

arXiv2021

Sparsest Univariate Learning Models Under Lipschitz Constraint

Beside the minimization of the prediction error, two of the most desirable properties of a regression scheme are stability and interpretability. Driven by these principles, we propose continuous-domain formulations for one-dimensional regression problems. In our first approach, we use the Lipschitz constant as a regularizer, which results in an implicit tuning of the overall robustness of the learned mapping. In our second approach, we control the Lipschitz constant explicitly using a user-defined upper-bound and make use of a sparsity-promoting regularizer to favor simpler (and, hence, more interpretable) solutions. The theoretical study of the latter formulation is motivated in part by its equivalence, which we prove, with the training of a Lipschitz-constrained two-layer univariate neural network with rectified linear unit (ReLU) activations and weight decay. By proving representer theorems, we show that both problems admit global minimizers that are continuous and piecewise-linear (CPWL) functions. Moreover, we propose efficient algorithms that find the sparsest solution of each problem: the CPWL mapping with the least number of linear regions. Finally, we illustrate numerically the outcome of our formulations.

cs.LG↗

arXiv2021

Sparsest Piecewise-Linear Regression of One-Dimensional Data

We study the problem of one-dimensional regression of data points with total-variation (TV) regularization (in the sense of measures) on the second derivative, which is known to promote piecewise-linear solutions with few knots. While there are efficient algorithms for determining such adaptive splines, the difficulty with TV regularization is that the solution is generally non-unique, an aspect that is often ignored in practice. In this paper, we present a systematic analysis that results in a complete description of the solution set with a clear distinction between the cases where the solution is unique and those, much more frequent, where it is not. For the latter scenario, we identify the sparsest solutions, i.e., those with the minimum number of knots, and we derive a formula to compute the minimum number of knots based solely on the data points. To achieve this, we first consider the problem of exact interpolation which leads to an easier theoretical analysis. Next, we relax the exact interpolation requirement to a regression setting, and we consider a penalized optimization problem with a strictly convex data-fidelity cost function. We show that the underlying penalized problem can be reformulated as a constrained problem, and thus that all our previous results still apply. Based on our theoretical analysis, we propose a simple and fast two-step algorithm, agnostic to uniqueness, to reach a sparsest solution of this penalized problem.

math.OC↗

arXiv2021

Convex Optimization in Sums of Banach Spaces

We characterize the solution of a broad class of convex optimization problems that address the reconstruction of a function from a finite number of linear measurements. The underlying hypothesis is that the solution is decomposable as a finite sum of components, where each component belongs to its own prescribed Banach space; moreover, the problem is regularized by penalizing some composite norm of the solution. We establish general conditions for existence and derive the generic parametric representation of the solution components. These representations fall into three categories depending on the underlying regularization norm: (i) a linear expansion in terms of predefined "kernels" when the component space is a reproducing kernel Hilbert space (RKHS), (ii) a non-linear (duality) mapping of a linear combination of measurement functionals when the component Banach space is strictly convex, and, (iii) an adaptive expansion in terms of a small number of atoms within a larger dictionary when the component Banach space is not strictly convex. Our approach generalizes and unifies a number of multi-kernel (RKHS) and sparse-dictionary learning techniques for compressed sensing available in the literature. It also yields the natural extension of the classical spline-fitting techniques in (semi-)RKHS to the abstract Banach-space setting.

math.OC↗

arXiv2021

Diffraction Tomography with Helmholtz Equation: Efficient and Robust Multigrid-Based Solver

Diffraction tomography is a noninvasive technique that estimates the refractive indices of unknown objects and involves an inverse-scattering problem governed by the wave equation. Recent works have shown the benefit of nonlinear models of wave propagation that account for multiple scattering and reflections. In particular, the Lippmann-Schwinger~(LiS) model defines an inverse problem to simulate the wave propagation. Although accurate, this model is hard to solve when the samples are highly contrasted or have a large physical size. In this work, we introduce instead a Helmholtz-based nonlinear model for inverse scattering. To solve the corresponding inverse problem, we propose a robust and efficient multigrid-based solver. Moreover, we show that our method is a suitable alternative to the LiS model, especially for strongly scattering objects. Numerical experiments on simulated and real data demonstrate the effectiveness of the Helmholtz model, as well as the efficiency of the proposed multigrid method.

eess.IV↗

arXiv2021

Shortest-support Multi-Spline Bases for Generalized Sampling

Generalized sampling consists in the recovery of a function $f$, from the samples of the responses of a collection of linear shift-invariant systems to the input $f$. The reconstructed function is typically a member of a finitely generated integer-shift-invariant space that can reproduce polynomials up to a given degree $M$. While this property allows for an approximation power of order $(M+1)$, it comes with a tradeoff on the length of the support of the basis functions. Specifically, we prove that the sum of the length of the support of the generators is at least $(M+1)$. Following this result, we introduce the notion of shortest basis of degree $M$, which is motivated by our desire to minimize the computational costs. We then demonstrate that any basis of shortest support generates a Riesz basis. Finally, we introduce a recursive algorithm to construct the shortest-support basis for any multi-spline space. It provides a generalization of both polynomial and Hermite B-splines. This framework paves the way for novel applications such as fast derivative sampling with arbitrarily high approximation power.

math.NA↗

arXiv2021

Principled Design and Implementation of Steerable Detectors

We provide a complete pipeline for the detection of patterns of interest in an image. In our approach, the patterns are assumed to be adequately modeled by a known template, and are located at unknown positions and orientations that we aim at retrieving. We propose a continuous-domain additive image model, where the analyzed image is the sum of the patterns to localize and a background with self-similar isotropic power-spectrum. We are then able to compute the optimal filter fulfilling the SNR criterion based on one single template and background pair: it strongly responds to the template while being optimally decoupled from the background model. In addition, we constrain our filter to be steerable, which allows for a fast template detection together with orientation estimation. In practice, the implementation requires to discretize a continuous-domain formulation on polar grids, which is performed using quadratic radial B-splines. We demonstrate the practical usefulness of our method on a variety of template approximation and pattern detection experiments. We show that the detection performance drastically improves when we exploit the statistics of the background via its power-spectrum decay, which we refer to as spectral-shaping. The proposed scheme outperforms state-of-the-art steerable methods by up to 50% of absolute detection performance.

eess.IV↗

arXiv2021

Continuous-Domain Formulation of Inverse Problems for Composite Sparse-Plus-Smooth Signals

We present a novel framework for the reconstruction of 1D composite signals assumed to be a mixture of two additive components, one sparse and the other smooth, given a finite number of linear measurements. We formulate the reconstruction problem as a continuous-domain regularized inverse problem with multiple penalties. We prove that these penalties induce reconstructed signals that indeed take the desired form of the sum of a sparse and a smooth component. We then discretize this problem using Riesz bases, which yields a discrete problem that can be solved by standard algorithms. Our discretization is exact in the sense that we are solving the continuous-domain problem over the search space specified by our bases without any discretization error. We propose a complete algorithmic pipeline and demonstrate its feasibility on simulated data.

eess.SP↗

arXiv2021

Biomedical Image Reconstruction: From the Foundations to Deep Neural Networks

This tutorial covers biomedical image reconstruction, from the foundational concepts of system modeling and direct reconstruction to modern sparsity and learning-based approaches. Imaging is a critical tool in biological research and medicine, and most imaging systems necessarily use an image-reconstruction algorithm to create an image; the design of these algorithms has been a topic of research since at least the 1960's. In the last few years, machine learning-based approaches have shown impressive performance on image reconstruction problems, triggering a wave of enthusiasm and creativity around the paradigm of learning. Our goal is to unify this body of research, identifying common principles and reusable building blocks across decades and among diverse imaging modalities. We first describe system modeling, emphasizing how a few building blocks can be used to describe a broad range of imaging modalities. We then discuss reconstruction algorithms, grouping them into three broad generations. The first are the classical direct methods, including Tikhonov regularization; the second are the variational methods based on sparsity and the theory of compressive sensing; and the third are the learning-based (also called data-driven) methods, especially those using deep convolutional neural networks. There are strong links between these generations: classical (first-generation) methods appear as modules inside the latter two, and the former two are used to inspire new designs for learning-based (third-generation) methods. As a result, a solid understanding of all of three generations is necessary for the design of state-of-the-art algorithms.

eess.IV↗

arXiv2021

Optimal-transport-based metric for SMLM

We propose the use of Flat Metric to assess the performance of reconstruction methods for single-molecule localization microscopy (SMLM) in scenarios where the ground-truth is available. Flat Metric is intimately related to the concept of optimal transport between measures of different mass, providing solid mathematical foundations for SMLM evaluation and integrating both localization and detection performance. In this paper, we provide the foundations of Flat Metric and validate this measure by applying it to controlled synthetic examples and to data from the SMLM 2016 Challenge.

eess.IV↗

arXiv2021

Time-Dependent Deep Image Prior for Dynamic MRI

We propose a novel unsupervised deep-learning-based algorithm for dynamic magnetic resonance imaging (MRI) reconstruction. Dynamic MRI requires rapid data acquisition for the study of moving organs such as the heart. Existing reconstruction methods suffer from restrictions either in the model design or in the absence of ground-truth data, resulting in low image quality. We introduce a generalized version of the deep-image-prior approach, which optimizes the network weights to fit a sequence of sparsely acquired dynamic MRI measurements. Our method needs neither prior training nor additional data. In particular, for cardiac images, it does not require the marking of heartbeats or the reordering of spokes. The key ingredients of our method are threefold: 1) a fixed low-dimensional manifold that encodes the temporal variations of images; 2) a network that maps the manifold into a more expressive latent space; and 3) a convolutional neural network that generates a dynamic series of MRI images from the latent variables and that favors their consistency with the measurements in k-space. Our method outperforms the state-of-the-art methods quantitatively and qualitatively in both retrospective and real fetal cardiac datasets. To the best of our knowledge, this is the first unsupervised deep-learning-based method that can reconstruct the continuous variation of dynamic MRI sequences with high spatial resolution.

eess.IV↗

arXiv2020

Multi-Kernel Regression with Sparsity Constraint

In this paper, we provide a Banach-space formulation of supervised learning with generalized total-variation (gTV) regularization. We identify the class of kernel functions that are admissible in this framework. Then, we propose a variation of supervised learning in a continuous-domain hybrid search space with gTV regularization. We show that the solution admits a multi-kernel expansion with adaptive positions. In this representation, the number of active kernels is upper-bounded by the number of data points while the gTV regularization imposes an $\ell_1$ penalty on the kernel coefficients. Finally, we illustrate numerically the outcome of our theory.

cs.LG↗

arXiv2020

Robust Phase Unwrapping via Deep Image Prior for Quantitative Phase Imaging

Quantitative phase imaging (QPI) is an emerging label-free technique that produces images containing morphological and dynamical information without contrast agents. Unfortunately, the phase is wrapped in most imaging system. Phase unwrapping is the computational process that recovers a more informative image. It is particularly challenging with thick and complex samples such as organoids. Recent works that rely on supervised training show that deep learning is a powerful method to unwrap the phase; however, supervised approaches require large and representative datasets which are difficult to obtain for complex biological samples. Inspired by the concept of deep image priors, we propose a deep-learning-based method that does not need any training set. Our framework relies on an untrained convolutional neural network to accurately unwrap the phase while ensuring the consistency of the measurements. We experimentally demonstrate that the proposed method faithfully recovers the phase of complex samples on both real and simulated data. Our work paves the way to reliable phase imaging of thick and complex samples with QPI.

eess.IV↗