Searcharxiv⌕ Search

arXiv subjects

Konstantin Usevich

Publications and source records attributed to Konstantin Usevich.

At least 37 records · Page 2Linked to original sources

Extended L-ensembles: a new representation for Determinantal Point Processes

Determinantal point processes (DPPs) are a class of repulsive point processes, popular for their relative simplicity. They are traditionally defined via their marginal distributions, but a subset of DPPs called "L-ensembles" have tractable likelihoods and are thus particularly easy to work with. Indeed, in many applications, DPPs are more naturally defined based on the L-ensemble formulation rather than through the marginal kernel. The fact that not all DPPs are L-ensembles is unfortunate, but there is a unifying description. We introduce here extended L-ensembles, and show that all DPPs are extended L-ensembles (and vice-versa). Extended L-ensembles have very simple likelihood functions, contain L-ensembles and projection DPPs as special cases. From a theoretical standpoint, they fix some pathologies in the usual formalism of DPPs, for instance the fact that projection DPPs are not L-ensembles. From a practical standpoint, they extend the set of kernel functions that may be used to define DPPs: we show that conditional positive definite kernels are good candidates for defining DPPs, including DPPs that need no spatial scale parameter. Finally, extended L-ensembles are based on so-called ``saddle-point matrices'', and we prove an extension of the Cauchy-Binet theorem for such matrices that may be of independent interest.

math.ST↗

Determinantal Point Processes in the Flat Limit

Determinantal point processes (DPPs) are repulsive point processes where the interaction between points depends on the determinant of a positive-semi definite matrix. In this paper, we study the limiting process of L-ensembles based on kernel matrices, when the kernel function becomes flat (so that every point interacts with every other point, in a sense). We show that these limiting processes are best described in the formalism of extended L-ensembles and partial projection DPPs, and the exact limit depends mostly on the smoothness of the kernel function. In some cases, the limiting process is even universal, meaning that it does not depend on specifics of the kernel function, but only on its degree of smoothness. Since flat-limit DPPs are still repulsive processes, this implies that practically useful families of DPPs exist that do not require a spatial length-scale parameter.

math.PR↗

Low-rank tensor recovery for Jacobian-based Volterra identification of parallel Wiener-Hammerstein systems

We consider the problem of identifying a parallel Wiener-Hammerstein structure from Volterra kernels. Methods based on Volterra kernels typically resort to coupled tensor decompositions of the kernels. However, in the case of parallel Wiener-Hammerstein systems, such methods require nontrivial constraints on the factors of the decompositions. In this paper, we propose an entirely different approach: by using special sampling (operating) points for the Jacobian of the nonlinear map from past inputs to the output, we can show that the Jacobian matrix becomes a linear projection of a tensor whose rank is equal to the number of branches. This representation allows us to solve the identification problem as a tensor recovery problem.

eess.SP↗

Tensor-based framework for training flexible neural networks

Activation functions (AFs) are an important part of the design of neural networks (NNs), and their choice plays a predominant role in the performance of a NN. In this work, we are particularly interested in the estimation of flexible activation functions using tensor-based solutions, where the AFs are expressed as a weighted sum of predefined basis functions. To do so, we propose a new learning algorithm which solves a constrained coupled matrix-tensor factorization (CMTF) problem. This technique fuses the first and zeroth order information of the NN, where the first-order information is contained in a Jacobian tensor, following a constrained canonical polyadic decomposition (CPD). The proposed algorithm can handle different decomposition bases. The goal of this method is to compress large pretrained NN models, by replacing subnetworks, {\em i.e.,} one or multiple layers of the original network, by a new flexible layer. The approach is applied to a pretrained convolutional neural network (CNN) used for character classification.

cs.LG↗

Jacobi-type algorithm for low rank orthogonal approximation of symmetric tensors and its convergence analysis

In this paper, we propose a Jacobi-type algorithm to solve the low rank orthogonal approximation problem of symmetric tensors. This algorithm includes as a special case the well-known Jacobi CoM2 algorithm for the approximate orthogonal diagonalization problem of symmetric tensors. We first prove the weak convergence of this algorithm, \textit{i.e.} any accumulation point is a stationary point. Then we study the global convergence of this algorithm under a gradient based ordering for a special case: the best rank-2 orthogonal approximation of 3rd order symmetric tensors, and prove that an accumulation point is the unique limit point under some conditions. Numerical experiments are presented to show the efficiency of this algorithm.

math.NA↗

Coupled Tensor Decomposition for Hyperspectral and Multispectral Image Fusion with Inter-Image Variability

Coupled tensor approximation has recently emerged as a promising approach for the fusion of hyperspectral and multispectral images, reconciling state of the art performance with strong theoretical guarantees. However, tensor-based approaches previously proposed assume that the different observed images are acquired under exactly the same conditions. A recent work proposed to accommodate inter-image spectral variability in the image fusion problem using a matrix factorization-based formulation, but did not account for spatially-localized variations. Moreover, it lacks theoretical guarantees and has a high associated computational complexity. In this paper, we consider the image fusion problem while accounting for both spatially and spectrally localized changes in an additive model. We first study how the general identifiability of the model is impacted by the presence of such changes. Then, assuming that the high-resolution image and the variation factors admit a Tucker decomposition, two new algorithms are proposed -- one purely algebraic, and another based on an optimization procedure. Theoretical guarantees for the exact recovery of the high-resolution image are provided for both algorithms. Experimental results show that the proposed method outperforms state-of-the-art methods in the presence of spectral and spatial variations between the images, at a smaller computational cost.

eess.SP↗

On the convergence of Jacobi-type algorithms for Independent Component Analysis

Jacobi-type algorithms for simultaneous approximate diagonalization of real (or complex) symmetric tensors have been widely used in independent component analysis (ICA) because of their good performance. One natural way of choosing the index pairs in Jacobi-type algorithms is the classical cyclic ordering, while the other way is based on the Riemannian gradient in each iteration. In this paper, we mainly review in an accessible manner our recent results in a series of papers about weak and global convergence of these Jacobi-type algorithms. These results are mainly based on the Lojasiewicz gradient inequality.

math.NA↗

Hyperspectral Super-Resolution with Coupled Tucker Approximation: Recoverability and SVD-based algorithms

We propose a novel approach for hyperspectral super-resolution, that is based on low-rank tensor approximation for a coupled low-rank multilinear (Tucker) model. We show that the correct recovery holds for a wide range of multilinear ranks. For coupled tensor approximation, we propose two SVD-based algorithms that are simple and fast, but with a performance comparable to the state-of-the-art methods. The approach is applicable to the case of unknown spatial degradation and to the pansharpening problem.

eess.SP↗

On approximate diagonalization of third order symmetric tensors by orthogonal transformations

In this paper, we study the approximate orthogonal diagonalization problem of third order symmetric tensors. We define several classes of approximately diagonal tensors, including the ones corresponding to the stationary points of this problem. We study the relationships between these classes, and other well-known objects, such as tensor Z-eigenvalue and Z-eigenvector. We also prove results on convergence of the cyclic Jacobi (or Jacobi CoM2) algorithm.

math.NA↗

Decoupling multivariate polynomials: interconnections between tensorizations

Decoupling multivariate polynomials is useful for obtaining an insight into the workings of a nonlinear mapping, performing parameter reduction, or approximating nonlinear functions. Several different tensor-based approaches have been proposed independently for this task, involving different tensor representations of the functions, and ultimately leading to a canonical polyadic decomposition. We first show that the involved tensors are related by a linear transformation, and that their CP decompositions and uniqueness properties are closely related. This connection provides a way to better assess which of the methods should be favored in certain problem settings, and may be a starting point to unify the two approaches. Second, we show that taking into account the previously ignored intrinsic structure in the tensor decompositions improves the uniqueness properties of the decompositions and thus enlarges the applicability range of the methods.

math.NA↗

Structured low-rank matrix completion for forecasting in time series analysis

In this paper we consider the low-rank matrix completion problem with specific application to forecasting in time series analysis. Briefly, the low-rank matrix completion problem is the problem of imputing missing values of a matrix under a rank constraint. We consider a matrix completion problem for Hankel matrices and a convex relaxation based on the nuclear norm. Based on new theoretical results and a number of numerical and real examples, we investigate the cases when the proposed approach can work. Our results highlight the importance of choosing a proper weighting scheme for the known observations.

stat.ME↗

Globally convergent Jacobi-type algorithms for simultaneous orthogonal symmetric tensor diagonalization

In this paper, we consider a family of Jacobi-type algorithms for simultaneous orthogonal diagonalization problem of symmetric tensors. For the Jacobi-based algorithm of [SIAM J. Matrix Anal. Appl., 2(34):651--672, 2013], we prove its global convergence for simultaneous orthogonal diagonalization of symmetric matrices and 3rd-order tensors. We also propose a new Jacobi-based algorithm in the general setting and prove its global convergence for sufficiently smooth functions.

math.NA↗

Identifiability of an X-rank decomposition of polynomial maps

In this paper, we study a polynomial decomposition model that arises in problems of system identification, signal processing and machine learning. We show that this decomposition is a special case of the X-rank decomposition --- a powerful novel concept in algebraic geometry that generalizes the tensor CP decomposition. We prove new results on generic/maximal rank and on identifiability of a particular polynomial decomposition model. In the paper, we try to make results and basic tools accessible for general audience (assuming no knowledge of algebraic geometry or its prerequisites).

cs.IT↗

Variable projection methods for approximate (greatest) common divisor computations

We consider the problem of finding for a given $N$-tuple of polynomials (real or complex) the closest $N$-tuple that has a common divisor of degree at least $d$. Extended weighted Euclidean seminorm of the coefficients is used as a measure of closeness. Two equivalent representations of the problem are considered: (i) direct parameterization over the common divisors and quotients (image representation), and (ii) Sylvester low-rank approximation (kernel representation). We use the duality between least-squares and least-norm problems to show that (i) and (ii) are closely related to mosaic Hankel low-rank approximation. This allows us to apply to the approximate common divisor problem recent results on complexity and accuracy of computations for mosaic Hankel low-rank approximation. We develop optimization methods based on the variable projection principle both for image and kernel representation. These methods have linear complexity in the degrees of the polynomials for small and large $d$. We provide a software implementation of the developed methods, which is based on a software package for structured low-rank approximation.

math.OC↗

Adjusted least squares fitting of algebraic hypersurfaces

We consider the problem of fitting a set of points in Euclidean space by an algebraic hypersurface. We assume that points on a true hypersurface, described by a polynomial equation, are corrupted by zero mean independent Gaussian noise, and we estimate the coefficients of the true polynomial equation. The adjusted least squares estimator accounts for the bias present in the ordinary least squares estimator. The adjusted least squares estimator is based on constructing a quasi-Hankel matrix, which is a bias-corrected matrix of moments. For the case of unknown noise variance, the estimator is defined as a solution of a polynomial eigenvalue problem. In this paper, we present new results on invariance properties of the adjusted least squares estimator and an improved algorithm for computing the estimator for an arbitrary set of monomials in the polynomial equation.

stat.CO↗

Quasi-Hankel low-rank matrix completion: a convex relaxation

The completion of matrices with missing values under the rank constraint is a non-convex optimization problem. A popular convex relaxation is based on minimization of the nuclear norm (sum of singular values) of the matrix. For this relaxation, an important question is whether the two optimization problems lead to the same solution. This question was addressed in the literature mostly in the case of random positions of missing elements and random known elements. In this contribution, we analyze the case of structured matrices with fixed pattern of missing values, in particular, the case of Hankel and quasi-Hankel matrix completion, which appears as a subproblem in the computation of symmetric tensor canonical polyadic decomposition. We extend existing results on completion of rank-one real Hankel matrices to completion of rank-r complex Hankel and quasi-Hankel matrices.

math.OC↗

Multivariate and 2D Extensions of Singular Spectrum Analysis with the Rssa Package

Implementation of multivariate and 2D extensions of Singular Spectrum Analysis (SSA) by means of the R-package Rssa is considered. The extensions include MSSA for simultaneous analysis and forecasting of several time series and 2D-SSA for analysis of digital images. A new extension of 2D-SSA analysis called Shaped 2D-SSA is introduced for analysis of images of arbitrary shape, not necessary rectangular. It is shown that implementation of Shaped 2D-SSA can serve as a base for implementation of MSSA and other generalizations. Efficient implementation of operations with Hankel and Hankel-block-Hankel matrices through FFT is suggested. Examples with code fragments in R, which explain the methodology and demonstrate the proper use of Rssa, are presented.

stat.ME↗