arXiv subjects

Armenak Petrosyan

Publications and source records attributed to Armenak Petrosyan.

12 recordsLinked to original sources

arXiv2026

Layerwise Decoupling for Stable Structured Sparsification of Fully Connected Layers

We propose a decoupled, layerwise method for structurally sparsifying the fully connected layers of pretrained neural networks. Rather than penalizing all layers jointly, our approach extracts shallow two-layer subnetworks, normalizes the inner weights, and applies a structured group penalty to the outer weight matrix of each block, processing layers sequentially to prune neurons and reduce the width of each layer. We prove that the constrained decoupled objective is equivalent at optimality to a specific joint penalty on the inner and outer weights, for any positively homogeneous activation, and thus admits a clean projected and proximal formulation. Our central finding is that this decoupled reformulation is more robust than coupled methods. In numerical experiments it provides a wider usable range of the regularization strength and a lower rate of catastrophic over-pruning than the tested joint baseline while maintaining comparable accuracy. We establish these properties in controlled classification and sparse-recovery studies, and examine their scope in a high-dimensional PINN stress test and in the feed-forward layers of OPT-1.3B.

cs.LG↗

arXiv2024

Reconstructing Graph Signals from Noisy Dynamical Samples

We investigate the dynamical sampling space-time trade-off problem within a graph setting. Specifically, we derive necessary and sufficient conditions for space-time sampling that enable the reconstruction of an initial band-limited signal on a graph. Additionally, we develop and test numerical algorithms for approximating the optimal placement of sensors on the graph to minimize the mean squared error when recovering signals from time-space measurements corrupted by i.i.d.~additive noise. Our numerical experiments demonstrate that our approach outperforms previously proposed algorithms for related problems.

cs.IT↗

arXiv2023

Orthogonally weighted $\ell_{2,1}$ regularization for rank-aware joint sparse recovery: algorithm and analysis

We propose and analyze an efficient algorithm for solving the joint sparse recovery problem using a new regularization-based method, named orthogonally weighted $\ell_{2,1}$ ($\mathit{ow}\ell_{2,1}$), which is specifically designed to take into account the rank of the solution matrix. This method has applications in feature extraction, matrix column selection, and dictionary learning, and it is distinct from commonly used $\ell_{2,1}$ regularization and other existing regularization-based approaches because it can exploit the full rank of the row-sparse solution matrix, a key feature in many applications. We provide a proof of the method's rank-awareness, establish the existence of solutions to the proposed optimization problem, and develop an efficient algorithm for solving it, whose convergence is analyzed. We also present numerical experiments to illustrate the theory and demonstrate the effectiveness of our method on real-life problems.

math.NA↗

arXiv2022

Nonconvex regularization for sparse neural networks

Convex $\ell_1$ regularization using an infinite dictionary of neurons has been suggested for constructing neural networks with desired approximation guarantees, but can be affected by an arbitrary amount of over-parametrization. This can lead to a loss of sparsity and result in networks with too many active neurons for the given data, in particular if the number of data samples is large. As a remedy, in this paper, a nonconvex regularization method is investigated in the context of shallow ReLU networks: We prove that in contrast to the convex approach, any resulting (locally optimal) network is finite even in the presence of infinite data (i.e., if the data distribution is known and the limiting case of infinite samples is considered). Moreover, we show that approximation guarantees and existing bounds on the network size for finite data are maintained.

math.OC↗

arXiv2021

Greedy Shallow Networks: An Approach for Constructing and Training Neural Networks

We present a greedy-based approach to construct an efficient single hidden layer neural network with the ReLU activation that approximates a target function. In our approach we obtain a shallow network by utilizing a greedy algorithm with the prescribed dictionary provided by the available training data and a set of possible inner weights. To facilitate the greedy selection process we employ an integral representation of the network, based on the ridgelet transform, that significantly reduces the cardinality of the dictionary and hence promotes feasibility of the greedy selection. Our approach allows for the construction of efficient architectures which can be treated either as improved initializations to be used in place of random-based alternatives, or as fully-trained networks in certain cases, thus potentially nullifying the need for backpropagation training. Numerical experiments demonstrate the tenability of the proposed concept and its advantages compared to the conventional techniques for selecting architectures and initializations for neural networks.

cs.LG↗

arXiv2020

Neural network integral representations with the ReLU activation function

In this effort, we derive a formula for the integral representation of a shallow neural network with the ReLU activation function. We assume that the outer weighs admit a finite $L_1$-norm with respect to Lebesgue measure on the sphere. For univariate target functions we further provide a closed-form formula for all possible representations. Additionally, in this case our formula allows one to explicitly solve the least $L_1$-norm neural network representation for a given function.

cs.LG↗

arXiv2019

A Weighted $\ell_1$-Minimization Approach For Wavelet Reconstruction of Signals and Images

In this effort, we propose a convex optimization approach based on weighted $\ell_1$-regularization for reconstructing objects of interest, such as signals or images, that are sparse or compressible in a wavelet basis. We recover the wavelet coefficients associated to the functional representation of the object of interest by solving our proposed optimization problem. We give a specific choice of weights and show numerically that the chosen weights admit efficient recovery of objects of interest from either a set of sub-samples or a noisy version. Our method not only exploits sparsity but also helps promote a particular kind of structured sparsity often exhibited by many signals and images. Furthermore, we illustrate the effectiveness of the proposed convex optimization problem by providing numerical examples using both orthonormal wavelets and a frame of wavelets. We also provide an adaptive choice of weights which is a modification of the iteratively reweighted $\ell_1$-minimization method.

eess.IV↗

arXiv2019

Local-to-global frames and applications to dynamical sampling problem

In this paper we consider systems of vectors in a Hilbert space $\mathcal{H}$ of the form $\{g_{jk}: j \in J, \, k\in K\}\subset \mathcal{H}$ where $J$ and $K$ are countable sets of indices. We find conditions under which the local reconstruction properties of such a system extend to global stable recovery properties on the whole space. As a particular case, we obtain new local-to-global results for systems of type $\{A^ng\}_{g\in\mathcal{G},0\leq n\leq L }$ arising in the dynamical sampling problem.

math.FA↗

arXiv2019

An Operator theoretic approach to the convergence of rearranged Fourier series

This article studies the rearrangement problem for Fourier series introduced by P.L. Ulyanov, who conjectured that every continuous function on the torus admits a rearrangement of its Fourier coefficients such that the rearranged partial sums of the Fourier series converge uniformly to the function. The main theorem here gives several new equivalences to this conjecture in terms of the convergence of the rearranged Fourier series in the strong (equivalently in this case, weak) operator topologies on $B(L_2(T))$. Additionally, a new framework for further investigation is introduced by considering convergence for subspaces of $L_2$, which leads to many methods for attempting to prove or disprove Ulyanov's conjecture. In this framework, we provide characterizations of unconditional convergence of the Fourier series in the SOT and WOT. These considerations also give rise to some interesting questions regarding weaker versions of the rearrangement problem. Along the way, we consider some interesting questions related to the classical theory of trigonometric polynomials. All of the results here admit natural extensions to arbitrary dimensions.

math.CA↗

arXiv2019

Reconstruction of jointly sparse vectors via manifold optimization

In this paper, we consider the challenge of reconstructing jointly sparse vectors from linear measurements. Firstly, we show that by utilizing the rank of the output data matrix we can reduce the problem to a full column rank case. This result reveals a reduction in the computational complexity of the original problem and enables a simple implementation of joint sparse recovery algorithms for full-rank setting. Secondly, we propose a new method for joint sparse recovery in the form of a non-convex optimization problem on a non-compact Stiefel manifold. In our numerical experiments our method outperforms the commonly used $\ell_{2,1}$ minimization in the sense that much fewer measurements are required for accurate sparse reconstructions. We postulate this approach possesses the desirable rank aware property, that is, being able to take advantage of the rank of the unknown matrix to improve the recovery.

math.NA↗

arXiv2019

Frames Induced by the Action of Continuous Powers of an Operator

We investigate systems of the form $\{A^tg:g\in\mathcal{G},t\in[0,L]\}$ where $A \in B(\mathcal{H})$ is a normal operator in a separable Hilbert space $\mathcal{H}$, $\mathcal{G}\subset \mathcal{H}$ is a countable set, and $L$ is a positive real number. Although the main goal of this work is to study the frame properties of $\{A^tg:g\in\mathcal{G},t\in[0,L]\}$, as intermediate steps, we explore the completeness and Bessel properties of such systems from a theoretical perspective, which are of interest by themselves. Beside the theoretical appeal of investigating such systems, their connections to dynamical and mobile sampling make them fundamental for understanding and solving several major problems in engineering and science.

math.FA↗

arXiv2016

Dynamical sampling and systems from iterative actions of operators

We review some of the recent developments and prove new results concerning frames and Bessel systems generated by iterations of the form $\{A^ng: g\in G,\, n=0,1,2,\dots \}$, where $A$ is a bounded linear operators on a separable complex Hilbert space $ \mathcal{H} $ and $G$ is a countable set of vectors in $ \mathcal{H} $. The system of iterations mentioned above was motivated from the so called dynamical sampling problem. In dynamical sampling, an unknown function $f$ and its future states $A^nf$ are coarsely sampled at each time level $n$, $0\leq n< L$, where $A$ is an evolution operator that drives the system. The goal is to recover $f$ from these space-time samples.

math.FA↗