arXiv subjects

Paolo Di Lorenzo

Publications and source records attributed to Paolo Di Lorenzo.

89 records · Page 5Linked to original sources

arXiv2020

Distributed Stochastic Nonconvex Optimization and Learning based on Successive Convex Approximation

We study distributed stochastic nonconvex optimization in multi-agent networks. We introduce a novel algorithmic framework for the distributed minimization of the sum of the expected value of a smooth (possibly nonconvex) function (the agents' sum-utility) plus a convex (possibly nonsmooth) regularizer. The proposed method hinges on successive convex approximation (SCA) techniques, leveraging dynamic consensus as a mechanism to track the average gradient among the agents, and recursive averaging to recover the expected gradient of the sum-utility function. Almost sure convergence to (stationary) solutions of the nonconvex problem is established. Finally, the method is applied to distributed stochastic training of neural networks. Numerical results confirm the theoretical claims, and illustrate the advantages of the proposed method with respect to other methods available in the literature.

eess.SP↗

arXiv2018

Observing and Tracking Bandlimited Graph Processes

One of the most crucial challenges in graph signal processing is the sampling of bandlimited graph signals, i.e., signals that are sparse in a well-defined graph Fourier domain. So far, the prior art is mostly focused on (sub)sampling single snapshots of graph signals ignoring their evolution over time. However, time can bring forth new insights, since many real signals like sensor measurements, biological, financial, and network signals in general, have intrinsic correlations in both domains. In this work, {we fill this lacuna} by jointly considering the graph-time nature of graph signals, named \emph{graph processes} for two main tasks: \emph{i)} observability of graph processes; and \emph{ii)} tracking of graph processes via Kalman filtering; both from a (possibly time-varying) subset of nodes. A detailed mathematical analysis ratifies the proposed methods and provides insights into the role played by the different actors, such as the graph topology, the process bandwidth, and the sampling strategy. Moreover, (sub)optimal sampling strategies that jointly exploit the nature of the graph structure and graph process are proposed. Several numerical tests on both synthetic and real data validate our theoretical findings and illustrate the performance of the proposed methods in coping with time-varying graph signals.

eess.SP↗

arXiv2018

Graph topology inference based on sparsifying transform learning

Graph-based representations play a key role in machine learning. The fundamental step in these representations is the association of a graph structure to a dataset. In this paper, we propose a method that aims at finding a block sparse representation of the graph signal leading to a modular graph whose Laplacian matrix admits the found dictionary as its eigenvectors. The role of sparsity here is to induce a band-limited representation or, equivalently, a modular structure of the graph. The proposed strategy is composed of two optimization steps: i) learning an orthonormal sparsifying transform from the data; ii) recovering the Laplacian, and then topology, from the transform. The first step is achieved through an iterative algorithm whose alternating intermediate solutions are expressed in closed form. The second step recovers the Laplacian matrix from the sparsifying transform through a convex optimization method. Numerical results corroborate the effectiveness of the proposed methods over both synthetic data and real brain data, used for inferring the brain functionality network through experiments conducted over patients affected by epilepsy.

eess.SP↗

arXiv2017

Adaptive Graph Signal Processing: Algorithms and Optimal Sampling Strategies

The goal of this paper is to propose novel strategies for adaptive learning of signals defined over graphs, which are observed over a (randomly time-varying) subset of vertices. We recast two classical adaptive algorithms in the graph signal processing framework, namely, the least mean squares (LMS) and the recursive least squares (RLS) adaptive estimation strategies. For both methods, a detailed mean-square analysis illustrates the effect of random sampling on the adaptive reconstruction capability and the steady-state performance. Then, several probabilistic sampling strategies are proposed to design the sampling probability at each node in the graph, with the aim of optimizing the tradeoff between steady-state performance, graph sampling rate, and convergence rate of the adaptive algorithms. Finally, a distributed RLS strategy is derived and is shown to be convergent to its centralized counterpart. Numerical simulations carried out over both synthetic and real data illustrate the good performance of the proposed sampling and reconstruction strategies for (possibly distributed) adaptive learning of signals defined over graphs.

cs.LG↗

arXiv2017

On the Graph Fourier Transform for Directed Graphs

The analysis of signals defined over a graph is relevant in many applications, such as social and economic networks, big data or biological networks, and so on. A key tool for analyzing these signals is the so called Graph Fourier Transform (GFT). Alternative definitions of GFT have been suggested in the literature, based on the eigen-decomposition of either the graph Laplacian or adjacency matrix. In this paper, we address the general case of directed graphs and we propose an alternative approach that builds the graph Fourier basis as the set of orthonormal vectors that minimize a continuous extension of the graph cut size, known as the Lovász extension. To cope with the non-convexity of the problem, we propose two alternative iterative optimization methods, properly devised for handling orthogonality constraints. Finally, we extend the method to minimize a continuous relaxation of the balanced cut size. The formulated problem is again non-convex and we propose an efficient solution method based on an explicit-implicit gradient algorithm.

math.SP↗

arXiv2017

Stochastic Training of Neural Networks via Successive Convex Approximations

This paper proposes a new family of algorithms for training neural networks (NNs). These are based on recent developments in the field of non-convex optimization, going under the general name of successive convex approximation (SCA) techniques. The basic idea is to iteratively replace the original (non-convex, highly dimensional) learning problem with a sequence of (strongly convex) approximations, which are both accurate and simple to optimize. Differently from similar ideas (e.g., quasi-Newton algorithms), the approximations can be constructed using only first-order information of the neural network function, in a stochastic fashion, while exploiting the overall structure of the learning problem for a faster convergence. We discuss several use cases, based on different choices for the loss function (e.g., squared loss and cross-entropy loss), and for the regularization of the NN's weights. We experiment on several medium-sized benchmark problems, and on a large-scale dataset involving simulated physical data. The results show how the algorithm outperforms state-of-the-art techniques, providing faster convergence to a better minimum. Additionally, we show how the algorithm can be easily parallelized over multiple computational units without hindering its performance. In particular, each computational unit can optimize a tailored surrogate function defined on a randomly assigned subset of the input variables, whose dimension can be selected depending entirely on the available computational power.

stat.ML↗

arXiv2017

A Framework for Parallel and Distributed Training of Neural Networks

The aim of this paper is to develop a general framework for training neural networks (NNs) in a distributed environment, where training data is partitioned over a set of agents that communicate with each other through a sparse, possibly time-varying, connectivity pattern. In such distributed scenario, the training problem can be formulated as the (regularized) optimization of a non-convex social cost function, given by the sum of local (non-convex) costs, where each agent contributes with a single error term defined with respect to its local dataset. To devise a flexible and efficient solution, we customize a recently proposed framework for non-convex optimization over networks, which hinges on a (primal) convexification-decomposition technique to handle non-convexity, and a dynamic consensus procedure to diffuse information among the agents. Several typical choices for the training criterion (e.g., squared loss, cross entropy, etc.) and regularization (e.g., $\ell_2$ norm, sparsity inducing penalties, etc.) are included in the framework and explored along the paper. Convergence to a stationary solution of the social non-convex problem is guaranteed under mild assumptions. Additionally, we show a principled way allowing each agent to exploit a possible multi-core architecture (e.g., a local cloud) in order to parallelize its local optimization step, resulting in strategies that are both distributed (across the agents) and parallel (inside each agent) in nature. A comprehensive set of experimental results validate the proposed approach.

stat.ML↗

arXiv2016

Adaptive Least Mean Squares Estimation of Graph Signals

The aim of this paper is to propose a least mean squares (LMS) strategy for adaptive estimation of signals defined over graphs. Assuming the graph signal to be band-limited, over a known bandwidth, the method enables reconstruction, with guaranteed performance in terms of mean-square error, and tracking from a limited number of observations over a subset of vertices. A detailed mean square analysis provides the performance of the proposed method, and leads to several insights for designing useful sampling strategies for graph signals. Numerical results validate our theoretical findings, and illustrate the performance of the proposed method. Furthermore, to cope with the case where the bandwidth is not known beforehand, we propose a method that performs a sparse online estimation of the signal support in the (graph) frequency domain, which enables online adaptation of the graph sampling strategy. Finally, we apply the proposed method to build the power spatial density cartography of a given operational region in a cognitive network environment.

cs.LG↗

arXiv2016

Signals on Graphs: Uncertainty Principle and Sampling

In many applications, the observations can be represented as a signal defined over the vertices of a graph. The analysis of such signals requires the extension of standard signal processing tools. In this work, first, we provide a class of graph signals that are maximally concentrated on the graph domain and on its dual. Then, building on this framework, we derive an uncertainty principle for graph signals and illustrate the conditions for the recovery of band-limited signals from a subset of samples. We show an interesting link between uncertainty principle and sampling and propose alternative signal recovery algorithms, including a generalization to frame-based reconstruction methods. After showing that the performance of signal recovery algorithms is significantly affected by the location of samples, we suggest and compare a few alternative sampling strategies. Finally, we provide the conditions for perfect recovery of a useful signal corrupted by sparse noise, showing that this problem is also intrinsically related to vertex-frequency localization properties.

cs.DM↗

arXiv2016

Joint Optimization of Radio Resources and Code Partitioning in Mobile Edge Computing

The aim of this paper is to propose a computation offloading strategy for mobile edge computing. We exploit the concept of call graph, which models a generic computer program as a set of procedures related to each other through a weighted directed graph. Our goal is to derive the optimal partition of the call graph establishing which procedures are to be executed locally or remotely. The main novelty of our work is that the optimal partition is obtained jointly with the selection of radio parameters, e.g., transmit power and constellation size, in order to minimize the energy consumption at the mobile handset, under a latency constraint taking into account transmit time and execution time. We consider both single and multi-channel transmission strategies and we prove that a globally optimal solution can be achieved in both cases. Finally, we propose a suboptimal strategy aimed at solving a relaxed version of the original problem in order to tradeoff complexity and performance of the proposed framework. Finally, several numerical results illustrate under what conditions in terms of call graph topology, communication strategy, and computation parameters, the proposed offloading strategy provides large performance gains.

cs.NI↗

arXiv2016

NEXT: In-Network Nonconvex Optimization

We study nonconvex distributed optimization in multi-agent networks with time-varying (nonsymmetric) connectivity. We introduce the first algorithmic framework for the distributed minimization of the sum of a smooth (possibly nonconvex and nonseparable) function - the agents' sum-utility - plus a convex (possibly nonsmooth and nonseparable) regularizer. The latter is usually employed to enforce some structure in the solution, typically sparsity. The proposed method hinges on successive convex approximation techniques while leveraging dynamic consensus as a mechanism to distribute the computation among the agents: each agent first solves (possibly inexactly) a local convex approximation of the nonconvex original problem, and then performs local averaging operations. Asymptotic convergence to (stationary) solutions of the nonconvex problem is established. Our algorithmic framework is then customized to a variety of convex and nonconvex problems in several fields, including signal processing, communications, networking, and machine learning. Numerical results show that the new method compares favorably to existing distributed algorithms on both convex and nonconvex problems.

cs.DC↗

arXiv2015

Uncertainty Principle and Sampling of Signals Defined on Graphs

In many applications, from sensor to social networks, gene regulatory networks or big data, observations can be represented as a signal defined over the vertices of a graph. Building on the recently introduced Graph Fourier Transform, the first contribution of this paper is to provide an uncertainty principle for signals on graph. As a by-product of this theory, we show how to build a dictionary of maximally concentrated signals on vertex/frequency domains. Then, we establish a direct relation between uncertainty principle and sampling, which forms the basis for a sampling theorem of signals defined on graph. Based on this theory, we show that, besides sampling rate, the samples' location plays a key role in the performance of signal recovery algorithms. Hence, we suggest a few alternative sampling strategies and compare them with recently proposed methods.

cs.IT↗

arXiv2014

Distributed Estimation and Control of Algebraic Connectivity over Random Graphs

In this paper we propose a distributed algorithm for the estimation and control of the connectivity of ad-hoc networks in the presence of a random topology. First, given a generic random graph, we introduce a novel stochastic power iteration method that allows each node to estimate and track the algebraic connectivity of the underlying expected graph. Using results from stochastic approximation theory, we prove that the proposed method converges almost surely (a.s.) to the desired value of connectivity even in the presence of imperfect communication scenarios. The estimation strategy is then used as a basic tool to adapt the power transmitted by each node of a wireless network, in order to maximize the network connectivity in the presence of realistic Medium Access Control (MAC) protocols or simply to drive the connectivity toward a desired target value. Numerical results corroborate our theoretical findings, thus illustrating the main features of the algorithm and its robustness to fluctuations of the network graph due to the presence of random link failures.

cs.DC↗

arXiv2014

Diffusion Adaptation Strategies for Distributed Estimation over Gaussian Markov Random Fields

The aim of this paper is to propose diffusion strategies for distributed estimation over adaptive networks, assuming the presence of spatially correlated measurements distributed according to a Gaussian Markov random field (GMRF) model. The proposed methods incorporate prior information about the statistical dependency among observations, while at the same time processing data in real-time and in a fully decentralized manner. A detailed mean-square analysis is carried out in order to prove stability and evaluate the steady-state performance of the proposed strategies. Finally, we also illustrate how the proposed techniques can be easily extended in order to incorporate thresholding operators for sparsity recovery applications. Numerical results show the potential advantages of using such techniques for distributed learning in adaptive networks deployed over GMRF.

eess.SY↗

arXiv2013

Distributed Detection and Estimation in Wireless Sensor Networks

In this article we consider the problems of distributed detection and estimation in wireless sensor networks. In the first part, we provide a general framework aimed to show how an efficient design of a sensor network requires a joint organization of in-network processing and communication. Then, we recall the basic features of consensus algorithm, which is a basic tool to reach globally optimal decisions through a distributed approach. The main part of the paper starts addressing the distributed estimation problem. We show first an entirely decentralized approach, where observations and estimations are performed without the intervention of a fusion center. Then, we consider the case where the estimation is performed at a fusion center, showing how to allocate quantization bits and transmit powers in the links between the nodes and the fusion center, in order to accommodate the requirement on the maximum estimation variance, under a constraint on the global transmit power. We extend the approach to the detection problem. Also in this case, we consider the distributed approach, where every node can achieve a globally optimal decision, and the case where the decision is taken at a central node. In the latter case, we show how to allocate coding bits and transmit power in order to maximize the detection probability, under constraints on the false alarm rate and the global transmit power. Then, we generalize consensus algorithms illustrating a distributed procedure that converges to the projection of the observation vector onto a signal subspace. We then address the issue of energy consumption in sensor networks, thus showing how to optimize the network topology in order to minimize the energy necessary to achieve a global consensus. Finally, we address the problem of matching the topology of the network to the graph describing the statistical dependencies among the observed variables.

cs.DC↗

arXiv2013

Kepler Photometry of Four Radio-Loud AGN in 2010-2012

We have used Kepler photometry to characterize variability in four radio-loud active galactic nuclei (three quasars and one object tentatively identified as a Seyfert 1.5 galaxy) on timescales from minutes to months, comparable to the light crossing time of the accretion disk around the central supermassive black hole or the base of the relativistic jet. Kepler's almost continuous observations provide much better temporal coverage than is possible from ground-based observations. We report the first such data analyzed for quasars. We have constructed power spectral densities using 8 Kepler quarters of long-cadence (30-minute) data for three AGN, 6 quarters for one AGN and 2 quarters of short-cadence (1-minute) data for all four AGN. On timescales longer than about 0.2-0.6 day, we find red noise with mean power-law slopes ranging from -1.8 to -1.2, consistent with the variability originating in turbulence either behind a shock or within an accretion disk. Each AGN has a range of red noise slopes which vary slightly by month and quarter of observation. No quasi-periodic oscillations of astrophysical origin were detected. We detected several days-long flares when brightness increased by 3% - 7% in two objects. No flares on timescales of minutes to hours were detected. Our observations imply that the duty cycle for enhanced activity in these radio-loud AGN is small. These well-sampled AGN light curves provide an impetus to develop more detailed models of turbulence in jets and instabilities in accretion disks.

astro-ph.HE↗

arXiv2012

Sparse Distributed Learning Based on Diffusion Adaptation

This article proposes diffusion LMS strategies for distributed estimation over adaptive networks that are able to exploit sparsity in the underlying system model. The approach relies on convex regularization, common in compressive sensing, to enhance the detection of sparsity via a diffusive process over the network. The resulting algorithms endow networks with learning abilities and allow them to learn the sparse structure from the incoming data in real-time, and also to track variations in the sparsity of the model. We provide convergence and mean-square performance analysis of the proposed method and show under what conditions it outperforms the unregularized diffusion version. We also show how to adaptively select the regularization parameter. Simulation results illustrate the advantage of the proposed filters for sparse data recovery.

cs.LG↗