arXiv subjects

Vahid Tarokh

Publications and source records attributed to Vahid Tarokh.

At least 145 records · Page 8Linked to original sources

arXiv2017

On Optimal Generalizability in Parametric Learning

We consider the parametric learning problem, where the objective of the learner is determined by a parametric loss function. Employing empirical risk minimization with possibly regularization, the inferred parameter vector will be biased toward the training samples. Such bias is measured by the cross validation procedure in practice where the data set is partitioned into a training set used for training and a validation set, which is not used in training and is left to measure the out-of-sample performance. A classical cross validation strategy is the leave-one-out cross validation (LOOCV) where one sample is left out for validation and training is done on the rest of the samples that are presented to the learner, and this process is repeated on all of the samples. LOOCV is rarely used in practice due to the high computational complexity. In this paper, we first develop a computationally efficient approximate LOOCV (ALOOCV) and provide theoretical guarantees for its performance. Then we use ALOOCV to provide an optimization algorithm for finding the regularizer in the empirical risk minimization framework. In our numerical experiments, we illustrate the accuracy and efficiency of ALOOCV as well as our proposed framework for the optimization of the regularizer.

stat.ML↗

arXiv2017

Convergence of Limited Communications Gradient Methods

Distributed optimization increasingly plays a central role in economical and sustainable operation of cyber-physical systems. Nevertheless, the complete potential of the technology has not yet been fully exploited in practice due to communication limitations posed by the real-world infrastructures. This work investigates fundamental properties of distributed optimization based on gradient methods, where gradient information is communicated using limited number of bits. In particular, a general class of quantized gradient methods are studied where the gradient direction is approximated by a finite quantization set. Sufficient and necessary conditions are provided on such a quantization set to guarantee that the methods minimize any convex objective function with Lipschitz continuous gradient and a nonempty and bounded set of optimizers. A lower bound on the cardinality of the quantization set is provided, along with specific examples of minimal quantizations. Convergence rate results are established that connect the fineness of the quantization and the number of iterations needed to reach a predefined solution accuracy. Generalizations of the results to a relevant class of constrained problems using projections are considered. Finally, the results are illustrated by simulations of practical systems.

math.OC↗

arXiv2017

On the Spectral Norms of Pseudo-Wigner and Related Matrices

We investigate the spectral norms of symmetric $N \times N$ matrices from two pseudo-random ensembles. The first is the pseudo-Wigner ensemble introduced in "Pseudo-Wigner Matrices" by Soloveychik, Xiang and Tarokh and the second is its sample covariance-type analog defined in this work. Both ensembles are defined through the concept of $r$-independence by controlling the amount of randomness in the underlying matrices, and can be constructed from dual BCH codes. We show that when the measure of randomness $r$ grows as $N^ρ$, where $ρ\in (0,1]$ and $\varepsilon > 0$, the norm of the matrices is almost surely within $o\left(\frac{\log^{1+\varepsilon} N}{N^{\min[ρ,2/3]}}\right)$ distance from $1$. Numerical simulations verifying the obtained results are provided.

math.PR↗

arXiv2017

Dictionary Learning and Sparse Coding-based Denoising for High-Resolution Task Functional Connectivity MRI Analysis

We propose a novel denoising framework for task functional Magnetic Resonance Imaging (tfMRI) data to delineate the high-resolution spatial pattern of the brain functional connectivity via dictionary learning and sparse coding (DLSC). In order to address the limitations of the unsupervised DLSC-based fMRI studies, we utilize the prior knowledge of task paradigm in the learning step to train a data-driven dictionary and to model the sparse representation. We apply the proposed DLSC-based method to Human Connectome Project (HCP) motor tfMRI dataset. Studies on the functional connectivity of cerebrocerebellar circuits in somatomotor networks show that the DLSC-based denoising framework can significantly improve the prominent connectivity patterns, in comparison to the temporal non-local means (tNLM)-based denoising method as well as the case without denoising, which is consistent and neuroscientifically meaningful within motor area. The promising results show that the proposed method can provide an important foundation for the high-resolution functional connectivity analysis, and provide a better approach for fMRI preprocessing.

cs.LG↗

arXiv2017

Nonlinear Sequential Accepts and Rejects for Identification of Top Arms in Stochastic Bandits

We address the M-best-arm identification problem in multi-armed bandits. A player has a limited budget to explore K arms (M<K), and once pulled, each arm yields a reward drawn (independently) from a fixed, unknown distribution. The goal is to find the top M arms in the sense of expected reward. We develop an algorithm which proceeds in rounds to deactivate arms iteratively. At each round, the budget is divided by a nonlinear function of remaining arms, and the arms are pulled correspondingly. Based on a decision rule, the deactivated arm at each round may be accepted or rejected. The algorithm outputs the accepted arms that should ideally be the top M arms. We characterize the decay rate of the misidentification probability and establish that the nonlinear budget allocation proves to be useful for different problem environments (described by the number of competitive arms). We provide comprehensive numerical experiments showing that our algorithm outperforms the state-of-the-art using suitable nonlinearity.

stat.ML↗

arXiv2017

A Note on the Concentration of Spectral Measure of Wigner's Matrices

In this short note we derive concentration inequalities for the empirical absolute moments of square symmetric matrices with independent symmetrically distributed +/-1 entries. Most of the previous results of this type are limited to functions with bounded Lipschitz constant, and therefore exclude the moments from consideration. Based on the fine analysis of the distribution of the largest eigenvalues developed by Soshnikov and Sinai, we extend the measure concentration tools of Talagrand to encompass power functions.

math.PR↗

arXiv2017

On Sequential Elimination Algorithms for Best-Arm Identification in Multi-Armed Bandits

We consider the best-arm identification problem in multi-armed bandits, which focuses purely on exploration. A player is given a fixed budget to explore a finite set of arms, and the rewards of each arm are drawn independently from a fixed, unknown distribution. The player aims to identify the arm with the largest expected reward. We propose a general framework to unify sequential elimination algorithms, where the arms are dismissed iteratively until a unique arm is left. Our analysis reveals a novel performance measure expressed in terms of the sampling mechanism and number of eliminated arms at each round. Based on this result, we develop an algorithm that divides the budget according to a nonlinear function of remaining arms at each round. We provide theoretical guarantees for the algorithm, characterizing the suitable nonlinearity for different problem environments described by the number of competitive arms. Matching the theoretical results, our experiments show that the nonlinear algorithm outperforms the state-of-the-art. We finally study the side-observation model, where pulling an arm reveals the rewards of its related arms, and we establish improved theoretical guarantees in the pure-exploration setting.

stat.ML↗

arXiv2017

The Number of Independent Sets in Hexagonal Graphs

A new method is proposed to derive rigorous bounds on η, the growth rate of the logarithm of the number of independent sets on a hexagonal lattice. Specifically, we prove that 1.546440708536001 <= η <= 1.5513, which improves upon the best known 1.5463 <= η <= 1.5527 due to Nagy and Zeger. Our lower bound matches the numerical estimate of Baxter up to 9 digits after the decimal point.

cs.IT↗

arXiv2017

Capacity of Hexagonal Checkerboard Codes

In this paper, we propose a new method to bound the capacity of checkerboard codes on the hexagonal lattice. This produces rigorous bounds that are tighter than those commonly known.

cs.IT↗

arXiv2016

SLANTS: Sequential Adaptive Nonlinear Modeling of Vector Time Series

We propose a method for adaptive nonlinear sequential modeling of vector-time series data. Data is modeled as a nonlinear function of past values corrupted by noise, and the underlying non-linear function is assumed to be approximately expandable in a spline basis. We cast the modeling of data as finding a good fit representation in the linear span of multi-dimensional spline basis, and use a variant of l1-penalty regularization in order to reduce the dimensionality of representation. Using adaptive filtering techniques, we design our online algorithm to automatically tune the underlying parameters based on the minimization of the regularized sequential prediction error. We demonstrate the generality and flexibility of the proposed approach on both synthetic and real-world datasets. Moreover, we analytically investigate the performance of our algorithm by obtaining both bounds of the prediction errors, and consistency results for variable selection.

stat.ME↗

arXiv2016

Bridging AIC and BIC: a new criterion for autoregression

We introduce a new criterion to determine the order of an autoregressive model fitted to time series data. It has the benefits of the two well-known model selection techniques, the Akaike information criterion and the Bayesian information criterion. When the data is generated from a finite order autoregression, the Bayesian information criterion is known to be consistent, and so is the new criterion. When the true order is infinity or suitably high with respect to the sample size, the Akaike information criterion is known to be efficient in the sense that its prediction performance is asymptotically equivalent to the best offered by the candidate models; in this case, the new criterion behaves in a similar manner. Different from the two classical criteria, the proposed criterion adaptively achieves either consistency or efficiency depending on the underlying true model. In practice where the observed time series is given without any prior information about the model specification, the proposed order selection criterion is more flexible and robust compared with classical approaches. Numerical results are presented demonstrating the adaptivity of the proposed technique when applied to various datasets.

math.ST↗

arXiv2016

Multiple Change Point Analysis: Fast Implementation And Strong Consistency

One of the main challenges in identifying structural changes in stochastic processes is to carry out analysis for time series with dependency structure in a computationally tractable way. Another challenge is that the number of true change points is usually unknown, requiring a suitable model selection criterion to arrive at informative conclusions. To address the first challenge, we model the data generating process as a segment-wise autoregression, which is composed of several segments (time epochs), each of which modeled by an autoregressive model. We propose a multi-window method that is both effective and efficient for discovering the structural changes. The proposed approach was motivated by transforming a segment-wise autoregression into a multivariate time series that is asymptotically segment-wise independent and identically distributed. To address the second challenge, we derive theoretical guarantees for (almost surely) selecting the true number of change points of segment-wise independent multivariate time series. Specifically, under mild assumptions, we show that a Bayesian Information Criterion (BIC)-like criterion gives a strongly consistent selection of the optimal number of change points, while an Akaike Information Criterion (AIC)-like criterion cannot. Finally, we demonstrate the theory and strength of the proposed algorithms by experiments on both synthetic and real-world data, including the Eastern US temperature data and the El Nino data from 1854 to 2015. The experiment leads to some interesting discoveries about temporal variability of the summer-time temperature over the Eastern US, and about the most dominant factor of ocean influence on climate.

stat.ME↗

arXiv2016

Distributed Resource Allocation Using One-Way Communication with Applications to Power Networks

Typical coordination schemes for future power grids require two-way communications. Since the number of end power-consuming devices is large, the bandwidth requirements for such two-way communication schemes may be prohibitive. Motivated by this observation, we study distributed coordination schemes that require only one-way limited communications. In particular, we investigate how dual descent distributed optimization algorithm can be employed in power networks using one-way communication. In this iterative algorithm, system coordinators broadcast coordinating (or pricing) signals to the users/devices who update power consumption based on the received signal. Then system coordinators update the coordinating signals based on the physical measurement of the aggregate power usage. We provide conditions to guarantee the feasibility of the aggregated power usage at each iteration so as to avoid blackout. Furthermore, we prove the convergence of algorithms under these conditions, and establish its rate of convergence. We illustrate the performance of our algorithms using numerical simulations. These results show that one-way limited communication may be viable for coordinating/operating the future smart grids.

eess.SY↗

arXiv2016

Robustness Analysis for an Online Decentralized Descent Power allocation algorithm

As independent service providers shift from conventional energy to renewable energy sources, the power distribution system will likely experience increasingly significant fluctuation in supply, given the uncertain and intermittent nature of renewable sources like wind and solar energy. These fluctuations in power generation, coupled with time-varying consumer demands of electricity and the massive scale of power distribution networks present the need to not only design real-time decentralized power allocation algorithms, but also characterize how effective they are given fast-changing consumer demands and power generation capacities. In this paper, we present an Online Decentralized Dual Descent (OD3) power allocation algorithm and determine (in the worst case) how much of observed social welfare and price volatility can be explained by fluctuations in generation capacity and consumer demand. Convergence properties and performance guarantees of the OD3 algorithm are analyzed by characterizing the difference between the online decision and the optimal decision. The theoretical results in the paper are validated and illustrated by numerical experiments using real data.

eess.SY↗

arXiv2015

Data-Driven Learning of the Number of States in Multi-State Autoregressive Models

In this work, we consider the class of multi-state autoregressive processes that can be used to model non-stationary time-series of interest. In order to capture different autoregressive (AR) states underlying an observed time series, it is crucial to select the appropriate number of states. We propose a new model selection technique based on the Gap statistics, which uses a null reference distribution on the stable AR filters to check whether adding a new AR state significantly improves the performance of the model. To that end, we define a new distance measure between AR filters based on mean squared prediction error (MSPE), and propose an efficient method to generate random stable filters that are uniformly distributed in the coefficient space. Numerical results are provided to evaluate the performance of the proposed approach.

stat.ML↗

arXiv2015

Comment on "Asymptotic Achievability of the Cramér-Rao Bound for Noisy Compressive Sampling"

In [1], we proved the asymptotic achievability of the Cramér-Rao bound in the compressive sensing setting in the linear sparsity regime. In the proof, we used an erroneous closed-form expression of $ασ^2$ for the genie-aided Cramér-Rao bound $σ^2 \textrm{Tr} (\mathbf{A}^*_\mathcal{I} \mathbf{A}_\mathcal{I})^{-1}$ from Lemma 3.5, which appears in Eqs. (20) and (29). The proof, however, holds if one avoids replacing $σ^2 \textrm{Tr} (\mathbf{A}^*_\mathcal{I} \mathbf{A}_\mathcal{I})^{-1}$ by the expression of Lemma 3.5, and hence the claim of the Main Theorem stands true. In Chapter 2 of the Ph. D. dissertation by Behtash Babadi [2], this error was fixed and a more detailed proof in the non-asymptotic regime was presented. A draft of Chapter 2 of [2] is included in this note, verbatim. We would like to refer the interested reader to the full dissertation, which is electronically archived in the ProQuest database [2], and a draft of which can be accessed through the author's homepage under: http://ece.umd.edu/~behtash/babadi_thesis_2011.pdf.

cs.IT↗

arXiv2015

Learning the Number of Autoregressive Mixtures in Time Series Using the Gap Statistics

Using a proper model to characterize a time series is crucial in making accurate predictions. In this work we use time-varying autoregressive process (TVAR) to describe non-stationary time series and model it as a mixture of multiple stable autoregressive (AR) processes. We introduce a new model selection technique based on Gap statistics to learn the appropriate number of AR filters needed to model a time series. We define a new distance measure between stable AR filters and draw a reference curve that is used to measure how much adding a new AR filter improves the performance of the model, and then choose the number of AR filters that has the maximum gap with the reference curve. To that end, we propose a new method in order to generate uniform random stable AR filters in root domain. Numerical results are provided demonstrating the performance of the proposed approach.

stat.ML↗

arXiv2015

Complementary Lattice Arrays for Coded Aperture Imaging

In this work, we consider complementary lattice arrays in order to enable a broader range of designs for coded aperture imaging systems. We provide a general framework and methods that generate richer and more flexible designs than existing ones. Besides this, we review and interpret the state-of-the-art uniformly redundant arrays (URA) designs, broaden the related concepts, and further propose some new design methods.

cs.IT↗