arXiv subjects

Bodhisattva Sen

Publications and source records attributed to Bodhisattva Sen.

At least 55 records · Page 3Linked to original sources

arXiv2018

A machine--vision method for automatic classification of stellar halo substructure

Tidal debris structures formed from disrupted satellites contain important clues about the assembly histories of galaxies. To date, studies of these structures have been hampered by reliance on by-eye identification and morphological classification which leaves their interpretation significantly uncertain. In this work we present a new machine-vision technique based on the Subspace-Constrained Mean Shift (SCMS) algorithm which can perform these tasks automatically. SCMS finds the location of the high-density `ridges' that define substructure morphology. After identification, the coefficients of an orthogonal series density estimator are used to classify points on the ridges as part of a continuum between shell-like or stream-like debris, from which a global morphological classification can be determined. We dub this procedure Subspace Constrained Unsupervised Detection of Structure (SCUDS). By applying this tool to controlled N--body simulations of minor mergers we demonstrate that the extracted classifications correspond to the well--understood underlying physics of phase mixing. The application of SCUDS to resolved stellar population data from near-future surveys will inform our understanding of the buildup of galaxies stellar halos.

astro-ph.GA↗

arXiv2018

Composite Difference-Max Programs for Modern Statistical Estimation Problems

Many modern statistical estimation problems are defined by three major components: a statistical model that postulates the dependence of an output variable on the input features; a loss function measuring the error between the observed output and the model predicted output; and a regularizer that controls the overfitting and/or variable selection in the model. We study the sampling version of this generic statistical estimation problem where the model parameters are estimated by empirical risk minimization, which involves the minimization of the empirical average of the loss function at the data points weighted by the model regularizer. In our setup we allow all three component functions discussed above to be of the difference-of-convex (dc) type and illustrate them with a host of commonly used examples, including those in continuous piecewise affine regression and in deep learning (where the activation functions are piecewise affine). We describe a nonmonotone majorization-minimization (MM) algorithm for solving the unified nonconvex, nondifferentiable optimization problem which is formulated as a specially structured composite dc program of the pointwise max type, and present convergence results to a directional stationary solution. An efficient semismooth Newton method is proposed to solve the dual of the MM subproblems. Numerical results are presented to demonstrate the effectiveness of the proposed algorithm and the superiority of continuous piecewise affine regression over the standard linear model.

math.OC↗

arXiv2018

On Degrees of Freedom of Projection Estimators with Applications to Multivariate Nonparametric Regression

In this paper, we consider the nonparametric regression problem with multivariate predictors. We provide a characterization of the degrees of freedom and divergence for estimators of the unknown regression function, which are obtained as outputs of linearly constrained quadratic optimization procedures, namely, minimizers of the least squares criterion with linear constraints and/or quadratic penalties. As special cases of our results, we derive explicit expressions for the degrees of freedom in many nonparametric regression problems, e.g., bounded isotonic regression, multivariate (penalized) convex regression, and additive total variation regularization. Our theory also yields, as special cases, known results on the degrees of freedom of many well-studied estimators in the statistics literature, such as ridge regression, Lasso and generalized Lasso. Our results can be readily used to choose the tuning parameter(s) involved in the estimation procedure by minimizing the Stein's unbiased risk estimate. As a by-product of our analysis we derive an interesting connection between bounded isotonic regression and isotonic regression on a general partially ordered set, which is of independent interest.

math.ST↗

arXiv2018

Nonparametric Shape-restricted Regression

We consider the problem of nonparametric regression under shape constraints. The main examples include isotonic regression (with respect to any partial order), unimodal/convex regression, additive shape-restricted regression, and constrained single index model. We review some of the theoretical properties of the least squares estimator (LSE) in these problems, emphasizing on the adaptive nature of the LSE. In particular, we study the behavior of the risk of the LSE, and its pointwise limiting distribution theory, with special emphasis to isotonic regression. We survey various methods for constructing pointwise confidence intervals around these shape-restricted functions. We also briefly discuss the computation of the LSE and indicate some open research problems and future directions.

math.ST↗

arXiv2018

Adaptive Risk Bounds in Univariate Total Variation Denoising and Trend Filtering

We study trend filtering, a relatively recent method for univariate nonparametric regression. For a given positive integer $r$, the $r$-th order trend filtering estimator is defined as the minimizer of the sum of squared errors when we constrain (or penalize) the sum of the absolute $r$-th order discrete derivatives of the fitted function at the design points. For $r=1$, the estimator reduces to total variation regularization which has received much attention in the statistics and image processing literature. In this paper, we study the performance of the trend filtering estimator for every positive integer $r$, both in the constrained and penalized forms. Our main results show that in the strong sparsity setting when the underlying function is a (discrete) spline with few "knots", the risk (under the global squared error loss) of the trend filtering estimator (with an appropriate choice of the tuning parameter) achieves the parametric $n^{-1}$ rate, up to a logarithmic (multiplicative) factor. Our results therefore provide support for the use of trend filtering, for every $r$, in the strong sparsity setting.

math.ST↗

arXiv2018

Optimal Inference with a Multidimensional Multiscale Statistic

We observe a stochastic process $Y$ on $[0,1]^d$ ($d\geq 1$) satisfying $dY(t)=n^{1/2}f(t)dt$ + $dW(t)$, $t \in [0,1]^d$, where $n \geq 1$ is a given scale parameter (`sample size'), $W$ is the standard Brownian sheet on $[0,1]^d$ and $f \in L_1([0,1]^d)$ is the unknown function of interest. We propose a multivariate multiscale statistic in this setting and prove its almost sure finiteness; this extends the work of Dümbgen and Spokoiny (2001) who proposed the analogous statistic for $d=1$. We use the proposed multiscale statistic to construct optimal tests for testing $f=0$ versus (i) appropriate Hölder classes of functions, and (ii) alternatives of the form $f=μ_n \mathbb{I}_{B_n}$, where $B_n$ is an axis-aligned hyperrectangle in $[0,1]^d$ and $μ_n \in \mathbb{R}$; $μ_n$ and $B_n$ unknown. In the process we generalize Theorem 6.1 of Dümbgen and Spokoiny (2001) about stochastic processes with sub-Gaussian increments on a pseudometric space, which is of independent interest.

math.ST↗

arXiv2016

Divide and Conquer in Non-standard Problems and the Super-efficiency Phenomenon

We study how the divide and conquer principle --- partition the available data into subsamples, compute an estimate from each subsample and combine these appropriately to form the final estimator --- works in non-standard problems where rates of convergence are typically slower than $\sqrt{n}$ and limit distributions are non-Gaussian, with a special emphasis on the least squares estimator (and its inverse) of a monotone regression function. We find that the pooled estimator, obtained by averaging non-standard estimates across the mutually exclusive subsamples, outperforms the non-standard estimator based on the entire sample in the sense of pointwise inference. We also show that, under appropriate conditions, if the number of subsamples is allowed to increase at appropriate rates, the pooled estimator is asymptotically normally distributed with a variance that is empirically estimable from the subsample-level estimates. Further, in the context of monotone function estimation we show that this gain in pointwise efficiency comes at a price --- the pooled estimator's performance, in a uniform sense (maximal risk) over a class of models worsens as the number of subsamples increases, leading to a version of the super-efficiency phenomenon. In the process, we develop analytical results for the order of the bias in isotonic regression, which are of independent interest.

math.ST↗

arXiv2016

On Univariate Convex Regression

We find the local rate of convergence of the least squares estimator (LSE) of a one dimensional convex regression function when (a) a certain number of derivatives vanish at the point of interest, and (b) the true regression function is locally affine. In each case we derive the limiting distribution of the LSE and its derivative. The pointwise limiting distributions depend on the second and third derivatives at 0 of the "invelope function" of the integral of a two-sided Brownian motion with polynomial drifts. We also investigate the inconsistency of the LSE and the unboundedness of its derivative at the boundary of the domain of the covariate space. An estimator of the argmin of the convex regression function is proposed and its asymptotic distribution is derived. Further, we present some new results on the characterization of the convex LSE that may be of independent interest.

stat.ME↗

arXiv2015

A Consistent Bootstrap Procedure for the Maximum Score Estimator

In this paper we study the applicability of the bootstrap to do inference on Manski's maximum score estimator under the full generality of the model. We propose three new, model-based bootstrap procedures for this problem and show their consistency. Simulation experiments are carried out to evaluate their performance and to compare them with subsampling methods. Additionally, we prove a uniform convergence theorem for triangular arrays of random variables coming from binary choice models, which may be of independent interest.

stat.AP↗

arXiv2015

Estimation of a Two-component Mixture Model with Applications to Multiple Testing

We consider a two-component mixture model with one known component. We develop methods for estimating the mixing proportion and the unknown distribution nonparametrically, given i.i.d.~data from the mixture model, using ideas from shape restricted function estimation. We establish the consistency of our estimators. We find the rate of convergence and asymptotic limit of the estimator for the mixing proportion. Completely automated distribution-free honest finite sample lower confidence bounds are developed for the mixing proportion. Connection to the problem of multiple testing is discussed. The identifiability of the model, and the estimation of the density of the unknown distribution are also addressed. We compare the proposed estimators, which are easily implementable, with some of the existing procedures through simulation studies and analyse two data sets, one arising from an application in astronomy and the other from a microarray experiment.

stat.ME↗

arXiv2015

On matrix estimation under monotonicity constraints

We consider the problem of estimating an unknown $n_1 \times n_2$ matrix $\mathbf{θ^*}$ from noisy observations under the constraint that $\mathbfθ^*$ is nondecreasing in both rows and columns. We consider the least squares estimator (LSE) in this setting and study its risk properties. We show that the worst case risk of the LSE is $n^{-1/2}$, up to multiplicative logarithmic factors, where $n = n_1 n_2$ and that the LSE is minimax rate optimal (up to logarithmic factors). We further prove that for some special $\mathbfθ^*$, the risk of the LSE could be much smaller than $n^{-1/2}$; in fact, it could even be parametric i.e., $n^{-1}$ up to logarithmic factors. Such parametric rates occur when the number of "rectangular" blocks of $\mathbfθ^*$ is bounded from above by a constant. We derive, as a consequence, an interesting adaptation property of the LSE which we term variable adaptation -- the LSE performs as well as the oracle estimator when estimating a matrix that is constant along each row/column. Our proofs borrow ideas from empirical process theory and convex geometry and are of independent interest.

math.ST↗

arXiv2015

On a Nonparametric Notion of Residual and its Applications

Let $(X, \mathbf{Z})$ be a continuous random vector in $\mathbb{R} \times \mathbb{R}^d$, $d \ge 1$. In this paper, we define the notion of a nonparametric residual of $X$ on $\mathbf{Z}$ that is always independent of the predictor $\mathbf{Z}$. We study its properties and show that the proposed notion of residual matches with the usual residual (error) in a multivariate normal regression model. Given a random vector $(X, Y, \mathbf{Z})$ in $\mathbb{R} \times \mathbb{R} \times \mathbb{R}^d$, we use this notion of residual to show that the conditional independence between $X$ and $Y$, given $\mathbf{Z}$, is equivalent to the mutual independence of the residuals (of $X$ on $\mathbf{Z}$ and $Y$ on $\mathbf{Z}$) and $\mathbf{Z}$. This result is used to develop a test for conditional independence. We propose a bootstrap scheme to approximate the critical value of this test. We compare the proposed test, which is easily implementable, with some of the existing procedures through a simulation study.

stat.ME↗

arXiv2015

A Computational Framework for Multivariate Convex Regression and its Variants

We study the nonparametric least squares estimator (LSE) of a multivariate convex regression function. The LSE, given as the solution to a quadratic program with $O(n^2)$ linear constraints ($n$ being the sample size), is difficult to compute for large problems. Exploiting problem specific structure, we propose a scalable algorithmic framework based on the augmented Lagrangian method to compute the LSE. We develop a novel approach to obtain smooth convex approximations to the fitted (piecewise affine) convex LSE and provide formal bounds on the quality of approximation. When the number of samples is not too large compared to the dimension of the predictor, we propose a regularization scheme --- Lipschitz convex regression --- where we constrain the norm of the subgradients, and study the rates of convergence of the obtained LSE. Our algorithmic framework is simple and flexible and can be easily adapted to handle variants: estimation of a non-decreasing/non-increasing convex/concave (with or without a Lipschitz bound) function. We perform numerical studies illustrating the scalability of the proposed algorithm.

stat.CO↗

arXiv2015

On risk bounds in isotonic and other shape restricted regression problems

We consider the problem of estimating an unknown $θ\in {\mathbb{R}}^n$ from noisy observations under the constraint that $θ$ belongs to certain convex polyhedral cones in ${\mathbb{R}}^n$. Under this setting, we prove bounds for the risk of the least squares estimator (LSE). The obtained risk bound behaves differently depending on the true sequence $θ$ which highlights the adaptive behavior of $θ$. As special cases of our general result, we derive risk bounds for the LSE in univariate isotonic and convex regression. We study the risk bound in isotonic regression in greater detail: we show that the isotonic LSE converges at a whole range of rates from $\log n/n$ (when $θ$ is constant) to $n^{-2/3}$ (when $θ$ is uniformly increasing in a certain sense). We argue that the bound presents a benchmark for the risk of any estimator in isotonic regression by proving nonasymptotic local minimax lower bounds. We prove an analogue of our bound for model misspecification where the true $θ$ is not necessarily nondecreasing.

math.ST↗

arXiv2015

Reconstructing the Accretion History of the Galactic Stellar Halo from Chemical Abundance Ratio Distributions

Observational studies of halo stars during the last two decades have placed some limits on the quantity and nature of accreted dwarf galaxy contributions to the Milky Way stellar halo by typically utilizing stellar phase-space information to identify the most recent halo accretion events. In this study we tested the prospects of using 2-D chemical abundance ratio distributions (CARDs) found in stars of the stellar halo to determine its formation history. First, we used simulated data from eleven "MW-like" halos to generate satellite template sets of 2-D CARDs of accreted dwarf satellites which are comprised of accreted dwarfs from various mass regimes and epochs of accretion. Next, we randomly drew samples of $\sim10^{3-4}$ mock observations of stellar chemical abundance ratios ([$α$/Fe], [Fe/H]) from those eleven halos to generate samples of the underlying densities for our CARDs to be compared to our templates in our analysis. Finally, we used the expectation-maximization algorithm to derive accretion histories in relation to the satellite template set (STS) used and the sample size. For certain STS used we typically can identify the relative mass contributions of all accreted satellites to within a factor of 2. We also find that this method is particularly sensitive to older accretion events involving low-luminous dwarfs e.g. ultra-faint dwarfs - precisely those events that are too ancient to be seen by phase-space studies of stars and too faint to be seen by high-z studies of the early Universe. Since our results only exploit two chemical dimensions and near-future surveys promise to provide $\sim6-9$ dimensions, we conclude that these new high-resolution spectroscopic surveys of the stellar halo will allow us to recover its accretion history - and the luminosity function of infalling dwarf galaxies - across cosmic time.

astro-ph.GA↗

arXiv2014

Global risk bounds and adaptation in univariate convex regression

We consider the problem of nonparametric estimation of a convex regression function $ϕ_0$. We study the risk of the least squares estimator (LSE) under the natural squared error loss. We show that the risk is always bounded from above by $n^{-4/5}$ modulo logarithmic factors while being much smaller when $ϕ_0$ is well-approximable by a piecewise affine convex function with not too many affine pieces (in which case, the risk is at most $1/n$ up to logarithmic factors). On the other hand, when $ϕ_0$ has curvature, we show that no estimator can have risk smaller than a constant multiple of $n^{-4/5}$ in a very strong sense by proving a "local" minimax lower bound. We also study the case of model misspecification where we show that the LSE exhibits the same global behavior provided the loss is measured from the closest convex projection of the true regression function. In the process of deriving our risk bounds, we prove new results for the metric entropy of local neighborhoods of the space of univariate convex functions. These results, which may be of independent interest, demonstrate the non-uniform nature of the space of univariate convex functions in sharp contrast to classical function spaces based on smoothness constraints.

math.ST↗

arXiv2014

Testing against a linear regression model using ideas from shape-restricted estimation

A formal likelihood ratio hypothesis test for the validity of a parametric regression function is proposed, using a large-dimensional, nonparametric double cone alternative. For example, the test against a constant function uses the alternative of increasing or decreasing regression functions, and the test against a linear function uses the convex or concave alternative. The proposed test is exact, unbiased and the critical value is easily computed. The power of the test increases to one as the sample size increases, under very mild assumptions -- even when the alternative is mis-specified. That is, the power of the test converges to one for any true regression function that deviates (in a non-degenerate way) from the parametric null hypothesis. We also formulate tests for the linear versus partial linear model, and consider the special case of the additive model. Simulations show that our procedure behaves well consistently when compared with other methods. Although the alternative fit is non-parametric, no tuning parameters are involved.

stat.ME↗

arXiv2014

On Testing Independence and Goodness-of-fit in Linear Models

We consider a linear regression model and propose an omnibus test to simultaneously check the assumption of independence between the error and the predictor variables, and the goodness-of-fit of the parametric model. Our approach is based on testing for independence between the residual obtained from the parametric fit and the predictor using the Hilbert--Schmidt independence criterion (Gretton et al. (2008)). The proposed method requires no user-defined regularization, is simple to compute, based merely on pairwise distances between points in the sample, and is consistent against all alternatives. We develop distribution theory for the proposed test statistic, both under the null and the alternative hypotheses, and devise a bootstrap scheme to approximate its null distribution. We prove the consistency of the bootstrap scheme. A simulation study shows that our method has better power than its main competitors. Two real datasets are analyzed to demonstrate the scope and usefulness of our method.

stat.ME↗