arXiv subjects

Arindam Banerjee

Publications and source records attributed to Arindam Banerjee.

At least 55 records · Page 3Linked to original sources

arXiv2021

Learning and Dynamical Models for Sub-seasonal Climate Forecasting: Comparison and Collaboration

Sub-seasonal climate forecasting (SSF) is the prediction of key climate variables such as temperature and precipitation on the 2-week to 2-month time horizon. Skillful SSF would have substantial societal value in areas such as agricultural productivity, hydrology and water resource management, and emergency planning for extreme events such as droughts and wildfires. Despite its societal importance, SSF has stayed a challenging problem compared to both short-term weather forecasting and long-term seasonal forecasting. Recent studies have shown the potential of machine learning (ML) models to advance SSF. In this paper, for the first time, we perform a fine-grained comparison of a suite of modern ML models with start-of-the-art physics-based dynamical models from the Subseasonal Experiment (SubX) project for SSF in the western contiguous United States. Additionally, we explore mechanisms to enhance the ML models by using forecasts from dynamical models. Empirical results illustrate that, on average, ML models outperform dynamical models while the ML models tend to be conservatives in their forecasts compared to the SubX models. Further, we illustrate that ML models make forecasting errors under extreme weather conditions, e.g., cold waves due to the polar vortex, highlighting the need for separate models for extreme events. Finally, we show that suitably incorporating dynamical model forecasts as inputs to ML models can substantially improve the forecasting performance of the ML models. The SSF dataset constructed for the work, dynamical model predictions, and code for the ML models are released along with the paper for the benefit of the broader machine learning community.

physics.ao-ph↗

arXiv2021

Ordinary and symbolic powers of edge ideals of weighted oriented graphs

Let $\mathcal{D}$ be a weighted oriented graph and $I(\mathcal{D})$ be its edge ideal. In this paper, we show that all the symbolic and ordinary powers of $I(\mathcal{D})$ coincide when $\mathcal{D}$ is a weighted oriented certain class of tree. Finally, we give necessary and sufficient conditions for the equality of ordinary and symbolic powers of naturally oriented lines.

math.AC↗

arXiv2021

Edge ideals of Erdös-Rényi random graphs : Linear resolution, unmixedness and regularity

We study the homological algebra of edge ideals of Erdös-Rényi random graphs. These random graphs are generated by deleting edges of a complete graph on $n$ vertices independently of each other with probability $1-p$. We focus on some aspects of these random edge ideals - linear resolution, unmixedness and algebraic invariants like the Castelnuovo-Mumford regularity, projective dimension and depth. We first show a double phase transition for existence of linear presentation and resolution and determine the critical windows as well. As a consequence, we obtain that except for a very specific choice of parameters (i.e., $n,p := p(n)$), with high probability, a random edge ideal has linear presentation if and only if it has linear resolution. This shows certain conjectures hold true for large random graphs with high probability even though the conjectures were shown to fail for determinstic graphs. Next, we study asymptotic behaviour of some algebraic invariants - the Castelnuovo-Mumford regularity, projective dimension and depth - of such random edge ideals in the sparse regime (i.e., $p = \fracλ{n}, λ\in (0,\infty)$). These invariants are studied using local weak convergence (or Benjamini-Schramm convergence) and relating them to invariants on Galton-Watson trees. We also show that when $p \to 0$ or $p \to 1$ fast enough, then with high probability the edge ideals are unmixed and for most other choices of $p$, these ideals are not unmixed with high probability. This is further progress towards the conjecture that random monomial ideals are unlikely to have Cohen-Macaulay property (see De Loera et al. 2019a,2019b) in the setting when the number of variables goes to infinity but the degree is fixed.

math.CO↗

arXiv2021

Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identification

Differentially private SGD (DP-SGD) is one of the most popular methods for solving differentially private empirical risk minimization (ERM). Due to its noisy perturbation on each gradient update, the error rate of DP-SGD scales with the ambient dimension $p$, the number of parameters in the model. Such dependence can be problematic for over-parameterized models where $p \gg n$, the number of training samples. Existing lower bounds on private ERM show that such dependence on $p$ is inevitable in the worst case. In this paper, we circumvent the dependence on the ambient dimension by leveraging a low-dimensional structure of gradient space in deep networks -- that is, the stochastic gradients for deep nets usually stay in a low dimensional subspace in the training process. We propose Projected DP-SGD that performs noise reduction by projecting the noisy gradients to a low-dimensional subspace, which is given by the top gradient eigenspace on a small public dataset. We provide a general sample complexity analysis on the public dataset for the gradient subspace identification problem and demonstrate that under certain low-dimensional assumptions the public sample complexity only grows logarithmically in $p$. Finally, we provide a theoretical analysis and empirical evaluations to show that our method can substantially improve the accuracy of DP-SGD in the high privacy regime (corresponding to low privacy loss $ε$).

cs.LG↗

arXiv2021

Generalized Hilbert-Kunz function of the Rees algebra of the face ring of a simplicial complex

Let $R$ be the face ring of a simplicial complex of dimension $d-1$ and ${\mathcal R}(\mathfrak{n})$ be the Rees algebra of the maximal homogeneous ideal $\mathfrak{n}$ of $R.$ We show that the generalized Hilbert-Kunz function $HK(s)=\ell({\mathcal R}(\mathfrak n)/(\mathfrak n, \mathfrak n t)^{[s]})$ is given by a polynomial for all large $s.$ We calculate it in many examples and also provide a Macaulay2 code for computing $HK(s).$

math.AC↗

arXiv2021

Experiments with Rich Regime Training for Deep Learning

In spite of advances in understanding lazy training, recent work attributes the practical success of deep learning to the rich regime with complex inductive bias. In this paper, we study rich regime training empirically with benchmark datasets, and find that while most parameters are lazy, there is always a small number of active parameters which change quite a bit during training. We show that re-initializing (resetting to their initial random values) the active parameters leads to worse generalization. Further, we show that most of the active parameters are in the bottom layers, close to the input, especially as the networks become wider. Based on such observations, we study static Layer-Wise Sparse (LWS) SGD, which only updates some subsets of layers. We find that only updating the top and bottom layers have good generalization and, as expected, only updating the top layers yields a fast algorithm. Inspired by this, we investigate probabilistic LWS-SGD, which mostly updates the top layers and occasionally updates the full network. We show that probabilistic LWS-SGD matches the generalization performance of vanilla SGD and the back-propagation time can be 2-5 times more efficient.

cs.LG↗

arXiv2021

Packing properties of cubic squarefree monomial ideals

The symbolic powers, in general, are not equal to the ordinary powers. Therefore, one interesting question here is for what classes of ideals ordinary and symbolic powers coincide? The answer to this question for squarefree monomial ideals may be packing property. In this paper, we classify all cubic path ideals for those the symbolic and ordinary powers coincide.

math.AC↗

arXiv2020

Practical Optimal Control of a Wave-Energy Converter in Regular Wave Environments

A generic formulation for the optimal control of a single wave-energy converter (WEC) is proposed. The formulation involves hard and soft constraints on the motion of the WEC to promote reduced damage and fatigue to the device during operation. Most of the WEC control literature ignores the cost of the control and could therefore result in generating less power than expected, or even negative power. Therefore, to ensure actual power gains in practice, we incorporate a penalty term in the objective function to approximate the cost of applying the control force. A discretization of the resulting optimal control problem is a quadratic optimization problem that can be solved efficiently using state-of-the-art solvers. Using hydrodynamic coefficients estimated by simulations made in WEC-Sim, numerical illustrations are provided of the trade-off between careful operation of the device and power generated. Finally, a demonstration of the real-time use of the approach is provided.

math.OC↗

arXiv2020

De-randomized PAC-Bayes Margin Bounds: Applications to Non-convex and Non-smooth Predictors

In spite of several notable efforts, explaining the generalization of deterministic non-smooth deep nets, e.g., ReLU-nets, has remained challenging. Existing approaches for deterministic non-smooth deep nets typically need to bound the Lipschitz constant of such deep nets but such bounds are quite large, may even increase with the training set size yielding vacuous generalization bounds. In this paper, we present a new family of de-randomized PAC-Bayes margin bounds for deterministic non-convex and non-smooth predictors, e.g., ReLU-nets. Unlike PAC-Bayes, which applies to Bayesian predictors, the de-randomized bounds apply to deterministic predictors like ReLU-nets. A specific instantiation of the bound depends on a trade-off between the (weighted) distance of the trained weights from the initialization and the effective curvature (`flatness') of the trained predictor. To get to these bounds, we first develop a de-randomization argument for non-convex but smooth predictors, e.g., linear deep networks (LDNs), which connects the performance of the deterministic predictor with a Bayesian predictor. We then consider non-smooth predictors which for any given input realized as a smooth predictor, e.g., ReLU-nets become some LDNs for any given input, but the realized smooth predictors can be different for different inputs. For such non-smooth predictors, we introduce a new PAC-Bayes analysis which takes advantage of the smoothness of the realized predictors, e.g., LDN, for a given input, and avoids dependency on the Lipschitz constant of the non-smooth predictor. After careful de-randomization, we get a bound for the deterministic non-smooth predictor. We also establish non-uniform sample complexity results based on such bounds. Finally, we present extensive empirical results of our bounds over changing training set size and randomness in labels.

cs.LG↗

arXiv2020

Gradient Boosted Normalizing Flows

By chaining a sequence of differentiable invertible transformations, normalizing flows (NF) provide an expressive method of posterior approximation, exact density evaluation, and sampling. The trend in normalizing flow literature has been to devise deeper, more complex transformations to achieve greater flexibility. We propose an alternative: Gradient Boosted Normalizing Flows (GBNF) model a density by successively adding new NF components with gradient boosting. Under the boosting framework, each new NF component optimizes a sample weighted likelihood objective, resulting in new components that are fit to the residuals of the previously trained components. The GBNF formulation results in a mixture model structure, whose flexibility increases as more components are added. Moreover, GBNFs offer a wider, as opposed to strictly deeper, approach that improves existing NFs at the cost of additional training---not more complex transformations. We demonstrate the effectiveness of this technique for density estimation and, by coupling GBNF with a variational autoencoder, generative modeling of images. Our results show that GBNFs outperform their non-boosted analog, and, in some cases, produce better results with smaller, simpler flows.

cs.LG↗

arXiv2020

Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Bounds

We study differentially private (DP) algorithms for stochastic non-convex optimization. In this problem, the goal is to minimize the population loss over a $p$-dimensional space given $n$ i.i.d. samples drawn from a distribution. We improve upon the population gradient bound of ${\sqrt{p}}/{\sqrt{n}}$ from prior work and obtain a sharper rate of $\sqrt[4]{p}/\sqrt{n}$. We obtain this rate by providing the first analyses on a collection of private gradient-based methods, including adaptive algorithms DP RMSProp and DP Adam. Our proof technique leverages the connection between differential privacy and adaptive data analysis to bound gradient estimation error at every iterate, which circumvents the worse generalization bound from the standard uniform convergence argument. Finally, we evaluate the proposed algorithms on two popular deep learning tasks and demonstrate the empirical advantages of DP adaptive gradient methods over standard DP SGD.

cs.LG↗

arXiv2020

Sub-Seasonal Climate Forecasting via Machine Learning: Challenges, Analysis, and Advances

Sub-seasonal climate forecasting (SSF) focuses on predicting key climate variables such as temperature and precipitation in the 2-week to 2-month time scales. Skillful SSF would have immense societal value, in areas such as agricultural productivity, water resource management, transportation and aviation systems, and emergency planning for extreme weather events. However, SSF is considered more challenging than either weather prediction or even seasonal prediction. In this paper, we carefully study a variety of machine learning (ML) approaches for SSF over the US mainland. While atmosphere-land-ocean couplings and the limited amount of good quality data makes it hard to apply black-box ML naively, we show that with carefully constructed feature representations, even linear regression models, e.g., Lasso, can be made to perform well. Among a broad suite of 10 ML approaches considered, gradient boosting performs the best, and deep learning (DL) methods show some promise with careful architecture choices. Overall, suitable ML methods are able to outperform the climatological baseline, i.e., predictions based on the 30-year average at a given location and time. Further, based on studying feature importance, ocean (especially indices based on climatic oscillations such as El Nino) and land (soil moisture) covariates are found to be predictive, whereas atmospheric covariates are not considered helpful.

cs.LG↗

arXiv2020

Variable-density buoyancy-driven turbulence with asymmetric initial density distribution

The effects of different initial density distributions on the evolution of buoyancy-driven homogeneous variable-density turbulence (HVDT) at low (0.05) and high (0.75) Atwood numbers are studied by using high-resolution direct numerical simulations. HVDT aims to mimic the acceleration-driven Rayleigh-Taylor and shock-driven Richtmyer-Meshkov instabilities and reveals new physics that arise from variable-density effects on the turbulent mixing. Here, the initial amounts of pure light and pure heavy flows are altered primarily to mimic the variable-density turbulence at the different locations of the Rayleigh-Taylor and Richtmyer-Meshkov instabilities' mixing layers where the amounts of the mixing fluids are not equal. It is found that for the low Atwood number cases, the asymmetric initial density distribution has limited effects on both global and local flow evolution for HVDT. However, at high Atwood number, both global flow evolution and the local flow structures are strongly affected by the initial composition ratio. The flow composed of more light fluid reaches higher turbulent levels and the local statistics reach their fully-developed behavior earlier in the time evolution. During the late time decay, where most of the flow is well-mixed, all parameters become independent of the initial composition ratio for both low and high Atwood number cases.

physics.flu-dyn↗

arXiv2020

Atwood and Reynolds numbers effects on the evolution of buoyancy-driven homogeneous variable-density turbulence

The evolution of buoyancy-driven homogeneous variable-density turbulence (HVDT) at Atwood numbers up to 0.75 and large Reynolds numbers is studied by using high-resolution Direct Numerical Simulations. To help understand the highly non-equilibrium nature of buoyancy-driven HVDT, the flow evolution is divided into four different regimes based on the behavior of turbulent kinetic energy derivatives. The results show that each regime has a unique type of dependency on both Atwood and Reynolds numbers. It is found that the local statistics of the flow based on the flow composition are more sensitive to Atwood and Reynolds numbers compared to those based on the entire flow. It is also observed that at higher Atwood numbers, different flow features reach their asymptotic Reynolds number behavior at different times. The energy spectrum defined based on the Favre fluctuations momentum has less large scale contamination from viscous effects for variable density flows with constant properties, compared to other forms used previously. The evolution of the energy spectrum highlights distinct dynamical features of the four flow regimes. Thus, the slope of the energy spectrum at intermediate to large scales evolves from -7/3 to -1, as a function of the production to dissipation ratio. The classical Kolmogorov spectrum emerges at intermediate to high scales at the highest Reynolds numbers examined, after the turbulence starts to decay. Finally, the similarities and differences between buoyancy-driven HVDT and the more conventional stationary turbulence are discussed and new strategies and tools for analysis are proposed.

physics.flu-dyn↗

arXiv2020

Structured Linear Contextual Bandits: A Sharp and Geometric Smoothed Analysis

Bandit learning algorithms typically involve the balance of exploration and exploitation. However, in many practical applications, worst-case scenarios needing systematic exploration are seldom encountered. In this work, we consider a smoothed setting for structured linear contextual bandits where the adversarial contexts are perturbed by Gaussian noise and the unknown parameter $θ^*$ has structure, e.g., sparsity, group sparsity, low rank, etc. We propose simple greedy algorithms for both the single- and multi-parameter (i.e., different parameter for each context) settings and provide a unified regret analysis for $θ^*$ with any assumed structure. The regret bounds are expressed in terms of geometric quantities such as Gaussian widths associated with the structure of $θ^*$. We also obtain sharper regret bounds compared to earlier work for the unstructured $θ^*$ setting as a consequence of our improved analysis. We show there is implicit exploration in the smoothed setting where a simple greedy algorithm works.

cs.LG↗

arXiv2019

Random Quadratic Forms with Dependence: Applications to Restricted Isometry and Beyond

Several important families of computational and statistical results in machine learning and randomized algorithms rely on uniform bounds on quadratic forms of random vectors or matrices. Such results include the Johnson-Lindenstrauss (J-L) Lemma, the Restricted Isometry Property (RIP), randomized sketching algorithms, and approximate linear algebra. The existing results critically depend on statistical independence, e.g., independent entries for random vectors, independent rows for random matrices, etc., which prevent their usage in dependent or adaptive modeling settings. In this paper, we show that such independence is in fact not needed for such results which continue to hold under fairly general dependence structures. In particular, we present uniform bounds on random quadratic forms of stochastic processes which are conditionally independent and sub-Gaussian given another (latent) process. Our setup allows general dependencies of the stochastic process on the history of the latent process and the latent process to be influenced by realizations of the stochastic process. The results are thus applicable to adaptive modeling settings and also allows for sequential design of random vectors and matrices. We also discuss stochastic process based forms of J-L, RIP, and sketching, to illustrate the generality of the results.

cs.LG↗

arXiv2019

Regularity of Edge Ideals Via Suspension

We study the Castelnuovo-Mumford regularity of powers of edge ideals. We prove that if G is a bipartite graph, then reg(I(G)^s) \leq 2s + reg I(G) - 2 for all s \geq 2, which is the best possible upper bound for any s. Suspension plays a key role in proof of the base case s =2.

math.AC↗

arXiv2019

Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization

While stochastic gradient descent (SGD) and variants have been surprisingly successful for training deep nets, several aspects of the optimization dynamics and generalization are still not well understood. In this paper, we present new empirical observations and theoretical results on both the optimization dynamics and generalization behavior of SGD for deep nets based on the Hessian of the training loss and associated quantities. We consider three specific research questions: (1) what is the relationship between the Hessian of the loss and the second moment of stochastic gradients (SGs)? (2) how can we characterize the stochastic optimization dynamics of SGD with fixed and adaptive step sizes and diagonal pre-conditioning based on the first and second moments of SGs? and (3) how can we characterize a scale-invariant generalization bound of deep nets based on the Hessian of the loss, which by itself is not scale invariant? We shed light on these three questions using theoretical results supported by extensive empirical observations, with experiments on synthetic data, MNIST, and CIFAR-10, with different batch sizes, and with different difficulty levels by synthetically adding random labels.

cs.LG↗