Searcharxiv⌕ Search

arXiv subjects

Wei Deng

Publications and source records attributed to Wei Deng.

At least 91 records · Page 5Linked to original sources
arXiv2022

Non-reversible Parallel Tempering for Deep Posterior Approximation

Parallel tempering (PT), also known as replica exchange, is the go-to workhorse for simulations of multi-modal distributions. The key to the success of PT is to adopt efficient swap schemes. The popular deterministic even-odd (DEO) scheme exploits the non-reversibility property and has successfully reduced the communication cost from $O(P^2)$ to $O(P)$ given sufficiently many $P$ chains. However, such an innovation largely disappears in big data due to the limited chains and few bias-corrected swaps. To handle this issue, we generalize the DEO scheme to promote non-reversibility and propose a few solutions to tackle the underlying bias caused by the geometric stopping time. Notably, in big data scenarios, we obtain an appealing communication cost $O(P\log P)$ based on the optimal window size. In addition, we also adopt stochastic gradient descent (SGD) with large and constant learning rates as exploration kernels. Such a user-friendly nature enables us to conduct approximation tasks for complex posteriors without much tuning costs.

cs.LG↗
arXiv2022

Frequency multiplication with toroidal mode number of kink/fishbone modes on a static HL-2A-like tokamak

In the presence of energetic particles (EPs), the Long-Lived Mode (LLM) frequency multiplication with n = 1,2,3 or higher is often observed on HL-2A, where n is the toroidal mode number. Hybrid kinetic-MHD model simulations of the energetic particle (EP) driven kink/fishbone modes on a static HL-2A-like tokamak using NIMROD code find that, when the background plasma pressure is relatively high, and the EP pressure and the beam energy are relatively low, the mode frequency increases almost linearly with EP pressure, and the frequency is proportional to n ("frequency multiplication"), even in absence of any equilibrium plasma rotation. In addition, the frequency multiplication persists as the safety factor at magnetic axis q0 varies. In absence of EPs, the growth rate of the 1/1 mode is the largest; however, as the EP pressure increases, the growth rate of 2/2 modes or 3/3 modes becomes dominant, suggesting that the higher-n modes are more vulnerable to EPs. These results may shed light on the understanding about the toroidal mode number dependence of kink/fishbone modes in the advanced scenarios of tokamaks with weak or reversed central magnetic shear.

physics.plasm-ph↗
arXiv2022

A Contour Stochastic Gradient Langevin Dynamics Algorithm for Simulations of Multi-modal Distributions

We propose an adaptively weighted stochastic gradient Langevin dynamics algorithm (SGLD), so-called contour stochastic gradient Langevin dynamics (CSGLD), for Bayesian learning in big data statistics. The proposed algorithm is essentially a \emph{scalable dynamic importance sampler}, which automatically \emph{flattens} the target distribution such that the simulation for a multi-modal distribution can be greatly facilitated. Theoretically, we prove a stability condition and establish the asymptotic convergence of the self-adapting parameter to a {\it unique fixed-point}, regardless of the non-convexity of the original energy function; we also present an error analysis for the weighted averaging estimators. Empirically, the CSGLD algorithm is tested on multiple benchmark datasets including CIFAR10 and CIFAR100. The numerical results indicate its superiority to avoid the local trap problem in training deep neural networks.

stat.ML↗
arXiv2022

Interacting Contour Stochastic Gradient Langevin Dynamics

We propose an interacting contour stochastic gradient Langevin dynamics (ICSGLD) sampler, an embarrassingly parallel multiple-chain contour stochastic gradient Langevin dynamics (CSGLD) sampler with efficient interactions. We show that ICSGLD can be theoretically more efficient than a single-chain CSGLD with an equivalent computational budget. We also present a novel random-field function, which facilitates the estimation of self-adapting parameters in big data and obtains free mode explorations. Empirically, we compare the proposed algorithm with popular benchmark methods for posterior sampling. The numerical results show a great potential of ICSGLD for large-scale uncertainty estimation tasks.

stat.ML↗
arXiv2021

On the Distribution of SINR for Widely Linear MMSE MIMO Systems with Rectilinear or Quasi-Rectilinear Signals

Although the widely linear least mean square error (WLMMSE) receiver has been an appealing option for multiple-input-multiple-output (MIMO) wireless systems, a statistical understanding on its pose-detection signal-to-interference-plus-noise ratio (SINR) in detail is still missing. To this end, we consider a WLMMSE MIMO transmission system with rectilinear or quasi-rectilinear (QR) signals over the uncorrelated Rayleigh fading channel and investigate the statistical properties of its SINR for an arbitrary antenna configuration with $N_t$ transmit antennas and $N_r$ receive ones. We first derive an analytic probability density function (PDF) of the SINR in terms of the confluent hypergeometric function of the second kind, for WLMMSE MIMO systems with an arbitrary $N_r$ and $N_t=2, 3$. For a more general case in practice, i.e., $N_t>3$, we resort to the moment generating function to obtain an approximate but closed form PDF under some mild conditions, which, as expected, is more Gaussian-like as $2N_r-N_t$ increases. The so-derived PDFs are able to provide key insights into the WLMMSE MIMO receiver in terms of the outage probability, the symbol error rate, and the diversity gain, all presented in closed form. In particular, its diversity gain and the gain improvement over the conventional LMMSE one are explicitly quantified as $N_r-(N_t-1)/2$ and $(N_t-1)/2$, respectively. Finally, Monte Carlo simulations support the analysis.

cs.IT↗
arXiv2021

Information Directed Sampling for Sparse Linear Bandits

Stochastic sparse linear bandits offer a practical model for high-dimensional online decision-making problems and have a rich information-regret structure. In this work we explore the use of information-directed sampling (IDS), which naturally balances the information-regret trade-off. We develop a class of information-theoretic Bayesian regret bounds that nearly match existing lower bounds on a variety of problem instances, demonstrating the adaptivity of IDS. To efficiently implement sparse IDS, we propose an empirical Bayesian approach for sparse posterior sampling using a spike-and-slab Gaussian-Laplace prior. Numerical results demonstrate significant regret reductions by sparse IDS relative to several baselines.

stat.ML↗
arXiv2021

Non-convex Learning via Replica Exchange Stochastic Gradient MCMC

Replica exchange Monte Carlo (reMC), also known as parallel tempering, is an important technique for accelerating the convergence of the conventional Markov Chain Monte Carlo (MCMC) algorithms. However, such a method requires the evaluation of the energy function based on the full dataset and is not scalable to big data. The naïve implementation of reMC in mini-batch settings introduces large biases, which cannot be directly extended to the stochastic gradient MCMC (SGMCMC), the standard sampling method for simulating from deep neural networks (DNNs). In this paper, we propose an adaptive replica exchange SGMCMC (reSGMCMC) to automatically correct the bias and study the corresponding properties. The analysis implies an acceleration-accuracy trade-off in the numerical discretization of a Markov jump process in a stochastic environment. Empirically, we test the algorithm through extensive experiments on various setups and obtain the state-of-the-art results on CIFAR10, CIFAR100, and SVHN in both supervised learning and semi-supervised learning tasks.

stat.ML↗
arXiv2021

Accelerating Convergence of Replica Exchange Stochastic Gradient MCMC via Variance Reduction

Replica exchange stochastic gradient Langevin dynamics (reSGLD) has shown promise in accelerating the convergence in non-convex learning; however, an excessively large correction for avoiding biases from noisy energy estimators has limited the potential of the acceleration. To address this issue, we study the variance reduction for noisy energy estimators, which promotes much more effective swaps. Theoretically, we provide a non-asymptotic analysis on the exponential acceleration for the underlying continuous-time Markov jump process; moreover, we consider a generalized Girsanov theorem which includes the change of Poisson measure to overcome the crude discretization based on the Gröwall's inequality and yields a much tighter error in the 2-Wasserstein ($\mathcal{W}_2$) distance. Numerically, we conduct extensive experiments and obtain the state-of-the-art results in optimization and uncertainty estimates for synthetic experiments and image data.

stat.ML↗
arXiv2021

DeepLight: Deep Lightweight Feature Interactions for Accelerating CTR Predictions in Ad Serving

Click-through rate (CTR) prediction is a crucial task in online display advertising. The embedding-based neural networks have been proposed to learn both explicit feature interactions through a shallow component and deep feature interactions using a deep neural network (DNN) component. These sophisticated models, however, slow down the prediction inference by at least hundreds of times. To address the issue of significantly increased serving delay and high memory usage for ad serving in production, this paper presents \emph{DeepLight}: a framework to accelerate the CTR predictions in three aspects: 1) accelerate the model inference via explicitly searching informative feature interactions in the shallow component; 2) prune redundant layers and parameters at intra-layer and inter-layer level in the DNN component; 3) promote the sparsity of the embedding layer to preserve the most discriminant signals. By combining the above efforts, the proposed approach accelerates the model inference by 46X on Criteo dataset and 27X on Avazu dataset without any loss on the prediction accuracy. This paves the way for successfully deploying complicated embedding-based neural networks in production for ad serving.

cs.LG↗
arXiv2020

An adaptive Hessian approximated stochastic gradient MCMC method

Bayesian approaches have been successfully integrated into training deep neural networks. One popular family is stochastic gradient Markov chain Monte Carlo methods (SG-MCMC), which have gained increasing interest due to their scalability to handle large datasets and the ability to avoid overfitting. Although standard SG-MCMC methods have shown great performance in a variety of problems, they may be inefficient when the random variables in the target posterior densities have scale differences or are highly correlated. In this work, we present an adaptive Hessian approximated stochastic gradient MCMC method to incorporate local geometric information while sampling from the posterior. The idea is to apply stochastic approximation to sequentially update a preconditioning matrix at each iteration. The preconditioner possesses second-order information and can guide the random walk of a sampler efficiently. Instead of computing and saving the full Hessian of the log posterior, we use limited memory of the sample and their stochastic gradients to approximate the inverse Hessian-vector multiplication in the updating formula. Moreover, by smoothly optimizing the preconditioning matrix, our proposed algorithm can asymptotically converge to the target distribution with a controllable bias under mild conditions. To reduce the training and testing computational burden, we adopt a magnitude-based weight pruning method to enforce the sparsity of the network. Our method is user-friendly and is scalable to standard SG-MCMC updating rules by implementing an additional preconditioner. The sparse approximation of inverse Hessian alleviates storage and computational complexities for large dimensional models. The bias introduced by stochastic approximation is controllable and can be analyzed theoretically. Numerical experiments are performed on several problems.

math.NA↗
arXiv2020

Bayesian Sparse learning with preconditioned stochastic gradient MCMC and its applications

In this work, we propose a Bayesian type sparse deep learning algorithm. The algorithm utilizes a set of spike-and-slab priors for the parameters in the deep neural network. The hierarchical Bayesian mixture will be trained using an adaptive empirical method. That is, one will alternatively sample from the posterior using preconditioned stochastic gradient Langevin Dynamics (PSGLD), and optimize the latent variables via stochastic approximation. The sparsity of the network is achieved while optimizing the hyperparameters with adaptive searching and penalizing. A popular SG-MCMC approach is Stochastic gradient Langevin dynamics (SGLD). However, considering the complex geometry in the model parameter space in non-convex learning, updating parameters using a universal step size in each component as in SGLD may cause slow mixing. To address this issue, we apply a computationally manageable preconditioner in the updating rule, which provides a step-size parameter to adapt to local geometric properties. Moreover, by smoothly optimizing the hyperparameter in the preconditioning matrix, our proposed algorithm ensures a decreasing bias, which is introduced by ignoring the correction term in preconditioned SGLD. According to the existing theoretical framework, we show that the proposed algorithm can asymptotically converge to the correct distribution with a controllable bias under mild conditions. Numerical tests are performed on both synthetic regression problems and learning the solutions of elliptic PDE, which demonstrate the accuracy and efficiency of present work.

math.NA↗
arXiv2020

NTIRE 2020 Challenge on Real-World Image Super-Resolution: Methods and Results

This paper reviews the NTIRE 2020 challenge on real world super-resolution. It focuses on the participating methods and final results. The challenge addresses the real world setting, where paired true high and low-resolution images are unavailable. For training, only one set of source input images is therefore provided along with a set of unpaired high-quality target images. In Track 1: Image Processing artifacts, the aim is to super-resolve images with synthetically generated image processing artifacts. This allows for quantitative benchmarking of the approaches \wrt a ground-truth image. In Track 2: Smartphone Images, real low-quality smart phone images have to be super-resolved. In both tracks, the ultimate goal is to achieve the best perceptual quality, evaluated using a human study. This is the second challenge on the subject, following AIM 2019, targeting to advance the state-of-the-art in super-resolution. To measure the performance we use the benchmark protocol from AIM 2019. In total 22 teams competed in the final testing phase, demonstrating new and innovative solutions to the problem.

eess.IV↗
arXiv2020

An Adaptive Empirical Bayesian Method for Sparse Deep Learning

We propose a novel adaptive empirical Bayesian method for sparse deep learning, where the sparsity is ensured via a class of self-adaptive spike-and-slab priors. The proposed method works by alternatively sampling from an adaptive hierarchical posterior distribution using stochastic gradient Markov Chain Monte Carlo (MCMC) and smoothly optimizing the hyperparameters using stochastic approximation (SA). We further prove the convergence of the proposed method to the asymptotically correct distribution under mild conditions. Empirical applications of the proposed method lead to the state-of-the-art performance on MNIST and Fashion MNIST with shallow convolutional neural networks and the state-of-the-art compression performance on CIFAR10 with Residual Networks. The proposed method also improves resistance to adversarial attacks.

stat.ML↗
arXiv2019

GRID: a Student Project to Monitor the Transient Gamma-Ray Sky in the Multi-Messenger Astronomy Era

The Gamma-Ray Integrated Detectors (GRID) is a space mission concept dedicated to monitoring the transient gamma-ray sky in the energy range from 10 keV to 2 MeV using scintillation detectors onboard CubeSats in low Earth orbits. The primary targets of GRID are the gamma-ray bursts (GRBs) in the local universe. The scientific goal of GRID is, in synergy with ground-based gravitational wave (GW) detectors such as LIGO and VIRGO, to accumulate a sample of GRBs associated with the merger of two compact stars and study jets and related physics of those objects. It also involves observing and studying other gamma-ray transients such as long GRBs, soft gamma-ray repeaters, terrestrial gamma-ray flashes, and solar flares. With multiple CubeSats in various orbits, GRID is unaffected by the Earth occultation and serves as a full-time and all-sky monitor. Assuming a horizon of 200 Mpc for ground-based GW detectors, we expect to see a few associated GW-GRB events per year. With about 10 CubeSats in operation, GRID is capable of localizing a faint GRB like 170817A with a 90% error radius of about 10 degrees, through triangulation and flux modulation. GRID is proposed and developed by students, with considerable contribution from undergraduate students, and will remain operated as a student project in the future. The current GRID collaboration involves more than 20 institutes and keeps growing. On August 29th, the first GRID detector onboard a CubeSat was launched into a Sun-synchronous orbit and is currently under test.

astro-ph.IM↗
arXiv2017

Using Deep Neural Networks to Automate Large Scale Statistical Analysis for Big Data Applications

Statistical analysis (SA) is a complex process to deduce population properties from analysis of data. It usually takes a well-trained analyst to successfully perform SA, and it becomes extremely challenging to apply SA to big data applications. We propose to use deep neural networks to automate the SA process. In particular, we propose to construct convolutional neural networks (CNNs) to perform automatic model selection and parameter estimation, two most important SA tasks. We refer to the resulting CNNs as the neural model selector and the neural model estimator, respectively, which can be properly trained using labeled data systematically generated from candidate models. Simulation study shows that both the selector and estimator demonstrate excellent performances. The idea and proposed framework can be further extended to automate the entire SA process and have the potential to revolutionize how SA is performed in big data analytics.

stat.ML↗
arXiv2017

Magnetized Reverse Shock: Density-fluctuation-induced Field Distortion, Polarization Degree Reduction, and Application to GRBs

The early optical afterglow emission of several gamma-ray bursts (GRBs) shows a high linear polarization degree (PD) of tens of percent, suggesting an ordered magnetic field in the emission region. The light curves are consistent with being of a reverse shock (RS) origin. However, the magnetization parameter, $σ$, of the outflow is unknown. If $σ$ is too small, an ordered field in the RS may be quickly randomized due to turbulence driven by various perturbations so that the PD may not be as high as observed. Here we use the "Athena++" relativistic MHD code to simulate a relativistic jet with an ordered magnetic field propagating into a clumpy ambient medium, with a focus on how density fluctuations may distort the ordered magnetic field and reduce PD in the RS emission for different $σ$ values. For a given density fluctuation, we discover a clear power-law relationship between the relative PD reduction and the $σ$ value of the outflow. Such a relation may be applied to estimate $σ$ of the GRB outflows using the polarization data of early afterglows.

astro-ph.HE↗
arXiv2016

Collision-induced magnetic reconnection and a unified interpretation of polarization properties of GRBs and blazars

The jet composition and energy dissipation mechanism of Gamma-ray bursts (GRBs) and Blazars are fundamental questions which remain not fully understood. One plausible model is to interpret the $γ$-ray emission of GRBs and optical emission of blazars as synchrotron radiation of electrons accelerated from the collision-induced magnetic dissipation regions in Poynting-flux-dominated jets. The polarization observation is an important and independent information to test this model. Based on our recent 3D relativistic MHD simulations of collision-induced magnetic dissipation of magnetically dominated blobs, here we perform calculations of the polarization properties of the emission in the dissipation region and apply the results to model the polarization observational data of GRB prompt emission and blazar optical emission. We show that the same numerical model with different input parameters can reproduce well the observational data of both GRBs and blazars, especially the $90^{\circ}$ polarization angle (PA) change in GRB 100826A and the $180^{\circ}$ PA swing in Blazar 3C279. This supports a unified model for GRB and blazar jets, suggesting that collision-induced magnetic reconnection is a common physical mechanism to power the relativistic jet emission from events with very different black hole masses.

astro-ph.HE↗