arXiv subjects

Lin Xiao

Publications and source records attributed to Lin Xiao.

At least 73 records · Page 4Linked to original sources

arXiv2020

Stochastic optimization with decision-dependent distributions

Stochastic optimization problems often involve data distributions that change in reaction to the decision variables. This is the case for example when members of the population respond to a deployed classifier by manipulating their features so as to improve the likelihood of being positively labeled. Recent works on performative prediction have identified an intriguing solution concept for such problems: find the decision that is optimal with respect to the static distribution that the decision induces. Continuing this line of work, we show that typical stochastic algorithms -- originally designed for static problems -- can be applied directly for finding such equilibria with little loss in efficiency. The reason is simple to explain: the main consequence of the distributional shift is that it corrupts algorithms with a bias that decays linearly with the distance to the solution. Using this perspective, we obtain sharp convergence guarantees for popular algorithms, such as stochastic gradient, clipped gradient, proximal point, and dual averaging methods, along with their accelerated and proximal variants. In realistic applications, deployment of a decision rule is often much more expensive than sampling. We show how to modify the aforementioned algorithms so as to maintain their sample efficiency while performing only logarithmically many deployments.

math.OC↗

arXiv2020

Improving Self-supervised Pre-training via a Fully-Explored Masked Language Model

Masked Language Model (MLM) framework has been widely adopted for self-supervised language pre-training. In this paper, we argue that randomly sampled masks in MLM would lead to undesirably large gradient variance. Thus, we theoretically quantify the gradient variance via correlating the gradient covariance with the Hamming distance between two different masks (given a certain text sequence). To reduce the variance due to the sampling of masks, we propose a fully-explored masking strategy, where a text sequence is divided into a certain number of non-overlapping segments. Thereafter, the tokens within one segment are masked for training. We prove, from a theoretical perspective, that the gradients derived from this new masking schema have a smaller variance and can lead to more efficient self-supervised training. We conduct extensive experiments on both continual pre-training and general pre-training from scratch. Empirical results confirm that this new masking strategy can consistently outperform standard random masking. Detailed efficiency analysis and ablation studies further validate the advantages of our fully-explored masking strategy under the MLM framework.

cs.CL↗

arXiv2020

Seeded growth of high-quality transition metal dichalcogenide single crystals via chemical vapor transport

Transition metal dichalcogenides (TMDs) are van der Waals layered materials with sizable and tunable bandgaps, offering promising platforms for two-dimensional electronics and optoelectronics. To this end, the bottleneck is how to acquire high-quality single crystals in a facile and efficient manner. As one of the most widely employed method of single-crystal growth, conventional chemical vapor transport (CVT) generally encountered problems including the excess nucleation that leads to small crystal clusters and slow growth rate. To address these issues, a seed crystal is introduced to suppress the nucleation and an inner tube is adopted as both a separator and a flow restrictor, favoring the growth of large-size and high-quality TMD single crystals successfully. Three examples are presented, the effective growth of millimeter-sized MoSe2 and MoTe2 single crystals, and the greatly shortened growth period for PtSe2 single crystal, all of which are synthesized in high quality according to detailed characterizations. The mechanism of seeded CVT is discussed. Furthermore, a phototransistor based on exfoliated multi-layered MoSe2 displays excellent photoresponse in ambient conditions, and considerably rapid rise and fall time of 110 and 125 us are obtained. This work paves the way for developing a facile and versatile method to synthesize high-quality TMD single crystals in laboratory, which could serve as favorable functional materials for potential low-dimensional optoelectronics.

cond-mat.mtrl-sci↗

arXiv2020

Jointly Modeling Intra- and Inter-transaction Dependencies with Hierarchical Attentive Transaction Embeddings for Next-item Recommendation

A transaction-based recommender system (TBRS) aims to predict the next item by modeling dependencies in transactional data. Generally, two kinds of dependencies considered are intra-transaction dependency and inter-transaction dependency. Most existing TBRSs recommend next item by only modeling the intra-transaction dependency within the current transaction while ignoring inter-transaction dependency with recent transactions that may also affect the next item. However, as not all recent transactions are relevant to the current and next items, the relevant ones should be identified and prioritized. In this paper, we propose a novel hierarchical attentive transaction embedding (HATE) model to tackle these issues. Specifically, a two-level attention mechanism integrates both item embedding and transaction embedding to build an attentive context representation that incorporates both intraand inter-transaction dependencies. With the learned context representation, HATE then recommends the next item. Experimental evaluations on two real-world transaction datasets show that HATE significantly outperforms the state-ofthe-art methods in terms of recommendation accuracy.

cs.IR↗

arXiv2020

Statistical Adaptive Stochastic Gradient Methods

We propose a statistical adaptive procedure called SALSA for automatically scheduling the learning rate (step size) in stochastic gradient methods. SALSA first uses a smoothed stochastic line-search procedure to gradually increase the learning rate, then automatically switches to a statistical method to decrease the learning rate. The line search procedure ``warms up'' the optimization process, reducing the need for expensive trial and error in setting an initial learning rate. The method for decreasing the learning rate is based on a new statistical test for detecting stationarity when using a constant step size. Unlike in prior work, our test applies to a broad class of stochastic gradient algorithms without modification. The combined method is highly robust and autonomous, and it matches the performance of the best hand-tuned learning rate schedules in our experiments on several deep learning tasks.

stat.ML↗

arXiv2020

Statistically Preconditioned Accelerated Gradient Method for Distributed Optimization

We consider the setting of distributed empirical risk minimization where multiple machines compute the gradients in parallel and a centralized server updates the model parameters. In order to reduce the number of communications required to reach a given accuracy, we propose a \emph{preconditioned} accelerated gradient method where the preconditioning is done by solving a local optimization problem over a subsampled dataset at the server. The convergence rate of the method depends on the square root of the relative condition number between the global and local loss functions. We estimate the relative condition number for linear prediction models by studying \emph{uniform} concentration of the Hessians over a bounded domain, which allows us to derive improved convergence rates for existing preconditioned gradient methods and our accelerated method. Experiments on real-world datasets illustrate the benefits of acceleration in the ill-conditioned regime.

math.OC↗

arXiv2019

Understanding the Role of Momentum in Stochastic Gradient Methods

The use of momentum in stochastic gradient methods has become a widespread practice in machine learning. Different variants of momentum, including heavy-ball momentum, Nesterov's accelerated gradient (NAG), and quasi-hyperbolic momentum (QHM), have demonstrated success on various tasks. Despite these empirical successes, there is a lack of clear understanding of how the momentum parameters affect convergence and various performance measures of different algorithms. In this paper, we use the general formulation of QHM to give a unified analysis of several popular algorithms, covering their asymptotic convergence conditions, stability regions, and properties of their stationary distributions. In addition, by combining the results on convergence rates and stationary distributions, we obtain sometimes counter-intuitive practical guidelines for setting the learning rate and momentum parameters.

cs.LG↗

arXiv2019

Joint Computation and Communication Design for UAV-Assisted Mobile Edge Computing in IoT

Unmanned aerial vehicle (UAV)-assisted mobile edge computing (MEC) system is a prominent concept, where a UAV equipped with a MEC server is deployed to serve a number of terminal devices (TDs) of Internet of Things (IoT) in a finite period. In this paper, each TD has a certain latency-critical computation task in each time slot to complete. Three computation strategies can be available to each TD. First, each TD can operate local computing by itself. Second, each TD can partially offload task bits to the UAV for computing. Third, each TD can choose to offload task bits to access point (AP) via UAV relaying. We propose a new optimization problem formulation that aims to minimize the total energy consumption including communication-related energy, computation-related energy and UAV's flight energy by optimizing the bits allocation, time slot scheduling and power allocation as well as UAV trajectory design. As the formulated problem is non-convex and difficult to find the optimal solution, we solve the problem by two parts, and obtain the near optimal solution with within a dozen of iterations. Finally, numerical results are given to validate the proposed algorithm, which is verified to be efficient and superior to the other benchmark cases.

eess.SP↗

arXiv2019

From low probability to high confidence in stochastic convex optimization

Standard results in stochastic convex optimization bound the number of samples that an algorithm needs to generate a point with small function value in expectation. More nuanced high probability guarantees are rare, and typically either rely on "light-tail" noise assumptions or exhibit worse sample complexity. In this work, we show that a wide class of stochastic optimization algorithms for strongly convex problems can be augmented with high confidence bounds at an overhead cost that is only logarithmic in the confidence level and polylogarithmic in the condition number. The procedure we propose, called proxBoost, is elementary and builds on two well-known ingredients: robust distance estimation and the proximal point method. We discuss consequences for both streaming (online) algorithms and offline algorithms based on empirical risk minimization.

math.OC↗

arXiv2019

Using Statistics to Automate Stochastic Optimization

Despite the development of numerous adaptive optimizers, tuning the learning rate of stochastic gradient methods remains a major roadblock to obtaining good practical performance in machine learning. Rather than changing the learning rate at each iteration, we propose an approach that automates the most common hand-tuning heuristic: use a constant learning rate until "progress stops," then drop. We design an explicit statistical test that determines when the dynamics of stochastic gradient descent reach a stationary distribution. This test can be performed easily during training, and when it fires, we decrease the learning rate by a constant multiplicative factor. Our experiments on several deep learning tasks demonstrate that this statistical adaptive stochastic approximation (SASA) method can automatically find good learning rate schedules and match the performance of hand-tuned methods using default settings of its parameters. The statistical testing helps to control the variance of this procedure and improves its robustness.

stat.ML↗

arXiv2019

Hyperbolic Interaction Model For Hierarchical Multi-Label Classification

Different from the traditional classification tasks which assume mutual exclusion of labels, hierarchical multi-label classification (HMLC) aims to assign multiple labels to every instance with the labels organized under hierarchical relations. Besides the labels, since linguistic ontologies are intrinsic hierarchies, the conceptual relations between words can also form hierarchical structures. Thus it can be a challenge to learn mappings from word hierarchies to label hierarchies. We propose to model the word and label hierarchies by embedding them jointly in the hyperbolic space. The main reason is that the tree-likeness of the hyperbolic space matches the complexity of symbolic data with hierarchical structures. A new Hyperbolic Interaction Model (HyperIM) is designed to learn the label-aware document representations and make predictions for HMLC. Extensive experiments are conducted on three benchmark datasets. The results have demonstrated that the new model can realistically capture the complex data structures and further improve the performance for HMLC comparing with the state-of-the-art methods. To facilitate future research, our code is publicly available.

cs.LG↗

arXiv2019

Label-aware Document Representation via Hybrid Attention for Extreme Multi-Label Text Classification

Extreme multi-label text classification (XMTC) aims at tagging a document with most relevant labels from an extremely large-scale label set. It is a challenging problem especially for the tail labels because there are only few training documents to build classifier. This paper is motivated to better explore the semantic relationship between each document and extreme labels by taking advantage of both document content and label correlation. Our objective is to establish an explicit label-aware representation for each document with a hybrid attention deep neural network model(LAHA). LAHA consists of three parts. The first part adopts a multi-label self-attention mechanism to detect the contribution of each word to labels. The second part exploits the label structure and document content to determine the semantic connection between words and labels in a same latent space. An adaptive fusion strategy is designed in the third part to obtain the final label-aware document representation so that the essence of previous two parts can be sufficiently integrated. Extensive experiments have been conducted on six benchmark datasets by comparing with the state-of-the-art methods. The results show the superiority of our proposed LAHA method, especially on the tail labels.

cs.LG↗

arXiv2019

A Stochastic Composite Gradient Method with Incremental Variance Reduction

We consider the problem of minimizing the composition of a smooth (nonconvex) function and a smooth vector mapping, where the inner mapping is in the form of an expectation over some random variable or a finite sum. We propose a stochastic composite gradient method that employs an incremental variance-reduced estimator for both the inner vector mapping and its Jacobian. We show that this method achieves the same orders of complexity as the best known first-order methods for minimizing expected-value and finite-sum nonconvex functions, despite the additional outer composition which renders the composite gradient estimator biased. This finding enables a much broader range of applications in machine learning to benefit from the low complexity of incremental variance-reduction methods.

math.OC↗

arXiv2019

The distance, supernova rate and supernova progenitors of NGC 6946

The distance to the fireworks galaxy NGC 6946 is highly uncertain. Recent distance estimates using the tip of the red giant branch of 7.7 to 7.8 Mpc are larger than the distance commonly assumed for studying supernovae in this galaxy of 5.9 Mpc. Here we use the high supernova rate of the galaxy to derive the star-formation rate and predict the galaxy's FUV flux. We also account for dust extinction by different methods to derive a distance of 7.9+/-4.0 Mpc for NGC 6946. We then use the new distance to re-evaluate the understanding of the supernova progenitors 2002hh, 2004et, 2017eaw, the possible electron capture event 2008S and the possible black-hole forming event N6946-BH1. For the latter two exotic events the new distance improves the consistency between the observed progenitors and the stellar models that give rise to these events. From our findings we strongly recommend that all future studies of NGC~6946 must use the greater distance to the galaxy of 7.72+/-0.32 Mpc of Anand et al. (2018).

astro-ph.GA↗

arXiv2018

Adaptive Stochastic Variance Reduction for Subsampled Newton Method with Cubic Regularization

The cubic regularized Newton method of Nesterov and Polyak has become increasingly popular for non-convex optimization because of its capability of finding an approximate local solution with second-order guarantee. Several recent works extended this method to the setting of minimizing the average of N smooth functions by replacing the exact gradients and Hessians with subsampled approximations. It has been shown that the total Hessian sample complexity can be reduced to be sublinear in N per iteration by leveraging stochastic variance reduction techniques. We present an adaptive variance reduction scheme for subsampled Newton method with cubic regularization, and show that the expected Hessian sample complexity is O(N + N^{2/3}ε^{-3/2}) for finding an (ε,ε^{1/2})-approximate local solution (in terms of first and second-order guarantees respectively). Moreover, we show that the same Hessian sample complexity retains with fixed sample sizes if exact gradients are used. The techniques of our analysis are different from previous works in that we do not rely on high probability bounds based on matrix concentration inequalities. Instead, we derive and utilize bounds on the 3rd and 4th order moments of the average of random matrices, which are of independent interest on their own.

math.OC↗

arXiv2018

Core-collapse supernovae ages and metallicities from emission-line diagnostics of nearby stellar populations

Massive stars are the main objects that illuminate H II regions and they evolve quickly to end their lives in core-collapse supernovae (CCSNe). Thus it is important to investigate the association between CCSNe and H II regions. In this paper, we present emission line diagnostics of the stellar populations around nearby CCSNe, that include their host H II regions, from the PMAS/PPAK Integral-field Supernova hosts COmpilation (PISCO). We then use BPASS stellar population models to determine the age, metallicity and gas parameters for H II regions associated with CCSNe, contrasting models that consider either single star evolution alone or incorporate interacting binaries. We find binary-star models, that allow for ionizing photon loss, provide a more realistic fit to the observed CCSN hosts with metallicities that are closer to those derived from the oxygen abundance in O3N2. We also find that type II and type Ibc SNe arise from progenitor stars of similar age, mostly from 7 to 45 Myr, which corresponds to stars with masses < 20 solar mass . However these two types SNe have little preference in their host environment metallicity measured by oxygen abundance or in progenitor initial mass. We note however that at lower metallicities supernovae are more likely to be of type II.

astro-ph.GA↗

arXiv2018

Secrecy Energy Efficiency Maximization for UAV-Enabled Mobile Relaying

This paper investigates the secrecy energy efficiency (SEE) maximization problem for unmanned aerial vehicle enabled mobile relaying system, where a high-mobility UAV is exploited to assist delivering the confidential information from a ground source to a legitimate ground destination with the direct link blocked, in the presence of a potential eavesdropper. We aim to maximize SEE of the UAV by jointly optimizing the communication scheduling, power allocation, and UAV trajectory over a finite time horizon. The formulated problem is a mixed-integer non-convex optimization problem that is challenging to be solved optimally. To make the problem tractable, we decompose the problem into three subproblems, and propose an efficient iterative algorithm that alternately optimizes the subproblems. In addition, two special cases are considered as benchmarks in this paper. Simulation results show that the proposed design significantly improves the SEE of the UAV, as compared to the benchmarks

cs.IT↗

arXiv2018

SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation

When function approximation is used, solving the Bellman optimality equation with stability guarantees has remained a major open problem in reinforcement learning for decades. The fundamental difficulty is that the Bellman operator may become an expansion in general, resulting in oscillating and even divergent behavior of popular algorithms like Q-learning. In this paper, we revisit the Bellman equation, and reformulate it into a novel primal-dual optimization problem using Nesterov's smoothing technique and the Legendre-Fenchel transformation. We then develop a new algorithm, called Smoothed Bellman Error Embedding, to solve this optimization problem where any differentiable function class may be used. We provide what we believe to be the first convergence guarantee for general nonlinear function approximation, and analyze the algorithm's sample complexity. Empirically, our algorithm compares favorably to state-of-the-art baselines in several benchmark control problems.

cs.LG↗