arXiv subjects

Tao Lin

Publications and source records attributed to Tao Lin.

At least 199 records · Page 11Linked to original sources

arXiv2020

On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them

We analyze the influence of adversarial training on the loss landscape of machine learning models. To this end, we first provide analytical studies of the properties of adversarial loss functions under different adversarial budgets. We then demonstrate that the adversarial loss landscape is less favorable to optimization, due to increased curvature and more scattered gradients. Our conclusions are validated by numerical analyses, which show that training under large adversarial budgets impede the escape from suboptimal random initialization, cause non-vanishing gradients and make the model find sharper minima. Based on these observations, we show that a periodic adversarial scheduling (PAS) strategy can effectively overcome these challenges, yielding better results than vanilla adversarial training while being much less sensitive to the choice of learning rate.

cs.LG↗

arXiv2020

A Game-Theoretic Analysis of the Empirical Revenue Maximization Algorithm with Endogenous Sampling

The Empirical Revenue Maximization (ERM) is one of the most important price learning algorithms in auction design: as the literature shows it can learn approximately optimal reserve prices for revenue-maximizing auctioneers in both repeated auctions and uniform-price auctions. However, in these applications the agents who provide inputs to ERM have incentives to manipulate the inputs to lower the outputted price. We generalize the definition of an incentive-awareness measure proposed by Lavi et al (2019), to quantify the reduction of ERM's outputted price due to a change of $m\ge 1$ out of $N$ input samples, and provide specific convergence rates of this measure to zero as $N$ goes to infinity for different types of input distributions. By adopting this measure, we construct an efficient, approximately incentive-compatible, and revenue-optimal learning algorithm using ERM in repeated auctions against non-myopic bidders, and show approximate group incentive-compatibility in uniform-price auctions.

cs.GT↗

arXiv2020

Masking as an Efficient Alternative to Finetuning for Pretrained Language Models

We present an efficient method of utilizing pretrained language models, where we learn selective binary masks for pretrained weights in lieu of modifying them through finetuning. Extensive evaluations of masking BERT and RoBERTa on a series of NLP tasks show that our masking scheme yields performance comparable to finetuning, yet has a much smaller memory footprint when several tasks need to be inferred simultaneously. Through intrinsic evaluations, we show that representations computed by masked language models encode information necessary for solving downstream tasks. Analyzing the loss landscape, we show that masking and finetuning produce models that reside in minima that can be connected by a line segment with nearly constant test accuracy. This confirms that masking can be utilized as an efficient alternative to finetuning.

cs.CL↗

arXiv2020

Error Analysis of Symmetric Linear/Bilinear Partially Penalized Immersed Finite Element Methods for Helmholtz Interface Problems

This article presents an error analysis of the symmetric linear/bilinear partially penalized immersed finite element (PPIFE) methods for interface problems of Helmholtz equations. Under the assumption that the exact solution possesses a usual piecewise $H^2$ regularity, the optimal error bounds for the PPIFE solutions are derived in an energy norm and the usual $L^2$ norm provided that the mesh size is sufficiently small. A numerical example is conducted to validate the theoretical conclusions.

math.NA↗

arXiv2020

Dynamic Model Pruning with Feedback

Deep neural networks often have millions of parameters. This can hinder their deployment to low-end devices, not only due to high memory requirements but also because of increased latency at inference. We propose a novel model compression method that generates a sparse trained model without additional overhead: by allowing (i) dynamic allocation of the sparsity pattern and (ii) incorporating feedback signal to reactivate prematurely pruned weights we obtain a performant sparse model in one single training pass (retraining is not needed, but can further improve the performance). We evaluate our method on CIFAR-10 and ImageNet, and show that the obtained sparse models can reach the state-of-the-art performance of dense models. Moreover, their performance surpasses that of models generated by all previously proposed pruning schemes.

cs.LG↗

arXiv2020

Extrapolation for Large-batch Training in Deep Learning

Deep learning networks are typically trained by Stochastic Gradient Descent (SGD) methods that iteratively improve the model parameters by estimating a gradient on a very small fraction of the training data. A major roadblock faced when increasing the batch size to a substantial fraction of the training data for improving training time is the persistent degradation in performance (generalization gap). To address this issue, recent work propose to add small perturbations to the model parameters when computing the stochastic gradients and report improved generalization performance due to smoothing effects. However, this approach is poorly understood; it requires often model-specific noise and fine-tuning. To alleviate these drawbacks, we propose to use instead computationally efficient extrapolation (extragradient) to stabilize the optimization trajectory while still benefiting from smoothing to avoid sharp minima. This principled approach is well grounded from an optimization perspective and we show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer. We demonstrate that in a variety of experiments the scheme allows scaling to much larger batch sizes than before whilst reaching or surpassing SOTA accuracy.

cs.LG↗

arXiv2020

An Enriched Immersed Finite Element Method for Interface Problems with Nonhomogeneous Jump Conditions

This article presents and analyzes a $p^{th}$-degree immersed finite element (IFE) method for elliptic interface problems with nonhomogeneous jump conditions. In this method, jump conditions are approximated optimally by basic IFE and enrichment IFE piecewise polynomial functions constructed by solving local Cauchy problems on interface elements. The proposed IFE method is based on a discontinuous Galerkin formulation on interface elements and a continuous Galerkin formulation on non-interface elements. This $p^{th}$-degree IFE method is proved to converge optimally under mesh refinement. In addition, this article addresses the stability of this IFE method and has established upper bounds for its condition numbers which are optimal with respect to the mesh size but suboptimal with respect to the contrast of the discontinuous coefficient.

math.NA↗

arXiv2020

Don't Use Large Mini-Batches, Use Local SGD

Mini-batch stochastic gradient methods (SGD) are state of the art for distributed training of deep neural networks. Drastic increases in the mini-batch sizes have lead to key efficiency and scalability gains in recent years. However, progress faces a major roadblock, as models trained with large batches often do not generalize well, i.e. they do not show good accuracy on new data. As a remedy, we propose a \emph{post-local} SGD and show that it significantly improves the generalization performance compared to large-batch training on standard benchmarks while enjoying the same efficiency (time-to-accuracy) and scalability. We further provide an extensive study of the communication efficiency vs. performance trade-offs associated with a host of \emph{local SGD} variants.

cs.LG↗

arXiv2020

Deep Collaborative Embedding for information cascade prediction

Recently, information cascade prediction has attracted increasing interest from researchers, but it is far from being well solved partly due to the three defects of the existing works. First, the existing works often assume an underlying information diffusion model, which is impractical in real world due to the complexity of information diffusion. Second, the existing works often ignore the prediction of the infection order, which also plays an important role in social network analysis. At last, the existing works often depend on the requirement of underlying diffusion networks which are likely unobservable in practice. In this paper, we aim at the prediction of both node infection and infection order without requirement of the knowledge about the underlying diffusion mechanism and the diffusion network, where the challenges are two-fold. The first is what cascading characteristics of nodes should be captured and how to capture them, and the second is that how to model the non-linear features of nodes in information cascades. To address these challenges, we propose a novel model called Deep Collaborative Embedding (DCE) for information cascade prediction, which can capture not only the node structural property but also two kinds of node cascading characteristics. We propose an auto-encoder based collaborative embedding framework to learn the node embeddings with cascade collaboration and node collaboration, in which way the non-linearity of information cascades can be effectively captured. The results of extensive experiments conducted on real-world datasets verify the effectiveness of our approach.

cs.SI↗

arXiv2019

On Clearing Prices in Matching Markets: A Simple Characterization without Duality

Duality of linear programming is a standard approach to the classical weighted maximum matching problem. From an economic perspective, the dual variables can be regarded as prices of products and payoffs of buyers in a two-sided matching market. Traditional duality-based algorithms, e.g., Hungarian, essentially aims at finding a set of prices that clears the market. Under such market-clearing prices, a maximum matching is formed when buyers buy their most preferred products respectively. We study the property of market-clearing prices without the use of duality, showing that: (1) the space of market-clearing prices is convex and closed under element-wise maximum and minimum operations; (2) any market-clearing prices induce all maximum matchings.

cs.DS↗

arXiv2019

A Trilinear Immersed Finite Element Method for Solving Elliptic Interface Problems

This article presents an immersed finite element (IFE) method for solving the typical three-dimensional second order elliptic interface problem with an interface-independent Cartesian mesh. The local IFE space on each interface element consists of piecewise trilinear polynomials which are constructed by extending polynomials from one subelement to the whole element according to the jump conditions of the interface problem. In this space, the IFE shape functions with the Lagrange degrees of freedom can always be constructed regardless of interface location and discontinuous coefficients. The proposed IFE space is proven to have the optimal approximation capabilities to the functions satisfying the jump conditions. A group of numerical examples with representative interface geometries are presented to demonstrate features of the proposed IFE method.

math.NA↗

arXiv2019

Exploring Interpretable LSTM Neural Networks over Multi-Variable Data

For recurrent neural networks trained on time series with target and exogenous variables, in addition to accurate prediction, it is also desired to provide interpretable insights into the data. In this paper, we explore the structure of LSTM recurrent neural networks to learn variable-wise hidden states, with the aim to capture different dynamics in multi-variable time series and distinguish the contribution of variables to the prediction. With these variable-wise hidden states, a mixture attention mechanism is proposed to model the generative process of the target. Then we develop associated training methods to jointly learn network parameters, variable and temporal importance w.r.t the prediction of the target variable. Extensive experiments on real datasets demonstrate enhanced prediction performance by capturing the dynamics of different variables. Meanwhile, we evaluate the interpretation results both qualitatively and quantitatively. It exhibits the prospect as an end-to-end framework for both forecasting and knowledge extraction over multi-variable data.

cs.LG↗

arXiv2019

FoundationDB Record Layer: A Multi-Tenant Structured Datastore

The FoundationDB Record Layer is an open source library that provides a record-oriented data store with semantics similar to a relational database implemented on top of FoundationDB, an ordered, transactional key-value store. The Record Layer provides a lightweight, highly extensible way to store structured data. It offers schema management and a rich set of query and indexing facilities, some of which are not usually found in traditional relational databases, such as nested record types, indexes on commit versions, and indexes that span multiple record types. The Record Layer is stateless and built for massive multi-tenancy, encapsulating and isolating all of a tenant's state, including indexes, into a separate logical database. We demonstrate how the Record Layer is used by CloudKit, Apple's cloud backend service, to provide powerful abstractions to applications serving hundreds of millions of users. CloudKit uses the Record Layer to host billions of independent databases, many with a common schema. Features provided by the Record Layer enable CloudKit to provide richer APIs and stronger semantics with reduced maintenance overhead and improved scalability.

cs.DB↗

arXiv2018

T-GCN: A Temporal Graph ConvolutionalNetwork for Traffic Prediction

Accurate and real-time traffic forecasting plays an important role in the Intelligent Traffic System and is of great significance for urban traffic planning, traffic management, and traffic control. However, traffic forecasting has always been considered an open scientific issue, owing to the constraints of urban road network topological structure and the law of dynamic change with time, namely, spatial dependence and temporal dependence. To capture the spatial and temporal dependence simultaneously, we propose a novel neural network-based traffic forecasting method, the temporal graph convolutional network (T-GCN) model, which is in combination with the graph convolutional network (GCN) and gated recurrent unit (GRU). Specifically, the GCN is used to learn complex topological structures to capture spatial dependence and the gated recurrent unit is used to learn dynamic changes of traffic data to capture temporal dependence. Then, the T-GCN model is employed to traffic forecasting based on the urban road network. Experiments demonstrate that our T-GCN model can obtain the spatio-temporal correlation from traffic data and the predictions outperform state-of-art baselines on real-world traffic datasets. Our tensorflow implementation of the T-GCN is available at https://github.com/lehaifeng/T-GCN.

cs.LG↗

arXiv2018

A method of detector and event visualization with Unity in JUNO

A visualization method based on Unity engine is proposed for the Jiangmen Underground Neutrino Observatory (JUNO) experiment. The method has been applied in development of a new event display tool named ELAINA (Event Live Animation with unIty for Neutrino Analysis), which provides an intuitive way for users to observe the detector geometry, to tune the reconstruction algorithm and to analyze the physics events. In comparison with the traditional ROOT-based event display, ELAINA provides better visual effects with the Unity engine. It is developed independently of the JUNO offline software but shares the same detector description and event data model in JUNO offline with interfaces. Users can easily download and run the event display on their local computers with different operation systems.

physics.ins-det↗

arXiv2018

Training DNNs with Hybrid Block Floating Point

The wide adoption of DNNs has given birth to unrelenting computing requirements, forcing datacenter operators to adopt domain-specific accelerators to train them. These accelerators typically employ densely packed full precision floating-point arithmetic to maximize performance per area. Ongoing research efforts seek to further increase that performance density by replacing floating-point with fixed-point arithmetic. However, a significant roadblock for these attempts has been fixed point's narrow dynamic range, which is insufficient for DNN training convergence. We identify block floating point (BFP) as a promising alternative representation since it exhibits wide dynamic range and enables the majority of DNN operations to be performed with fixed-point logic. Unfortunately, BFP alone introduces several limitations that preclude its direct applicability. In this work, we introduce HBFP, a hybrid BFP-FP approach, which performs all dot products in BFP and other operations in floating point. HBFP delivers the best of both worlds: the high accuracy of floating point at the superior hardware density of fixed point. For a wide variety of models, we show that HBFP matches floating point's accuracy while enabling hardware implementations that deliver up to 8.5x higher throughput.

cs.LG↗

arXiv2018

Multi-variable LSTM neural network for autoregressive exogenous model

In this paper, we propose multi-variable LSTM capable of accurate forecasting and variable importance interpretation for time series with exogenous variables. Current attention mechanism in recurrent neural networks mostly focuses on the temporal aspect of data and falls short of characterizing variable importance. To this end, the multi-variable LSTM equipped with tensorized hidden states is developed to learn hidden states for individual variables, which give rise to our mixture temporal and variable attention. Based on such attention mechanism, we infer and quantify variable importance. Extensive experiments using real datasets with Granger-causality test and the synthetic dataset with ground truth demonstrate the prediction performance and interpretability of multi-variable LSTM in comparison to a variety of baselines. It exhibits the prospect of multi-variable LSTM as an end-to-end framework for both forecasting and knowledge discovery.

cs.LG↗

arXiv2018

A Fixed Mesh Method With Immersed Finite Elements for Solving Interface Inverse Problems

We present a new fixed mesh algorithm for solving a class of interface inverse problems for the typical elliptic interface problems. These interface inverse problems are formulated as shape optimization prob- lems whose objective functionals depend on the shape of the interface. Regardless of the location of the interface, both the governing partial differential equations and the objective functional are discretized optimally, with respect to the involved polynomial space, by an immersed finite element (IFE) method on a fixed mesh. Furthermore, the formula for the gradient of the descritized objective function is de- rived within the IFE framework that can be computed accurately and efficiently through the discretized adjoint procedure. Features of this proposed IFE method based on a fixed mesh are demonstrated by its applications to three representative interface inverse problems: the interface inverse problem with an internal measurement on a sub-domain, a Dirichlet-Neumann type inverse problem whose data is given on the boundary, and a heat dissipation design problem.

math.NA↗