arXiv subjects

Chi Wang

Publications and source records attributed to Chi Wang.

At least 127 records · Page 7Linked to original sources

arXiv2021

Geometry Attention Transformer with Position-aware LSTMs for Image Captioning

In recent years, transformer structures have been widely applied in image captioning with impressive performance. For good captioning results, the geometry and position relations of different visual objects are often thought of as crucial information. Aiming to further promote image captioning by transformers, this paper proposes an improved Geometry Attention Transformer (GAT) model. In order to further leverage geometric information, two novel geometry-aware architectures are designed respectively for the encoder and decoder in our GAT. Besides, this model includes the two work modules: 1) a geometry gate-controlled self-attention refiner, for explicitly incorporating relative spatial information into image region representations in encoding steps, and 2) a group of position-LSTMs, for precisely informing the decoder of relative word position in generating caption texts. The experiment comparisons on the datasets MS COCO and Flickr30K show that our GAT is efficient, and it could often outperform current state-of-the-art image captioning models.

cs.CV↗

arXiv2021

Q-error Bounds of Random Uniform Sampling for Cardinality Estimation

Random uniform sampling has been studied in various statistical tasks but few of them have covered the Q-error metric for cardinality estimation (CE). In this paper, we analyze the confidence intervals of random uniform sampling with and without replacement for single-table CE. Results indicate that the upper Q-error bound depends on the sample size and true cardinality. Our bound gives a rule-of-thumb for how large a sample should be kept for single-table CE.

math.ST↗

arXiv2021

Attention-guided Temporally Coherent Video Object Matting

This paper proposes a novel deep learning-based video object matting method that can achieve temporally coherent matting results. Its key component is an attention-based temporal aggregation module that maximizes image matting networks' strength for video matting networks. This module computes temporal correlations for pixels adjacent to each other along the time axis in feature space, which is robust against motion noises. We also design a novel loss term to train the attention weights, which drastically boosts the video matting performance. Besides, we show how to effectively solve the trimap generation problem by fine-tuning a state-of-the-art video object segmentation network with a sparse set of user-annotated keyframes. To facilitate video matting and trimap generation networks' training, we construct a large-scale video matting dataset with 80 training and 28 validation foreground video clips with ground-truth alpha mattes. Experimental results show that our method can generate high-quality alpha mattes for various videos featuring appearance change, occlusion, and fast motion. Our code and dataset can be found at: https://github.com/yunkezhang/TCVOM

cs.CV↗

arXiv2021

An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models

The performance of fine-tuning pre-trained language models largely depends on the hyperparameter configuration. In this paper, we investigate the performance of modern hyperparameter optimization methods (HPO) on fine-tuning pre-trained language models. First, we study and report three HPO algorithms' performances on fine-tuning two state-of-the-art language models on the GLUE dataset. We find that using the same time budget, HPO often fails to outperform grid search due to two reasons: insufficient time budget and overfitting. We propose two general strategies and an experimental procedure to systematically troubleshoot HPO's failure cases. By applying the procedure, we observe that HPO can succeed with more appropriate settings in the search space and time budget; however, in certain cases overfitting remains. Finally, we make suggestions for future work. Our implementation can be found in https://github.com/microsoft/FLAML/tree/main/flaml/nlp/.

cs.CL↗

arXiv2021

ChaCha for Online AutoML

We propose the ChaCha (Champion-Challengers) algorithm for making an online choice of hyperparameters in online learning settings. ChaCha handles the process of determining a champion and scheduling a set of `live' challengers over time based on sample complexity bounds. It is guaranteed to have sublinear regret after the optimal configuration is added into consideration by an application-dependent oracle based on the champions. Empirically, we show that ChaCha provides good performance across a wide array of datasets when optimizing over featurization and hyperparameter decisions.

cs.LG↗

arXiv2021

FLAML: A Fast and Lightweight AutoML Library

We study the problem of using low computational cost to automate the choices of learners and hyperparameters for an ad-hoc training dataset and error metric, by conducting trials of different configurations on the given training data. We investigate the joint impact of multiple factors on both trial cost and model error, and propose several design guidelines. Following them, we build a fast and lightweight library FLAML which optimizes for low computational resource in finding accurate models. FLAML integrates several simple but effective search strategies into an adaptive system. It significantly outperforms top-ranked AutoML libraries on a large open source AutoML benchmark under equal, or sometimes orders of magnitude smaller budget constraints.

cs.LG↗

arXiv2020

Frugal Optimization for Cost-related Hyperparameters

The increasing demand for democratizing machine learning algorithms calls for hyperparameter optimization (HPO) solutions at low cost. Many machine learning algorithms have hyperparameters which can cause a large variation in the training cost. But this effect is largely ignored in existing HPO methods, which are incapable to properly control cost during the optimization process. To address this problem, we develop a new cost-frugal HPO solution. The core of our solution is a simple but new randomized direct-search method, for which we prove a convergence rate of $O(\frac{\sqrt{d}}{\sqrt{K}})$ and an $O(dε^{-2})$-approximation guarantee on the total cost. We provide strong empirical results in comparison with state-of-the-art HPO methods on large AutoML benchmarks.

cs.LG↗

arXiv2020

A Matrix Chernoff Bound for Markov Chains and Its Application to Co-occurrence Matrices

We prove a Chernoff-type bound for sums of matrix-valued random variables sampled via a regular (aperiodic and irreducible) finite Markov chain. Specially, consider a random walk on a regular Markov chain and a Hermitian matrix-valued function on its state space. Our result gives exponentially decreasing bounds on the tail distributions of the extreme eigenvalues of the sample mean matrix. Our proof is based on the matrix expander (regular undirected graph) Chernoff bound [Garg et al. STOC '18] and scalar Chernoff-Hoeffding bounds for Markov chains [Chung et al. STACS '12]. Our matrix Chernoff bound for Markov chains can be applied to analyze the behavior of co-occurrence statistics for sequential data, which have been common and important data signals in machine learning. We show that given a regular Markov chain with $n$ states and mixing time $τ$, we need a trajectory of length $O(τ(\log{(n)}+\log{(τ)})/ε^2)$ to achieve an estimator of the co-occurrence matrix with error bound $ε$. We conduct several experiments and the experimental results are consistent with the exponentially fast convergence rate from theoretical analysis. Our result gives the first bound on the convergence rate of the co-occurrence matrix and the first sample complexity analysis in graph representation learning.

stat.ML↗

arXiv2020

Faster Graph Embeddings via Coarsening

Graph embeddings are a ubiquitous tool for machine learning tasks, such as node classification and link prediction, on graph-structured data. However, computing the embeddings for large-scale graphs is prohibitively inefficient even if we are interested only in a small subset of relevant vertices. To address this, we present an efficient graph coarsening approach, based on Schur complements, for computing the embedding of the relevant vertices. We prove that these embeddings are preserved exactly by the Schur complement graph that is obtained via Gaussian elimination on the non-relevant vertices. As computing Schur complements is expensive, we give a nearly-linear time algorithm that generates a coarsened graph on the relevant vertices that provably matches the Schur complement in expectation in each iteration. Our experiments involving prediction tasks on graphs demonstrate that computing embeddings on the coarsened graph, rather than the entire graph, leads to significant time savings without sacrificing accuracy.

cs.LG↗

arXiv2020

Understanding the hardness of approximate query processing with joins

We study the hardness of Approximate Query Processing (AQP) of various types of queries involving joins over multiple tables of possibly different sizes. In the case where the query result is a single value (e.g., COUNT, SUM, and COUNT(DISTINCT)), we prove worst-case information-theoretic lower bounds for AQP problems that are given parameters $ε$ and $δ$, and return estimated results within a factor of 1+$ε$ of the true results with error probability at most $δ$. In particular, the lower bounds for cardinality estimation over joins under various settings are contained in our results. Informally, our results show that for various database queries with joins, unless restricted to the set of queries whose results are always guaranteed to be above a very large threshold, the amount of information an AQP algorithm needs for returning an accurate approximation is at least linear in the number of rows in the largest table. Similar lower bounds even hold for some special cases where additional information such as top-K heavy hitters and all frequency vectors are available. In the case of GROUP-BY where the query result is not a single number, we study the lower bound for the amount of information used by any approximation algorithm that does not report any non-existing group and does not miss groups of large total size. Our work extends the work of Alon, Gibbons, Matias, and Szegedy [AGMS99].We compare our lower bounds with the amount of information required by Bernoulli sampling to give an accurate approximation. For COUNT queries with joins over multiple tables of the same size, the upper bound matches the lower bound, unless the problem setting is restricted to the set of queries whose results are always guaranteed to be above a very large threshold.

cs.DB↗

arXiv2020

GEE-TGDR: A longitudinal feature selection algorithm and its application to lncRNA expression profiles for psoriasis patients treated with immune therapies

With the fast evolution of high-throughput technology, longitudinal gene expression experiments have become affordable and increasingly common in biomedical fields. Generalized estimating equation (GEE) approach is a widely used statistical method for the analysis of longitudinal data. Feature selection is imperative in longitudinal omics data analysis. Among a variety of existing feature selection methods, an embedded method, namely, threshold gradient descent regularization (TGDR) stands out due to its excellent characteristics. An alignment of GEE with TGDR is a promising area for the purpose of identifying relevant markers that can explain the dynamic changes of outcomes across time. In this study, we proposed a new novel feature selection algorithm for longitudinal outcomes:GEE-TGDR. In the GEE-TGDR method, the corresponding quasi-likelihood function of a GEE model is the objective function to be optimized and the optimization and feature selection are accomplished by the TGDR method. We applied the GEE-TGDR method a longitudinal lncRNA gene expression dataset that examined the treatment response of psoriasis patients to immune therapy. Under different working correlation structures, a list including 10 relevant lncRNAs were identified with a predictive accuracy of 80 % and meaningful biological interpretation. To conclude, a widespread application of the proposed GEE-TGDR method in omics data analysis is anticipated.

stat.AP↗

arXiv2020

ALEX: An Updatable Adaptive Learned Index

Recent work on "learned indexes" has changed the way we look at the decades-old field of DBMS indexing. The key idea is that indexes can be thought of as "models" that predict the position of a key in a dataset. Indexes can, thus, be learned. The original work by Kraska et al. shows that a learned index beats a B+Tree by a factor of up to three in search time and by an order of magnitude in memory footprint. However, it is limited to static, read-only workloads. In this paper, we present a new learned index called ALEX which addresses practical issues that arise when implementing learned indexes for workloads that contain a mix of point lookups, short range queries, inserts, updates, and deletes. ALEX effectively combines the core insights from learned indexes with proven storage and indexing techniques to achieve high performance and low memory footprint. On read-only workloads, ALEX beats the learned index from Kraska et al. by up to 2.2X on performance with up to 15X smaller index size. Across the spectrum of read-write workloads, ALEX beats B+Trees by up to 4.1X while never performing worse, with up to 2000X smaller index size. We believe ALEX presents a key step towards making learned indexes practical for a broader class of database workloads with dynamic updates.

cs.DB↗

arXiv2020

Qd-tree: Learning Data Layouts for Big Data Analytics

Corporations today collect data at an unprecedented and accelerating scale, making the need to run queries on large datasets increasingly important. Technologies such as columnar block-based data organization and compression have become standard practice in most commercial database systems. However, the problem of best assigning records to data blocks on storage is still open. For example, today's systems usually partition data by arrival time into row groups, or range/hash partition the data based on selected fields. For a given workload, however, such techniques are unable to optimize for the important metric of the number of blocks accessed by a query. This metric directly relates to the I/O cost, and therefore performance, of most analytical queries. Further, they are unable to exploit additional available storage to drive this metric down further. In this paper, we propose a new framework called a query-data routing tree, or qd-tree, to address this problem, and propose two algorithms for their construction based on greedy and deep reinforcement learning techniques. Experiments over benchmark and real workloads show that a qd-tree can provide physical speedups of more than an order of magnitude compared to current blocking schemes, and can reach within 2X of the lower bound for data skipping based on selectivity, while providing complete semantic descriptions of created blocks.

cs.DB↗

arXiv2020

Evolution of the Earth's Magnetosheath Turbulence: A statistical study based on MMS observations

Composed of shocked solar wind, the Earth's magnetosheath serves as a natural laboratory to study the transition of turbulence from low Alfv{é}n Mach number, $M_\mathrm{A}$, to high $M_\mathrm{A}$. The simultaneous observations of magnetic field and plasma moments with unprecedented high temporal resolution provided by NASA's \textit{Magnetospheric Multiscale} Mission enable us to study the magnetosheath turbulence at both magnetohydrodynamics (MHD) and sub-ion scales. Based on 1841 burst-mode segments of MMS-1 from 2015/09 to 2019/06, comprehensive patterns of the spatial evolution of magnetosheath turbulences are obtained: (1) from the sub-solar region to the flanks, $M_\mathrm{A}$ increases from $<$ 1 to $>$ 5. At MHD scales, the spectral indices of the magnetic-field and velocity spectra present a positive and negative correlation with $M_\mathrm{A}$. However, no obvious correlations between the spectral indices and $M_\mathrm{A}$ are found at sub-ion scales. (2) from the bow shock to the magnetopause, the turbulent sonic Mach number, $M_{\mathrm{turb}}$, generally decreases from $>$ 0.4 to $<$ 0.1. All spectra steepen at MHD scales and flatten at sub-ion scales, representing a positive/negative correlations with $M_\mathrm{turb}$. The break frequency increases by 0.1 Hz when approaching the magnetopause for the magnetic-field and velocity spectra, while it remains at 0.3 Hz for the density spectra. (3) In spite of some differences, similar results are found for the quasi-parallel and quasi-perpendicular magnetosheath. In addition, the spatial evolution of magnetosheath turbulence is found to be independent of the upstream solar wind conditions, e.g., the Z-component of the interplanetary magnetic field and the solar wind speed.

physics.space-ph↗

arXiv2020

TaxoExpan: Self-supervised Taxonomy Expansion with Position-Enhanced Graph Neural Network

Taxonomies consist of machine-interpretable semantics and provide valuable knowledge for many web applications. For example, online retailers (e.g., Amazon and eBay) use taxonomies for product recommendation, and web search engines (e.g., Google and Bing) leverage taxonomies to enhance query understanding. Enormous efforts have been made on constructing taxonomies either manually or semi-automatically. However, with the fast-growing volume of web content, existing taxonomies will become outdated and fail to capture emerging knowledge. Therefore, in many applications, dynamic expansions of an existing taxonomy are in great demand. In this paper, we study how to expand an existing taxonomy by adding a set of new concepts. We propose a novel self-supervised framework, named TaxoExpan, which automatically generates a set of pairs from the existing taxonomy as training data. Using such self-supervision data, TaxoExpan learns a model to predict whether a query concept is the direct hyponym of an anchor concept. We develop two innovative techniques in TaxoExpan: (1) a position-enhanced graph neural network that encodes the local structure of an anchor concept in the existing taxonomy, and (2) a noise-robust training objective that enables the learned model to be insensitive to the label noise in the self-supervision data. Extensive experiments on three large-scale datasets from different domains demonstrate both the effectiveness and the efficiency of TaxoExpan for taxonomy expansion.

cs.CL↗

arXiv2019

Towards Extracting Highlights From Recorded Live Videos: An Implicit Crowdsourcing Approach

Live streaming platforms need to store a lot of recorded live videos on a daily basis. An important problem is how to automatically extract highlights (i.e., attractive short video clips) from these massive, long recorded live videos. One approach is to directly apply a highlight extraction algorithm to video content. However, algorithmic approaches are either domain-specific, which require experts to spend a long time to design, or resource-intensive, which require a lot of training data and/or computing resources. In this paper, we propose Lightor, a novel implicit crowdsourcing approach to overcome these limitations. The key insight is to collect users' natural interactions with a live streaming platform, and then leverage them to detect highlights. Lightor consists of two major components. Highlight Initializer collects time-stamped chat messages from a live video and then uses them to predict approximate highlight positions. Highlight Extractor keeps track of how users interact with these approximate highlight positions and then refines these positions iteratively. We find that the collected user chat and interaction data are very noisy, and propose effective techniques to deal with noise. Lightor can be easily deployed into existing live streaming platforms, or be implemented as a web browser extension. We recruit hundreds of users from Amazon Mechanical Turk, and evaluate the performance of Lightor using two popular games in Twitch. The results show that Lightor can achieve high extraction precision with a small set of training data and low computing resources.

cs.DB↗

arXiv2019

Machine Learning Approach for Solar Wind Categorization

Solar wind classification is conducive to understand the physical processes ongoing at the Sun and solar wind evolution in the interplanetary space, and furthermore, it is helpful for early warning of space weather events. With rapid developments in the field of artificial intelligence, machine learning approaches are increasingly being used for pattern recognition. In this study, an approach from machine learning perspectives is developed to automatically classify the solar wind at 1 AU into four types: coronal-hole-origin, streamer-belt-origin, sector-reversal-region-origin, and ejecta. By exhaustive enumeration, an eight-dimensional scheme ($B_T$, $N_P$, $T_P$, $V_P$, $N_{αp}$, $T_{exp}/T_P$, $S_p$, and $M_f$) is found to perform the best among 8191 combinations of 13 solar wind parameters. 10 popular supervised machine learning models, namely $k$ Nearest Neighbors (KNN), Support Vector Machines with linear and Radial Basic Function kernels, Decision Tree, Random Forest, Adaptive Boosting, Neural Network, Gaussian Naive Bayes, Quadratic Discriminant Analysis, and Extreme Gradient Boosting, are applied to the labeled solar wind data sets. Among them, KNN classifier obtains the highest overall classification accuracy, 92.8%. It significantly improves the accuracy by 9.6% over existing manual schemes. No solar wind composition measurements are needed, permitting our classification scheme to be applied to most solar wind spacecraft data. Besides, two application examples indicate that solar wind classification is helpful for the risk evaluation of predicted magnetic storms and surface charging of geosynchronous spacecrafts.

physics.space-ph↗

arXiv2019

Hour-scale persistent negative anomaly of atmospheric electrostatic field near the epicenter before earthquake

Although earthquake prediction is a big challenge in the world, some simple observational tools can capture many physical signals and demonstrate that an earthquake (EQ) may be forthcoming in short period. Many researchers have studied the significant variation of atmospheric electrostatic field related to the forthcoming earthquake. However, until now, there is not a compelling physical mechanism which can explain why atmospheric electrostatic abnormal signal could appear just before an earthquake. Here we present a precursor signal and propose a brief physical interpretation. Under fair air conditions, if the near-surface atmospheric electrostatic field Ez (oriented down when it is positive) presents a very stable negative anomaly (from -100 V/m to -5000 V/m), it will forebode that an earthquake (seismicity from 3-8) would take place in the next several to tens of hours within a distance less than 100 km. We name this prediction technique as "DC Ez Determination"(DED). In addition, the mechanism of such abnormal quasi-static electric field before a forthcoming earthquake has been proposed here: (1) Radon gas releases from the rock clefts near the epicenter during seismogenic process. (2) α particles are produced due to the radioactive decay of Radon gas in the air. (3) α particle ionizing radiation creates more positive particles in the air, which is much more effective than that β and γ particles produced during Radon radioactive decay. (4) The new positive particles change formal positive atmospheric electric field (Ez) into stably negative near the earth-surface. (5) The closer the instrument is to the epicenter, the more likely it is to observe the negative Ez signal related to the earthquake. It is recommended to establish an instrument network to capture the reliable precursor and make some warnings before the earthquake disaster.

physics.geo-ph↗