arXiv subjects

Yunyi Zhang

Publications and source records attributed to Yunyi Zhang.

50 records · Page 3Linked to original sources

arXiv2021

Debiased and threshold ridge regression for linear model with heteroskedastic and dependent error

Focusing on a high dimensional linear model $y = Xβ+ ε$ with dependent, non-stationary, and heteroskedastic errors, this paper applies the debiased and threshold ridge regression method that gives a consistent estimator for linear combinations of $β$; and derives a Gaussian approximation theorem for the estimator. Besides, it proposes a dependent wild bootstrap algorithm to construct the estimator's confidence intervals and perform hypothesis testing. Numerical experiments on the proposed estimator and the bootstrap algorithm show that they have favorable finite sample performance. Research on a high dimensional linear model with dependent(non-stationary) errors is sparse, and our work should bring some new insights to this field.

math.ST↗

arXiv2021

Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Training

We study the problem of training named entity recognition (NER) models using only distantly-labeled data, which can be automatically obtained by matching entity mentions in the raw text with entity types in a knowledge base. The biggest challenge of distantly-supervised NER is that the distant supervision may induce incomplete and noisy labels, rendering the straightforward application of supervised learning ineffective. In this paper, we propose (1) a noise-robust learning scheme comprised of a new loss function and a noisy label removal step, for training NER models on distantly-labeled data, and (2) a self-training method that uses contextualized augmentations created by pre-trained language models to improve the generalization ability of the NER model. On three benchmark datasets, our method achieves superior performance, outperforming existing distantly-supervised NER models by significant margins.

cs.CL↗

arXiv2021

Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup

Contrastive learning has been applied successfully to learn vector representations of text. Previous research demonstrated that learning high-quality representations benefits from batch-wise contrastive loss with a large number of negatives. In practice, the technique of in-batch negative is used, where for each example in a batch, other batch examples' positives will be taken as its negatives, avoiding encoding extra negatives. This, however, still conditions each example's loss on all batch examples and requires fitting the entire large batch into GPU memory. This paper introduces a gradient caching technique that decouples backpropagation between contrastive loss and the encoder, removing encoder backward pass data dependency along the batch dimension. As a result, gradients can be computed for one subset of the batch at a time, leading to almost constant memory usage.

cs.LG↗

arXiv2021

Ridge Regression Revisited: Debiasing, Thresholding and Bootstrap

The success of the Lasso in the era of high-dimensional data can be attributed to its conducting an implicit model selection, i.e., zeroing out regression coefficients that are not significant. By contrast, classical ridge regression can not reveal a potential sparsity of parameters, and may also introduce a large bias under the high-dimensional setting. Nevertheless, recent work on the Lasso involves debiasing and thresholding, the latter in order to further enhance the model selection. As a consequence, ridge regression may be worth another look since -- after debiasing and thresholding -- it may offer some advantages over the Lasso, e.g., it can be easily computed using a closed-form expression. % and it has similar performance to threshold Lasso. In this paper, we define a debiased and thresholded ridge regression method, and prove a consistency result and a Gaussian approximation theorem. We further introduce a wild bootstrap algorithm to construct confidence regions and perform hypothesis testing for a linear combination of parameters. In addition to estimation, we consider the problem of prediction, and present a novel, hybrid bootstrap algorithm tailored for prediction intervals. Extensive numerical simulations further show that the debiased and thresholded ridge regression has favorable finite sample performance and may be preferable in some settings.

math.ST↗

arXiv2021

Bootstrap prediction intervals with asymptotic conditional validity and unconditional guarantees

It can be argued that optimal prediction should take into account all available data. Therefore, to evaluate a prediction interval's performance one should employ conditional coverage probability, conditioning on all available observations. Focusing on a linear model, we derive the asymptotic distribution of the difference between the conditional coverage probability of a nominal prediction interval and the conditional coverage probability of a prediction interval obtained via a residual-based bootstrap. Applying this result, we show that a prediction interval generated by the residual-based bootstrap has approximately 50% probability to yield conditional under-coverage. We then develop a new bootstrap algorithm that generates a prediction interval that asymptotically controls both the conditional coverage probability as well as the possibility of conditional under-coverage. We complement the asymptotic results with several finite-sample simulations.

math.ST↗

arXiv2020

Semi-parametric estimation and prediction intervals in state space models

Literatures in state space models focus on parametric inference and prediction, which fail if the state space model is not fully specified and the maximum likelihood estimation does not work. In this paper, we assume the state transition matrix and the distribution of state noises are unknown. Under this assumption, we provide methods to consistently estimate these terms. In addition, we introduce an algorithm to construct consistent prediction intervals for state vectors and future observations. We complement the asymptotic results with several numerical experiments.

math.ST↗

arXiv2020

Text Classification Using Label Names Only: A Language Model Self-Training Approach

Current text classification methods typically require a good number of human-labeled documents as training data, which can be costly and difficult to obtain in real applications. Humans can perform classification without seeing any labeled examples but only based on a small set of words describing the categories to be classified. In this paper, we explore the potential of only using the label name of each class to train classification models on unlabeled data, without using any labeled documents. We use pre-trained neural language models both as general linguistic knowledge sources for category understanding and as representation learning models for document classification. Our method (1) associates semantically related words with the label names, (2) finds category-indicative words and trains the model to predict their implied categories, and (3) generalizes the model via self-training. We show that our model achieves around 90% accuracy on four benchmark datasets including topic and sentiment classification without using any labeled documents but learning from unlabeled data supervised by at most 3 words (1 in most cases) per class as the label name.

cs.CL↗

arXiv2020

CoRel: Seed-Guided Topical Taxonomy Construction by Concept Learning and Relation Transferring

Taxonomy is not only a fundamental form of knowledge representation, but also crucial to vast knowledge-rich applications, such as question answering and web search. Most existing taxonomy construction methods extract hypernym-hyponym entity pairs to organize a "universal" taxonomy. However, these generic taxonomies cannot satisfy user's specific interest in certain areas and relations. Moreover, the nature of instance taxonomy treats each node as a single word, which has low semantic coverage. In this paper, we propose a method for seed-guided topical taxonomy construction, which takes a corpus and a seed taxonomy described by concept names as input, and constructs a more complete taxonomy based on user's interest, wherein each node is represented by a cluster of coherent terms. Our framework, CoRel, has two modules to fulfill this goal. A relation transferring module learns and transfers the user's interested relation along multiple paths to expand the seed taxonomy structure in width and depth. A concept learning module enriches the semantics of each concept node by jointly embedding the taxonomy and text. Comprehensive experiments conducted on real-world datasets show that Corel generates high-quality topical taxonomies and outperforms all the baselines significantly.

cs.CL↗

arXiv2020

Hierarchical Topic Mining via Joint Spherical Tree and Text Embedding

Mining a set of meaningful topics organized into a hierarchy is intuitively appealing since topic correlations are ubiquitous in massive text corpora. To account for potential hierarchical topic structures, hierarchical topic models generalize flat topic models by incorporating latent topic hierarchies into their generative modeling process. However, due to their purely unsupervised nature, the learned topic hierarchy often deviates from users' particular needs or interests. To guide the hierarchical topic discovery process with minimal user supervision, we propose a new task, Hierarchical Topic Mining, which takes a category tree described by category names only, and aims to mine a set of representative terms for each category from a text corpus to help a user comprehend his/her interested topics. We develop a novel joint tree and text embedding method along with a principled optimization procedure that allows simultaneous modeling of the category tree structure and the corpus generative process in the spherical space for effective category-representative term discovery. Our comprehensive experiments show that our model, named JoSH, mines a high-quality set of hierarchical topics with high efficiency and benefits weakly-supervised hierarchical text classification tasks.

cs.CL↗

arXiv2020

Empower Entity Set Expansion via Language Model Probing

Entity set expansion, aiming at expanding a small seed entity set with new entities belonging to the same semantic class, is a critical task that benefits many downstream NLP and IR applications, such as question answering, query understanding, and taxonomy construction. Existing set expansion methods bootstrap the seed entity set by adaptively selecting context features and extracting new entities. A key challenge for entity set expansion is to avoid selecting ambiguous context features which will shift the class semantics and lead to accumulative errors in later iterations. In this study, we propose a novel iterative set expansion framework that leverages automatically generated class names to address the semantic drift issue. In each iteration, we select one positive and several negative class names by probing a pre-trained language model, and further score each candidate entity based on selected class names. Experiments on two datasets show that our framework generates high-quality class names and outperforms previous state-of-the-art methods significantly.

cs.CL↗

arXiv2020

Complexity of Leading Digit Sequences

Let $S_{a,b}$ denote the sequence of leading digits of $a^n$ in base $b$. It is well known that if $a$ is not a rational power of $b$, then the sequence $S_{a,b}$ satisfies Benford's Law; that is, digit $d$ occurs in $S_{a,b}$ with frequency $\log_{b}(1+1/d)$, for $d=1,2,\dots,b-1$. In this paper, we investigate the \emph{complexity} of such sequences. We focus mainly on the \emph{block complexity}, $p_{a,b}(n)$, defined as the number of distinct blocks of length $n$ appearing in $S_{a,b}$. In our main result we determine $p_{a,b}(n)$ for all squarefree bases $b\ge 5$ and all rational numbers $a>0$ that are not integral powers of $b$. In particular, we show that, for all such pairs $(a,b)$, the complexity function $p_{a,b}(n)$ is \emph{affine}, i.e., satisfies $p_{a,b}(n)=c_{a,b} n + d_{a,b}$ for all $n\ge1$, with coefficients $c_{a,b}\ge1$ and $d_{a,b}\ge0$, given explicitly in terms of $a$ and $b$. We also show that the requirement that $b$ be squarefree cannot be dropped: If $b$ is not squarefree, then there exist integers $a$ with $1<a<b$ for which $p_{a,b}(n)$ is not of the above form. We use this result to obtain sharp upper and lower bounds for $p_{a,b}(n)$, and to determine the asymptotic behavior of this function as $b\to\infty$ through squarefree values. We also consider the question which linear functions $p(n)=cn+d$ arise as the complexity function $p_{a,b}(n)$ of some leading digit sequence $S_{a,b}$. We conclude with a discussion of other complexity measures for the sequences $S_{a,b}$ and some open problems.

math.NT↗

arXiv2020

Guiding Corpus-based Set Expansion by Auxiliary Sets Generation and Co-Expansion

Given a small set of seed entities (e.g., ``USA'', ``Russia''), corpus-based set expansion is to induce an extensive set of entities which share the same semantic class (Country in this example) from a given corpus. Set expansion benefits a wide range of downstream applications in knowledge discovery, such as web search, taxonomy construction, and query suggestion. Existing corpus-based set expansion algorithms typically bootstrap the given seeds by incorporating lexical patterns and distributional similarity. However, due to no negative sets provided explicitly, these methods suffer from semantic drift caused by expanding the seed set freely without guidance. We propose a new framework, Set-CoExpan, that automatically generates auxiliary sets as negative sets that are closely related to the target set of user's interest, and then performs multiple sets co-expansion that extracts discriminative features by comparing target set with auxiliary sets, to form multiple cohesive sets that are distinctive from one another, thus resolving the semantic drift issue. In this paper we demonstrate that by generating auxiliary sets, we can guide the expansion process of target set to avoid touching those ambiguous areas around the border with auxiliary sets, and we show that Set-CoExpan outperforms strong baseline methods significantly.

cs.CL↗

arXiv2018

$L_p$ and almost sure convergence of estimation on heavy tail index under random censoring

In this paper, we prove $L_p,\ p\geq 2$ and almost sure convergence of tail index estimator mentioned in \cite{grama2008} under random censoring and several assumptions. $p$th moment of the error of the estimator is proved to be of order $O\left(\frac{1}{\log^{mκ/2}n}\right)$ with given assumptions. We also perform several finite sample simulations to quantify performance of this estimator. Finite sample results show that the proposed estimator is effective in finding underlying tail index even when censor rate is high.

math.ST↗

arXiv2018

Monotone function estimator and its application

In this paper, the model $Y_i=g(Z_i),\ i=1,2,...,n$ with $Z_i$ being random variables with known distribution and $g(x)$ being unknown strictly increasing function is proposed and almost sure convergence of estimator for $g(x)$ is proved for i.i.d and short range dependent data. Confidence intervals and bands are constructed for i.i.d data theoretically and confidence intervals are introduced for short range dependent data through resampling. Besides, a test for equivalence of $g(x)$ to the desired function is proposed. Finite sample analysis and application of this model on an urban waste water treatment plant's data is demonstrated as well.

math.ST↗