arXiv subjects

Bin Yu

Publications and source records attributed to Bin Yu.

At least 127 records · Page 7Linked to original sources

arXiv2022

Seven Principles for Rapid-Response Data Science: Lessons Learned from Covid-19 Forecasting

In this article, we take a step back to distill seven principles out of our experience in the spring of 2020, when our 12-person rapid-response team used skills of data science and beyond to help distribute Covid PPE. This process included tapping into domain knowledge of epidemiology and medical logistics chains, curating a relevant data repository, developing models for short-term county-level death forecasting in the US, and building a website for sharing visualization (an automated AI machine). The principles are described in the context of working with Response4Life, a then-new nonprofit organization, to illustrate their necessity. Many of these principles overlap with those in standard data-science teams, but an emphasis is put on dealing with problems that require rapid response, often resembling agile software development.

stat.AP↗

arXiv2022

Provable Boolean Interaction Recovery from Tree Ensemble obtained via Random Forests

Random Forests (RF) are at the cutting edge of supervised machine learning in terms of prediction performance, especially in genomics. Iterative Random Forests (iRF) use a tree ensemble from iteratively modified RF to obtain predictive and stable non-linear or Boolean interactions of features. They have shown great promise for Boolean biological interaction discovery that is central to advancing functional genomics and precision medicine. However, theoretical studies into how tree-based methods discover Boolean feature interactions are missing. Inspired by the thresholding behavior in many biological processes, we first introduce a novel discontinuous nonlinear regression model, called the Locally Spiky Sparse (LSS) model. Specifically, the LSS model assumes that the regression function is a linear combination of piecewise constant Boolean interaction terms. Given an RF tree ensemble, we define a quantity called Depth-Weighted Prevalence (DWP) for a set of signed features S. Intuitively speaking, DWP(S) measures how frequently features in S appear together in an RF tree ensemble. We prove that, with high probability, DWP(S) attains a universal upper bound that does not involve any model coefficients, if and only if S corresponds to a union of Boolean interactions under the LSS model. Consequentially, we show that a theoretically tractable version of the iRF procedure, called LSSFind, yields consistent interaction discovery under the LSS model as the sample size goes to infinity. Finally, simulation results show that LSSFind recovers the interactions under the LSS model even when some assumptions are violated.

math.ST↗

arXiv2022

Instability, Computational Efficiency and Statistical Accuracy

Many statistical estimators are defined as the fixed point of a data-dependent operator, with estimators based on minimizing a cost function being an important special case. The limiting performance of such estimators depends on the properties of the population-level operator in the idealized limit of infinitely many samples. We develop a general framework that yields bounds on statistical accuracy based on the interplay between the deterministic convergence rate of the algorithm at the population level, and its degree of (in)stability when applied to an empirical object based on $n$ samples. Using this framework, we analyze both stable forms of gradient descent and some higher-order and unstable algorithms, including Newton's method and its cubic-regularized variant, as well as the EM algorithm. We provide applications of our general results to several concrete classes of models, including Gaussian mixture estimation, non-linear regression models, and informative non-response models. We exhibit cases in which an unstable algorithm can achieve the same statistical accuracy as a stable algorithm in exponentially fewer steps -- namely, with the number of iterations being reduced from polynomial to logarithmic in sample size $n$.

cs.LG↗

arXiv2022

Hierarchical Shrinkage: improving the accuracy and interpretability of tree-based methods

Tree-based models such as decision trees and random forests (RF) are a cornerstone of modern machine-learning practice. To mitigate overfitting, trees are typically regularized by a variety of techniques that modify their structure (e.g. pruning). We introduce Hierarchical Shrinkage (HS), a post-hoc algorithm that does not modify the tree structure, and instead regularizes the tree by shrinking the prediction over each node towards the sample means of its ancestors. The amount of shrinkage is controlled by a single regularization parameter and the number of data points in each ancestor. Since HS is a post-hoc method, it is extremely fast, compatible with any tree growing algorithm, and can be used synergistically with other regularization techniques. Extensive experiments over a wide variety of real-world datasets show that HS substantially increases the predictive performance of decision trees, even when used in conjunction with other regularization techniques. Moreover, we find that applying HS to each tree in an RF often improves accuracy, as well as its interpretability by simplifying and stabilizing its decision boundaries and SHAP values. We further explain the success of HS in improving prediction performance by showing its equivalence to ridge regression on a (supervised) basis constructed of decision stumps associated with the internal nodes of a tree. All code and models are released in a full-fledged package available on Github (github.com/csinva/imodels)

cs.LG↗

arXiv2021

A Novel Load Balancing Scheme for Mobile Edge Computing

To overcome long propagation delays for data exchange between the remote cloud data center and end devices in Mobile Cloud Computing (MCC), Mobile Edge Computing (MEC) is merging to push mobile computing, network control and storage to the network edges. A cloudlet in MEC is a mobility-enhanced small-scale cloud, which contains several MEC servers located in close proximity to mobile devices. The main purpose of a cloudlet is to stably provide services to mobile devices with low latency. When a cloudlet offers hundreds kinds of services to millions of mobile devices, it is critical to balance the loads so as to improve performance. In this paper, we propose a three-layer mobile hybrid hierarchical P2P (MHP2P) model as a cloudlet. MHP2P performs satisfactory service lookup efficiency and system scalability as well as high stability. More importantly, a load balance scheme is provided so that the loads of MHP2P can be well balanced with the increasing of MEC servers and query load. A large number of simulation experiments indicate that the proposed scheme is efficient for enhancing the load balancing performance in MHP2P based MEC systems.

cs.DC↗

arXiv2021

Sharp Analysis of Expectation-Maximization for Weakly Identifiable Models

We study a class of weakly identifiable location-scale mixture models for which the maximum likelihood estimates based on $n$ i.i.d. samples are known to have lower accuracy than the classical $n^{- \frac{1}{2}}$ error. We investigate whether the Expectation-Maximization (EM) algorithm also converges slowly for these models. We provide a rigorous characterization of EM for fitting a weakly identifiable Gaussian mixture in a univariate setting where we prove that the EM algorithm converges in order $n^{\frac{3}{4}}$ steps and returns estimates that are at a Euclidean distance of order ${ n^{- \frac{1}{8}}}$ and ${ n^{-\frac{1} {4}}}$ from the true location and scale parameter respectively. Establishing the slow rates in the univariate setting requires a novel localization argument with two stages, with each stage involving an epoch-based argument applied to a different surrogate EM operator at the population level. We demonstrate several multivariate ($d \geq 2$) examples that exhibit the same slow rates as the univariate case. We also prove slow statistical rates in higher dimensions in a special case, when the fitted covariance is constrained to be a multiple of the identity.

math.ST↗

arXiv2021

The Three Stages of Learning Dynamics in High-Dimensional Kernel Methods

To understand how deep learning works, it is crucial to understand the training dynamics of neural networks. Several interesting hypotheses about these dynamics have been made based on empirically observed phenomena, but there exists a limited theoretical understanding of when and why such phenomena occur. In this paper, we consider the training dynamics of gradient flow on kernel least-squares objectives, which is a limiting dynamics of SGD trained neural networks. Using precise high-dimensional asymptotics, we characterize the dynamics of the fitted model in two "worlds": in the Oracle World the model is trained on the population distribution and in the Empirical World the model is trained on a sampled dataset. We show that under mild conditions on the kernel and $L^2$ target regression function the training dynamics undergo three stages characterized by the behaviors of the models in the two worlds. Our theoretical results also mathematically formalize some interesting deep learning phenomena. Specifically, in our setting we show that SGD progressively learns more complex functions and that there is a "deep bootstrap" phenomenon: during the second stage, the test error of both worlds remain close despite the empirical training error being much smaller. Finally, we give a concrete example comparing the dynamics of two different kernels which shows that faster training is not necessary for better generalization.

stat.ML↗

arXiv2021

A cautionary tale on fitting decision trees to data from additive models: generalization lower bounds

Decision trees are important both as interpretable models amenable to high-stakes decision-making, and as building blocks of ensemble methods such as random forests and gradient boosting. Their statistical properties, however, are not well understood. The most cited prior works have focused on deriving pointwise consistency guarantees for CART in a classical nonparametric regression setting. We take a different approach, and advocate studying the generalization performance of decision trees with respect to different generative regression models. This allows us to elicit their inductive bias, that is, the assumptions the algorithms make (or do not make) to generalize to new data, thereby guiding practitioners on when and how to apply these methods. In this paper, we focus on sparse additive generative models, which have both low statistical complexity and some nonparametric flexibility. We prove a sharp squared error generalization lower bound for a large class of decision tree algorithms fitted to sparse additive models with $C^1$ component functions. This bound is surprisingly much worse than the minimax rate for estimating such sparse additive models. The inefficiency is due not to greediness, but to the loss in power for detecting global structure when we average responses solely over each leaf, an observation that suggests opportunities to improve tree-based algorithms, for example, by hierarchical shrinkage. To prove these bounds, we develop new technical machinery, establishing a novel connection between decision tree estimation and rate-distortion theory, a sub-field of information theory.

stat.ML↗

arXiv2021

On mixing enhancement by secondary baroclinic vorticity in shock-bubble interaction

To investigate the intrinsic mechanism for mixing enhancement by variable density behaviour, a canonical variable density (VD) mixing extracted from a supersonic streamwise vortex protocol, shock bubble interaction (SBI), is numerically studied and compared with a counterpart of passive scalar (PS) mixing. It is meaningful to observe that the maximum concentration decays much faster in VD SBI than in PS SBI regardless of the shock Mach number (Ma = 1.22 ~ 4). The quasi-Lamb-Oseen type velocity distribution in the PS SBI is found by analyzing the azimuthal velocity that stretches the bubble. Meanwhile, for the VD SBI, an additional stretching enhanced by the secondary baroclinic vorticity (SBV) production contributes to the faster-mixing decay. The underlying mechanism of the SBV-enhanced stretching is further revealed through the density and velocity difference between the shocked light bubble and the heavy ambient air. By combining the SBV-accelerated stretching model and the initial shock compression, a novel mixing time estimation for VD SBI is theoretically proposed by solving the advection-diffusion equation under a deformation field of an axisymmetric vortex with the additional SBV induced azimuthal velocity. Based on the mixing time model, a mixing enhancement number defined by the ratio of VD and PS mixing time further reveals the contribution from the variable-density effect, which implies a better control of density distribution for mixing enhancement in a supersonic streamwise vortex.

physics.flu-dyn↗

arXiv2021

Adaptive wavelet distillation from neural networks through interpretations

Recent deep-learning models have achieved impressive prediction performance, but often sacrifice interpretability and computational efficiency. Interpretability is crucial in many disciplines, such as science and medicine, where models must be carefully vetted or where interpretation is the goal itself. Moreover, interpretable models are concise and often yield computational efficiency. Here, we propose adaptive wavelet distillation (AWD), a method which aims to distill information from a trained neural network into a wavelet transform. Specifically, AWD penalizes feature attributions of a neural network in the wavelet domain to learn an effective multi-resolution wavelet transform. The resulting model is highly predictive, concise, computationally efficient, and has properties (such as a multi-scale structure) which make it easy to interpret. In close collaboration with domain experts, we showcase how AWD addresses challenges in two real-world settings: cosmological parameter inference and molecular-partner prediction. In both cases, AWD yields a scientifically interpretable and concise model which gives predictive performance better than state-of-the-art neural networks. Moreover, AWD identifies predictive features that are scientifically meaningful in the context of respective domains. All code and models are released in a full-fledged package available on Github (https://github.com/Yu-Group/adaptive-wavelets).

stat.ML↗

arXiv2021

Interpreting and improving deep-learning models with reality checks

Recent deep-learning models have achieved impressive predictive performance by learning complex functions of many variables, often at the cost of interpretability. This chapter covers recent work aiming to interpret models by attributing importance to features and feature groups for a single prediction. Importantly, the proposed attributions assign importance to interactions between features, in addition to features in isolation. These attributions are shown to yield insights across real-world domains, including bio-imaging, cosmology image and natural-language processing. We then show how these attributions can be used to directly improve the generalization of a neural network or to distill it into a simple model. Throughout the chapter, we emphasize the use of reality checks to scrutinize the proposed interpretation techniques.

stat.ML↗

arXiv2021

Transformation Importance with Applications to Cosmology

Machine learning lies at the heart of new possibilities for scientific discovery, knowledge generation, and artificial intelligence. Its potential benefits to these fields requires going beyond predictive accuracy and focusing on interpretability. In particular, many scientific problems require interpretations in a domain-specific interpretable feature space (e.g. the frequency domain) whereas attributions to the raw features (e.g. the pixel space) may be unintelligible or even misleading. To address this challenge, we propose TRIM (TRansformation IMportance), a novel approach which attributes importances to features in a transformed space and can be applied post-hoc to a fully trained model. TRIM is motivated by a cosmological parameter estimation problem using deep neural networks (DNNs) on simulated data, but it is generally applicable across domains/models and can be combined with any local interpretation method. In our cosmology example, combining TRIM with contextual decomposition shows promising results for identifying which frequencies a DNN uses, helping cosmologists to understand and validate that the model learns appropriate physical features rather than simulation artifacts.

stat.ML↗

arXiv2021

Scaling behavior of density gradient accelerated mixing rate in shock bubble interaction

Variable-density mixing in shock bubble interaction, a canonical flow of Richtermyer-Meshkov instability, is studied by the high-resolution simulation. While the dissipation mainly controls the passive scalar mixing rate, an objective definition of variable-density mixing rate characterizing the macroscopic mixing formation is still lacking, and the fundamental behavior of mixing rate evolution is not yet well understood. Here, we first show that the variable-density mixing of shock bubble interaction is distinctly different from the previous observations in the passive scalar mixing. The widely-accepted hyperbolic conservation of the first moment of concentration in the scalar mixing, i.e., the conservation of the mean concentration, is violated in variable-density flows. We further combine the compositional transport equation and the divergence relation for the miscible flows to provide the evidence that the existence of density gradient accelerated mixing rate, decomposed by the accelerated dissipation term and redistributed diffusion term, contributes to the anomalous decrease or increase of the mean concentration depending on Atwood number. Further analyzing a number of simulations for the cylindrical or spherical bubbles under a broad range of shock Mach numbers, Reynolds numbers, and Peclet numbers, the density gradient accelerated mixing rate exhibits weak dependent on Peclet numbers, and identifies an Atwood number range with high mixing rate, which can be theoretically predicted based on the mode of hyperbolic conservation violation behavior.

physics.flu-dyn↗

arXiv2021

Radio stars of the SKA

Radio emission from stars can be used, e.g., to study ionized winds or stellar flares. The radio emission is faint and studies have been limited to few objects. The Square Kilometer Array (SKA) will bring a survey ability to the topic of radio stars. In this paper we investigate what the SKA can detect, and what sensitivity will be required for deep surveys of the stellar Milky Way. We focus on the radio emission from OB stars, Be stars, flares from M dwarfs, and Ultra Compact HII regions. The stellar distribution in the Milky Way is simulated using the Besancon model, and various relations are used to predict their radio flux. We find that the full SKA will easily detect all UltraCompact HII regions. At the limit of 10 nJy at 5 GHz, the SKA can detect 1500 Be stars and 50 OB stars per square degree, out to several kpc. It can also detect flares from 4500 M dwarfs per square degree. At 100 nJy the numbers become about 8 times smaller. SKA surveys of the Galactic plane should be designed for high sensitivity. Deep imaging should consider the significant number of faint flares in the field, even outside the plane of the Milky Way.

astro-ph.SR↗

arXiv2021

Anosov flows on Dehn surgeries on the figure-eight knot

The purpose of this paper is to classify Anosov flows on the 3-manifolds obtained by Dehn surgeries on the figure-eight knot. This set of 3-manifolds is denoted by M(r) (r is a ratioanl number), which contains the first class of hyperbolic 3-manifolds admitting Anosov flows in history, discovered by Goodman. Combining with the classification of Anosov flows on the sol-manifold M(0) due to Plante, we have: 1. if r is an integer, up to topological equivalence, M(r) exactly carries a unique Anosov flow, which is constructed by Goodman by doing a Dehn-Fried-Goodman surgery on a suspension Anosov flow; 2. if r is not an integer, M(r) does not carry any Anosov flow. As a consequence of the second result, we get infinitely many closed orientable hyperbolic 3-manifolds which carry taut foliations but does not carry any Anosov flow. The fundamental tool in the proofs is the set of branched surfaces built by Schwider, which is used to carry essential laminations on M(r).

math.DS↗

arXiv2021

Fast mixing of Metropolized Hamiltonian Monte Carlo: Benefits of multi-step gradients

Hamiltonian Monte Carlo (HMC) is a state-of-the-art Markov chain Monte Carlo sampling algorithm for drawing samples from smooth probability densities over continuous spaces. We study the variant most widely used in practice, Metropolized HMC with the Störmer-Verlet or leapfrog integrator, and make two primary contributions. First, we provide a non-asymptotic upper bound on the mixing time of the Metropolized HMC with explicit choices of step-size and number of leapfrog steps. This bound gives a precise quantification of the faster convergence of Metropolized HMC relative to simpler MCMC algorithms such as the Metropolized random walk, or Metropolized Langevin algorithm. Second, we provide a general framework for sharpening mixing time bounds of Markov chains initialized at a substantial distance from the target distribution over continuous spaces. We apply this sharpening device to the Metropolized random walk and Langevin algorithms, thereby obtaining improved mixing time bounds from a non-warm initial distribution.

stat.ML↗

arXiv2020

On topological representation theory from quivers

In this work, we introduce {\em topological representations of a quiver} as a system consisting of topological spaces and its relationships determined by the quiver. Such a setting gives a natural connection between topological representations of a quiver and diagrams of topological spaces. First, we investigate the relation between the category of topological representations and that of linear representations of a quiver via $P(Γ)$-$\mathcal{TOP}^o$ and $kΓ$-Mod, concerning (positively) graded or vertex (positively) graded modules. Second, we discuss the homological theory of topological representations of quivers via $Γ$-limit $Lim^Γ$ and using it, define the homology groups of topological representations of quivers via $H_n$. It is found that some properties of a quiver can be read from homology groups. Third, we investigate the homotopy theory of topological representations of quivers. We define the homotopy equivalence between two morphisms in $\textbf{Top}\mathrm{-}\textbf{Rep}Γ$ and show that the parallel Homotopy Axiom also holds for top-representations based on the homotopy equivalence. Last, we mainly obtain the functor $At^Γ$ from $\textbf{Top}\mathrm{-}\textbf{Rep}Γ$ to $\textbf{Top}$ and show that $At^Γ$ preserves homotopy equivalence between morphisms. The relationship is established between the homotopy groups of a top-representation $(T,f)$ and the homotopy groups of $At^Γ(T,f)$.

math.RT↗

arXiv2020

Enriched Annotations for Tumor Attribute Classification from Pathology Reports with Limited Labeled Data

Precision medicine has the potential to revolutionize healthcare, but much of the data for patients is locked away in unstructured free-text, limiting research and delivery of effective personalized treatments. Generating large annotated datasets for information extraction from clinical notes is often challenging and expensive due to the high level of expertise needed for high quality annotations. To enable natural language processing for small dataset sizes, we develop a novel enriched hierarchical annotation scheme and algorithm, Supervised Line Attention (SLA), and apply this algorithm to predicting categorical tumor attributes from kidney and colon cancer pathology reports from the University of California San Francisco (UCSF). Whereas previous work only annotated document level labels, we in addition ask the annotators to enrich the traditional label by asking them to also highlight the relevant line or potentially lines for the final label, which leads to a 20% increase of annotation time required per document. With the enriched annotations, we develop a simple and interpretable machine learning algorithm that first predicts the relevant lines in the document and then predicts the tumor attribute. Our results show across the small dataset sizes of 32, 64, 128, and 186 labeled documents per cancer, SLA only requires half the number of labeled documents as state-of-the-art methods to achieve similar or better micro-f1 and macro-f1 scores for the vast majority of comparisons that we made. Accounting for the increased annotation time, this leads to a 40% reduction in total annotation time over the state of the art.

cs.CL↗