arXiv subjects

Yulong Wang

Publications and source records attributed to Yulong Wang.

At least 73 records · Page 4Linked to original sources

arXiv2022

Capital and Labor Income Pareto Exponents in the United States, 1916-2019

Accurately estimating income Pareto exponents is challenging due to limitations in data availability and the applicability of statistical methods. Using tabulated summaries of incomes from tax authorities and a recent estimation method, we estimate income Pareto exponents in U.S. for 1916-2019. We find that during the past three decades, the capital and labor income Pareto exponents have been stable at around 1.2 and 2. Our findings suggest that the top tail income and wealth inequality is higher and wealthy agents have twice as large an impact on the aggregate economy than previously thought but there is no clear trend post-1985.

econ.GN↗

arXiv2022

Error-based Knockoffs Inference for Controlled Feature Selection

Recently, the scheme of model-X knockoffs was proposed as a promising solution to address controlled feature selection under high-dimensional finite-sample settings. However, the procedure of model-X knockoffs depends heavily on the coefficient-based feature importance and only concerns the control of false discovery rate (FDR). To further improve its adaptivity and flexibility, in this paper, we propose an error-based knockoff inference method by integrating the knockoff features, the error-based feature importance statistics, and the stepdown procedure together. The proposed inference procedure does not require specifying a regression model and can handle feature selection with theoretical guarantees on controlling false discovery proportion (FDP), FDR, or k-familywise error rate (k-FWER). Empirical evaluations demonstrate the competitive performance of our approach on both simulated and real data.

stat.ME↗

arXiv2021

Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters

Deep learning based models have dominated the current landscape of production recommender systems. Furthermore, recent years have witnessed an exponential growth of the model scale--from Google's 2016 model with 1 billion parameters to the latest Facebook's model with 12 trillion parameters. Significant quality boost has come with each jump of the model capacity, which makes us believe the era of 100 trillion parameters is around the corner. However, the training of such models is challenging even within industrial scale data centers. This difficulty is inherited from the staggering heterogeneity of the training computation--the model's embedding layer could include more than 99.99% of the total model size, which is extremely memory-intensive; while the rest neural network is increasingly computation-intensive. To support the training of such huge models, an efficient distributed training system is in urgent need. In this paper, we resolve this challenge by careful co-design of both the optimization algorithm and the distributed system architecture. Specifically, in order to ensure both the training efficiency and the training accuracy, we design a novel hybrid training algorithm, where the embedding layer and the dense neural network are handled by different synchronization mechanisms; then we build a system called Persia (short for parallel recommendation training system with hybrid acceleration) to support this hybrid training algorithm. Both theoretical demonstration and empirical study up to 100 trillion parameters have conducted to justified the system design and implementation of Persia. We make Persia publicly available (at https://github.com/PersiaML/Persia) so that anyone would be able to easily train a recommender model at the scale of 100 trillion parameters.

cs.LG↗

arXiv2021

Three-dimensional Cooperative Localization of Commercial-Off-The-Shelf Sensors

Many location-based services use Received Signal Strength (RSS) measurements due to their universal availability. In this paper, we study the association of a large number of low-cost Internet-of-Things (IoT) sensors and their possible installation locations, which can enable various sensing and automation-related applications. We propose an efficient approach to solve the corresponding permutation combinatorial optimization problem, which integrates continuous space cooperative localization and permutation space likelihood ascent search. A convex relaxation-based optimization is designed to estimate the coarse locations of blindfolded devices in continuous 3D spaces, which are then projected to the feasible permutation space. An efficient Cramér-Rao Lower Bound based likelihood ascent search algorithm is proposed to refine the solution. Extensive experiments were conducted to evaluate the performance of the proposed approach, which show that the proposed approach significantly outperforms state-of-the-art combinatorial optimization algorithms and achieves close-to-100% accuracy with affordable execution time.

cs.IR↗

arXiv2021

Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinese

Although pre-trained models (PLMs) have achieved remarkable improvements in a wide range of NLP tasks, they are expensive in terms of time and resources. This calls for the study of training more efficient models with less computation but still ensures impressive performance. Instead of pursuing a larger scale, we are committed to developing lightweight yet more powerful models trained with equal or less computation and friendly to rapid deployment. This technical report releases our pre-trained model called Mengzi, which stands for a family of discriminative, generative, domain-specific, and multimodal pre-trained model variants, capable of a wide range of language and vision tasks. Compared with public Chinese PLMs, Mengzi is simple but more powerful. Our lightweight model has achieved new state-of-the-art results on the widely-used CLUE benchmark with our optimized pre-training and fine-tuning techniques. Without modifying the model architecture, our model can be easily employed as an alternative to existing PLMs. Our sources are available at https://github.com/Langboat/Mengzi.

cs.CL↗

arXiv2021

Analyzing Adversarial Robustness of Deep Neural Networks in Pixel Space: a Semantic Perspective

The vulnerability of deep neural networks to adversarial examples, which are crafted maliciously by modifying the inputs with imperceptible perturbations to misled the network produce incorrect outputs, reveals the lack of robustness and poses security concerns. Previous works study the adversarial robustness of image classifiers on image level and use all the pixel information in an image indiscriminately, lacking of exploration of regions with different semantic meanings in the pixel space of an image. In this work, we fill this gap and explore the pixel space of the adversarial image by proposing an algorithm to looking for possible perturbations pixel by pixel in different regions of the segmented image. The extensive experimental results on CIFAR-10 and ImageNet verify that searching for the modified pixel in only some pixels of an image can successfully launch the one-pixel adversarial attacks without requiring all the pixels of the entire image, and there exist multiple vulnerable points scattered in different regions of an image. We also demonstrate that the adversarial robustness of different regions on the image varies with the amount of semantic information contained.

cs.CV↗

arXiv2021

Semantic Segmentation by Improved Generative Adversarial Networks

While most existing segmentation methods usually combined the powerful feature extraction capabilities of CNNs with Conditional Random Fields (CRFs) post-processing, the result always limited by the fault of CRFs . Due to the notoriously slow calculation speeds and poor efficiency of CRFs, in recent years, CRFs post-processing has been gradually eliminated. In this paper, an improved Generative Adversarial Networks (GANs) for image semantic segmentation task (semantic segmentation by GANs, Seg-GAN) is proposed to facilitate further segmentation research. In addition, we introduce Convolutional CRFs (ConvCRFs) as an effective improvement solution for the image semantic segmentation task. Towards the goal of differentiating the segmentation results from the ground truth distribution and improving the details of the output images, the proposed discriminator network is specially designed in a full convolutional manner combined with cascaded ConvCRFs. Besides, the adversarial loss aggressively encourages the output image to be close to the distribution of the ground truth. Our method not only learns an end-to-end mapping from input image to corresponding output image, but also learns a loss function to train this mapping. The experiments show that our method achieves better performance than state-of-the-art methods.

cs.CV↗

arXiv2021

Threshold Regression with Nonparametric Sample Splitting

This paper develops a threshold regression model where an unknown relationship between two variables nonparametrically determines the threshold. We allow the observations to be cross-sectionally dependent so that the model can be applied to determine an unknown spatial border for sample splitting over a random field. We derive the uniform rate of convergence and the nonstandard limiting distribution of the nonparametric threshold estimator. We also obtain the root-n consistency and the asymptotic normality of the regression coefficient estimator. Our model has broad empirical relevance as illustrated by estimating the tipping point in social segregation problems as a function of demographic characteristics; and determining metropolitan area boundaries using nighttime light intensity collected from satellite imagery. We find that the new empirical results are substantially different from those in the existing studies.

econ.EM↗

arXiv2020

Testing Finite Moment Conditions for the Consistency and the Root-N Asymptotic Normality of the GMM and M Estimators

Common approaches to inference for structural and reduced-form parameters in empirical economic analysis are based on the consistency and the root-n asymptotic normality of the GMM and M estimators. The canonical consistency (respectively, root-n asymptotic normality) for these classes of estimators requires at least the first (respectively, second) moment of the score to be finite. In this article, we present a method of testing these conditions for the consistency and the root-n asymptotic normality of the GMM and M estimators. The proposed test controls size nearly uniformly over the set of data generating processes that are compatible with the null hypothesis. Simulation studies support this theoretical result. Applying the proposed test to the market share data from the Dominick's Finer Foods retail chain, we find that a common \textit{ad hoc} procedure to deal with zero market shares in analysis of differentiated products markets results in a failure to satisfy the conditions for both the consistency and the root-n asymptotic normality.

econ.EM↗

arXiv2020

Fixed-k Inference for Conditional Extremal Quantiles

We develop a new extreme value theory for repeated cross-sectional and panel data to construct asymptotically valid confidence intervals (CIs) for conditional extremal quantiles from a fixed number $k$ of nearest-neighbor tail observations. As a by-product, we also construct CIs for extremal quantiles of coefficients in linear random coefficient models. For any fixed $k$, the CIs are uniformly valid without parametric assumptions over a set of nonparametric data generating processes associated with various tail indices. Simulation studies show that our CIs exhibit superior small-sample coverage and length properties than alternative nonparametric methods based on asymptotic normality. Applying the proposed method to Natality Vital Statistics, we study factors of extremely low birth weights. We find that signs of major effects are the same as those found in preceding studies based on parametric models, but with different magnitudes.

econ.EM↗

arXiv2020

Nonparametric Tests of Tail Behavior in Stochastic Frontier Models

This article studies tail behavior for the error components in the stochastic frontier model, where one component has bounded support on one side, and the other has unbounded support on both sides. Under weak assumptions on the error components, we derive nonparametric tests that the unbounded component distribution has thin tails and that the component tails are equivalent. The tests are useful diagnostic tools for stochastic frontier analysis. A simulation study and an application to a stochastic cost frontier for 6,100 US banks from 1998 to 2005 are provided. The new tests reject the normal or Laplace distributional assumptions, which are commonly imposed in the existing literature.

econ.EM↗

arXiv2020

Data-Free Adversarial Perturbations for Practical Black-Box Attack

Neural networks are vulnerable to adversarial examples, which are malicious inputs crafted to fool pre-trained models. Adversarial examples often exhibit black-box attacking transferability, which allows that adversarial examples crafted for one model can fool another model. However, existing black-box attack methods require samples from the training data distribution to improve the transferability of adversarial examples across different models. Because of the data dependence, the fooling ability of adversarial perturbations is only applicable when training data are accessible. In this paper, we present a data-free method for crafting adversarial perturbations that can fool a target model without any knowledge about the training data distribution. In the practical setting of a black-box attack scenario where attackers do not have access to target models and training data, our method achieves high fooling rates on target models and outperforms other universal adversarial perturbation methods. Our method empirically shows that current deep learning models are still at risk even when the attackers do not have access to training data.

cs.CV↗

arXiv2020

Estimation and Inference about Tail Features with Tail Censored Data

This paper considers estimation and inference about tail features when the observations beyond some threshold are censored. We first show that ignoring such tail censoring could lead to substantial bias and size distortion, even if the censored probability is tiny. Second, we propose a new maximum likelihood estimator (MLE) based on the Pareto tail approximation and derive its asymptotic properties. Third, we provide a small sample modification to the MLE by resorting to Extreme Value theory. The MLE with this modification delivers excellent small sample performance, as shown by Monte Carlo simulations. We illustrate its empirical relevance by estimating (i) the tail index and the extreme quantiles of the US individual earnings with the Current Population Survey dataset and (ii) the tail index of the distribution of macroeconomic disasters and the coefficient of risk aversion using the dataset collected by Barro and Urs{ú}a (2008). Our new empirical findings are substantially different from the existing literature.

econ.EM↗

arXiv2020

Efficient Minimum Distance Estimation of Pareto Exponent from Top Income Shares

We propose an efficient estimation method for the income Pareto exponent when only certain top income shares are observable. Our estimator is based on the asymptotic theory of weighted sums of order statistics and the efficient minimum distance estimator. Simulations show that our estimator has excellent finite sample properties. We apply our estimation method to U.S. top income share data and find that the Pareto exponent has been stable at around 1.5 since 1985, suggesting that the rise in inequality during the last three decades is mainly driven by redistribution between the rich and poor, not among the rich.

math.ST↗

arXiv2020

6D Object Pose Regression via Supervised Learning on Point Clouds

This paper addresses the task of estimating the 6 degrees of freedom pose of a known 3D object from depth information represented by a point cloud. Deep features learned by convolutional neural networks from color information have been the dominant features to be used for inferring object poses, while depth information receives much less attention. However, depth information contains rich geometric information of the object shape, which is important for inferring the object pose. We use depth information represented by point clouds as the input to both deep networks and geometry-based pose refinement and use separate networks for rotation and translation regression. We argue that the axis-angle representation is a suitable rotation representation for deep learning, and use a geodesic loss function for rotation regression. Ablation studies show that these design choices outperform alternatives such as the quaternion representation and L2 loss, or regressing translation and rotation with the same network. Our simple yet effective approach clearly outperforms state-of-the-art methods on the YCB-video dataset. The implementation and trained model are avaliable at: https://github.com/GeeeG/CloudPose.

cs.CV↗

arXiv2019

Interpretable Disentanglement of Neural Networks by Extracting Class-Specific Subnetwork

We propose a novel perspective to understand deep neural networks in an interpretable disentanglement form. For each semantic class, we extract a class-specific functional subnetwork from the original full model, with compressed structure while maintaining comparable prediction performance. The structure representations of extracted subnetworks display a resemblance to their corresponding class semantic similarities. We also apply extracted subnetworks in visual explanation and adversarial example detection tasks by merely replacing the original full model with class-specific subnetworks. Experiments demonstrate that this intuitive operation can effectively improve explanation saliency accuracy for gradient-based explanation methods, and increase the detection rate for confidence score-based adversarial example detection methods.

cs.LG↗

arXiv2019

Pruning from Scratch

Network pruning is an important research field aiming at reducing computational costs of neural networks. Conventional approaches follow a fixed paradigm which first trains a large and redundant network, and then determines which units (e.g., channels) are less important and thus can be removed. In this work, we find that pre-training an over-parameterized model is not necessary for obtaining the target pruned structure. In fact, a fully-trained over-parameterized model will reduce the search space for the pruned structure. We empirically show that more diverse pruned structures can be directly pruned from randomly initialized weights, including potential models with better performance. Therefore, we propose a novel network pruning pipeline which allows pruning from scratch. In the experiments for compressing classification models on CIFAR10 and ImageNet datasets, our approach not only greatly reduces the pre-training burden of traditional pruning methods, but also achieves similar or even higher accuracy under the same computation budgets. Our results facilitate the community to rethink the effectiveness of existing techniques used for network pruning.

cs.CV↗

arXiv2018

A Hierarchical Recurrent Neural Network for Symbolic Melody Generation

In recent years, neural networks have been used to generate symbolic melodies. However, the long-term structure in the melody has posed great difficulty for designing a good model. In this paper, we present a hierarchical recurrent neural network for melody generation, which consists of three Long-Short-Term-Memory (LSTM) subnetworks working in a coarse-to-fine manner along time. Specifically, the three subnetworks generate bar profiles, beat profiles and notes in turn, and the output of the high-level subnetworks are fed into the low-level subnetworks, serving as guidance for generating the finer time-scale melody components in low-level subnetworks. Two human behavior experiments demonstrate the advantage of this structure over the single-layer LSTM which attempts to learn all hidden structures in melodies. Compared with the state-of-the-art models MidiNet and MusicVAE, the hierarchical recurrent neural network produces better melodies evaluated by humans.

cs.SD↗