arXiv subjects

Yu Cheng

Publications and source records attributed to Yu Cheng.

At least 397 records · Page 22Linked to original sources

arXiv2021

DSAM: A Distance Shrinking with Angular Marginalizing Loss for High Performance Vehicle Re-identificatio

Vehicle Re-identification (ReID) is an important yet challenging problem in computer vision. Compared to other visual objects like faces and persons, vehicles simultaneously exhibit much larger intraclass viewpoint variations and interclass visual similarities, making most exiting loss functions designed for face recognition and person ReID unsuitable for vehicle ReID. To obtain a high-performance vehicle ReID model, we present a novel Distance Shrinking with Angular Marginalizing (DSAM) loss function to perform hybrid learning in both the Original Feature Space (OFS) and the Feature Angular Space (FAS) using the local verification and the global identification information. Specifically, it shrinks the distance between samples of the same class locally in the Original Feature Space while keeps samples of different classes far away in the Feature Angular Space. The shrinking and marginalizing operations are performed during each iteration of the training process and are suitable for different SoftMax based loss functions. We evaluate the DSAM loss function on three large vehicle ReID datasets with detailed analyses and extensive comparisons with many competing vehicle ReID methods. Experimental results show that our DSAM loss enhances the SoftMax loss by a large margin on the PKU-VD1-Large dataset: 10.41% for mAP, 5.29% for cmc1, and 4.60% for cmc5. Moreover, the mAP is increased by 9.34% on the PKU-VehicleID dataset and 6.13% on the VeRi-776 dataset. Source code will be released to facilitate further studies in this research direction.

cs.CV↗

arXiv2021

VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation

Most existing video-and-language (VidL) research focuses on a single dataset, or multiple datasets of a single task. In reality, a truly useful VidL system is expected to be easily generalizable to diverse tasks, domains, and datasets. To facilitate the evaluation of such systems, we introduce Video-And-Language Understanding Evaluation (VALUE) benchmark, an assemblage of 11 VidL datasets over 3 popular tasks: (i) text-to-video retrieval; (ii) video question answering; and (iii) video captioning. VALUE benchmark aims to cover a broad range of video genres, video lengths, data volumes, and task difficulty levels. Rather than focusing on single-channel videos with visual information only, VALUE promotes models that leverage information from both video frames and their associated subtitles, as well as models that share knowledge across multiple tasks. We evaluate various baseline methods with and without large-scale VidL pre-training, and systematically investigate the impact of video input channels, fusion methods, and different video representations. We also study the transferability between tasks, and conduct multi-task learning under different settings. The significant gap between our best model and human performance calls for future study for advanced VidL models. VALUE is available at https://value-benchmark.github.io/.

cs.CV↗

arXiv2021

APo-VAE: Text Generation in Hyperbolic Space

Natural language often exhibits inherent hierarchical structure ingrained with complex syntax and semantics. However, most state-of-the-art deep generative models learn embeddings only in Euclidean vector space, without accounting for this structural property of language. In this paper, we investigate text generation in a hyperbolic latent space to learn continuous hierarchical representations. An Adversarial Poincare Variational Autoencoder (APo-VAE) is presented, where both the prior and variational posterior of latent variables are defined over a Poincare ball via wrapped normal distributions. By adopting the primal-dual formulation of KL divergence, an adversarial learning procedure is introduced to empower robust model training. Extensive experiments in language modeling and dialog-response generation tasks demonstrate the winning effectiveness of the proposed APo-VAE model over VAEs in Euclidean latent space, thanks to its superb capabilities in capturing latent language hierarchies in hyperbolic space.

cs.LG↗

arXiv2021

MaxVA: Fast Adaptation of Step Sizes by Maximizing Observed Variance of Gradients

Adaptive gradient methods such as RMSProp and Adam use exponential moving estimate of the squared gradient to compute adaptive step sizes, achieving better convergence than SGD in face of noisy objectives. However, Adam can have undesirable convergence behaviors due to unstable or extreme adaptive learning rates. Methods such as AMSGrad and AdaBound have been proposed to stabilize the adaptive learning rates of Adam in the later stage of training, but they do not outperform Adam in some practical tasks such as training Transformers \cite{transformer}. In this paper, we propose an adaptive learning rate principle, in which the running mean of squared gradient in Adam is replaced by a weighted mean, with weights chosen to maximize the estimated variance of each coordinate. This results in a faster adaptation to the local gradient variance, which leads to more desirable empirical convergence behaviors than Adam. We prove the proposed algorithm converges under mild assumptions for nonconvex stochastic optimization problems, and demonstrate the improved efficacy of our adaptive averaging approach on machine translation, natural language understanding and large-batch pretraining of BERT. The code is available at https://github.com/zhuchen03/MaxVA.

cs.LG↗

arXiv2021

Flavor-changing Majoron interactions with leptons

When the Standard Model Higgs sector is extended with a complex singlet that breaks global lepton number symmetry spontaneously, a massless Goldstone boson called the Majoron J arises. In addition to increasing Higgs invisible decay through mixing, the Majoron can generally have flavor-changing interactions with fermions. We find that type-III seesaw model poses such interesting properties with both charged leptons and neutrinos. This opens up new channels to search for the Majoron. We use the experimental data such as muonium-anti-muonium oscillation and flavor-changing neutrino and charged lepton decays to put constraints on the couplings. As a novel way to reveal the chiral properties of these interactions, we propose an experimentally measurable polarization asymmetry of flavor-changing l to l' J decays.

hep-ph↗

arXiv2021

Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encoding

Transformer has become ubiquitous in the deep learning field. One of the key ingredients that destined its success is the self-attention mechanism, which allows fully-connected contextual encoding over input tokens. However, despite its effectiveness in modeling short sequences, self-attention suffers when handling inputs with extreme long-range dependencies, as its complexity grows quadratically with respect to the sequence length. Therefore, long sequences are often encoded by Transformer in chunks using a sliding window. In this paper, we propose Cluster-Former, a novel clustering-based sparse Transformer to perform attention across chunked sequences. The proposed framework is pivoted on two unique types of Transformer layer: Sliding-Window Layer and Cluster-Former Layer, which encode local sequence information and global context jointly and iteratively. This new design allows information integration beyond local windows, which is especially beneficial for question answering (QA) tasks that rely on long-range dependencies. Experiments show that Cluster-Former achieves state-of-the-art performance on several major QA benchmarks.

cs.CL↗

arXiv2021

EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets

Heavily overparameterized language models such as BERT, XLNet and T5 have achieved impressive success in many NLP tasks. However, their high model complexity requires enormous computation resources and extremely long training time for both pre-training and fine-tuning. Many works have studied model compression on large NLP models, but only focusing on reducing inference time while still requiring an expensive training process. Other works use extremely large batch sizes to shorten the pre-training time, at the expense of higher computational resource demands. In this paper, inspired by the Early-Bird Lottery Tickets recently studied for computer vision tasks, we propose EarlyBERT, a general computationally-efficient training algorithm applicable to both pre-training and fine-tuning of large-scale language models. By slimming the self-attention and fully-connected sub-layers inside a transformer, we are the first to identify structured winning tickets in the early stage of BERT training. We apply those tickets towards efficient BERT training, and conduct comprehensive pre-training and fine-tuning experiments on GLUE and SQuAD downstream tasks. Our results show that EarlyBERT achieves comparable performance to standard BERT, with 35~45% less training time. Code is available at https://github.com/VITA-Group/EarlyBERT.

cs.CL↗

arXiv2021

Star Formation in a Strongly Magnetized Cloud

We study star formation in the Center Ridge 1 (CR1) clump in the Vela C giant molecular cloud, selected as a high column density region that shows the lowest level of dust continuum polarization angle dispersion, likely indicating that the magnetic field is relatively strong. We observe the source with the ALMA 7m-array at 1.05~mm and 1.3~mm wavelengths, which enable measurements of dust temperature, core mass and astrochemical deuteration. A relatively modest number of eleven dense cores are identified via their dust continuum emission, with masses spanning from 0.17 to 6.7 Msun. Overall CR1 has a relatively low compact dense gas fraction compared with other typical clouds with similar column densities, which may be a result of the strong magnetic field and/or the very early evolutionary stage of this region. The deuteration ratios, Dfrac, of the cores, measured with N2H+(3-2) and N2D+(3-2) lines, span from 0.011 to 0.85, with the latter being one of the highest values yet detected. The level of deuteration appears to decrease with evolution from prestellar to protostellar phase. A linear filament, running approximately parallel with the large scale magnetic field orientation, is seen connecting the two most massive cores, each having CO bipolar outflows aligned orthogonally to the filament. The filament contains the most deuterated core, likely to be prestellar and located midway between the protostars. The observations permit measurement of the full deuteration structure of the filament along its length, which we present. We also discuss the kinematics and dynamics of this structure, as well as of the dense core population.

astro-ph.GA↗

arXiv2021

The Core Mass Function Across Galactic Environments. III. Massive Protoclusters

The stellar initial mass function (IMF) is fundamental for many areas of astrophysics, but its origin remains poorly understood. It may be inherited from the core mass function (CMF) or arise as a result of more chaotic, competitive accretion. Dense, gravitationally bound cores are seen in molecular clouds and some observations have suggested that the CMF is similar in shape to the IMF, though translated to higher masses by a factor of $\sim3$. Here we measure the CMF in 28 dense clumps within 3.5 kpc that are likely to be central regions of massive protoclusters, observed via $1.3\:{\rm{mm}}$ dust continuum emission by the ALMAGAL project. We identify 222 cores using the dendrogram algorithm with masses ranging from 0.04 to $252\:M_{\odot}$. We apply completeness corrections for flux and number recovery, estimated from core insertion and recovery experiments. At higher masses, the final derived CMF is well described by a single power law of the form $dN/d\:{\textrm{log}}\:M\propto\:M^{-α}$ with $α\simeq0.94\pm0.08$. However, we find evidence of a break in this power-law behavior between $\sim5$ and $15\:M_{\odot}$, which is, to our knowledge, the first time such a break has been found in distant ($\gtrsim 1$ kpc) regions by ALMA. We compare this massive protocluster CMF with those derived using the same methods in the G286 protocluster and a sample of Infrared Dark Clouds. The massive protocluster CMF is significantly different, i.e., containing more massive cores, which is a potential indication of the role of environment on the CMF and IMF.

astro-ph.GA↗

arXiv2021

Gauge invariance of the vector meson mass in the Coleman-Weinberg model

We revisit the problem of the gauge invariance in the Coleman-Weinberg model in which a $U(1)$ gauge symmetry is driven spontaneously broken by radiative corrections. It was noticed in previous work that masses in this model are not gauge invariant at one-loop order. In our analysis, we use the dressed propagators of scalars which include a resummation of the one-loop self-energy correction to the tree-level propagator. We calculate the one-loop self-energy correction to the vector meson using these dressed propagators. We find that the pole mass of the vector meson calculated using the dressed propagator is gauge invariant at the vacuum determined using the effective potential calculated with a resummation of daisy diagrams.

hep-ph↗

arXiv2021

Robust Learning of Fixed-Structure Bayesian Networks in Nearly-Linear Time

We study the problem of learning Bayesian networks where an $ε$-fraction of the samples are adversarially corrupted. We focus on the fully-observable case where the underlying graph structure is known. In this work, we present the first nearly-linear time algorithm for this problem with a dimension-independent error guarantee. Previous robust algorithms with comparable error guarantees are slower by at least a factor of $(d/ε)$, where $d$ is the number of variables in the Bayesian network and $ε$ is the fraction of corrupted samples. Our algorithm and analysis are considerably simpler than those in previous work. We achieve this by establishing a direct connection between robust learning of Bayesian networks and robust mean estimation. As a subroutine in our algorithm, we develop a robust mean estimation algorithm whose runtime is nearly-linear in the number of nonzeros in the input samples, which may be of independent interest.

cs.LG↗

arXiv2021

Sparsification of Directed Graphs via Cut Balance

In this paper, we consider the problem of designing cut sparsifiers and sketches for directed graphs. To bypass known lower bounds, we allow the sparsifier/sketch to depend on the balance of the input graph, which smoothly interpolates between undirected and directed graphs. We give nearly matching upper and lower bounds for both for-all (cf. Benczúr and Karger, STOC 1996) and for-each (Andoni et al., ITCS 2016) cut sparsifiers/sketches as a function of cut balance, defined the maximum ratio of the cut value in the two directions of a directed graph (Ene et al., STOC 2016). We also show an interesting application of digraph sparsification via cut balance by using it to give a very short proof of a celebrated maximum flow result of Karger and Levine (STOC 2002).

cs.DS↗

arXiv2021

Universal coupled theory for metamaterial Bound states in the continuum

In this paper, we present a novel universal coupled theory for metamaterial Bound states in the continuum (BIC) or quasi-Bound states in the continuum (quasi-BIC) which provides ultra-high Q resonance for metamaterial devices. Our theory analytically calculates the coupling of two bright modes with phase. Our method has much more accuracy for ultra-strong coupling comparing with the previous theory (the coupling of one bright mode and one dark mode). Therefore, our theory is much more suitable for BIC or quasi-BIC and we can accurately predict the transmission spectrum of metamaterial BIC or quasi-BIC for the first time.

physics.optics↗

arXiv2021

CUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation Learning

This work concerns video-language pre-training and representation learning. In this now ubiquitous training scheme, a model first performs pre-training on paired videos and text (e.g., video clips and accompanied subtitles) from a large uncurated source corpus, before transferring to specific downstream tasks. This two-stage training process inevitably raises questions about the generalization ability of the pre-trained model, which is particularly pronounced when a salient domain gap exists between source and target data (e.g., instructional cooking videos vs. movies). In this paper, we first bring to light the sensitivity of pre-training objectives (contrastive vs. reconstructive) to domain discrepancy. Then, we propose a simple yet effective framework, CUPID, to bridge this domain gap by filtering and adapting source data to the target data, followed by domain-focused pre-training. Comprehensive experiments demonstrate that pre-training on a considerably small subset of domain-focused data can effectively close the source-target domain gap and achieve significant performance gain, compared to random sampling or even exploiting the full pre-training dataset. CUPID yields new state-of-the-art performance across multiple video-language and video tasks, including text-to-video retrieval [72, 37], video question answering [36], and video captioning [72], with consistent performance lift over different pre-training methods.

cs.CV↗

arXiv2021

Automated Mechanism Design for Classification with Partial Verification

We study the problem of automated mechanism design with partial verification, where each type can (mis)report only a restricted set of types (rather than any other type), induced by the principal's limited verification power. We prove hardness results when the revelation principle does not necessarily hold, as well as when types have even minimally different preferences. In light of these hardness results, we focus on truthful mechanisms in the setting where all types share the same preference over outcomes, which is motivated by applications in, e.g., strategic classification. We present a number of algorithmic and structural results, including an efficient algorithm for finding optimal deterministic truthful mechanisms, which also implies a faster algorithm for finding optimal randomized truthful mechanisms via a characterization based on convexity. We then consider a more general setting, where the principal's cost is a function of the combination of outcomes assigned to each type. In particular, we focus on the case where the cost function is submodular, and give generalizations of essentially all our results in the classical setting where the cost function is additive. Our results provide a relatively complete picture for automated mechanism design with partial verification.

cs.GT↗

arXiv2021

Graph and Temporal Convolutional Networks for 3D Multi-person Pose Estimation in Monocular Videos

Despite the recent progress, 3D multi-person pose estimation from monocular videos is still challenging due to the commonly encountered problem of missing information caused by occlusion, partially out-of-frame target persons, and inaccurate person detection. To tackle this problem, we propose a novel framework integrating graph convolutional networks (GCNs) and temporal convolutional networks (TCNs) to robustly estimate camera-centric multi-person 3D poses that do not require camera parameters. In particular, we introduce a human-joint GCN, which, unlike the existing GCN, is based on a directed graph that employs the 2D pose estimator's confidence scores to improve the pose estimation results. We also introduce a human-bone GCN, which models the bone connections and provides more information beyond human joints. The two GCNs work together to estimate the spatial frame-wise 3D poses and can make use of both visible joint and bone information in the target frame to estimate the occluded or missing human-part information. To further refine the 3D pose estimation, we use our temporal convolutional networks (TCNs) to enforce the temporal and human-dynamics constraints. We use a joint-TCN to estimate person-centric 3D poses across frames, and propose a velocity-TCN to estimate the speed of 3D joints to ensure the consistency of the 3D pose estimation in consecutive frames. Finally, to estimate the 3D human poses for multiple persons, we propose a root-TCN that estimates camera-centric 3D poses without requiring camera parameters. Quantitative and qualitative evaluations demonstrate the effectiveness of the proposed method.

cs.CV↗

arXiv2021

Monocular 3D Multi-Person Pose Estimation by Integrating Top-Down and Bottom-Up Networks

In monocular video 3D multi-person pose estimation, inter-person occlusion and close interactions can cause human detection to be erroneous and human-joints grouping to be unreliable. Existing top-down methods rely on human detection and thus suffer from these problems. Existing bottom-up methods do not use human detection, but they process all persons at once at the same scale, causing them to be sensitive to multiple-persons scale variations. To address these challenges, we propose the integration of top-down and bottom-up approaches to exploit their strengths. Our top-down network estimates human joints from all persons instead of one in an image patch, making it robust to possible erroneous bounding boxes. Our bottom-up network incorporates human-detection based normalized heatmaps, allowing the network to be more robust in handling scale variations. Finally, the estimated 3D poses from the top-down and bottom-up networks are fed into our integration network for final 3D poses. Besides the integration of top-down and bottom-up networks, unlike existing pose discriminators that are designed solely for single person, and consequently cannot assess natural inter-person interactions, we propose a two-person pose discriminator that enforces natural two-person interactions. Lastly, we also apply a semi-supervised method to overcome the 3D ground-truth data scarcity. Our quantitative and qualitative evaluations show the effectiveness of our method compared to the state-of-the-art baselines.

cs.CV↗

arXiv2021

UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training

Vision-and-language pre-training has achieved impressive success in learning multimodal representations between vision and language. To generalize this success to non-English languages, we introduce UC2, the first machine translation-augmented framework for cross-lingual cross-modal representation learning. To tackle the scarcity problem of multilingual captions for image datasets, we first augment existing English-only datasets with other languages via machine translation (MT). Then we extend the standard Masked Language Modeling and Image-Text Matching training objectives to multilingual setting, where alignment between different languages is captured through shared visual context (i.e, using image as pivot). To facilitate the learning of a joint embedding space of images and all languages of interest, we further propose two novel pre-training tasks, namely Masked Region-to-Token Modeling (MRTM) and Visual Translation Language Modeling (VTLM), leveraging MT-enhanced translated data. Evaluation on multilingual image-text retrieval and multilingual visual question answering benchmarks demonstrates that our proposed framework achieves new state-of-the-art on diverse non-English benchmarks while maintaining comparable performance to monolingual pre-trained models on English tasks.

cs.CV↗