arXiv subjects

Yi Xu

Publications and source records attributed to Yi Xu.

At least 343 records · Page 19Linked to original sources

arXiv2022

Vision-Language Pre-Training with Triple Contrastive Learning

Vision-language representation learning largely benefits from image-text alignment through contrastive losses (e.g., InfoNCE loss). The success of this alignment strategy is attributed to its capability in maximizing the mutual information (MI) between an image and its matched text. However, simply performing cross-modal alignment (CMA) ignores data potential within each modality, which may result in degraded representations. For instance, although CMA-based models are able to map image-text pairs close together in the embedding space, they fail to ensure that similar inputs from the same modality stay close by. This problem can get even worse when the pre-training data is noisy. In this paper, we propose triple contrastive learning (TCL) for vision-language pre-training by leveraging both cross-modal and intra-modal self-supervision. Besides CMA, TCL introduces an intra-modal contrastive objective to provide complementary benefits in representation learning. To take advantage of localized and structural information from image and text input, TCL further maximizes the average MI between local regions of image/text and their global summary. To the best of our knowledge, ours is the first work that takes into account local structure information for multi-modality representation learning. Experimental evaluations show that our approach is competitive and achieves the new state of the art on various common down-stream vision-language tasks such as image-text retrieval and visual question answering.

cs.CV↗

arXiv2022

Multi-modal Alignment using Representation Codebook

Aligning signals from different modalities is an important step in vision-language representation learning as it affects the performance of later stages such as cross-modality fusion. Since image and text typically reside in different regions of the feature space, directly aligning them at instance level is challenging especially when features are still evolving during training. In this paper, we propose to align at a higher and more stable level using cluster representation. Specifically, we treat image and text as two "views" of the same entity, and encode them into a joint vision-language coding space spanned by a dictionary of cluster centers (codebook). We contrast positive and negative samples via their cluster assignments while simultaneously optimizing the cluster centers. To further smooth out the learning process, we adopt a teacher-student distillation paradigm, where the momentum teacher of one view guides the student learning of the other. We evaluated our approach on common vision language benchmarks and obtain new SoTA on zero-shot cross modality retrieval while being competitive on various other transfer tasks.

cs.CV↗

arXiv2022

Adaptive Trajectory Prediction via Transferable GNN

Pedestrian trajectory prediction is an essential component in a wide range of AI applications such as autonomous driving and robotics. Existing methods usually assume the training and testing motions follow the same pattern while ignoring the potential distribution differences (e.g., shopping mall and street). This issue results in inevitable performance decrease. To address this issue, we propose a novel Transferable Graph Neural Network (T-GNN) framework, which jointly conducts trajectory prediction as well as domain alignment in a unified framework. Specifically, a domain-invariant GNN is proposed to explore the structural motion knowledge where the domain-specific knowledge is reduced. Moreover, an attention-based adaptive knowledge learning module is further proposed to explore fine-grained individual-level feature representations for knowledge transfer. By this way, disparities across different trajectory domains will be better alleviated. More challenging while practical trajectory prediction experiments are designed, and the experimental results verify the superior performance of our proposed model. To the best of our knowledge, our work is the pioneer which fills the gap in benchmarks and techniques for practical pedestrian trajectory prediction across different domains.

cs.CV↗

arXiv2022

EMPRESS. VI. Outflows Investigated in Low-Mass Galaxies with $M_*=10^4-10^7~M_\odot$: Weak Feedback in Low-Mass Galaxies?

We study emission line profiles of 21 nearby low-mass ($M_*=10^4-10^7~M_\odot$) galaxies in deep medium-high resolution spectra taken with Magellan/MagE. These low-mass galaxies are actively star-forming systems with high specific star-formation rates of $\mathrm{sSFR}\sim100-1000~\mathrm{Gyr}^{-1}$ that are well above the star-formation main sequence and its extrapolation. We identify broad-line components of H$α$ and [OIII]$λ5007$ emission in 14 out of the 21 galaxies that cannot be explained by the MagE instrumental profile or the natural broadening of line emission. We conduct double Gaussian profile fitting to the emission of the 14 galaxies, and find that the broad-line components have line widths significantly larger than those of the narrow-line components, indicative of galactic outflows. The board-line components have moderately large line widths of $\sim 100$ km s$^{-1}$. We estimate the maximum outflow velocities $v_\mathrm{max}$ and obtain values of $\simeq 60-200$ km s$^{-1}$, which are found to be comparable to or slightly larger than the escape velocities. Positive correlations of $v_\mathrm{max}$ with star-formation rates, stellar masses, and circular velocities, extend down into this low-mass regime. Broad- to narrow-line flux ratios BNRs are generally found to be smaller than those of massive galaxies. The small $v_\mathrm{max}$ and BNRs suggest that the mass loading factors $η$ can be as small as 0.1 - 1 or below, in contrast to the large $η$ of energy-driven outflows predicted by numerical simulations.

astro-ph.GA↗

arXiv2022

Deformable VisTR: Spatio temporal deformable attention for video instance segmentation

Video instance segmentation (VIS) task requires classifying, segmenting, and tracking object instances over all frames in a video clip. Recently, VisTR has been proposed as end-to-end transformer-based VIS framework, while demonstrating state-of-the-art performance. However, VisTR is slow to converge during training, requiring around 1000 GPU hours due to the high computational cost of its transformer attention module. To improve the training efficiency, we propose Deformable VisTR, leveraging spatio-temporal deformable attention module that only attends to a small fixed set of key spatio-temporal sampling points around a reference point. This enables Deformable VisTR to achieve linear computation in the size of spatio-temporal feature maps. Moreover, it can achieve on par performance as the original VisTR with 10$\times$ less GPU training hours. We validate the effectiveness of our method on the Youtube-VIS benchmark. Code is available at https://github.com/skrya/DefVIS.

cs.CV↗

arXiv2022

Effective Model Sparsification by Scheduled Grow-and-Prune Methods

Deep neural networks (DNNs) are effective in solving many real-world problems. Larger DNN models usually exhibit better quality (e.g., accuracy) but their excessive computation results in long inference time. Model sparsification can reduce the computation and memory cost while maintaining model quality. Most existing sparsification algorithms unidirectionally remove weights, while others randomly or greedily explore a small subset of weights in each layer for pruning. The limitations of these algorithms reduce the level of achievable sparsity. In addition, many algorithms still require pre-trained dense models and thus suffer from large memory footprint. In this paper, we propose a novel scheduled grow-and-prune (GaP) methodology without having to pre-train a dense model. It addresses the shortcomings of the previous works by repeatedly growing a subset of layers to dense and then pruning them back to sparse after some training. Experiments show that the models pruned using the proposed methods match or beat the quality of the highly optimized dense models at 80% sparsity on a variety of tasks, such as image classification, objective detection, 3D object part segmentation, and translation. They also outperform other state-of-the-art (SOTA) methods for model sparsification. As an example, a 90% non-uniform sparse ResNet-50 model obtained via GaP achieves 77.9% top-1 accuracy on ImageNet, improving the previous SOTA results by 1.5%. Code available at: https://github.com/boone891214/GaP.

cs.CV↗

arXiv2022

Rethinking Adam: A Twofold Exponential Moving Average Approach

Adaptive gradient methods, e.g. \textsc{Adam}, have achieved tremendous success in machine learning. Scaling the learning rate element-wisely by a certain form of second moment estimate of gradients, such methods are able to attain rapid training of modern deep neural networks. Nevertheless, they are observed to suffer from compromised generalization ability compared with stochastic gradient descent (\textsc{SGD}) and tend to be trapped in local minima at an early stage during training. Intriguingly, we discover that substituting the gradient in the second raw moment estimate term with its momentumized version in \textsc{Adam} can resolve the issue. The intuition is that gradient with momentum contains more accurate directional information and therefore its second moment estimation is a more favorable option for learning rate scaling than that of the raw gradient. Thereby we propose \textsc{AdaMomentum} as a new optimizer reaching the goal of training fast while generalizing much better. We further develop a theory to back up the improvement in generalization and provide convergence guarantees under both convex and nonconvex settings. Extensive experiments on a wide range of tasks and models demonstrate that \textsc{AdaMomentum} exhibits state-of-the-art performance and superior training stability consistently.

cs.LG↗

arXiv2022

Thermal coupled cluster theory for SU(2) systems

Coupled cluster (CC) has established itself as a powerful theory to study correlated quantum many-body systems. Finite-temperature generalizations of CC theory have attracted considerable interest and have been shown to work as nicely as the ground-state theory. However, most of these recent developments address only fermionic or bosonic systems. The distinct structure of the $su(2)$ algebra requires the development of a similar thermal CC theory for spin degrees of freedom. In this paper, we provide a formulation of our thermofield-inspired thermal CC for SU(2) systems. We apply the thermal CC to the Lipkin-Meshkov-Glick system as well as the one-dimensional transverse field Ising model as benchmark applications to highlight the accuracy of thermal CC in the study of finite-temperature phase diagrams in SU(2) systems.

cond-mat.str-el↗

arXiv2021

A Novel Convergence Analysis for Algorithms of the Adam Family

Since its invention in 2014, the Adam optimizer has received tremendous attention. On one hand, it has been widely used in deep learning and many variants have been proposed, while on the other hand their theoretical convergence property remains to be a mystery. It is far from satisfactory in the sense that some studies require strong assumptions about the updates, which are not necessarily applicable in practice, while other studies still follow the original problematic convergence analysis of Adam, which was shown to be not sufficient to ensure convergence. Although rigorous convergence analysis exists for Adam, they impose specific requirements on the update of the adaptive step size, which are not generic enough to cover many other variants of Adam. To address theses issues, in this extended abstract, we present a simple and generic proof of convergence for a family of Adam-style methods (including Adam, AMSGrad, Adabound, etc.). Our analysis only requires an increasing or large "momentum" parameter for the first-order moment, which is indeed the case used in practice, and a boundness condition on the adaptive factor of the step size, which applies to all variants of Adam under mild conditions of stochastic gradients. We also establish a variance diminishing result for the used stochastic gradient estimators. Indeed, our analysis of Adam is so simple and generic that it can be leveraged to establish the convergence for solving a broader family of non-convex optimization problems, including min-max, compositional, and bilevel optimization problems. For the full (earlier) version of this extended abstract, please refer to arXiv:2104.14840.

cs.LG↗

arXiv2021

Boosting Unsupervised Domain Adaptation with Soft Pseudo-label and Curriculum Learning

By leveraging data from a fully labeled source domain, unsupervised domain adaptation (UDA) improves classification performance on an unlabeled target domain through explicit discrepancy minimization of data distribution or adversarial learning. As an enhancement, category alignment is involved during adaptation to reinforce target feature discrimination by utilizing model prediction. However, there remain unexplored problems about pseudo-label inaccuracy incurred by wrong category predictions on target domain, and distribution deviation caused by overfitting on source domain. In this paper, we propose a model-agnostic two-stage learning framework, which greatly reduces flawed model predictions using soft pseudo-label strategy and avoids overfitting on source domain with a curriculum learning strategy. Theoretically, it successfully decreases the combined risk in the upper bound of expected error on the target domain. At the first stage, we train a model with distribution alignment-based UDA method to obtain soft semantic label on target domain with rather high confidence. To avoid overfitting on source domain, at the second stage, we propose a curriculum learning strategy to adaptively control the weighting between losses from the two domains so that the focus of the training stage is gradually shifted from source distribution to target distribution with prediction confidence boosted on the target domain. Extensive experiments on two well-known benchmark datasets validate the universal effectiveness of our proposed framework on promoting the performance of the top-ranked UDA algorithms and demonstrate its consistent superior performance.

cs.CV↗

arXiv2021

PoP-Net: Pose over Parts Network for Multi-Person 3D Pose Estimation from a Depth Image

In this paper, a real-time method called PoP-Net is proposed to predict multi-person 3D poses from a depth image. PoP-Net learns to predict bottom-up part representations and top-down global poses in a single shot. Specifically, a new part-level representation, called Truncated Part Displacement Field (TPDF), is introduced which enables an explicit fusion process to unify the advantages of bottom-up part detection and global pose detection. Meanwhile, an effective mode selection scheme is introduced to automatically resolve the conflicting cases between global pose and part detections. Finally, due to the lack of high-quality depth datasets for developing multi-person 3D pose estimation, we introduce Multi-Person 3D Human Pose Dataset (MP-3DHP) as a new benchmark. MP-3DHP is designed to enable effective multi-person and background data augmentation in model training, and to evaluate 3D human pose estimators under uncontrolled multi-person scenarios. We show that PoP-Net achieves the state-of-the-art results both on MP-3DHP and on the widely used ITOP dataset, and has significant advantages in efficiency for multi-person processing. To demonstrate one of the applications of our algorithm pipeline, we also show results of virtual avatars driven by our calculated 3D joint positions. MP-3DHP Dataset and the evaluation code have been made available at: https://github.com/oppo-us-research/PoP-Net.

cs.CV↗

arXiv2021

SLA$^2$P: Self-supervised Anomaly Detection with Adversarial Perturbation

Anomaly detection is a fundamental yet challenging problem in machine learning due to the lack of label information. In this work, we propose a novel and powerful framework, dubbed as SLA$^2$P, for unsupervised anomaly detection. After extracting representative embeddings from raw data, we apply random projections to the features and regard features transformed by different projections as belonging to distinct pseudo classes. We then train a classifier network on these transformed features to perform self-supervised learning. Next we add adversarial perturbation to the transformed features to decrease their softmax scores of the predicted labels and design anomaly scores based on the predictive uncertainties of the classifier on these perturbed features. Our motivation is that because of the relatively small number and the decentralized modes of anomalies, 1) the pseudo label classifier's training concentrates more on learning the semantic information of normal data rather than anomalous data; 2) the transformed features of the normal data are more robust to the perturbations than those of the anomalies. Consequently, the perturbed transformed features of anomalies fail to be classified well and accordingly have lower anomaly scores than those of the normal samples. Extensive experiments on image, text and inherently tabular benchmark datasets back up our findings and indicate that SLA$^2$P achieves state-of-the-art results on unsupervised anomaly detection tasks consistently.

cs.LG↗

arXiv2021

Self-Supervised Pre-Training for Transformer-Based Person Re-Identification

Transformer-based supervised pre-training achieves great performance in person re-identification (ReID). However, due to the domain gap between ImageNet and ReID datasets, it usually needs a larger pre-training dataset (e.g. ImageNet-21K) to boost the performance because of the strong data fitting ability of the transformer. To address this challenge, this work targets to mitigate the gap between the pre-training and ReID datasets from the perspective of data and model structure, respectively. We first investigate self-supervised learning (SSL) methods with Vision Transformer (ViT) pretrained on unlabelled person images (the LUPerson dataset), and empirically find it significantly surpasses ImageNet supervised pre-training models on ReID tasks. To further reduce the domain gap and accelerate the pre-training, the Catastrophic Forgetting Score (CFS) is proposed to evaluate the gap between pre-training and fine-tuning data. Based on CFS, a subset is selected via sampling relevant data close to the down-stream ReID data and filtering irrelevant data from the pre-training dataset. For the model structure, a ReID-specific module named IBN-based convolution stem (ICS) is proposed to bridge the domain gap by learning more invariant features. Extensive experiments have been conducted to fine-tune the pre-training models under supervised learning, unsupervised domain adaptation (UDA), and unsupervised learning (USL) settings. We successfully downscale the LUPerson dataset to 50% with no performance degradation. Finally, we achieve state-of-the-art performance on Market-1501 and MSMT17. For example, our ViT-S/16 achieves 91.3%/89.9%/89.6% mAP accuracy on Market1501 for supervised/UDA/USL ReID. Codes and models will be released to https://github.com/michuanhaohao/TransReID-SSL.

cs.CV↗

arXiv2021

Pitch Angle Scattering of Fast Particles by Low Frequency Magnetic Fluctuations

The adiabatic invariance of the magnetic moment during particle motion is of fundamental importance to the dynamics of magnetized plasma. The related rate of pitch angle scattering is investigated here for fast particles that thermally stream through static magnetic perturbations. For a uniform magnetic field with a localized perturbation it is found that the curvature parameter $κ^2=\min(R_c/ρ_L)$ does not predict the level of pitch angle scattering. Instead, based on numerical integration of particle orbits in prescribed magnetic fields, we derive predictions for the particle scattering rates, which can be characterized by the relative perturbation amplitude $A\simeq δB/B$, and the particle Larmor radius normalized by the field-aligned wavelength of the perturbations, $ρ_L/λ_{\|}$. Particles with $ρ_L/λ_{\|}\simeq1$, are subject to strong pitch angle scattering, while the level of scattering vanishes for both the limits of $ρ_L/λ_{\|} \ll 1$ and $ρ_L/λ_{\|} \gg 1$. The results are summarized in terms of a scattering operator, suitable for including the described scattering in basic kinetic models.

physics.plasm-ph↗

arXiv2021

An Online Method for A Class of Distributionally Robust Optimization with Non-Convex Objectives

In this paper, we propose a practical online method for solving a class of distributionally robust optimization (DRO) with non-convex objectives, which has important applications in machine learning for improving the robustness of neural networks. In the literature, most methods for solving DRO are based on stochastic primal-dual methods. However, primal-dual methods for DRO suffer from several drawbacks: (1) manipulating a high-dimensional dual variable corresponding to the size of data is time expensive; (2) they are not friendly to online learning where data is coming sequentially. To address these issues, we consider a class of DRO with an KL divergence regularization on the dual variables, transform the min-max problem into a compositional minimization problem, and propose practical duality-free online stochastic methods without requiring a large mini-batch size. We establish the state-of-the-art complexities of the proposed methods with and without a Polyak-Łojasiewicz (PL) condition of the objective. Empirical studies on large-scale deep learning tasks (i) demonstrate that our method can speed up the training by more than 2 times than baseline methods and save days of training time on a large-scale dataset with $\sim$ 265K images, and (ii) verify the supreme performance of DRO over Empirical Risk Minimization (ERM) on imbalanced datasets. Of independent interest, the proposed method can be also used for solving a family of stochastic compositional problems with state-of-the-art complexities.

cs.LG↗

arXiv2021

EMPRESS. IV. Extremely Metal-Poor Galaxies (EMPGs) Including Very Low-Mass Primordial Systems with M*=10^4--10^5 M_sun and 2--3% (O/H)_sun: High (Fe/O) Suggestive of Metal Enrichment by Hypernovae/Pair-Instability Supernovae

We present Keck/LRIS follow-up spectroscopy for 13 photometric candidates of extremely metal poor galaxies (EMPGs) selected by a machine-learning technique applied to the deep (~26 AB mag) optical and wide-area (~500 deg^2) Subaru imaging data in the EMPRESS survey. Nine out of the 13 candidates are EMPGs with an oxygen abundance (O/H) less than ~10% solar value (O/H)_sun, and four sources are contaminants of moderately metal-rich galaxies or no emission-line objects. Notably, two out of the nine EMPGs have extremely-low stellar masses and oxygen abundances of 5*10^4--7*10^5 M_sun and 2--3% (O/H)_sun, respectively. With a sample of five EMPGs with (Fe/O) measurements, two (three) of which are taken from this study (the literature), we confirm that two EMPGs with the lowest (O/H) ratios of ~2% (O/H)_sun show high (Fe/O) ratios of ~0.1, close to the solar abundance ratio. Comparing galaxy chemical enrichment models, we find that the two EMPGs cannot be explained by a scenario of metal-poor gas accretion/episodic star-formation history due to their low (N/O) ratios. We conclude that the two EMPGs can be reproduced by an inclusion of bright hypernovae and/or hypothetical pair-instability supernovae (SNe) preferentially produced in a metal-poor environment. This conclusion implies that primordial galaxies at z~10 could have a high abundance of Fe that is not originated from Type Ia SNe with delays, and that Fe may not serve as a cosmic clock for primordial galaxies.

astro-ph.GA↗

arXiv2021

Generating tightly focused perfect optical vortex for ultra-secure optical encryption

Light's orbital angular momentum (OAM) with inherent mode orthogonality has been suggested as a new way to the optical encryption. However, the dependence of annular intensity profiles on the topological charge complicates nanoscale light-matter interactions and hampers the ultra-secure encryption application. In this paper, we demonstrate ultra-secure image encryption by tightly focusing perfect optical vortex (POV) beams with controllable annular intensity profiles and OAM states. A simple scheme composed of single spatial light modulator is proposed to generate radius-controllable POV in tightly focused beams. Such focused POV beams with identical intensity profiles but varied OAM states are applied to disorder-coupled gold nanorod aggregates to selectively excite electromagnetic hot spots for encoding information through photothermal deformation. As such, ultra-secure image encryption in OAM states of POV beams in combination with different polarizations can be achieved. Our results lay the ground for diverse nanophotonic applications harnessing the OAM division of POV beams.

physics.optics↗

arXiv2021

Magic Pyramid: Accelerating Inference with Early Exiting and Token Pruning

Pre-training and then fine-tuning large language models is commonly used to achieve state-of-the-art performance in natural language processing (NLP) tasks. However, most pre-trained models suffer from low inference speed. Deploying such large models to applications with latency constraints is challenging. In this work, we focus on accelerating the inference via conditional computations. To achieve this, we propose a novel idea, Magic Pyramid (MP), to reduce both width-wise and depth-wise computation via token pruning and early exiting for Transformer-based models, particularly BERT. The former manages to save the computation via removing non-salient tokens, while the latter can fulfill the computation reduction by terminating the inference early before reaching the final layer, if the exiting condition is met. Our empirical studies demonstrate that compared to previous state of arts, MP is not only able to achieve a speed-adjustable inference but also to surpass token pruning and early exiting by reducing up to 70% giga floating point operations (GFLOPs) with less than 0.5% accuracy drop. Token pruning and early exiting express distinctive preferences to sequences with different lengths. However, MP is capable of achieving an average of 8.06x speedup on two popular text classification tasks, regardless of the sizes of the inputs.

cs.CL↗