arXiv subjects

Yi Zhu

Publications and source records attributed to Yi Zhu.

At least 145 records · Page 8Linked to original sources

arXiv2023

ViewCo: Discovering Text-Supervised Segmentation Masks via Multi-View Semantic Consistency

Recently, great success has been made in learning visual representations from text supervision, facilitating the emergence of text-supervised semantic segmentation. However, existing works focus on pixel grouping and cross-modal semantic alignment, while ignoring the correspondence among multiple augmented views of the same image. To overcome such limitation, we propose multi-\textbf{View} \textbf{Co}nsistent learning (ViewCo) for text-supervised semantic segmentation. Specifically, we first propose text-to-views consistency modeling to learn correspondence for multiple views of the same input image. Additionally, we propose cross-view segmentation consistency modeling to address the ambiguity issue of text supervision by contrasting the segment features of Siamese visual encoders. The text-to-views consistency benefits the dense assignment of the visual features by encouraging different crops to align with the same text, while the cross-view segmentation consistency modeling provides additional self-supervision, overcoming the limitation of ambiguous text supervision for segmentation masks. Trained with large-scale image-text data, our model can directly segment objects of arbitrary categories in a zero-shot manner. Extensive experiments show that ViewCo outperforms state-of-the-art methods on average by up to 2.9\%, 1.6\%, and 2.4\% mIoU on PASCAL VOC2012, PASCAL Context, and COCO, respectively.

cs.CV↗

arXiv2023

SuperScaler: Supporting Flexible DNN Parallelization via a Unified Abstraction

With the growing model size, deep neural networks (DNN) are increasingly trained over massive GPU accelerators, which demands a proper parallelization plan that transforms a DNN model into fine-grained tasks and then schedules them to GPUs for execution. Due to the large search space, the contemporary parallelization plan generators often rely on empirical rules that couple transformation and scheduling, and fall short in exploring more flexible schedules that yield better memory usage and compute efficiency. This tension can be exacerbated by the emerging models with increasing complexity in their structure and model size. SuperScaler is a system that facilitates the design and generation of highly flexible parallelization plans. It formulates the plan design and generation into three sequential phases explicitly: model transformation, space-time scheduling, and data dependency preserving. Such a principled approach decouples multiple seemingly intertwined factors and enables the composition of highly flexible parallelization plans. As a result, SuperScaler can not only generate empirical parallelization plans, but also construct new plans that achieve up to 3.5X speedup compared to state-of-the-art solutions like DeepSpeed, Megatron and Alpa, for emerging DNN models like Swin-Transformer and AlphaFold2, as well as well-optimized models like GPT-3.

cs.DC↗

arXiv2023

Pushing limits of photovoltaics and photodetection using radial junction nanowire devices

Nanowire devices have long been proposed as an efficient alternative to their planar counterparts for different optoelectronic applications. Unfortunately, challenges related to the growth and characterization of doping and p-n junction formation in nanowire devices (along axial or radial axis) have significantly impeded their development. The problems are further amplified if a p-n junction has to be implemented radially. Therefore, even though radial junction devices are expected to be on par with their axial junction counterparts, there are minimal reports on high-performance radial junction nanowire optoelectronic devices. This paper summarizes our recent results on the simulation and fabrication of radial junction nanowire solar cells and photodetectors, which have shown unprecedented performance and clearly demonstrate the importance of radial junction for optoelectronic applications. Our simulation results show that the proposed radial junction device is both optically and electrically optimal for solar cell and photodetector applications, especially if the absorber quality is extremely low. The radial junction nanowire solar cells could achieve a 17.2% efficiency, whereas the unbiased radial junction photodetector could show sensitivity down to a single photon level using an absorber with a lifetime of less than 50 ps. In comparison, the axial junction planar device made using same substrate as absorber showed less than 1% solar cell efficiency and almost no photodetection at 0 V. This study is conclusive experimental proof of the superiority of radial junction nanowire devices over their thin film or axial junction counterparts, especially when absorber lifetime is extremely low. The proposed device holds huge promise for III-V based photovoltaics and photodetectors.

cond-mat.mtrl-sci↗

arXiv2023

MixGen: A New Multi-Modal Data Augmentation

Data augmentation is a necessity to enhance data efficiency in deep learning. For vision-language pre-training, data is only augmented either for images or for text in previous works. In this paper, we present MixGen: a joint data augmentation for vision-language representation learning to further improve data efficiency. It generates new image-text pairs with semantic relationships preserved by interpolating images and concatenating text. It's simple, and can be plug-and-played into existing pipelines. We evaluate MixGen on four architectures, including CLIP, ViLT, ALBEF and TCL, across five downstream vision-language tasks to show its versatility and effectiveness. For example, adding MixGen in ALBEF pre-training leads to absolute performance improvements on downstream tasks: image-text retrieval (+6.2% on COCO fine-tuned and +5.3% on Flicker30K zero-shot), visual grounding (+0.9% on RefCOCO+), visual reasoning (+$0.9% on NLVR2), visual question answering (+0.3% on VQA2.0), and visual entailment (+0.4% on SNLI-VE).

cs.CV↗

arXiv2022

SPT: Semi-Parametric Prompt Tuning for Multitask Prompted Learning

Pre-trained large language models can efficiently interpolate human-written prompts in a natural way. Multitask prompted learning can help generalization through a diverse set of tasks at once, thus enhancing the potential for more effective downstream fine-tuning. To perform efficient multitask-inference in the same batch, parameter-efficient fine-tuning methods such as prompt tuning have been proposed. However, the existing prompt tuning methods may lack generalization. We propose SPT, a semi-parametric prompt tuning method for multitask prompted learning. The novel component of SPT is a memory bank from where memory prompts are retrieved based on discrete prompts. Extensive experiments, such as (i) fine-tuning a full language model with SPT on 31 different tasks from 8 different domains and evaluating zero-shot generalization on 9 heldout datasets under 5 NLP task categories and (ii) pretraining SPT on the GLUE datasets and evaluating fine-tuning on the SuperGLUE datasets, demonstrate effectiveness of SPT.

cs.CL↗

arXiv2022

What Makes for Good Tokenizers in Vision Transformer?

The architecture of transformers, which recently witness booming applications in vision tasks, has pivoted against the widespread convolutional paradigm. Relying on the tokenization process that splits inputs into multiple tokens, transformers are capable of extracting their pairwise relationships using self-attention. While being the stemming building block of transformers, what makes for a good tokenizer has not been well understood in computer vision. In this work, we investigate this uncharted problem from an information trade-off perspective. In addition to unifying and understanding existing structural modifications, our derivation leads to better design strategies for vision tokenizers. The proposed Modulation across Tokens (MoTo) incorporates inter-token modeling capability through normalization. Furthermore, a regularization objective TokenProp is embraced in the standard training regime. Through extensive experiments on various transformer architectures, we observe both improved performance and intriguing properties of these two plug-and-play designs with negligible computational overhead. These observations further indicate the importance of the commonly-omitted designs of tokenizers in vision transformer.

cs.CV↗

arXiv2022

CoupAlign: Coupling Word-Pixel with Sentence-Mask Alignments for Referring Image Segmentation

Referring image segmentation aims at localizing all pixels of the visual objects described by a natural language sentence. Previous works learn to straightforwardly align the sentence embedding and pixel-level embedding for highlighting the referred objects, but ignore the semantic consistency of pixels within the same object, leading to incomplete masks and localization errors in predictions. To tackle this problem, we propose CoupAlign, a simple yet effective multi-level visual-semantic alignment method, to couple sentence-mask alignment with word-pixel alignment to enforce object mask constraint for achieving more accurate localization and segmentation. Specifically, the Word-Pixel Alignment (WPA) module performs early fusion of linguistic and pixel-level features in intermediate layers of the vision and language encoders. Based on the word-pixel aligned embedding, a set of mask proposals are generated to hypothesize possible objects. Then in the Sentence-Mask Alignment (SMA) module, the masks are weighted by the sentence embedding to localize the referred object, and finally projected back to aggregate the pixels for the target. To further enhance the learning of the two alignment modules, an auxiliary loss is designed to contrast the foreground and background pixels. By hierarchically aligning pixels and masks with linguistic features, our CoupAlign captures the pixel coherence at both visual and semantic levels, thus generating more accurate predictions. Extensive experiments on popular datasets (e.g., RefCOCO and G-Ref) show that our method achieves consistent improvements over state-of-the-art methods, e.g., about 2% oIoU increase on the validation and testing set of RefCOCO. Especially, CoupAlign has remarkable ability in distinguishing the target from multiple objects of the same class.

cs.CV↗

arXiv2022

Visual Prompt Tuning for Test-time Domain Adaptation

Models should be able to adapt to unseen data during test-time to avoid performance drops caused by inevitable distribution shifts in real-world deployment scenarios. In this work, we tackle the practical yet challenging test-time adaptation (TTA) problem, where a model adapts to the target domain without accessing the source data. We propose a simple recipe called \textit{Data-efficient Prompt Tuning} (DePT) with two key ingredients. First, DePT plugs visual prompts into the vision Transformer and only tunes these source-initialized prompts during adaptation. We find such parameter-efficient finetuning can efficiently adapt the model representation to the target domain without overfitting to the noise in the learning objective. Second, DePT bootstraps the source representation to the target domain by memory bank-based online pseudo-labeling. A hierarchical self-supervised regularization specially designed for prompts is jointly optimized to alleviate error accumulation during self-training. With much fewer tunable parameters, DePT demonstrates not only state-of-the-art performance on major adaptation benchmarks VisDA-C, ImageNet-C, and DomainNet-126, but also superior data efficiency, i.e., adaptation with only 1\% or 10\% data without much performance degradation compared to 100\% data. In addition, DePT is also versatile to be extended to online or multi-source TTA settings.

cs.CV↗

arXiv2022

Edge Deep Learning Enabled Freezing of Gait Detection in Parkinson's Patients

This paper presents the design of a wireless sensor network for detecting and alerting the freezing of gait (FoG) symptoms in patients with Parkinson's disease. Three sensor nodes, each integrating a 3-axis accelerometer, can be placed on a patient at ankle, thigh, and truck. Each sensor node can independently detect FoG using an on-device deep learning (DL) model, featuring a squeeze and excitation convolutional neural network (CNN). In a validation using a public dataset, the prototype developed achieved a FoG detection sensitivity of 88.8% and an F1 score of 85.34%, using less than 20 k trainable parameters per sensor node. Once FoG is detected, an auditory signal will be generated to alert users, and the alarm signal will also be sent to mobile phones for further actions if needed. The sensor node can be easily recharged wirelessly by inductive coupling. The system is self-contained and processes all user data locally without streaming data to external devices or the cloud, thus eliminating the cybersecurity risks and power penalty associated with wireless data transmission. The developed methodology can be used in a wide range of applications.

eess.SP↗

arXiv2022

Depositing boron on Cu(111): Borophene or boride?

Large-area single-crystal surface structures were successfully prepared on Cu(111) substrate with boron deposition, which is critical for prospective applications. However, the proposed borophene structures do not match the scanning tunneling microscopy (STM) results very well, while the proposed copper boride is at odds with the traditional knowledge that ordered copper-rich borides normally do not exist due to small difference in electronegativity and large difference in atomic size. To clarify the controversy and elucidate the formation mechanism of the unexpected copper boride, we conducted systematic STM, X-ray photoelectron spectroscopy and angle-resolved photoemission spectroscopy investigations, confirming the synthesis of two-dimensional copper boride rather than borophene on Cu(111) after boron deposition under ultrahigh vacuum. First-principles calculations with defective surface models further indicate that boron atoms tend to react with Cu atoms near terrace edges or defects, which in turn shapes the intermediate structures of copper boride and leads to the formation of stable Cu-B monolayer via large-scale surface reconstruction eventually.

cond-mat.mtrl-sci↗

arXiv2022

Stability and large-time behavior on 3D incompressible MHD equations with partial dissipation near a background magnetic field

Physical experiments and numerical simulations have observed a remarkable stabilizing phenomenon: a background magnetic field stabilizes and damps electrically conducting fluids. This paper intends to establish this phenomenon as a mathematically rigorous fact on a magnetohydrodynamic (MHD) system with anisotropic dissipation in $\mathbb R^3$. The velocity equation in this system is the 3D Navier-Stokes equation with dissipation only in the $x_1$-direction while the magnetic field obeys the induction equation with magnetic diffusion in two horizontal directions. We establish that any perturbation near the background magnetic field $(0,1,0)$ is globally stable in the Sobolev setting $H^3(\mathbb R^3)$. In addition, explicit decay rates in $H^2(\mathbb R^3)$ are also obtained. When there is no presence of the magnetic field, the 3D anisotropic Navier-Stokes equation in $\mathbb R^3$ is not well understood and the small data global well-posedness remains an intriguing open problem. This paper reveals the mechanism of how the magnetic field generates enhanced dissipation and helps stabilize the fluid.

math.AP↗

arXiv2022

Global solutions to 3D incompressible MHD system with dissipation in only one direction

The small data global well-posedness of the 3D incompressible Navier-Stokes equations in $\mathbb R^3$ with only one-directional dissipation remains an outstanding open problem. The dissipation in just one direction, say $\partial_1^2 u$ is simply insufficient in controlling the nonlinearity in the whole space $\mathbb R^3$. The beautiful work of Paicu and Zhang \cite{ZHANG1} solved the case when the spatial domain is bounded in the $x_1$-direction by observing a crucial Poincaré type inequality. Motivated by this Navier-Stokes open problem and by experimental observations on the stabilizing effects of background magnetic fields, this paper intends to understand the global well-posedness and stability of a special 3D magnetohydrodynamic (MHD) system near a background magnetic field. The spatial domain is $\mathbb R^3$ and the velocity in this MHD system obeys the 3D Navier-Stokes with only one-directional dissipation. With no Poincaré type inequality, this problem appears to be impossible. By discovering the mathematical mechanism of the experimentally observed stabilizing effect and introducing several innovative techniques to deal with the derivative loss difficulties, we are able to bound the Navier-Stokes nonlinearity and solve the desired global well-posedness and stability problem.

math.AP↗

arXiv2022

Harnessing Interpretable Machine Learning for Holistic Inverse Design of Origami

This work harnesses interpretable machine learning methods to address the challenging inverse design problem of origami-inspired systems. We show that a decision tree-random forest method is particularly suitable for fitting origami databases, containing both design features and functional performance, to generate human-understandable decision rules for the inverse design of functional origami. First, the tree method is unique because it can handle complex interactions between categorical features and continuous features, allowing it to compare different origami patterns for a design. Second, this interpretable method can tackle multi-objective problems for designing functional origami with multiple and multi-physical performance targets. Finally, the method can extend existing shape-fitting algorithms for origami to consider non-geometrical performance. The proposed framework enables holistic inverse design of origami, considering both shape and function, to build novel reconfigurable structures for various applications such as metamaterials, deployable structures, soft robots, biomedical devices, and many more.

cond-mat.soft↗

arXiv2022

Pixel-level Correspondence for Self-Supervised Learning from Video

While self-supervised learning has enabled effective representation learning in the absence of labels, for vision, video remains a relatively untapped source of supervision. To address this, we propose Pixel-level Correspondence (PiCo), a method for dense contrastive learning from video. By tracking points with optical flow, we obtain a correspondence map which can be used to match local features at different points in time. We validate PiCo on standard benchmarks, outperforming self-supervised baselines on multiple dense prediction tasks, without compromising performance on image classification.

cs.CV↗

arXiv2022

Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-Tailed Recognition

Existing out-of-distribution (OOD) detection methods are typically benchmarked on training sets with balanced class distributions. However, in real-world applications, it is common for the training sets to have long-tailed distributions. In this work, we first demonstrate that existing OOD detection methods commonly suffer from significant performance degradation when the training set is long-tail distributed. Through analysis, we posit that this is because the models struggle to distinguish the minority tail-class in-distribution samples, from the true OOD samples, making the tail classes more prone to be falsely detected as OOD. To solve this problem, we propose Partial and Asymmetric Supervised Contrastive Learning (PASCL), which explicitly encourages the model to distinguish between tail-class in-distribution samples and OOD samples. To further boost in-distribution classification accuracy, we propose Auxiliary Branch Finetuning, which uses two separate branches of BN and classification layers for anomaly detection and in-distribution classification, respectively. The intuition is that in-distribution and OOD anomaly data have different underlying distributions. Our method outperforms previous state-of-the-art method by $1.29\%$, $1.45\%$, $0.69\%$ anomaly detection false positive rate (FPR) and $3.24\%$, $4.06\%$, $7.89\%$ in-distribution classification accuracy on CIFAR10-LT, CIFAR100-LT, and ImageNet-LT, respectively. Code and pre-trained models are available at https://github.com/amazon-research/long-tailed-ood-detection.

cs.CV↗

arXiv2022

Global well-posedness for 2D non-resistive compressible MHD system in periodic domain

This paper focuses on the 2D compressible magnetohydrodynamic (MHD) equations without magnetic diffusion in a periodic domain. We present a systematic approach to establishing the global existence of smooth solutions when the initial data is close to a background magnetic field. In addition, stability and large-time decay rates are also obtained. When there is no magnetic diffusion, the magnetic field and the density are governed by forced transport equations and the problem considered here is difficult. This paper implements several key observations and ideas to maximize the enhanced dissipation due to hidden structures and interactions. In particular, the weak smoothing and stabilization generated by the background magnetic field and the extra regularization in the divergence part of the velocity field are fully exploited. Compared with the previous works, this paper appears to be the first to investigate such system on bounded domains and the first to solve this problem by pure energy estimates, which help reduce the complexity in other approaches. In addition, this paper combines the well-posedness with the precise large-time behavior, a strategy that can be extended to higher dimensions.

math.AP↗

arXiv2022

Traveling edge states in massive Dirac equations along slowly varying edges

Topologically protected wave motion has attracted considerable interest due to its novel properties and potential applications in many different fields. In this work, we study edge modes and traveling edge states via the linear Dirac equations with so-called domain wall masses. The unidirectional edge state provides a heuristic approach to more general traveling edge states through the localized behavior along slowly varying edges. We show the leading asymptotic solutions of two typical edge states that follow the circular and curved edges with small curvature by analytic and quantitative arguments.

math.AP↗

arXiv2022

ADAPT: Vision-Language Navigation with Modality-Aligned Action Prompts

Vision-Language Navigation (VLN) is a challenging task that requires an embodied agent to perform action-level modality alignment, i.e., make instruction-asked actions sequentially in complex visual environments. Most existing VLN agents learn the instruction-path data directly and cannot sufficiently explore action-level alignment knowledge inside the multi-modal inputs. In this paper, we propose modAlity-aligneD Action PrompTs (ADAPT), which provides the VLN agent with action prompts to enable the explicit learning of action-level modality alignment to pursue successful navigation. Specifically, an action prompt is defined as a modality-aligned pair of an image sub-prompt and a text sub-prompt, where the former is a single-view observation and the latter is a phrase like ''walk past the chair''. When starting navigation, the instruction-related action prompt set is retrieved from a pre-built action prompt base and passed through a prompt encoder to obtain the prompt feature. Then the prompt feature is concatenated with the original instruction feature and fed to a multi-layer transformer for action prediction. To collect high-quality action prompts into the prompt base, we use the Contrastive Language-Image Pretraining (CLIP) model which has powerful cross-modality alignment ability. A modality alignment loss and a sequential consistency loss are further introduced to enhance the alignment of the action prompt and enforce the agent to focus on the related prompt sequentially. Experimental results on both R2R and RxR show the superiority of ADAPT over state-of-the-art methods.

cs.CV↗