arXiv subjects

Haoyu He

Publications and source records attributed to Haoyu He.

49 records · Page 3Linked to original sources

arXiv2023

EcoFormer: Energy-Saving Attention with Linear Complexity

Transformer is a transformative framework that models sequential data and has achieved remarkable performance on a wide range of tasks, but with high computational and energy cost. To improve its efficiency, a popular choice is to compress the models via binarization which constrains the floating-point values into binary ones to save resource consumption owing to cheap bitwise operations significantly. However, existing binarization methods only aim at minimizing the information loss for the input distribution statistically, while ignoring the pairwise similarity modeling at the core of the attention. To this end, we propose a new binarization paradigm customized to high-dimensional softmax attention via kernelized hashing, called EcoFormer, to map the original queries and keys into low-dimensional binary codes in Hamming space. The kernelized hash functions are learned to match the ground-truth similarity relations extracted from the attention map in a self-supervised way. Based on the equivalence between the inner product of binary codes and the Hamming distance as well as the associative property of matrix multiplication, we can approximate the attention in linear complexity by expressing it as a dot-product of binary codes. Moreover, the compact binary representations of queries and keys enable us to replace most of the expensive multiply-accumulate operations in attention with simple accumulations to save considerable on-chip energy footprint on edge devices. Extensive experiments on both vision and language tasks show that EcoFormer consistently achieves comparable performance with standard attentions while consuming much fewer resources. For example, based on PVTv2-B0 and ImageNet-1K, Ecoformer achieves a 73% on-chip energy footprint reduction with only a 0.33% performance drop compared to the standard attention. Code is available at https://github.com/ziplab/EcoFormer.

cs.CV↗

arXiv2022

MAP-Music2Vec: A Simple and Effective Baseline for Self-Supervised Music Audio Representation Learning

The deep learning community has witnessed an exponentially growing interest in self-supervised learning (SSL). However, it still remains unexplored how to build a framework for learning useful representations of raw music waveforms in a self-supervised manner. In this work, we design Music2Vec, a framework exploring different SSL algorithmic components and tricks for music audio recordings. Our model achieves comparable results to the state-of-the-art (SOTA) music SSL model Jukebox, despite being significantly smaller with less than 2% of parameters of the latter. The model will be released on Huggingface(Please refer to: https://huggingface.co/m-a-p/music2vec-v1)

cs.SD↗

arXiv2022

Mesa: A Memory-saving Training Framework for Transformers

There has been an explosion of interest in designing high-performance Transformers. While Transformers have delivered significant performance improvements, training such networks is extremely memory intensive owing to storing all intermediate activations that are needed for gradient computation during backpropagation, especially for long sequences. To this end, we present Mesa, a memory-saving training framework for Transformers. Specifically, Mesa uses exact activations during forward pass while storing a low-precision version of activations to reduce memory consumption during training. The low-precision activations are then dequantized during back-propagation to compute gradients. Besides, to address the heterogeneous activation distributions in the multi-head self-attention layers, we propose a head-wise activation quantization strategy, which quantizes activations based on the statistics of each head to minimize the approximation error. To further boost training efficiency, we learn quantization parameters by running estimates. More importantly, by re-investing the saved memory in employing a larger batch size or scaling up model size, we may further improve the performance under constrained computational resources. Extensive experiments on ImageNet, CIFAR-100 and ADE20K demonstrate that Mesa can achieve flexible memory-savings (up to 50%) during training while achieving comparable or even better performance. Code is available at https://github.com/ziplab/Mesa.

cs.CV↗

arXiv2022

An Optical Parametric Amplifier via $ χ^{(2)} $ in AlGaAs Waveguides

We report parametric gain by utilizing $ χ^{(2)} $ non-linearities in a semiconductor Bragg Reflection Waveguide (BRW) waveguide chip. Under the two-mode degenerate type II phase matching, it can be shown that more than 18 dBs of parametric gain for both TE and TM modes is tenable in 100s of micrometers of device length. Polarization insensitive parametric gain can be attained within the 1550 nm region of the spectrum. These AlGaAs BRW waveguides exhibit sub-photon per pulse sensitivity. This is in sharp contrast to other types of parametric gain devices which utilize $ χ^{(3)} $, where the pump wavelength is in the vicinity of the signal wavelength. This sensitivity, which reached 0.1~photon/pulse, can usher a new era for on-chip quantum information processing using compact, micrometer-scale devices.

physics.optics↗

arXiv2021

Less is More: Pay Less Attention in Vision Transformers

Transformers have become one of the dominant architectures in deep learning, particularly as a powerful alternative to convolutional neural networks (CNNs) in computer vision. However, Transformer training and inference in previous works can be prohibitively expensive due to the quadratic complexity of self-attention over a long sequence of representations, especially for high-resolution dense prediction tasks. To this end, we present a novel Less attention vIsion Transformer (LIT), building upon the fact that the early self-attention layers in Transformers still focus on local patterns and bring minor benefits in recent hierarchical vision Transformers. Specifically, we propose a hierarchical Transformer where we use pure multi-layer perceptrons (MLPs) to encode rich local patterns in the early stages while applying self-attention modules to capture longer dependencies in deeper layers. Moreover, we further propose a learned deformable token merging module to adaptively fuse informative patches in a non-uniform manner. The proposed LIT achieves promising performance on image recognition tasks, including image classification, object detection and instance segmentation, serving as a strong backbone for many vision tasks. Code is available at: https://github.com/zhuang-group/LIT

cs.CV↗

arXiv2021

A $χ^{(2)}$-based AlGaAs Phase Sensitive Amplifier with Record Gain, Noise and Sensitivity

Phase sensitive amplifiers (PSAs) have the potential to empower substantial advances in emerging generations of optical communication systems as well as classical and quantum on-chip signal processing. The core building block of a PSA is a nonlinear medium. While the second-order nonlinearity ($χ^{(2)}$) is stronger than the third-order nonlinearity ($χ^{(3)}$), it is used less often in semiconductors for parametric amplification owing to the challenges of effectively phase matching the interacting waves as well as two-photon absorption of the pump. In this work, we demonstrate the successful design, fabrication, and characterization of the first $χ^{(2)}$-based semiconductor PSA using an efficient phase matching approach and a pulsed pump, based on an aluminium gallium arsenide (AlGaAs) waveguide platform. Non-centrosymmetric semiconductors such as AlGaAs offer appreciable $χ^{(2)}$. Such waveguides also achieve more than one order of magnitude greater pump field confinement when compared to other materials with large $χ^{(2)}$ such as Periodically Poled Lithium Niobate (PPLN). Our AlGaAs PSA achieves an on-chip in-phase gain, a sensitivity of 0.005 photons per pulse, and approaches theoretical minimal noise figure (NF) of 0~dB. With the capability of operating on signal states with sub-single photons per pulse, our PSA could usher in a new era of on-chip quantum circuits.

physics.optics↗

arXiv2021

Distiller: A Systematic Study of Model Distillation Methods in Natural Language Processing

We aim to identify how different components in the KD pipeline affect the resulting performance and how much the optimal KD pipeline varies across different datasets/tasks, such as the data augmentation policy, the loss function, and the intermediate representation for transferring the knowledge between teacher and student. To tease apart their effects, we propose Distiller, a meta KD framework that systematically combines a broad range of techniques across different stages of the KD pipeline, which enables us to quantify each component's contribution. Within Distiller, we unify commonly used objectives for distillation of intermediate representations under a universal mutual information (MI) objective and propose a class of MI-$α$ objective functions with better bias/variance trade-off for estimating the MI between the teacher and the student. On a diverse set of NLP datasets, the best Distiller configurations are identified via large-scale hyperparameter optimization. Our experiments reveal the following: 1) the approach used to distill the intermediate representations is the most important factor in KD performance, 2) among different objectives for intermediate distillation, MI-$α$ performs the best, and 3) data augmentation provides a large boost for small training datasets or small student networks. Moreover, we find that different datasets/tasks prefer different KD algorithms, and thus propose a simple AutoDistiller algorithm that can recommend a good KD pipeline for a new dataset.

cs.CL↗

arXiv2021

Scalable Vision Transformers with Hierarchical Pooling

The recently proposed Visual image Transformers (ViT) with pure attention have achieved promising performance on image recognition tasks, such as image classification. However, the routine of the current ViT model is to maintain a full-length patch sequence during inference, which is redundant and lacks hierarchical representation. To this end, we propose a Hierarchical Visual Transformer (HVT) which progressively pools visual tokens to shrink the sequence length and hence reduces the computational cost, analogous to the feature maps downsampling in Convolutional Neural Networks (CNNs). It brings a great benefit that we can increase the model capacity by scaling dimensions of depth/width/resolution/patch size without introducing extra computational complexity due to the reduced sequence length. Moreover, we empirically find that the average pooled visual tokens contain more discriminative information than the single class token. To demonstrate the improved scalability of our HVT, we conduct extensive experiments on the image classification task. With comparable FLOPs, our HVT outperforms the competitive baselines on ImageNet and CIFAR-100 datasets. Code is available at https://github.com/MonashAI/HVT

cs.CV↗

arXiv2021

Progressive One-shot Human Parsing

Prior human parsing models are limited to parsing humans into classes pre-defined in the training data, which is not flexible to generalize to unseen classes, e.g., new clothing in fashion analysis. In this paper, we propose a new problem named one-shot human parsing (OSHP) that requires to parse human into an open set of reference classes defined by any single reference example. During training, only base classes defined in the training set are exposed, which can overlap with part of reference classes. In this paper, we devise a novel Progressive One-shot Parsing network (POPNet) to address two critical challenges , i.e., testing bias and small sizes. POPNet consists of two collaborative metric learning modules named Attention Guidance Module and Nearest Centroid Module, which can learn representative prototypes for base classes and quickly transfer the ability to unseen classes during testing, thereby reducing testing bias. Moreover, POPNet adopts a progressive human parsing framework that can incorporate the learned knowledge of parent classes at the coarse granularity to help recognize the descendant classes at the fine granularity, thereby handling the small sizes issue. Experiments on the ATR-OS benchmark tailored for OSHP demonstrate POPNet outperforms other representative one-shot segmentation models by large margins and establishes a strong baseline. Source code can be found at https://github.com/Charleshhy/One-shot-Human-Parsing.

cs.CV↗

arXiv2020

Non-classical Semiconductor Photon Sources Enhancing the Performance of Classical Target Detection Systems

We demonstrate and analyze how deploying non-classical intensity correlations obtained from a monolithic semiconductor quantum photon source can enhance classical target detection systems. This is demonstrated by examining the advantages offered by the utilization of the non-classical correlations in a correlation based target detection protocol. We experimentally demonstrate that under the same condition, the target contrast obtained from the protocol when non-classical correlations are utilized exhibits an improvement of up to 17.79dB over the best classical intensity correlation-based target detection protocol, under 29.69dB channel loss and excess noise 13.40dB stronger than the probe signal. We also assessed how the strong frequency correlations within the non-classical photon pairs can be used to further enhance this protocol.

physics.optics↗

arXiv2020

Enhancing LIDAR performance metrics using continuous-wave photon-pair sources

In order to enhance LIDAR performance metrics such as target detection sensitivity, noise resilience and ranging accuracy, we exploit the strong temporal correlation within the photon pairs generated in continuous-wave pumped semiconductor waveguides. The enhancement attained through the use of such non-classical sources is measured and compared to a corresponding target detection scheme based on simple photon-counting detection. The performances of both schemes are quantified by the estimation uncertainty and Fisher information of the probe photon transmission, which is a widely adopted sensing figure of merit. The target detection experiments are conducted with high probe channel loss ($\simeq 1-5\times10^{-5}$) and formidable environment noise up to 36 dB stronger than the detected probe power of $1.64\times 10^{-5}$ pW. The experimental result shows significant advantages offered by the enhanced scheme with up to 26.3 dB higher performance in terms of estimation uncertainty, which is equivalent to a reduction of target detection time by a factor of 430 or 146 (21.6 dB) times more resilience to noise. We also experimentally demonstrated ranging with these non-classical photon pairs generated with continuous-wave pump in the presence of strong noise and loss, achieving $\approx$5 cm distance resolution that is limited by the temporal resolution of the detectors.

physics.optics↗

arXiv2019

Grapy-ML: Graph Pyramid Mutual Learning for Cross-dataset Human Parsing

Human parsing, or human body part semantic segmentation, has been an active research topic due to its wide potential applications. In this paper, we propose a novel GRAph PYramid Mutual Learning (Grapy-ML) method to address the cross-dataset human parsing problem, where the annotations are at different granularities. Starting from the prior knowledge of the human body hierarchical structure, we devise a graph pyramid module (GPM) by stacking three levels of graph structures from coarse granularity to fine granularity subsequently. At each level, GPM utilizes the self-attention mechanism to model the correlations between context nodes. Then, it adopts a top-down mechanism to progressively refine the hierarchical features through all the levels. GPM also enables efficient mutual learning. Specifically, the network weights of the first two levels are shared to exchange the learned coarse-granularity information across different datasets. By making use of the multi-granularity labels, Grapy-ML learns a more discriminative feature representation and achieves state-of-the-art performance, which is demonstrated by extensive experiments on the three popular benchmarks, e.g. CIHP dataset. The source code is publicly available at https://github.com/Charleshhy/Grapy-ML.

cs.CV↗

arXiv2015

Two Polarization Entangled Sources from the Same Semiconductor Chip

Generating nonclassical states of photons such as polarization entangled states on a monolithic chip is a crucial step towards practical applications of optical quantum information processing such as quantum computing and quantum key distribution. Here we demonstrate two polarization entangled photon sources in a single monolithic semiconductor waveguide. The first source is achieved through the concurrent utilization of two spontaneous parametric down-conversion (SPDC) processes, namely type-0 and type-I SPDC processes. The chip can also generate polarization entangled photons via the type-II SPDC process, enabling the generation of both co-polarized and cross-polarized entangled photons in the same device. In both cases, polarization entanglement is generated directly on the chip without the use of any off-chip compensation, interferometry or bandpass filtering. This enables direct, chip-based generation of both Bell states $(|H,H\rangle+|V,V\rangle)/\sqrt{2}$ and $(|H,V\rangle+|V,H\rangle)/\sqrt{2}$ simultaneously utilizing the same pump source. In addition, based on compound semiconductors, this chip can be directly integrated with it own pump laser. This technique ushers an era of self-contained, integrated, electrically pumped, room-temperature polarization entangled photon sources.

quant-ph↗