arXiv subjects

Hao Liu

Publications and source records attributed to Hao Liu.

At least 487 records · Page 27Linked to original sources

arXiv2023

Towards Better Few-Shot and Finetuning Performance with Forgetful Causal Language Models

Large language models (LLM) trained using the next-token-prediction objective, such as GPT3 and PaLM, have revolutionized natural language processing in recent years by showing impressive zero-shot and few-shot capabilities across a wide range of tasks. In this work, we propose a simple technique that significantly boosts the performance of LLMs without adding computational cost. Our key observation is that, by performing the next token prediction task with randomly selected past tokens masked out, we can improve the quality of the learned representations for downstream language understanding tasks. We hypothesize that randomly masking past tokens prevents over-attending to recent tokens and encourages attention to tokens in the distant past. We find that our method, Forgetful Causal Masking (FCM), significantly improves both few-shot and finetuning performance of PaLM. We further consider a simple extension, T-FCM, which introduces bidirectional context to causal language model without altering the sequence order, and further improves finetuning performance.

cs.CL↗

arXiv2022

Two-Sample Test for Stochastic Block Models via Maximum Entry-wise Deviation

The stochastic block model is a popular tool for detecting community structures in network data. Detecting the difference between two community structures is an important issue for stochastic block models. However, the two-sample test has been a largely under-explored domain, and too little work has been devoted to it. In this article, based on the maximum entry--wise deviation of the two centered and rescaled adjacency matrices, we propose a novel test statistic to test two samples of stochastic block models. We prove that the null distribution of the proposed test statistic converges in distribution to a Gumbel distribution, and we show the change of the two samples from stochastic block models can be tested via the proposed method. Then, we show that the proposed test has an asymptotic power guarantee against alternative models. One noticeable advantage of the proposed test statistic is that the number of communities can be allowed to grow linearly up to a logarithmic factor. Further, we extend the proposed method to the degree-corrected stochastic block model. Both simulation studies and real-world data examples indicate that the proposed method works well.

stat.ME↗

arXiv2022

High Dimensional Binary Classification under Label Shift: Phase Transition and Regularization

Label Shift has been widely believed to be harmful to the generalization performance of machine learning models. Researchers have proposed many approaches to mitigate the impact of the label shift, e.g., balancing the training data. However, these methods often consider the underparametrized regime, where the sample size is much larger than the data dimension. The research under the overparametrized regime is very limited. To bridge this gap, we propose a new asymptotic analysis of the Fisher Linear Discriminant classifier for binary classification with label shift. Specifically, we prove that there exists a phase transition phenomenon: Under certain overparametrized regime, the classifier trained using imbalanced data outperforms the counterpart with reduced balanced data. Moreover, we investigate the impact of regularization to the label shift: The aforementioned phase transition vanishes as the regularization becomes strong.

cs.LG↗

arXiv2022

Search for gamma-ray spectral lines with the DArk Matter Particle Explorer

The DArk Matter Particle Explorer (DAMPE) is well suitable for searching for monochromatic and sharp $γ$-ray structures in the GeV$-$TeV range thanks to its unprecedented high energy resolution. In this work, we search for $γ$-ray line structures using five years of DAMPE data. To improve the sensitivity, we develop two types of dedicated data sets (including the BgoOnly data which is the first time to be used in the data analysis for the calorimeter-based gamma-ray observatories) and adopt the signal-to-noise ratio optimized regions of interest (ROIs) for different DM density profiles. No line signals or candidates are found between 10 and 300 GeV in the Galaxy. The constraints on the velocity-averaged cross section for $χχ\to γγ$ and the decay lifetime for $χ\to γν$, both at 95% confidence level, have been calculated and the systematic uncertainties have been taken into account. Comparing to the previous Fermi-LAT results, though DAMPE has an acceptance smaller by a factor of $\sim 10$, similar constraints on the DM parameters are achieved and below 100 GeV the lower limits on the decay lifetime are even stronger by a factor of a few. Our results demonstrate the potential of high-energy-resolution observations on dark matter detection.

astro-ph.HE↗

arXiv2022

A superconducting nanowire photon number resolving four-quadrant detector-based Gigabit deep-space laser communication receiver prototype

Deep space explorations require transferring huge amounts of data quickly from very distant targets. Laser communication is a promising technology that can offer a data rate of magnitude faster than conventional microwave communication due to the fundamentally narrow divergence of light. This study demonstrated a photon-sensitive receiver prototype with over Gigabit data rate, immunity to strong background photon noise, and simultaneous tracking ability. The advantages are inherited from a joint-optimized superconducting nanowire single-photon detector (SNSPD) array, designed into a four-quadrant structure with each quadrant capable of resolving six photons. Installed in a free-space coupled and low-vibration cryostat, the system detection efficiency reached 72.7%, the detector efficiency was 97.5%, and the total photon counting rate was 1.6 Gcps. Additionally, communication performance was tested for pulse position modulation (PPM) format. A series of signal processing methods were introduced to maximize the performance of the forward error correction (FEC) code. Consequently, the receiver exhibits a faster data rate and better sensitivity by about twofold (1.76 photons/bit at 800 Mbps and 3.40 photons/bit at 1.2 Gbps) compared to previously reported results (3.18 photon/bit at 622 Mbps for the Lunar Laser Communication Demonstration). Furthermore, communications in strong background noise and with simultaneous tracking ability were demonstrated aimed at the challenges of daylight operation and accurate tracking of dim beacon light in deep space scenarios.

physics.ins-det↗

arXiv2022

Elastica Models for Color Image Regularization

One classical approach to regularize color is to tream them as two dimensional surfaces embedded in a five dimensional spatial-chromatic space. In this case, a natural regularization term arises as the image surface area. Choosing the chromatic coordinates as dominating over the spatial ones, the image spatial coordinates could be thought of as a paramterization of the image surface manifold in a three dimensional color space. Minimizing the area of the image manifold leads to the Beltrami flow or mean curvature flow of the image surface in the 3D color space, while minimizing the elastica of the image surface yields an additional interesting regularization. Recently, the authors proposed a color elastica model, which minimizes both the surface area and elastica of the image manifold. In this paper, we propose to modify the color elastica and introduce two new models for color image regularization. The revised measures are motivated by the relations between the color elastica model, Euler's elastica model and the total variation model for gray level images. Compared to our previous color elastica model, the new models are direct extensions of Euler's elastica model to color images. The proposed models are nonlinear and challenging to minimize. To overcome this difficulty, two operator-splitting methods are suggested. Specifically, nonlinearities are decoupled by introducing new vector- and matrix-valued variables. Then, the minimization problems are converted to solving initial value problems which are time-discretized by operator splitting. Each subproblem, after splitting either, has a closed-form solution or can be solved efficiently. The effectiveness and advantages of the proposed models are demonstrated by comprehensive experiments. The benefits of incorporating the elastica of the image surface as regularization terms compared to common alternatives are empirically validated.

cs.CV↗

arXiv2022

Removal of point source leakage from time-order data filtering

Time-ordered data (TOD) from ground-based CMB experiments are generally filtered before map-making to remove or reduce the contamination from the ground and the atmospheric emissions. However, when the observation region contains strong point sources, the filtering process will result in considerable leakage around the point sources in a measured CMB map, and leave spurious polarization signals. Therefore, such signals need to be assessed and removed before CMB science exploitation. In this work, we present a new method that we call "template fitting" and can effectively remove these leakage signals in pixel domain, not only satisfying the requirement for measuring primordial gravitational waves from CMB-$B$ modes, but also avoiding time-consuming operations on TOD.

astro-ph.CO↗

arXiv2022

Accelerating spherical harmonic transforms for a large number of sky maps

The spherical harmonic transform is a powerful tool in the analysis of spherical data sets, such as the cosmic microwave background data. In this work, we present a new scheme for the spherical harmonic transforms that supports both CPU and GPU computations, which is specially efficient on a large number of sky maps. By comparing our implementation with the standard Libsharp-HEALPix program, we demonstrate 2-10 times speedup for the CPU implementation, and up to 30 times speedup when a state-of-the-art GPU is employed. This new scheme's software package is available via an open source GitHub repository.

astro-ph.IM↗

arXiv2022

A Fine-grained Interpretability Evaluation Benchmark for Neural NLP

While there is increasing concern about the interpretability of neural models, the evaluation of interpretability remains an open problem, due to the lack of proper evaluation datasets and metrics. In this paper, we present a novel benchmark to evaluate the interpretability of both neural models and saliency methods. This benchmark covers three representative NLP tasks: sentiment analysis, textual similarity and reading comprehension, each provided with both English and Chinese annotated data. In order to precisely evaluate the interpretability, we provide token-level rationales that are carefully annotated to be sufficient, compact and comprehensive. We also design a new metric, i.e., the consistency between the rationales before and after perturbations, to uniformly evaluate the interpretability on different types of tasks. Based on this benchmark, we conduct experiments on three typical models with three saliency methods, and unveil their strengths and weakness in terms of interpretability. We will release this benchmark https://www.luge.ai/#/luge/task/taskDetail?taskId=15 and hope it can facilitate the research in building trustworthy systems.

cs.CL↗

arXiv2022

PAI3D: Painting Adaptive Instance-Prior for 3D Object Detection

3D object detection is a critical task in autonomous driving. Recently multi-modal fusion-based 3D object detection methods, which combine the complementary advantages of LiDAR and camera, have shown great performance improvements over mono-modal methods. However, so far, no methods have attempted to utilize the instance-level contextual image semantics to guide the 3D object detection. In this paper, we propose a simple and effective Painting Adaptive Instance-prior for 3D object detection (PAI3D) to fuse instance-level image semantics flexibly with point cloud features. PAI3D is a multi-modal sequential instance-level fusion framework. It first extracts instance-level semantic information from images, the extracted information, including objects categorical label, point-to-object membership and object position, are then used to augment each LiDAR point in the subsequent 3D detection network to guide and improve detection performance. PAI3D outperforms the state-of-the-art with a large margin on the nuScenes dataset, achieving 71.4 in mAP and 74.2 in NDS on the test split. Our comprehensive experiments show that instance-level image semantics contribute the most to the performance gain, and PAI3D works well with any good-quality instance segmentation models and any modern point cloud 3D encoders, making it a strong candidate for deployment on autonomous vehicles.

cs.CV↗

arXiv2022

Global Classical Solutions to the viscous two-phase flow model with slip Boundary Conditions in 3D Exterior Domains

We consider the two-phase flow model with slip boundary condition in a 3D exterior domains whose boundary is smooth. We establish the global existence of classical solutions of this system provided that the initial energy is suitably small. Moreover, the pressure has large oscillations and contains vacuum states when the initial pressure allows large oscillations and a vacuum. Finally, we also obtain the large-time behavior of the classical solutions.

math.AP↗

arXiv2022

Multimodal Masked Autoencoders Learn Transferable Representations

Building scalable models to learn from diverse, multimodal data remains an open challenge. For vision-language data, the dominant approaches are based on contrastive learning objectives that train a separate encoder for each modality. While effective, contrastive learning approaches introduce sampling bias depending on the data augmentations used, which can degrade performance on downstream tasks. Moreover, these methods are limited to paired image-text data, and cannot leverage widely-available unpaired data. In this paper, we investigate whether a large multimodal model trained purely via masked token prediction, without using modality-specific encoders or contrastive learning, can learn transferable representations for downstream tasks. We propose a simple and scalable network architecture, the Multimodal Masked Autoencoder (M3AE), which learns a unified encoder for both vision and language data via masked token prediction. We provide an empirical study of M3AE trained on a large-scale image-text dataset, and find that M3AE is able to learn generalizable representations that transfer well to downstream tasks. Surprisingly, we find that M3AE benefits from a higher text mask ratio (50-90%), in contrast to BERT whose standard masking ratio is 15%, due to the joint training of two data modalities. We also provide qualitative analysis showing that the learned representation incorporates meaningful information from both image and language. Lastly, we demonstrate the scalability of M3AE with larger model size and training time, and its flexibility to train on both paired image-text data as well as unpaired data.

cs.CV↗

arXiv2022

Palm up: Playing in the Latent Manifold for Unsupervised Pretraining

Large and diverse datasets have been the cornerstones of many impressive advancements in artificial intelligence. Intelligent creatures, however, learn by interacting with the environment, which changes the input sensory signals and the state of the environment. In this work, we aim to bring the best of both worlds and propose an algorithm that exhibits an exploratory behavior whilst it utilizes large diverse datasets. Our key idea is to leverage deep generative models that are pretrained on static datasets and introduce a dynamic model in the latent space. The transition dynamics simply mixes an action and a random sampled latent. It then applies an exponential moving average for temporal persistency, the resulting latent is decoded to image using pretrained generator. We then employ an unsupervised reinforcement learning algorithm to explore in this environment and perform unsupervised representation learning on the collected data. We further leverage the temporal information of this data to pair data points as a natural supervision for representation learning. Our experiments suggest that the learned representations can be successfully transferred to downstream tasks in both vision and reinforcement learning domains.

cs.LG↗

arXiv2022

Performance forecasts for the primordial gravitational wave detection pipelines for AliCPT-1

AliCPT is the first Chinese cosmic microwave background (CMB) experiment which will make the most precise measurements of the CMB polarization in the northern hemisphere. The key science goal for AliCPT is the detection of primordial gravitational waves (PGWs). It is well known that an epoch of cosmic inflation, in the very early universe, can produce PGWs, which leave an imprint on the CMB in form of odd parity $B$-mode polarization. In this work, we study the performance of the component separation and parameter estimation pipelines in context of constraining the value of the tensor-to-scalar ratio. Based on the simulated data for one observation season, we compare five different pipelines with different working principles. Three pipelines perform component separation at map or spectra level before estimating $r$ from the cleaned spectra, while the other two pipelines performs a global fit for both foreground parameters and $r$. We also test different methods to account for the effects of time stream filtering systematics. This work shows that our pipelines provide consistent and robust constraints on the tensor-to-scalar ratio and a consistent sensitivity $σ(r) \sim 0.02$. This showcases the potential of precise $B$-mode polarization measurement with AliCPT-1. AliCPT will provide a powerful opportunity to detect PGWs, which is complementary with various ground-based CMB experiments in the southern hemisphere.

astro-ph.CO↗

arXiv2022

A Novel Max Pressure Algorithm Based on Traffic Delay

This paper considers a novel travel-delay-based Max Pressure algorithm for control of arbitrary transportation networks with signalized intersections. The traditional number-of-vehicle-based Max Pressure (Original-MP) algorithm has received tremendous attention recently due to its ease of implementation and scalability to large network scenarios. The Original-MP algorithm also has a desirable property called maximum stability, which means a demand scenario can be accommodated by this method as long as it can be accommodated by any existing control policy. However, implementation of the Original-MP Max Pressure algorithm may be difficult in practice as estimation of queue lengths at intersections requires significant measurement infrastructure. The Original-MP framework also uses a point queue model to represent the vehicle transition between links, which does not consider the position of the vehicles, even though this may significantly impact the control performance. In addition, intersection approaches with low travel demand can incur arbitrarily large delays due to having short queues. The proposed travel-delay based Max Pressure model overcomes these drawbacks while inheriting the maximum stability feature of the Original-MP. It also is shown to outperform several benchmark Max Pressure variants in a battery of simulation tests. Lastly, the proposed algorithm can be implemented in a Connected Vehicle (CV) environment, in which a subset of vehicles serve as mobile probes. The results show the proposed travel-delay-based model generates lower delay with non-full penetration rate than the benchmark models with full penetration rate under certain traffic conditions.

math.OC↗

arXiv2022

Practical Adversarial Attacks on Spatiotemporal Traffic Forecasting Models

Machine learning based traffic forecasting models leverage sophisticated spatiotemporal auto-correlations to provide accurate predictions of city-wide traffic states. However, existing methods assume a reliable and unbiased forecasting environment, which is not always available in the wild. In this work, we investigate the vulnerability of spatiotemporal traffic forecasting models and propose a practical adversarial spatiotemporal attack framework. Specifically, instead of simultaneously attacking all geo-distributed data sources, an iterative gradient-guided node saliency method is proposed to identify the time-dependent set of victim nodes. Furthermore, we devise a spatiotemporal gradient descent based scheme to generate real-valued adversarial traffic states under a perturbation constraint. Meanwhile, we theoretically demonstrate the worst performance bound of adversarial traffic forecasting attacks. Extensive experiments on two real-world datasets show that the proposed two-step framework achieves up to $67.8\%$ performance degradation on various advanced spatiotemporal forecasting models. Remarkably, we also show that adversarial training with our proposed attacks can significantly improve the robustness of spatiotemporal traffic forecasting models. Our code is available in \url{https://github.com/luckyfan-cs/ASTFA}.

cs.LG↗

arXiv2022

TaCo: Textual Attribute Recognition via Contrastive Learning

As textual attributes like font are core design elements of document format and page style, automatic attributes recognition favor comprehensive practical applications. Existing approaches already yield satisfactory performance in differentiating disparate attributes, but they still suffer in distinguishing similar attributes with only subtle difference. Moreover, their performance drop severely in real-world scenarios where unexpected and obvious imaging distortions appear. In this paper, we aim to tackle these problems by proposing TaCo, a contrastive framework for textual attribute recognition tailored toward the most common document scenes. Specifically, TaCo leverages contrastive learning to dispel the ambiguity trap arising from vague and open-ended attributes. To realize this goal, we design the learning paradigm from three perspectives: 1) generating attribute views, 2) extracting subtle but crucial details, and 3) exploiting valued view pairs for learning, to fully unlock the pre-training potential. Extensive experiments show that TaCo surpasses the supervised counterparts and advances the state-of-the-art remarkably on multiple attribute recognition tasks. Online services of TaCo will be made available.

cs.CV↗

arXiv2022

Asymptotic behavior of conservative solutions to the Hunter-Saxton equation

In this paper we study the large time asymptotic behavior of (energy) conservative solutions to the Hunter-Saxton equation in a generalized framework that consists of the evolutions of solution and its energy measure. We describe the large time asymptotic expansions of the conservative solutions, and rigorously verify the validity of the leading order term in $L^{\infty}(\mathbb{R})$ and ${\dot{H}}^1(\mathbb{R})$ spaces respectively. The leading order term is given by a kink-wave that is determined by the total energy of the system only. As a corollary, we also show that the singular part of the energy measure converges to zero, as the time goes to either positive or negative infinity. Under some natural decay rate assumptions on the tails of the initial energy measure, we rigorously provide the optimal error estimates in $L^{\infty}(\mathbb{R})$ and ${\dot{H}}^1(\mathbb{R})$. As the time goes to infinity, the pointwise convergence and pointwise growth rate for the solution are also obtained under the same assumptions on the initial data. The proofs of our results rely heavily on the elaborate analysis of the generalized characteristics designed for the measure-valued initial data, and explicit formulae for conservative solutions.

math.AP↗