Searcharxiv⌕ Search

arXiv subjects

Jue Wang

Publications and source records attributed to Jue Wang.

At least 109 records · Page 6Linked to original sources
arXiv2023

Fluid Antenna-Assisted MIMO Transmission Exploiting Statistical CSI

In conventional multiple-input multiple-output (MIMO) communication systems, the positions of antennas are fixed. To take full advantage of spatial degrees of freedom, a new technology called fluid antenna (FA) is proposed to obtain higher achievable rate and diversity gain. Most existing works on FA exploit instantaneous channel state information (CSI). However, in FA-assisted systems, it is difficult to obtain instantaneous CSI since changes in the antenna position will lead to channel variation. In this letter, we investigate a FA-assisted MIMO system using relatively slow-varying statistical CSI. Specifically, in the criterion of rate maximization, we propose an algorithmic framework for transmit precoding and transmit/receive FAs position designs with statistical CSI. Simulation results show that our proposed algorithm in FA-assisted systems significantly outperforms baselines in terms of rate performance.

cs.IT↗
arXiv2023

Infinite forecast combinations based on Dirichlet process

Forecast combination integrates information from various sources by consolidating multiple forecast results from the target time series. Instead of the need to select a single optimal forecasting model, this paper introduces a deep learning ensemble forecasting model based on the Dirichlet process. Initially, the learning rate is sampled with three basis distributions as hyperparameters to convert the infinite mixture into a finite one. All checkpoints are collected to establish a deep learning sub-model pool, and weight adjustment and diversity strategies are developed during the combination process. The main advantage of this method is its ability to generate the required base learners through a single training process, utilizing the decaying strategy to tackle the challenge posed by the stochastic nature of gradient descent in determining the optimal learning rate. To ensure the method's generalizability and competitiveness, this paper conducts an empirical analysis using the weekly dataset from the M4 competition and explores sensitivity to the number of models to be combined. The results demonstrate that the ensemble model proposed offers substantial improvements in prediction accuracy and stability compared to a single benchmark model.

cs.LG↗
arXiv2023

Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time

Large language models (LLMs) with hundreds of billions of parameters have sparked a new wave of exciting AI applications. However, they are computationally expensive at inference time. Sparsity is a natural approach to reduce this cost, but existing methods either require costly retraining, have to forgo LLM's in-context learning ability, or do not yield wall-clock time speedup on modern hardware. We hypothesize that contextual sparsity, which are small, input-dependent sets of attention heads and MLP parameters that yield approximately the same output as the dense model for a given input, can address these issues. We show that contextual sparsity exists, that it can be accurately predicted, and that we can exploit it to speed up LLM inference in wall-clock time without compromising LLM's quality or in-context learning ability. Based on these insights, we propose DejaVu, a system that uses a low-cost algorithm to predict contextual sparsity on the fly given inputs to each layer, along with an asynchronous and hardware-aware implementation that speeds up LLM inference. We validate that DejaVu can reduce the inference latency of OPT-175B by over 2X compared to the state-of-the-art FasterTransformer, and over 6X compared to the widely used Hugging Face implementation, without compromising model quality. The code is available at https://github.com/FMInference/DejaVu.

cs.LG↗
arXiv2023

Minimalist and High-Performance Semantic Segmentation with Plain Vision Transformers

In the wake of Masked Image Modeling (MIM), a diverse range of plain, non-hierarchical Vision Transformer (ViT) models have been pre-trained with extensive datasets, offering new paradigms and significant potential for semantic segmentation. Current state-of-the-art systems incorporate numerous inductive biases and employ cumbersome decoders. Building upon the original motivations of plain ViTs, which are simplicity and generality, we explore high-performance `minimalist' systems to this end. Our primary purpose is to provide simple and efficient baselines for practical semantic segmentation with plain ViTs. Specifically, we first explore the feasibility and methodology for achieving high-performance semantic segmentation using the last feature map. As a result, we introduce the PlainSeg, a model comprising only three 3$\times$3 convolutions in addition to the transformer layers (either encoder or decoder). In this process, we offer insights into two underlying principles: (i) high-resolution features are crucial to high performance in spite of employing simple up-sampling techniques and (ii) the slim transformer decoder requires a much larger learning rate than the wide transformer decoder. On this basis, we further present the PlainSeg-Hier, which allows for the utilization of hierarchical features. Extensive experiments on four popular benchmarks demonstrate the high performance and efficiency of our methods. They can also serve as powerful tools for assessing the transfer ability of base models in semantic segmentation. Code is available at \url{https://github.com/ydhongHIT/PlainSeg}.

cs.CV↗
arXiv2023

Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Prompt

While the numerous parameters in Large Language Models (LLMs) contribute to their superior performance, this massive scale makes them inefficient and memory-hungry. Thus, they are hard to deploy on commodity hardware, such as one single GPU. Given the memory and power constraints of such devices, model compression methods are widely employed to reduce both the model size and inference latency, which essentially trades off model quality in return for improved efficiency. Thus, optimizing this accuracy-efficiency trade-off is crucial for the LLM deployment on commodity hardware. In this paper, we introduce a new perspective to optimize this trade-off by prompting compressed models. Specifically, we first observe that for certain questions, the generation quality of a compressed LLM can be significantly improved by adding carefully designed hard prompts, though this isn't the case for all questions. Based on this observation, we propose a soft prompt learning method where we expose the compressed model to the prompt learning process, aiming to enhance the performance of prompts. Our experimental analysis suggests our soft prompt strategy greatly improves the performance of the 8x compressed LLaMA-7B model (with a joint 4-bit quantization and 50% weight pruning compression), allowing them to match their uncompressed counterparts on popular benchmarks. Also, we demonstrate that these learned prompts can be transferred across various datasets, tasks, and compression levels. Hence with this transferability, we can stitch the soft prompt to a newly compressed model to improve the test-time accuracy in an ``in-situ'' way.

cs.CL↗
arXiv2023

Holistic Evaluation of Language Models

Language models (LMs) are becoming the foundation for almost all major language technologies, but their capabilities, limitations, and risks are not well understood. We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models. First, we taxonomize the vast space of potential scenarios (i.e. use cases) and metrics (i.e. desiderata) that are of interest for LMs. Then we select a broad subset based on coverage and feasibility, noting what's missing or underrepresented (e.g. question answering for neglected English dialects, metrics for trustworthiness). Second, we adopt a multi-metric approach: We measure 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency) for each of 16 core scenarios when possible (87.5% of the time). This ensures metrics beyond accuracy don't fall to the wayside, and that trade-offs are clearly exposed. We also perform 7 targeted evaluations, based on 26 targeted scenarios, to analyze specific aspects (e.g. reasoning, disinformation). Third, we conduct a large-scale evaluation of 30 prominent language models (spanning open, limited-access, and closed models) on all 42 scenarios, 21 of which were not previously used in mainstream LM evaluation. Prior to HELM, models on average were evaluated on just 17.9% of the core HELM scenarios, with some prominent models not sharing a single scenario in common. We improve this to 96.0%: now all 30 models have been densely benchmarked on the same core scenarios and metrics under standardized conditions. Our evaluation surfaces 25 top-level findings. For full transparency, we release all raw model prompts and completions publicly for further analysis, as well as a general modular toolkit. We intend for HELM to be a living benchmark for the community, continuously updated with new scenarios, metrics, and models.

cs.CL↗
arXiv2023

Correlated Insulator and Chern Insulators in Pentalayer Rhombohedral Stacked Graphene

Rhombohedral stacked multilayer graphene is an ideal platform to search for correlated electron phenomena, due to its pair of flat bands touching at zero energy and further tunability by an electric field. Furthermore, its valley-dependent Berry phase at zero energy points to possible topological states when the pseudospin symmetry is broken by electron correlation. However, experimental explorations of these opportunities are very limited so far, due to a lack of devices with optimized layer numbers and configurations. Here we present electron transport measurements of hBN-encapsulated pentalayer graphene at down to 100 milli-Kelvin. We observed a correlated insulating state with >MOhm resistance at zero charge density and zero displacement field, where the tight-binding calculation predicts a metallic ground state. By increasing the displacement field, we observed a Chern insulator state with C = -5 and two other states with C = -3 at a low magnetic field of ~1 Tesla. At high displacement fields and charge densities, we observed isospin-polarized quarter- and half-metals. Therefore, rhombohedral-stacked pentalayer graphene is the first graphene system to exhibit two different types of Fermi-surface instabilities: driven by a pair of flat bands touching at zero energy, and by the Stoner mechanism in a single flat band. Our results demonstrate a new direction to explore intertwined electron correlation and topology phenomena in natural graphitic materials without the need of moiré superlattice engineering.

cond-mat.mes-hall↗
arXiv2023

Orbital Multiferroicity in Pentalayer Rhombohedral Graphene

Ferroic orders describe spontaneous polarization of spin, charge, and lattice degrees of freedom in materials. Materials featuring multiple ferroic orders, known as multiferroics, play important roles in multi-functional electrical and magnetic device applications. 2D materials with honeycomb lattices offer exciting opportunities to engineer unconventional multiferroicity, where the ferroic orders are driven purely by the orbital degrees of freedom but not electron spin. These include ferro-valleytricity corresponding to the electron valley and ferro-orbital-magnetism supported by quantum geometric effects. Such orbital multiferroics could offer strong valley-magnetic couplings and large responses to external fields-enabling device applications such as multiple-state memory elements, and electric control of valley and magnetic states. Here we report orbital multiferroicity in pentalayer rhombohedral graphene using low temperature magneto-transport measurements. We observed anomalous Hall signals Rxy with an exceptionally large Hall angle (tanΘH > 0.6) and orbital magnetic hysteresis at hole doping. There are four such states with different valley polarizations and orbital magnetizations, forming a valley-magnetic quartet. By sweeping the gate electric field E we observed a butterfly-shaped hysteresis of Rxy connecting the quartet. This hysteresis indicates a ferro-valleytronic order that couples to the composite field E\cdot B, but not the individual fields. Tuning E would switch each ferroic order independently, and achieve non-volatile switching of them together. Our observations demonstrate a new type of multiferroics and point to electrically tunable ultra-low power valleytronic and magnetic devices.

cond-mat.mes-hall↗
arXiv2023

Nonradiating sources of the biharmonic wave equation

This paper offers an extensive exploration of nonradiating sources for the two- and three-dimensional biharmonic wave equations. Various equivalent characterizations are derived to reveal the nature of a nonradiating source. Additionally, we establish the connection between nonradiating sources in the biharmonic wave equation and those in the Helmholtz equation as well as the modified Helmholtz equation. Several illustrative examples are explicitly constructed to showcase the existence of nonradiating sources. One significant implication of the existence of nonradiating sources is that it undermines the uniqueness of the inverse source problem when utilizing boundary data at a fixed frequency.

math.AP↗
arXiv2023

Motion-Guided Masking for Spatiotemporal Representation Learning

Several recent works have directly extended the image masked autoencoder (MAE) with random masking into video domain, achieving promising results. However, unlike images, both spatial and temporal information are important for video understanding. This suggests that the random masking strategy that is inherited from the image MAE is less effective for video MAE. This motivates the design of a novel masking algorithm that can more efficiently make use of video saliency. Specifically, we propose a motion-guided masking algorithm (MGM) which leverages motion vectors to guide the position of each mask over time. Crucially, these motion-based correspondences can be directly obtained from information stored in the compressed format of the video, which makes our method efficient and scalable. On two challenging large-scale video benchmarks (Kinetics-400 and Something-Something V2), we equip video MAE with our MGM and achieve up to +$1.3\%$ improvement compared to previous state-of-the-art methods. Additionally, our MGM achieves equivalent performance to previous video MAE using up to $66\%$ fewer training epochs. Lastly, we show that MGM generalizes better to downstream transfer learning and domain adaptation tasks on the UCF101, HMDB51, and Diving48 datasets, achieving up to +$4.9\%$ improvement compared to baseline methods.

cs.CV↗
arXiv2023

The Snapshot Problem for the Wave equation

By definition, a wave is a $C^\infty$ solution $u(x,t)$ of the wave equation on $\mathbb R^n$, and a snapshot of the wave $u$ at time $t$ is the function $u_t$ on $\mathbb R^n$ given by $u_t(x)=u(x,t)$. We show that there are infinitely many waves with given $C^\infty$ snapshots $f_0$ and $f_1$ at times $t=0$ and $t=1$ respectively, but that all such waves have the same snapshots at integer times. We present a necessary condition for the uniqueness, and a compatibility condition for the existence, of a wave $u$ to have three given snapshots at three different times, and we show how this compatibility condition leads to the problem of small denominators and Liouville numbers. We extend our results to shifted wave equations on noncompact symmetric spaces. Finally, we consider the two-snapshot problem and corresponding small denominator results for the shifted wave equation on the $n$-sphere.

math.AP↗
arXiv2023

Evidence for Exciton Crystals in a 2D Semiconductor Heterotrilayer

Two-dimensional (2D) transition metal dichalcogenides (TMDC) and their moiré interfaces have been demonstrated for correlated electron states, including Mott insulators and electron/hole crystals commensurate with moiré superlattices. Here we present spectroscopic evidences for ordered bosons - interlayer exciton crystals in a WSe2/MoSe2/WSe2 trilayer, where the enhanced Coulomb interactions over those in heterobilayers have been predicted to result in exciton ordering. While the dipolar interlayer excitons in the heterobilayer may be ordered by the periodic moiré traps, their mutual repulsion results in de-trapping at exciton density n_ex larger than 10^11 cm^-2 to form mobile exciton gases and further to electron-hole plasmas, both accompanied by broadening in photoluminescence (PL) peaks and large increases in mobility. In contrast, ordered interlayer excitons in the trilayer are characterized by negligible mobility and by sharper PL peaks persisting to n_ex approximately 10^12 cm^-2. We present evidences for the predicted quadrupolar exciton crystal and its transitions to dipolar excitons either with increasing n_ex or by an applied electric field. These ordered interlayer excitons may serve as models for the exploration of quantum phase transitions and quantum coherent phenomena.

cond-mat.mes-hall↗
arXiv2023

Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models

The quality of training data impacts the performance of pre-trained large language models (LMs). Given a fixed budget of tokens, we study how to best select data that leads to good downstream model performance across tasks. We develop a new framework based on a simple hypothesis: just as humans acquire interdependent skills in a deliberate order, language models also follow a natural order when learning a set of skills from their training data. If such an order exists, it can be utilized for improved understanding of LMs and for data-efficient training. Using this intuition, our framework formalizes the notion of a skill and of an ordered set of skills in terms of the associated data. First, using both synthetic and real data, we demonstrate that these ordered skill sets exist, and that their existence enables more advanced skills to be learned with less data when we train on their prerequisite skills. Second, using our proposed framework, we introduce an online data sampling algorithm, Skill-It, over mixtures of skills for both continual pre-training and fine-tuning regimes, where the objective is to efficiently learn multiple skills in the former and an individual skill in the latter. On the LEGO synthetic in the continual pre-training setting, Skill-It obtains 36.5 points higher accuracy than random sampling. On the Natural Instructions dataset in the fine-tuning setting, Skill-It reduces the validation loss on the target skill by 13.6% versus training on data associated with the target skill itself. We apply our skills framework on the recent RedPajama dataset to continually pre-train a 3B-parameter LM, achieving higher accuracy on the LM Evaluation Harness with 1B tokens than the baseline approach of sampling uniformly over data sources with 3B tokens.

cs.CL↗
arXiv2023

DeepOIS: Gyroscope-Guided Deep Optical Image Stabilizer Compensation

Mobile captured images can be aligned using their gyroscope sensors. Optical image stabilizer (OIS) terminates this possibility by adjusting the images during the capturing. In this work, we propose a deep network that compensates the motions caused by the OIS, such that the gyroscopes can be used for image alignment on the OIS cameras. To achieve this, first, we record both videos and gyroscopes with an OIS camera as training data. Then, we convert gyroscope readings into motion fields. Second, we propose a Fundamental Mixtures motion model for rolling shutter cameras, where an array of rotations within a frame are extracted as the ground-truth guidance. Third, we train a convolutional neural network with gyroscope motions as input to compensate for the OIS motion. Once finished, the compensation network can be applied for other scenes, where the image alignment is purely based on gyroscopes with no need for images contents, delivering strong robustness. Experiments show that our results are comparable with that of non-OIS cameras, and outperform image-based alignment results with a relatively large margin. Code and dataset are available at https://github.com/lhaippp/DeepOIS

cs.CV↗
arXiv2023

High-Quality Real-Time Rendering Using Subpixel Sampling Reconstruction

Generating high-quality, realistic rendering images for real-time applications generally requires tracing a few samples-per-pixel (spp) and using deep learning-based approaches to denoise the resulting low-spp images. Existing denoising methods have yet to achieve real-time performance at high resolutions due to the physically-based sampling and network inference time costs. In this paper, we propose a novel Monte Carlo sampling strategy to accelerate the sampling process and a corresponding denoiser, subpixel sampling reconstruction (SSR), to obtain high-quality images. Extensive experiments demonstrate that our method significantly outperforms previous approaches in denoising quality and reduces overall time costs, enabling real-time rendering capabilities at 2K resolution.

cs.CV↗
arXiv2023

Patch-based 3D Natural Scene Generation from a Single Example

We target a 3D generative model for general natural scenes that are typically unique and intricate. Lacking the necessary volumes of training data, along with the difficulties of having ad hoc designs in presence of varying scene characteristics, renders existing setups intractable. Inspired by classical patch-based image models, we advocate for synthesizing 3D scenes at the patch level, given a single example. At the core of this work lies important algorithmic designs w.r.t the scene representation and generative patch nearest-neighbor module, that address unique challenges arising from lifting classical 2D patch-based framework to 3D generation. These design choices, on a collective level, contribute to a robust, effective, and efficient model that can generate high-quality general natural scenes with both realistic geometric structure and visual appearance, in large quantities and varieties, as demonstrated upon a variety of exemplar scenes.

cs.GR↗
arXiv2023

Improving Fast Adversarial Training with Prior-Guided Knowledge

Fast adversarial training (FAT) is an efficient method to improve robustness. However, the original FAT suffers from catastrophic overfitting, which dramatically and suddenly reduces robustness after a few training epochs. Although various FAT variants have been proposed to prevent overfitting, they require high training costs. In this paper, we investigate the relationship between adversarial example quality and catastrophic overfitting by comparing the training processes of standard adversarial training and FAT. We find that catastrophic overfitting occurs when the attack success rate of adversarial examples becomes worse. Based on this observation, we propose a positive prior-guided adversarial initialization to prevent overfitting by improving adversarial example quality without extra training costs. This initialization is generated by using high-quality adversarial perturbations from the historical training process. We provide theoretical analysis for the proposed initialization and propose a prior-guided regularization method that boosts the smoothness of the loss function. Additionally, we design a prior-guided ensemble FAT method that averages the different model weights of historical models using different decay rates. Our proposed method, called FGSM-PGK, assembles the prior-guided knowledge, i.e., the prior-guided initialization and model weights, acquired during the historical training process. Evaluations of four datasets demonstrate the superiority of the proposed method.

cs.LG↗
arXiv2023

CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior

Speech-driven 3D facial animation has been widely studied, yet there is still a gap to achieving realism and vividness due to the highly ill-posed nature and scarcity of audio-visual data. Existing works typically formulate the cross-modal mapping into a regression task, which suffers from the regression-to-mean problem leading to over-smoothed facial motions. In this paper, we propose to cast speech-driven facial animation as a code query task in a finite proxy space of the learned codebook, which effectively promotes the vividness of the generated motions by reducing the cross-modal mapping uncertainty. The codebook is learned by self-reconstruction over real facial motions and thus embedded with realistic facial motion priors. Over the discrete motion space, a temporal autoregressive model is employed to sequentially synthesize facial motions from the input speech signal, which guarantees lip-sync as well as plausible facial expressions. We demonstrate that our approach outperforms current state-of-the-art methods both qualitatively and quantitatively. Also, a user study further justifies our superiority in perceptual quality.

cs.CV↗