arXiv subjects

Junru Wu

Publications and source records attributed to Junru Wu.

At least 19 recordsLinked to original sources

arXiv2026

Extracting the pairing gap from van Hove singularities in rf spectra of the Fermi Hubbard model

We show that van Hove singularities in rf spectra of the 3D attractive Fermi Hubbard model provide a robust route to extracting the pairing gap. Four types of singularities are classified, and their spectral positions are shown to depend solely on the pairing gap $\Delta$ and chemical potential $\mu$ through simple algebraic relations. Measuring two well-resolved singularities therefore determines both parameters without requiring full spectral fitting. Numerical simulations incorporating phenomenological lifetime and scattering broadenings confirm that these features remain visible in both momentum-integrated and $k_z$-integrated spectra, and become more pronounced at stronger coupling where conventional back-bending methods lose sensitivity. At half filling, particle-hole symmetry fixes $\mu$, reducing the extraction to a single singularity measurement. These results establish vHS analysis as a practical spectroscopic diagnostic for pairing in quantum-simulated 3D Fermi Hubbard systems.

cond-mat.quant-gas↗

arXiv2026

Rf spectra and pseudogap in ultracold Fermi gases across the BCS-BEC crossover from pairing fluctuation theory

The pseudogap phenomenon is a hallmark of strongly interacting Fermi systems, from high-temperature superconductors to ultracold atomic gases, yet its precise origin remains debated. Here we calculate the spectral function and rf spectra of ultracold atomic gases across the BCS-BEC crossover to quantitatively investigate the pairing mechanism of the pseudogap. We advance our pairing fluctuation theory by incorporating particle-hole fluctuations, which renormalize the effective interaction in the particle-particle channel. To achieve quantitative accuracy, we employ a full numerical convolution for the pair susceptibility and self-energy, moving beyond previous analytic pseudogap approximations. This convolution approach automatically captures two critical effects: (i) the full spectral broadening of fermions due to finite pair lifetime, and (ii) the previously neglected pair-hole scattering effect, which manifests as a substantial Hartree energy. We calculate the spectral function, and use rf spectral intensity maps and energy distribution curves to determine the quasiparticle dispersion. From these, we extract the pseudogap $\Delta$, Hartree energy, and chemical potential, mapping their evolution across the crossover. Our results show that the pseudogap emerges continuously as the system moves from the BCS regime toward BEC. Furthermore, the pair spectral function reveals that pairs become diffusive at energies above 2$\Delta$, indicating that the pair lifetime is governed by virtual binding and unbinding processes. Our calculations achieve quantitative agreement with recent experiments across the BCS-BEC crossover, including at unitarity, providing strong support for a pairing-based origin of the pseudogap as described by our pairing fluctuation theory.

cond-mat.quant-gas↗

arXiv2026

Spectral study of the pseudogap in unitary Fermi gases

The existence of a pseudogap in unitary Fermi gases has recently been established and measured experimentally [Li et al., Nature 626, 288 (2024)]. This lends strong support for the pairing origin as the mechanism of the pseudogap in Fermi superfluids. Here we present a spectral study of unitary Fermi gases, and show how the data can be understood quantitatively, when compared with theoretically calculated momentum-resolved rf or microwave spectra, and the pseudogap extracted from the spectra. We use an iterative treatment of the fermion self energy and hence the spectral function, beyond previous pseudogap approximation, based on a pairing fluctuation theory that incorporates both particle-particle and particle-hole T matrices, with self-consistent self energy feedback. Our results not only provide a microscopic explanation of the experimental data but also strengthen the support for both the pairing-induced pseudogap physics and the pairing fluctuation theory of Fermi superfluidity.

cond-mat.quant-gas↗

arXiv2025

Effects of particle-hole fluctuations on the superfluid transition in two-dimensional atomic Fermi gases

Proper treatment of the many-body interactions is of paramount importance in our understanding of strongly correlated systems. Here we investigate the effects of particle-hole fluctuations on the Berezinskii-Kosterlitz-Thouless (BKT) transition in two-dimensional Fermi gases throughout the entire BCS-BEC crossover. We include self-consistently in the self energy treatment the entire particle-hole $T$ matrix, which constitutes a renormalization of the bare interaction that appears in the particle-particle scattering $T$ matrix, leading to a screening of the pairing interaction and hence a dramatic reduction of the pairing gap and the transition temperature. The BKT transition temperature $T_\text{BKT}$ is determined by the critical phase space density, for which the pair density and pair mass are determined using a pairing fluctuation theory, which accommodates self-consistently the important self-energy feedback in the treatment of finite-momentum pairing fluctuations. The screening strength varies continuously from its maximum in the BCS limit to essentially zero in BEC limit. In the unitary regime, it leads to an interaction-dependent shift of $T_\text{BKT}$ towards the BEC regime. This shift is crucial in an attempt to explain experimental data quantitatively, which often depends on the interaction strength. Our findings are consistent with available experimental results in the unitary and BEC regimes and with quantum Monte Carlo simulations in the BCS and unitary regimes.

cond-mat.quant-gas↗

arXiv2025

PrivDFS: Private Inference via Distributed Feature Sharing against Data Reconstruction Attacks

In this paper, we introduce PrivDFS, a distributed feature-sharing framework for input-private inference in image classification. A single holistic intermediate representation in split inference gives diffusion-based Data Reconstruction Attacks (DRAs) sufficient signal to reconstruct the input with high fidelity. PrivDFS restructures this vulnerability by fragmenting the representation and processing the fragments independently across a majority-honest set of servers. As a result, each branch observes only an incomplete and reconstruction-insufficient view of the input. To realize this, PrivDFS employs learnable binary masks that partition the intermediate representation into sparse and largely non-overlapping feature shares, each processed by a separate server, while a lightweight fusion module aggregates their predictions on the client. This design preserves full task accuracy when all branches are combined, yet sharply limits the reconstructive power available to any individual server. PrivDFS applies seamlessly to both ResNet-based CNNs and Vision Transformers. Across CIFAR-10/100, CelebA, and ImageNet-1K, PrivDFS induces a pronounced collapse in DRA performance, e.g., on CIFAR-10, PSNR drops from 23.25 -> 12.72 and SSIM from 0.963 -> 0.260, while maintaining accuracy within 1% of non-private split inference. These results establish structural feature partitioning as a practical and architecture-agnostic approach to reducing reconstructive leakage in cloud-based vision inference.

cs.LG↗

arXiv2025

From Curated Data to Scalable Models: Continual Pre-training of Dense and MoE Large Language Models for Tibetan

Large language models (LLMs) have achieved remarkable success across a wide range of natural language processing tasks, yet their performance remains heavily biased toward high-resource languages. Tibetan, despite its cultural significance and large speaker population, is still substantially underrepresented. In this work, we present a comprehensive pipeline for advancing Tibetan language modeling through large-scale data curation and continual pre-training. We construct a 72 GB high-quality Tibetan corpus, the largest to date, and adapt Qwen2.5-7B through balanced multilingual continual pre-training with Tibetan, Chinese, and English, followed by multilingual instruction tuning. To further scale capacity efficiently, we extend the dense model to a 50B-A10B Mixture-of-Experts architecture. Due to the absence of standardized Tibetan benchmarks, we build multiple evaluation datasets via high-quality translation and human verification. Experimental results show that both dense and MoE models consistently outperform existing open-source and Tibetan-focused models of similar scale across diverse tasks. Our work advances Tibetan-centric LLM research and provides transferable insights for extending LLMs to other low-resource languages. We will release the model weights, evaluation benchmarks, and detailed data processing documentation in the follow-up.

cs.CL↗

arXiv2025

Harnessing Pairwise Ranking Prompting Through Sample-Efficient Ranking Distillation

While Pairwise Ranking Prompting (PRP) with Large Language Models (LLMs) is one of the most effective zero-shot document ranking methods, it has a quadratic computational complexity with respect to the number of documents to be ranked, as it requires an enumeration over all possible document pairs. Consequently, the outstanding ranking performance of PRP has remained unreachable for most real-world ranking applications. In this work, we propose to harness the effectiveness of PRP through pairwise distillation. Specifically, we distill a pointwise student ranker from pairwise teacher labels generated by PRP, resulting in an efficient student model that retains the performance of PRP with substantially lower computational costs. Furthermore, we find that the distillation process can be made sample-efficient: with only 2% of pairs, we are able to obtain the same performance as using all pairs for teacher labels. Thus, our novel approach provides a solution to harness the ranking performance of PRP without incurring high computational costs during both distillation and serving.

cs.IR↗

arXiv2025

Adapting Decoder-Based Language Models for Diverse Encoder Downstream Tasks

Decoder-based transformers, while revolutionizing language modeling and scaling to immense sizes, have not completely overtaken encoder-heavy architectures in natural language processing. Specifically, encoder-only models remain dominant in tasks like classification, regression, and ranking. This is primarily due to the inherent structure of decoder-based models, which limits their direct applicability to these tasks. In this paper, we introduce Gemma Encoder, adapting the powerful Gemma decoder model to an encoder architecture, thereby unlocking its potential for a wider range of non-generative applications. To optimize the adaptation from decoder to encoder, we systematically analyze various pooling strategies, attention mechanisms, and hyperparameters (e.g., dropout rate). Furthermore, we benchmark Gemma Encoder against established approaches on the GLUE benchmarks, and MS MARCO ranking benchmark, demonstrating its effectiveness and versatility.

cs.CL↗

arXiv2024

RRM: Robust Reward Model Training Mitigates Reward Hacking

Reward models (RMs) play a pivotal role in aligning large language models (LLMs) with human preferences. However, traditional RM training, which relies on response pairs tied to specific prompts, struggles to disentangle prompt-driven preferences from prompt-independent artifacts, such as response length and format. In this work, we expose a fundamental limitation of current RM training methods, where RMs fail to effectively distinguish between contextual signals and irrelevant artifacts when determining preferences. To address this, we introduce a causal framework that learns preferences independent of these artifacts and propose a novel data augmentation technique designed to eliminate them. Extensive experiments show that our approach successfully filters out undesirable artifacts, yielding a more robust reward model (RRM). Our RRM improves the performance of a pairwise reward model trained on Gemma-2-9b-it, on RewardBench, increasing accuracy from 80.61% to 84.15%. Additionally, we train two DPO policies using both the RM and RRM, demonstrating that the RRM significantly enhances DPO-aligned policies, improving MT-Bench scores from 7.27 to 8.31 and length-controlled win-rates in AlpacaEval-2 from 33.46% to 52.49%.

cs.CL↗

arXiv2024

LAMPO: Large Language Models as Preference Machines for Few-shot Ordinal Classification

We introduce LAMPO, a novel paradigm that leverages Large Language Models (LLMs) for solving few-shot multi-class ordinal classification tasks. Unlike conventional methods, which concatenate all demonstration examples with the test instance and prompt LLMs to produce the pointwise prediction, our framework uses the LLM as a preference machine that makes a relative comparative decision between the test instance and each demonstration. A self-supervised method is then introduced to aggregate these binary comparisons into the final ordinal decision. LAMPO addresses several limitations inherent in previous methods, including context length constraints, ordering biases, and challenges associated with absolute point-wise estimation. Extensive experiments on seven public datasets demonstrate LAMPO's remarkably competitive performance across a diverse spectrum of applications (e.g., movie review analysis and hate speech detection). Notably, in certain applications, the improvement can be substantial, exceeding 20% in an absolute term. Moreover, we believe LAMPO represents an interesting addition to the non-parametric application layered on top of LLMs, as it supports black-box LLMs without necessitating the outputting of LLM's internal states (e.g., embeddings), as seen in previous approaches.

cs.LG↗

arXiv2024

Principled Architecture-aware Scaling of Hyperparameters

Training a high-quality deep neural network requires choosing suitable hyperparameters, which is a non-trivial and expensive process. Current works try to automatically optimize or design principles of hyperparameters, such that they can generalize to diverse unseen scenarios. However, most designs or optimization methods are agnostic to the choice of network structures, and thus largely ignore the impact of neural architectures on hyperparameters. In this work, we precisely characterize the dependence of initializations and maximal learning rates on the network architecture, which includes the network depth, width, convolutional kernel size, and connectivity patterns. By pursuing every parameter to be maximally updated with the same mean squared change in pre-activations, we can generalize our initialization and learning rates across MLPs (multi-layer perception) and CNNs (convolutional neural network) with sophisticated graph topologies. We verify our principles with comprehensive experiments. More importantly, our strategy further sheds light on advancing current benchmarks for architecture design. A fair comparison of AutoML algorithms requires accurate network rankings. However, we demonstrate that network rankings can be easily changed by better training networks in benchmarks with our architecture-aware learning rates and initialization.

cs.LG↗

arXiv2024

LiPO: Listwise Preference Optimization through Learning-to-Rank

Aligning language models (LMs) with curated human feedback is critical to control their behaviors in real-world applications. Several recent policy optimization methods, such as DPO and SLiC, serve as promising alternatives to the traditional Reinforcement Learning from Human Feedback (RLHF) approach. In practice, human feedback often comes in a format of a ranked list over multiple responses to amortize the cost of reading prompt. Multiple responses can also be ranked by reward models or AI feedback. There lacks such a thorough study on directly fitting upon a list of responses. In this work, we formulate the LM alignment as a \textit{listwise} ranking problem and describe the LiPO framework, where the policy can potentially learn more effectively from a ranked list of plausible responses given the prompt. This view draws an explicit connection to Learning-to-Rank (LTR), where most existing preference optimization work can be mapped to existing ranking objectives. Following this connection, we provide an examination of ranking objectives that are not well studied for LM alignment with DPO and SLiC as special cases when list size is two. In particular, we highlight a specific method, LiPO-$\lambda$, which leverages a state-of-the-art \textit{listwise} ranking objective and weights each preference pair in a more advanced manner. We show that LiPO-$\lambda$ can outperform DPO variants and SLiC by a clear margin on several preference alignment tasks with both curated and real rankwise preference data.

cs.CL↗

arXiv2024

Quantum Geometry, Anomalous Scaling, and Strong Pseudogap Superfluidity in a Flat-Band Lieb Lattice

Flat-band systems such as magic-angle twisted bilayer graphene host strong-correlation superconductivity at vanishingly weak coupling, yet how quantum geometry and pairing fluctuations conspire to drive this phenomenon remains an open question. We investigate finite-temperature superfluidity in a quasi-two-dimensional Lieb lattice using a pairing fluctuation theory with a band-uniform attractive interaction $g<0$ that isolates the intrinsic quantum geometric contributions. Quantum geometry significantly amplifies superfluidity; the geometric pair hopping integral surpasses its conventional counterpart, and the geometric superfluid density becomes the dominant in-plane transport component. When the Fermi level enters the flat band, the BCS paradigm breaks down entirely; the pairing gap and $T_\text{c}$ shift from exponential to anomalous power-law scaling $\Delta, T_\text{c} \propto |g|^\nu$ ($\nu>1$), and the superfluid density inherits an unconventional power-law temperature dependence at low temperatures. In the 2D limit ($t_z=0$), the pseudogap at $T_\text{c}$ nearly saturates the zero-temperature gap even at $|g|/t=0.001$, placing the system in a strong-pseudogap regime that would otherwise require unitary or BEC-scale interactions. These findings establish a microscopic mechanism for flat-band enhanced superfluidity and offer testable predictions for ultracold atom experiments.

cond-mat.quant-gas↗

arXiv2023

Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels

Zero-shot text rankers powered by recent LLMs achieve remarkable ranking performance by simply prompting. Existing prompts for pointwise LLM rankers mostly ask the model to choose from binary relevance labels like "Yes" and "No". However, the lack of intermediate relevance label options may cause the LLM to provide noisy or biased answers for documents that are partially relevant to the query. We propose to incorporate fine-grained relevance labels into the prompt for LLM rankers, enabling them to better differentiate among documents with different levels of relevance to the query and thus derive a more accurate ranking. We study two variants of the prompt template, coupled with different numbers of relevance levels. Our experiments on 8 BEIR data sets show that adding fine-grained relevance labels significantly improves the performance of LLM rankers.

cs.IR↗

arXiv2023

Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting

Ranking documents using Large Language Models (LLMs) by directly feeding the query and candidate documents into the prompt is an interesting and practical problem. However, researchers have found it difficult to outperform fine-tuned baseline rankers on benchmark datasets. We analyze pointwise and listwise ranking prompts used by existing methods and argue that off-the-shelf LLMs do not fully understand these challenging ranking formulations. In this paper, we propose to significantly reduce the burden on LLMs by using a new technique called Pairwise Ranking Prompting (PRP). Our results are the first in the literature to achieve state-of-the-art ranking performance on standard benchmarks using moderate-sized open-sourced LLMs. On TREC-DL 2019&2020, PRP based on the Flan-UL2 model with 20B parameters performs favorably with the previous best approach in the literature, which is based on the blackbox commercial GPT-4 that has 50x (estimated) model size, while outperforming other LLM-based solutions, such as InstructGPT which has 175B parameters, by over 10% for all ranking metrics. By using the same prompt template on seven BEIR tasks, PRP outperforms supervised baselines and outperforms the blackbox commercial ChatGPT solution by 4.2% and pointwise LLM-based solutions by more than 10% on average NDCG@10. Furthermore, we propose several variants of PRP to improve efficiency and show that it is possible to achieve competitive results even with linear complexity.

cs.IR↗

arXiv2022

Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics

This work targets designing a principled and unified training-free framework for Neural Architecture Search (NAS), with high performance, low cost, and in-depth interpretation. NAS has been explosively studied to automate the discovery of top-performer neural networks, but suffers from heavy resource consumption and often incurs search bias due to truncated training or approximations. Recent NAS works start to explore indicators that can predict a network's performance without training. However, they either leveraged limited properties of deep networks, or the benefits of their training-free indicators are not applied to more extensive search methods. By rigorous correlation analysis, we present a unified framework to understand and accelerate NAS, by disentangling "TEG" characteristics of searched networks - Trainability, Expressivity, Generalization - all assessed in a training-free manner. The TEG indicators could be scaled up and integrated with various NAS search methods, including both supernet and single-path approaches. Extensive studies validate the effective and efficient guidance from our TEG-NAS framework, leading to both improved search accuracy and over 56% reduction in search time cost. Moreover, we visualize search trajectories on three landscapes of "TEG" characteristics, observing that while a good local minimum is easier to find on NAS-Bench-201 given its simple topology, balancing "TEG" characteristics is much harder on the DARTS search space due to its complex landscape geometry. Our code is available at https://github.com/VITA-Group/TEGNAS.

cs.LG↗

arXiv2022

Scaling Multimodal Pre-Training via Cross-Modality Gradient Harmonization

Self-supervised pre-training recently demonstrates success on large-scale multimodal data, and state-of-the-art contrastive learning methods often enforce the feature consistency from cross-modality inputs, such as video/audio or video/text pairs. Despite its convenience to formulate and leverage in practice, such cross-modality alignment (CMA) is only a weak and noisy supervision, since two modalities can be semantically misaligned even they are temporally aligned. For example, even in the commonly adopted instructional videos, a speaker can sometimes refer to something that is not visually present in the current frame; and the semantic misalignment would only be more unpredictable for the raw videos from the internet. We conjecture that might cause conflicts and biases among modalities, and may hence prohibit CMA from scaling up to training with larger and more heterogeneous data. This paper first verifies our conjecture by observing that, even in the latest VATT pre-training using only instructional videos, there exist strong gradient conflicts between different CMA losses within the same video, audio, text triplet, indicating them as the noisy source of supervision. We then propose to harmonize such gradients, via two techniques: (i) cross-modality gradient realignment: modifying different CMA loss gradients for each sample triplet, so that their gradient directions are more aligned; and (ii) gradient-based curriculum learning: leveraging the gradient conflict information on an indicator of sample noisiness, to develop a curriculum learning strategy to prioritize training on less noisy sample triplets. Applying those techniques to pre-training VATT on the HowTo100M dataset, we consistently improve its performance on different downstream tasks. Moreover, we are able to scale VATT pre-training to more complicated non-narrative Youtube8M dataset to further improve the state-of-the-arts.

cs.CV↗

arXiv2022

Grasping the Arrow of Time from the Singularity: Decoding Micromotion in Low-dimensional Latent Spaces from StyleGAN

The disentanglement of StyleGAN latent space has paved the way for realistic and controllable image editing, but does StyleGAN know anything about temporal motion, as it was only trained on static images? To study the motion features in the latent space of StyleGAN, in this paper, we hypothesize and demonstrate that a series of meaningful, natural, and versatile small, local movements (referred to as "micromotion", such as expression, head movement, and aging effect) can be represented in low-rank spaces extracted from the latent space of a conventionally pre-trained StyleGAN-v2 model for face generation, with the guidance of proper "anchors" in the form of either short text or video clips. Starting from one target face image, with the editing direction decoded from the low-rank space, its micromotion features can be represented as simple as an affine transformation over its latent feature. Perhaps more surprisingly, such micromotion subspace, even learned from just single target face, can be painlessly transferred to other unseen face images, even those from vastly different domains (such as oil painting, cartoon, and sculpture faces). It demonstrates that the local feature geometry corresponding to one type of micromotion is aligned across different face subjects, and hence that StyleGAN-v2 is indeed "secretly" aware of the subject-disentangled feature variations caused by that micromotion. We present various successful examples of applying our low-dimensional micromotion subspace technique to directly and effortlessly manipulate faces, showing high robustness, low computational overhead, and impressive domain transferability. Our codes are available at https://github.com/wuqiuche/micromotion-StyleGAN.

cs.CV↗