arXiv subjects

Lei Feng

Publications and source records attributed to Lei Feng.

At least 73 records · Page 4Linked to original sources

arXiv2025

An Invariant Latent Space Perspective on Language Model Inversion

Language model inversion (LMI), i.e., recovering hidden prompts from outputs, emerges as a concrete threat to user privacy and system security. We recast LMI as reusing the LLM's own latent space and propose the Invariant Latent Space Hypothesis (ILSH): (1) diverse outputs from the same source prompt should preserve consistent semantics (source invariance), and (2) input<->output cyclic mappings should be self-consistent within a shared latent space (cyclic invariance). Accordingly, we present Inv^2A, which treats the LLM as an invariant decoder and learns only a lightweight inverse encoder that maps outputs to a denoised pseudo-representation. When multiple outputs are available, they are sparsely concatenated at the representation layer to increase information density. Training proceeds in two stages: contrastive alignment (source invariance) and supervised reinforcement (cyclic invariance). An optional training-free neighborhood search can refine local performance. Across 9 datasets covering user and system prompt scenarios, Inv^2A outperforms baselines by an average of 4.77% BLEU score while reducing dependence on large inverse corpora. Our analysis further shows that prevalent defenses provide limited protection, underscoring the need for stronger strategies. The source code and data involved in this paper can be found in https://github.com/yyy01/Invariant_Attacker.

cs.LG↗

arXiv2025

Exploring Weak-to-Strong Generalization for CLIP-based Classification

Aligning large-scale commercial models with user intent is crucial to preventing harmful outputs. Current methods rely on human supervision but become impractical as model complexity increases. When models surpass human knowledge, providing accurate feedback becomes challenging and inefficient. A novel solution proposed recently is using a weaker model to supervise a stronger model. This concept leverages the ability of weaker models to perform evaluations, thereby reducing the workload on human supervisors. Previous work has shown the effectiveness of weak-to-strong generalization in the context of language-only models. Extending this concept to vision-language models leverages these insights, adapting the proven benefits to a multi-modal context. In our study, we explore weak-to-strong generalization for CLIP-based classification. We propose a method, class prototype learning (CPL), which aims to enhance the classification capabilities of the CLIP model, by learning more representative prototypes for each category. Our findings indicate that, despite using a simple loss function under weak supervision, CPL yields robust improvements in targeted scenarios, particularly when pretraining is limited. Extensive experiments demonstrate that our approach is effective under these settings, achieving a 3.67% improvement over strong baseline methods.

cs.CV↗

arXiv2025

Defending Multimodal Backdoored Models by Repulsive Visual Prompt Tuning

Multimodal contrastive learning models (e.g., CLIP) can learn high-quality representations from large-scale image-text datasets, while they exhibit significant vulnerabilities to backdoor attacks, raising serious safety concerns. In this paper, we reveal that CLIP's vulnerabilities primarily stem from its tendency to encode features beyond in-dataset predictive patterns, compromising its visual feature resistivity to input perturbations. This makes its encoded features highly susceptible to being reshaped by backdoor triggers. To address this challenge, we propose Repulsive Visual Prompt Tuning (RVPT), a novel defense approach that employs deep visual prompt tuning with a specially designed feature-repelling loss. Specifically, RVPT adversarially repels the encoded features from deeper layers while optimizing the standard cross-entropy loss, ensuring that only predictive features in downstream tasks are encoded, thereby enhancing CLIP's visual feature resistivity against input perturbations and mitigating its susceptibility to backdoor attacks. Unlike existing multimodal backdoor defense methods that typically require the availability of poisoned data or involve fine-tuning the entire model, RVPT leverages few-shot downstream clean samples and only tunes a small number of parameters. Empirical results demonstrate that RVPT tunes only 0.27\% of the parameters in CLIP, yet it significantly outperforms state-of-the-art defense methods, reducing the attack success rate from 89.70\% to 2.76\% against the most advanced multimodal attacks on ImageNet and effectively generalizes its defensive capabilities across multiple datasets.

cs.CV↗

arXiv2025

Universal Relations in Long-range Quantum Spin Chains

Understanding the emergence of novel collective behaviors in strongly interacting systems lies at the heart of quantum many-body physics. Valuable insight comes from examining how few-body correlations manifest in many-body systems, embodying the ``from few to many'' philosophy. An intriguing example is the set of universal relations in ultracold atomic gases, which connect a wide range of observables to a single quantity known as the contact. In this Letter, we demonstrate that universal relations manifest in a distinct class of quantum many-body systems, long-range quantum spin chains, which belong to a completely new universality class. Using effective field theory and the operator product expansion, we establish connections between the asymptotic behavior of equal-time spin correlation functions, the dynamical structure factor, and the contact density. The theoretical predictions for equal-time correlators are explicitly verified through numerical simulations based on matrix product states. Our results could be readily tested in state-of-the-art trapped-ion systems.

cond-mat.quant-gas↗

arXiv2025

Enhancing Sample Selection Against Label Noise by Cutting Mislabeled Easy Examples

Sample selection is a prevalent approach in learning with noisy labels, aiming to identify confident samples for training. Although existing sample selection methods have achieved decent results by reducing the noise rate of the selected subset, they often overlook that not all mislabeled examples harm the model's performance equally. In this paper, we demonstrate that mislabeled examples correctly predicted by the model early in the training process are particularly harmful to model performance. We refer to these examples as Mislabeled Easy Examples (MEEs). To address this, we propose Early Cutting, which introduces a recalibration step that employs the model's later training state to re-select the confident subset identified early in training, thereby avoiding misleading confidence from early learning and effectively filtering out MEEs. Experiments on the CIFAR, WebVision, and full ImageNet-1k datasets demonstrate that our method effectively improves sample selection and model performance by reducing MEEs.

cs.LG↗

arXiv2025

Neural Network Reprogrammability: A Unified Theme on Model Reprogramming, Prompt Tuning, and Prompt Instruction

As large-scale pre-trained foundation models continue to expand in size and capability, efficiently adapting them to specific downstream tasks has become increasingly critical. Despite substantial progress, existing adaptation approaches have evolved largely in isolation, without a clear understanding of their interrelationships. This survey introduces neural network reprogrammability as a unifying framework that bridges mainstream model adaptation techniques--model reprogramming, prompt tuning, and prompt instruction--previously fragmented research areas yet converges on a shared principle: repurposing a pre-trained model by manipulating information at the interfaces while keeping the model parameters frozen. These methods exploit neural networks' sensitivity to manipulation on different interfaces, be it through perturbing inputs, inserting tokens into intermediate layers, or providing task-specific examples in context, to redirect model behaviors towards desired outcomes. We then present a taxonomy that categorizes such information manipulation-based adaptation approaches across four key dimensions: manipulation format (fixed or learnable), location (interfaces where manipulations occur), operator (how they are applied), and output alignment requirement (post-processing needed to align outputs with downstream tasks). Notably, this framework applies consistently across data modalities, independent of specific model architectures. Moreover, viewing established techniques like in-context learning and chain-of-thought prompting through this lens reveals both their theoretical connections and practical distinctions. We further analyze remaining technical challenges and ethical considerations, positioning neural network reprogrammability as a fundamental paradigm for efficient model adaptation. We lastly identify promising research directions emerging from this integrative viewpoint.

cs.LG↗

arXiv2025

Rethinking Toxicity Evaluation in Large Language Models: A Multi-Label Perspective

Large language models (LLMs) have achieved impressive results across a range of natural language processing tasks, but their potential to generate harmful content has raised serious safety concerns. Current toxicity detectors primarily rely on single-label benchmarks, which cannot adequately capture the inherently ambiguous and multi-dimensional nature of real-world toxic prompts. This limitation results in biased evaluations, including missed toxic detections and false positives, undermining the reliability of existing detectors. Additionally, gathering comprehensive multi-label annotations across fine-grained toxicity categories is prohibitively costly, further hindering effective evaluation and development. To tackle these issues, we introduce three novel multi-label benchmarks for toxicity detection: \textbf{Q-A-MLL}, \textbf{R-A-MLL}, and \textbf{H-X-MLL}, derived from public toxicity datasets and annotated according to a detailed 15-category taxonomy. We further provide a theoretical proof that, on our released datasets, training with pseudo-labels yields better performance than directly learning from single-label supervision. In addition, we develop a pseudo-label-based toxicity detection method. Extensive experimental results show that our approach significantly surpasses advanced baselines, including GPT-4o and DeepSeek, thus enabling more accurate and reliable evaluation of multi-label toxicity in LLM-generated content.

cs.CL↗

arXiv2025

IRNet: Iterative Refinement Network for Noisy Partial Label Learning

Partial label learning (PLL) is a typical weakly supervised learning, where each sample is associated with a set of candidate labels. Its basic assumption is that the ground-truth label must be in the candidate set, but this assumption may not be satisfied due to the unprofessional judgment of annotators. Therefore, we relax this assumption and focus on a more general task, noisy PLL, where the ground-truth label may not exist in the candidate set. To address this challenging task, we propose a novel framework called ``Iterative Refinement Network (IRNet)'', aiming to purify noisy samples through two key modules (i.e., noisy sample detection and label correction). To achieve better performance, we exploit smoothness constraints to reduce prediction errors in these modules. Through theoretical analysis, we prove that IRNet is able to reduce the noise level of the dataset and eventually approximate the Bayes optimal classifier. Meanwhile, IRNet is a plug-in strategy that can be integrated with existing PLL approaches. Experimental results on multiple benchmark datasets show that IRNet outperforms state-of-the-art approaches on noisy PLL. Our source code is available at: https://github.com/zeroQiaoba/IRNet.

cs.CV↗

arXiv2025

Long-Tailed Out-of-Distribution Detection with Refined Separate Class Learning

Out-of-distribution (OOD) detection is crucial for deploying robust machine learning models. However, when training data follows a long-tailed distribution, the model's ability to accurately detect OOD samples is significantly compromised, due to the confusion between OOD samples and head/tail classes. To distinguish OOD samples from both head and tail classes, the separate class learning (SCL) approach has emerged as a promising solution, which separately conduct head-specific and tail-specific class learning. To this end, we examine the limitations of existing works of SCL and reveal that the OOD detection performance is notably influenced by the use of static scaling temperature value and the presence of uninformative outliers. To mitigate these limitations, we propose a novel approach termed Refined Separate Class Learning (RSCL), which leverages dynamic class-wise temperature adjustment to modulate the temperature parameter for each in-distribution class and informative outlier mining to identify diverse types of outliers based on their affinity with head and tail classes. Extensive experiments demonstrate that RSCL achieves superior OOD detection performance while improving the classification accuracy on in-distribution data.

cs.LG↗

arXiv2025

Mitigating Strategy-Selection Bias in Reasoning for More Effective Test-Time Scaling

Test-time scaling (TTS) has been shown to improve the performance of large language models (LLMs) by sampling and aggregating diverse reasoning paths. However, existing research has overlooked a critical issue: selection bias of reasoning strategies during scaling. Specifically, when generating reasoning processes, LLMs tend to follow certain strategies (e.g., algebraic solutions for math problems) while neglecting other valid alternatives (e.g., geometric solutions), resulting in insufficient exploration of the solution space. To further understand the impact of this bias, we present a theoretical analysis that reveals when it undermines the effectiveness of test-time scaling. Motivated by this theoretical insight, we introduce TTS-Uniform, a framework designed to mitigate the selection bias of reasoning strategies. It (i) identifies potential strategies, (ii) uniformly allocates the sampling budget across them, and (iii) filters out unstable strategies prior to aggregation. Experimental results show that TTS-Uniform significantly enhances scaling effectiveness across multiple mainstream LLMs and benchmark datasets.

cs.AI↗

arXiv2025

Test-Time Multimodal Backdoor Detection by Contrastive Prompting

While multimodal contrastive learning methods (e.g., CLIP) can achieve impressive zero-shot classification performance, recent research has revealed that these methods are vulnerable to backdoor attacks. To defend against backdoor attacks on CLIP, existing defense methods focus on either the pre-training stage or the fine-tuning stage, which would unfortunately cause high computational costs due to numerous parameter updates and are not applicable in black-box settings. In this paper, we provide the first attempt at a computationally efficient backdoor detection method to defend against backdoored CLIP in the \emph{inference} stage. We empirically find that the visual representations of backdoored images are \emph{insensitive} to \emph{benign} and \emph{malignant} changes in class description texts. Motivated by this observation, we propose BDetCLIP, a novel test-time backdoor detection method based on contrastive prompting. Specifically, we first prompt a language model (e.g., GPT-4) to produce class-related description texts (benign) and class-perturbed random texts (malignant) by specially designed instructions. Then, the distribution difference in cosine similarity between images and the two types of class description texts can be used as the criterion to detect backdoor samples. Extensive experiments validate that our proposed BDetCLIP is superior to state-of-the-art backdoor detection methods, in terms of both effectiveness and efficiency. Our codes are publicly available at: https://github.com/Purshow/BDetCLIP.

cs.CV↗

arXiv2025

A fully-programmable integrated photonic processor for both domain-specific and general-purpose computing

A variety of complicated computational scenarios have made unprecedented demands on the computing power and energy efficiency of electronic computing systems, including solving intractable nondeterministic polynomial-time (NP)-complete problems and dealing with large-scale artificial intelligence models. Optical computing emerges as a promising paradigm to meet these challenges, whereas current optical computing architectures have limited versatility. Their applications are usually either constrained to a specialized domain or restricted to general-purpose matrix computation. Here, we implement a fully-programmable integrated photonic processor that can be configured to tackle both specific computational problems and general-purpose matrix computation. We achieve complete end-to-end control of the photonic processor by utilizing a self-developed integrated programmable optoelectronic computing platform. For domain-specific computing, our photonic processor can efficiently solve two kinds of NP-complete problems: subset sum problem (far more than 2^N different instances) and exact cover problem. For general-purpose computation, we experimentally demonstrate high-precision optical dot product and further realize accurate image edge detection and MNIST handwritten image classification task with an accuracy of 97%. Our work enhances the versatility and capability of optical computing architecture, paving the way for its practical application in future high-performance and complex computing scenarios.

physics.optics↗

arXiv2025

What Makes "Good" Distractors for Object Hallucination Evaluation in Large Vision-Language Models?

Large Vision-Language Models (LVLMs), empowered by the success of Large Language Models (LLMs), have achieved impressive performance across domains. Despite the great advances in LVLMs, they still suffer from the unavailable object hallucination issue, which tends to generate objects inconsistent with the image content. The most commonly used Polling-based Object Probing Evaluation (POPE) benchmark evaluates this issue by sampling negative categories according to category-level statistics, \textit{e.g.}, category frequencies and co-occurrence. However, with the continuous advancement of LVLMs, the POPE benchmark has shown diminishing effectiveness in assessing object hallucination, as it employs a simplistic sampling strategy that overlooks image-specific information and restricts distractors to negative object categories only. In this paper, we introduce the Hallucination searching-based Object Probing Evaluation (HOPE) benchmark, aiming to generate the most misleading distractors (\textit{i.e.}, non-existent objects or incorrect image descriptions) that can trigger hallucination in LVLMs, which serves as a means to more rigorously assess their immunity to hallucination. To explore the image-specific information, the content-aware hallucination searching leverages Contrastive Language-Image Pre-Training (CLIP) to approximate the predictive behavior of LVLMs by selecting negative objects with the highest predicted likelihood as distractors. To expand the scope of hallucination assessment, the description-based hallucination searching constructs highly misleading distractors by pairing true objects with false descriptions. Experimental results show that HOPE leads to a precision drop of at least 9\% and up to 23\% across various state-of-the-art LVLMs, significantly outperforming POPE in exposing hallucination vulnerabilities. The code is available at https://github.com/xiemk/HOPE.

cs.CV↗

arXiv2025

Efficient Preparation of Fermionic Superfluids in an Optical Dipole Trap through Reinforcement Learning

We demonstrate a reinforcement learning (RL) based control framework for optimizing evaporative cooling in the preparation of strongly interacting degenerate Fermi gases of Li6. Using a Soft Actor-Critic (SAC) algorithm, the system autonomously explores a high-dimensional parameter space to learn optimal cooling trajectories. Compared to conventional exponential ramps, our method achieves up to 130% improvement in atomic density within a 0.5 second, revealing non-trivial control strategies that balance fast evaporation and thermalization. While our current optimization focuses on the evaporation stage, future integration of other cooling stages, such as grey molasses cooling, could further extend RL to the full preparation pipeline. Our result highlights the promise of RL as a general tool for closed-loop quantum control and automated calibration in complex atomic physics experiments.

cond-mat.quant-gas↗

arXiv2025

Energy-Efficient RSMA-enabled Low-altitude MEC Optimization Via Generative AI-enhanced Deep Reinforcement Learning

The growing demand for low-latency computing in 6G is driving the use of UAV-based low-altitude mobile edge computing (MEC) systems. However, limited spectrum often leads to severe uplink interference among ground terminals (GTs). In this paper, we investigate a rate-splitting multiple access (RSMA)-enabled low-altitude MEC system, where a UAV-based edge server assists multiple GTs in concurrently offloading their tasks over a shared uplink. We formulate a joint optimization problem involving the UAV 3D trajectory, RSMA decoding order, task offloading decisions, and resource allocation, aiming to mitigate multi-user interference and maximize energy efficiency. Given the high dimensionality, non-convex nature, and dynamic characteristics of this optimization problem, we propose a generative AI-enhanced deep reinforcement learning (DRL) framework to solve it efficiently. Specifically, we embed a diffusion model into the actor network to generate high-quality action samples, improving exploration in hybrid action spaces and avoiding local optima. In addition, a priority-based RSMA decoding strategy is designed to facilitate efficient successive interference cancellation with low complexity. Simulation results demonstrate that the proposed method for low-altitude MEC systems outperforms baseline methods, and that integrating GDM with RSMA can achieve significantly improved energy efficiency performance.

cs.NI↗

arXiv2025

Universal Bound States in Long-range Spin Chains with an Impurity

Understanding how quasi-particles interact with impurities is crucial for unveiling novel properties of quantum many-body systems. A prominent example is the enhanced scattering between electrons and magnetic impurities in the low-energy limit, which gives rise to the Kondo effect. In this letter, motivated by recent developments in quantum simulation platforms, we investigate the universal behavior of long-range quantum spin chains with a single local impurity, focusing on systems that conserve the magnon number. Using effective field theory, we show that distinct classes of universal three-magnon states can emerge when the impurity-mediated two-magnon interaction is on resonance. When the long-range coupling decays as $1/r^α$, we find that (i) for $α\in (2,2.89)$, the system exhibits Efimov effects, with the three-body binding energy forming a geometric series $\ln|E^{(n)}_{\text{3-body}}|\sim-n (α-1) π/s_0(α)$, (ii) for $α=2$, the system shows semi-super Efimov effects with $\ln|E^{(n)}_{\text{3-body}}|\sim-(nπ-θ)^2/8$. Our theoretical prediction is validated by the numerical solution of the Skorniakov-Ter-Martirosian equation. Our results could be tested experimentally in the future on quantum simulation platforms.

quant-ph↗

arXiv2025

TVG-SLAM: Robust Gaussian Splatting SLAM with Tri-view Geometric Constraints

Recent advances in 3D Gaussian Splatting (3DGS) have enabled RGB-only SLAM systems to achieve high-fidelity scene representation. However, the heavy reliance of existing systems on photometric rendering loss for camera tracking undermines their robustness, especially in unbounded outdoor environments with severe viewpoint and illumination changes. To address these challenges, we propose TVG-SLAM, a robust RGB-only 3DGS SLAM system that leverages a novel tri-view geometry paradigm to ensure consistent tracking and high-quality mapping. We introduce a dense tri-view matching module that aggregates reliable pairwise correspondences into consistent tri-view matches, forming robust geometric constraints across frames. For tracking, we propose Hybrid Geometric Constraints, which leverage tri-view matches to construct complementary geometric cues alongside photometric loss, ensuring accurate and stable pose estimation even under drastic viewpoint shifts and lighting variations. For mapping, we propose a new probabilistic initialization strategy that encodes geometric uncertainty from tri-view correspondences into newly initialized Gaussians. Additionally, we design a Dynamic Attenuation of Rendering Trust mechanism to mitigate tracking drift caused by mapping latency. Experiments on multiple public outdoor datasets show that our TVG-SLAM outperforms prior RGB-only 3DGS-based SLAM systems. Notably, in the most challenging dataset, our method improves tracking robustness, reducing the average Absolute Trajectory Error (ATE) by 69.0\% while achieving state-of-the-art rendering quality. The implementation of our method will be released as open-source.

cs.CV↗

arXiv2025

Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts

Model reprogramming adapts pretrained models to downstream tasks by modifying only the input and output spaces. Visual reprogramming (VR) is one instance for vision tasks that adds a trainable noise pattern (i.e., a visual prompt) to input images to facilitate downstream classification. The existing VR approaches for CLIP train a single visual prompt using all descriptions of different downstream classes. However, the limited learning capacity may result in (1) a failure to capture diverse aspects of the descriptions (e.g., shape, color, and texture), and (2) a possible bias toward less informative attributes that do not help distinguish between classes. In this paper, we introduce a decoupling-and-reweighting framework. Our decoupled visual prompts (DVP) are optimized using descriptions grouped by explicit causes (DVP-cse) or unsupervised clusters (DVP-cls). Then, we integrate the outputs of these visual prompts with a probabilistic reweighting matrix (PRM) that measures their contributions to each downstream class. Theoretically, DVP lowers the empirical risk bound. Experimentally, DVP outperforms baselines on average across 11 downstream datasets. Notably, the DVP-PRM integration enables insights into how individual visual prompts influence classification decisions, providing a probabilistic framework for understanding reprogramming. Our code is available at https://github.com/tmlr-group/DecoupledVP.

cs.LG↗