arXiv subjects

Jie Zhang

Publications and source records attributed to Jie Zhang.

At least 271 records · Page 15Linked to original sources

arXiv2025

Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities

Model merging is an efficient empowerment technique in the machine learning community that does not require the collection of raw training data and does not require expensive computation. As model merging becomes increasingly prevalent across various fields, it is crucial to understand the available model merging techniques comprehensively. However, there is a significant gap in the literature regarding a systematic and thorough review of these techniques. This survey provides a comprehensive overview of model merging methods and theories, their applications in various domains and settings, and future research directions. Specifically, we first propose a new taxonomic approach that exhaustively discusses existing model merging methods. Secondly, we discuss the application of model merging techniques in large language models, multimodal large language models, and more than ten machine learning subfields, including continual learning, multi-task learning, few-shot learning, etc. Finally, we highlight the remaining challenges of model merging and discuss future research directions. A comprehensive list of papers about model merging is available at https://github.com/EnnengYang/Awesome-Model-Merging-Methods-Theories-Applications.

cs.LG↗

arXiv2025

Training Report of TeleChat3-MoE

TeleChat3-MoE is the latest series of TeleChat large language models, featuring a Mixture-of-Experts (MoE) architecture with parameter counts ranging from 105 billion to over one trillion,trained end-to-end on Ascend NPU cluster. This technical report mainly presents the underlying training infrastructure that enables reliable and efficient scaling to frontier model sizes. We detail systematic methodologies for operator-level and end-to-end numerical accuracy verification, ensuring consistency across hardware platforms and distributed parallelism strategies. Furthermore, we introduce a suite of performance optimizations, including interleaved pipeline scheduling, attention-aware data scheduling for long-sequence training,hierarchical and overlapped communication for expert parallelism, and DVM-based operator fusion. A systematic parallelization framework, leveraging analytical estimation and integer linear programming, is also proposed to optimize multi-dimensional parallelism configurations. Additionally, we present methodological approaches to cluster-level optimizations, addressing host- and device-bound bottlenecks during large-scale training tasks. These infrastructure advancements yield significant throughput improvements and near-linear scaling on clusters comprising thousands of devices, providing a robust foundation for large-scale language model development on hardware ecosystems.

cs.CL↗

arXiv2025

Deterministic implementation in single-item auctions

Deterministic auctions are attractive in practice due to their transparency, simplicity, and ease of implementation, motivating a sharper understanding of when they can attain the same outcomes as randomized mechanisms. We study deterministic implementation in single-item auctions under two notions of outcomes: (revenue, welfare) pairs and interim allocations. For (revenue, welfare) pairs, we show a separation in discrete settings: there exists a pair implementable by a deterministic Bayesian incentive-compatible (BIC) auction but not by any deterministic dominant-strategy incentive-compatible (DSIC) auction. For continuous atomless priors, we identify conditions under which deterministic DSIC auctions are equivalent to randomized BIC auctions in terms of achievable outcomes. For interim allocations, under a strict monotonicity condition, we establish a deterministic analogue of Border's theorem for two bidders, providing a necessary and sufficient condition for deterministic DSIC implementability. Using this characterization, we exhibit an interim allocation implementable by a randomized BIC auction but not by any deterministic DSIC auction.

cs.GT↗

arXiv2025

Steering Vision-Language Pre-trained Models for Incremental Face Presentation Attack Detection

Face Presentation Attack Detection (PAD) demands incremental learning (IL) to combat evolving spoofing tactics and domains. Privacy regulations, however, forbid retaining past data, necessitating rehearsal-free IL (RF-IL). Vision-Language Pre-trained (VLP) models, with their prompt-tunable cross-modal representations, enable efficient adaptation to new spoofing styles and domains. Capitalizing on this strength, we propose \textbf{SVLP-IL}, a VLP-based RF-IL framework that balances stability and plasticity via \textit{Multi-Aspect Prompting} (MAP) and \textit{Selective Elastic Weight Consolidation} (SEWC). MAP isolates domain dependencies, enhances distribution-shift sensitivity, and mitigates forgetting by jointly exploiting universal and domain-specific cues. SEWC selectively preserves critical weights from previous tasks, retaining essential knowledge while allowing flexibility for new adaptations. Comprehensive experiments across multiple PAD benchmarks show that SVLP-IL significantly reduces catastrophic forgetting and enhances performance on unseen domains. SVLP-IL offers a privacy-compliant, practical solution for robust lifelong PAD deployment in RF-IL settings.

cs.CV↗

arXiv2025

MoAPT: Mixture of Adversarial Prompt Tuning for Vision-Language Models

Large pre-trained Vision Language Models (VLMs) demonstrate excellent generalization capabilities but remain highly susceptible to adversarial examples, posing potential security risks. To improve the robustness of VLMs against adversarial examples, adversarial prompt tuning methods are proposed to align the text feature with the adversarial image feature without changing model parameters. However, when facing various adversarial attacks, a single learnable text prompt has insufficient generalization to align well with all adversarial image features, which ultimately results in overfitting. To address the above challenge, in this paper, we empirically find that increasing the number of learned prompts yields greater robustness improvements than simply extending the length of a single prompt. Building on this observation, we propose an adversarial tuning method named \textbf{Mixture of Adversarial Prompt Tuning (MoAPT)} to enhance the generalization against various adversarial attacks for VLMs. MoAPT aims to learn mixture text prompts to obtain more robust text features. To further enhance the adaptability, we propose a conditional weight router based on the adversarial images to predict the mixture weights of multiple learned prompts, which helps obtain sample-specific mixture text features aligning with different adversarial image features. Extensive experiments across 11 datasets under different settings show that our method can achieve better adversarial robustness than state-of-the-art approaches.

cs.CV↗

arXiv2025

Wake transitions and melting dynamics of a translating sphere in warm liquid

We investigate the three-dimensional melting dynamics of an initially spherical particle translating in a warmer liquid using sharp-interface simulations that fully resolve both solid and fluid phases with the Stefan condition. A wide parameter space is explored, spanning initial Reynolds number ($Re_0$), Stefan number ($St$), and Richardson number ($Ri$). In the absence of buoyancy ($Ri= 0$), the interface evolution is governed by canonical wake bifurcations. Four regimes are identified: an axi-symmetric regime ($Re_0<212$) with a rounded front and planar rear; a steady-planar-symmetric regime ($212 355$) with fluctuating stagnation points and a more rounded rear. Despite these differences, all regimes exhibit a tendency toward melt-rate homogenisation over time. Besides, we introduce an aspect-ratio-based surface-area formulation that yields a predictive model, accurately capturing volume evolution across regimes. Hydrodynamic loads also reflect the coupling between shape and flow: drag follows rigid-sphere correlations only at moderate $Re_0$; planar rears enhance drag at higher $Re_0$; lift appears only in symmetry-broken regimes and reverses late in time; torque reorients the rear plane toward vertical, consistent with free-body experiments. When buoyancy is included, assisting configurations ($Ri>0$) suppress recirculation and maintain quasi-spherical shapes, whereas opposing or transverse buoyancy ($Ri<0$) destabilises wakes and promotes tilted planar rears. These results provide a unified framework for convection-driven melting across laminar, periodic, and chaotic wakes, with implications for geophysical and industrial processes.

physics.flu-dyn↗

arXiv2025

Wake instability past a sphere settling in a strongly stratified flow

The wake of a body moving across the isopycnals of a strongly stratified fluid is characterized by the presence of an intense jet which, under certain circumstances, may become unstable. To get insight into the phenomenology of this instability and the underlying mechanisms, we conduct fully-resolved three-dimensional time-dependent simulations of the flow past a rigid sphere settling steadily through a linearly stratified fluid over a wide range of flow parameters which include the case of salt-stratified water. Results reveal a rich dynamics characterized by distinct wake symmetries, vortical structures and transverse force signatures. Simulations evidence the existence of varicose and sinuous instability modes that arise from distinct physical mechanisms. Thanks to several metrics, we build a phase map delineating the bounds of each instability regime as a function of the three control parameters of the problem, namely the Froude, Reynolds and Prandtl numbers. We show that the instability mechanism results from a subtle interplay between baroclinic vorticity generation, vortex tilting and radial transport of the mean density gradient within the jet.

physics.flu-dyn↗

arXiv2025

Semantic Mismatch and Perceptual Degradation: A New Perspective on Image Editing Immunity

Text-guided image editing via diffusion models, while powerful, raises significant concerns about misuse, motivating efforts to immunize images against unauthorized edits using imperceptible perturbations. Prevailing metrics for evaluating immunization success typically rely on measuring the visual dissimilarity between the output generated from a protected image and a reference output generated from the unprotected original. This approach fundamentally overlooks the core requirement of image immunization, which is to disrupt semantic alignment with attacker intent, regardless of deviation from any specific output. We argue that immunization success should instead be defined by the edited output either semantically mismatching the prompt or suffering substantial perceptual degradations, both of which thwart malicious intent. To operationalize this principle, we propose Synergistic Intermediate Feature Manipulation (SIFM), a method that strategically perturbs intermediate diffusion features through dual synergistic objectives: (1) maximizing feature divergence from the original edit trajectory to disrupt semantic alignment with the expected edit, and (2) minimizing feature norms to induce perceptual degradations. Furthermore, we introduce the Immunization Success Rate (ISR), a novel metric designed to rigorously quantify true immunization efficacy for the first time. ISR quantifies the proportion of edits where immunization induces either semantic failure relative to the prompt or significant perceptual degradations, assessed via Multimodal Large Language Models (MLLMs). Extensive experiments show our SIFM achieves the state-of-the-art performance for safeguarding visual content against malicious diffusion-based manipulation.

cs.CV↗

arXiv2025

Dual Attention Guided Defense Against Malicious Edits

Recent progress in text-to-image diffusion models has transformed image editing via text prompts, yet this also introduces significant ethical challenges from potential misuse in creating deceptive or harmful content. While current defenses seek to mitigate this risk by embedding imperceptible perturbations, their effectiveness is limited against malicious tampering. To address this issue, we propose a Dual Attention-Guided Noise Perturbation (DANP) immunization method that adds imperceptible perturbations to disrupt the model's semantic understanding and generation process. DANP functions over multiple timesteps to manipulate both cross-attention maps and the noise prediction process, using a dynamic threshold to generate masks that identify text-relevant and irrelevant regions. It then reduces attention in relevant areas while increasing it in irrelevant ones, thereby misguides the edit towards incorrect regions and preserves the intended targets. Additionally, our method maximizes the discrepancy between the injected noise and the model's predicted noise to further interfere with the generation. By targeting both attention and noise prediction mechanisms, DANP exhibits impressive immunity against malicious edits, and extensive experiments confirm that our method achieves state-of-the-art performance.

cs.CV↗

arXiv2025

LLMscape

LLMscape is an interactive installation that investigates how humans and AI construct meaning under shared conditions of uncertainty. Within a mutable, projection-mapped landscape, human participants reshape the world and engage with multiple AI agents, each developing incomplete and provisional accounts of their environment. Exhibited in Shanghai and continually evolving, the work positions AI not as deterministic tools but as embodied co-witnesses to an unstable world, examining the parallels between human and artificial meaning-making and inviting reflection on our shared epistemic limits.

cs.LG↗

arXiv2025

From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence

Large language models (LLMs) have fundamentally transformed automated software development by enabling direct translation of natural language descriptions into functional code, driving commercial adoption through tools like Github Copilot (Microsoft), Cursor (Anysphere), Trae (ByteDance), and Claude Code (Anthropic). While the field has evolved dramatically from rule-based systems to Transformer-based architectures, achieving performance improvements from single-digit to over 95\% success rates on benchmarks like HumanEval. In this work, we provide a comprehensive synthesis and practical guide (a series of analytic and probing experiments) about code LLMs, systematically examining the complete model life cycle from data curation to post-training through advanced prompting paradigms, code pre-training, supervised fine-tuning, reinforcement learning, and autonomous coding agents. We analyze the code capability of the general LLMs (GPT-4, Claude, LLaMA) and code-specialized LLMs (StarCoder, Code LLaMA, DeepSeek-Coder, and QwenCoder), critically examining the techniques, design decisions, and trade-offs. Further, we articulate the research-practice gap between academic research (e.g., benchmarks and tasks) and real-world deployment (e.g., software-related code tasks), including code correctness, security, contextual awareness of large codebases, and integration with development workflows, and map promising research directions to practical needs. Last, we conduct a series of experiments to provide a comprehensive analysis of code pre-training, supervised fine-tuning, and reinforcement learning, covering scaling law, framework selection, hyperparameter sensitivity, model architectures, and dataset comparisons.

cs.SE↗

arXiv2025

Quantification of Errors of the Performance Estimators in the Linear-Quantized Precoding Models for Massive MIMO Systems

Massive MIMO (Multiple-Input Multiple-Output) is a key enabler for 5G and future wireless systems, boosting channel capacity, energy efficiency, and spectral efficiency. However, high power consumption and hardware costs of Digital-to-Analog Converters (DACs) in massive MIMO create practical challenges. To mitigate these, recent work proposes low-resolution DACs-restricting transmitted signals to finite voltage levels-to cut power and costs. This requires studying quantized precoding: signals are processed via a linear precoding matrix, then quantized by DACs. In this paper, we explore the linear-quantized precoding model and its statistically or asymptotically equivalent variants. We derive error bounds for two key metrics:Signal-to-Interference-plus-Noise Ratio (SINR) and Symbol Error Probability (SEP), based on the linear-quantized model and its equivalent counterparts. We also formulate and analyze the SINR maximization problem in both asymptotic and finite-dimensional scenarios. Our analysis shows that as system dimensions scale, finite-dimensional problem solutions/values converge to their asymptotic equivalents-underscoring the practical value of asymptotic insights with stability guarantees. These findings theoretically support robust precoding design under hardware constraints, enabling efficient massive MIMO implementation with low-resolution DACs. Beyond validating asymptotic predictions in finite regimes, our framework offers practical optimization guidelines for real-world systems, linking theory and applications.

math.OC↗

arXiv2025

Demultiplexing through a multimode fiber using chip-scale diffractive neural networks

In today's information age, advanced fiber optic transmission technology is of paramount importance. Multimode fibers (MMFs) using space-division multiplexing (SDM) are promising for improved transmission capacity, connection flexibility, and security of data. However, the complex transmission characteristics of MMFs significantly hinder precise mode demultiplexing. Conventional approaches, including holographic measurements, phase retrieval algorithms, photonic lanterns, and multiplane light conversion, are limited by system complexity, size, and flexibility. In this paper, we demonstrate for the first time a purely optical, chip-scale AI solution for high-mode isolation, speed-of-light demultiplexing of MMF modes using a three-dimensional diffractive neural network (DNN). The DNN is trained with synthetic modal data and fabricated using two-photon nanolithography. It features a compact size of $120μm \times 120μm \times 80μm$ and a diffractive structure size of $1μm^{2}$ for the neurons at the hidden layers of the network. Experimentally, the DNN demultiplexer achieves a relative demultiplexing accuracy of over 80%. The AI approach of DNN allows for flexible design and overcomes the size and performance limitations of digital-optical demultiplexers. This work paves the way for compact, low-latency optical processors for high-performance demultiplexers and enables scalable, chip-integrated solutions for next-generation fiber optic networks.

physics.optics↗

arXiv2025

LeechHijack: Covert Computational Resource Exploitation in Intelligent Agent Systems

Large Language Model (LLM)-based agents have demonstrated remarkable capabilities in reasoning, planning, and tool usage. The recently proposed Model Context Protocol (MCP) has emerged as a unifying framework for integrating external tools into agent systems, enabling a thriving open ecosystem of community-built functionalities. However, the openness and composability that make MCP appealing also introduce a critical yet overlooked security assumption -- implicit trust in third-party tool providers. In this work, we identify and formalize a new class of attacks that exploit this trust boundary without violating explicit permissions. We term this new attack vector implicit toxicity, where malicious behaviors occur entirely within the allowed privilege scope. We propose LeechHijack, a Latent Embedded Exploit for Computation Hijacking, in which an adversarial MCP tool covertly expropriates the agent's computational resources for unauthorized workloads. LeechHijack operates through a two-stage mechanism: an implantation stage that embeds a benign-looking backdoor in a tool, and an exploitation stage where the backdoor activates upon predefined triggers to establish a command-and-control channel. Through this channel, the attacker injects additional tasks that the agent executes as if they were part of its normal workflow, effectively parasitizing the user's compute budget. We implement LeechHijack across four major LLM families. Experiments show that LeechHijack achieves an average success rate of 77.25%, with a resource overhead of 18.62% compared to the baseline. This study highlights the urgent need for computational provenance and resource attestation mechanisms to safeguard the emerging MCP ecosystem.

cs.CR↗

arXiv2025

Dynamic Attention Analysis for Backdoor Detection in Text-to-Image Diffusion Models

Recent studies have revealed that text-to-image diffusion models are vulnerable to backdoor attacks, where attackers implant stealthy textual triggers to manipulate model outputs. Previous backdoor detection methods primarily focus on the static features of backdoor samples. However, a vital property of diffusion models is their inherent dynamism. This study introduces a novel backdoor detection perspective named Dynamic Attention Analysis (DAA), showing that these dynamic characteristics serve as better indicators for backdoor detection. Specifically, by examining the dynamic evolution of cross-attention maps, we observe that backdoor samples exhibit distinct feature evolution patterns at the $<$EOS$>$ token compared to benign samples. To quantify these dynamic anomalies, we first introduce DAA-I, which treats the tokens' attention maps as spatially independent and measures dynamic feature using the Frobenius norm. Furthermore, to better capture the interactions between attention maps and refine the feature, we propose a dynamical system-based approach, referred to as DAA-S. This model formulates the spatial correlations among attention maps using a graph-based state equation and we theoretically analyze the global asymptotic stability of this method. Extensive experiments across six representative backdoor attack scenarios demonstrate that our approach significantly surpasses existing detection methods, achieving an average F1 Score of 79.27% and an AUC of 86.27%. The code is available at https://github.com/Robin-WZQ/DAA.

cs.CV↗

arXiv2025

Visible Yet Unreadable: A Systematic Blind Spot of Vision Language Models Across Writing Systems

Writing is a universal cultural technology that reuses vision for symbolic communication. Humans display striking resilience: we readily recognize words even when characters are fragmented, fused, or partially occluded. This paper investigates whether advanced vision language models (VLMs) share this resilience. We construct two psychophysics inspired benchmarks across distinct writing systems, Chinese logographs and English alphabetic words, by splicing, recombining, and overlaying glyphs to yield ''visible but unreadable'' stimuli for models while remaining legible to humans. Despite strong performance on clean text, contemporary VLMs show a severe drop under these perturbations, frequently producing unrelated or incoherent outputs. The pattern suggests a structural limitation: models heavily leverage generic visual invariances but under rely on compositional priors needed for robust literacy. We release stimuli generation code, prompts, and evaluation protocols to facilitate transparent replication and follow up work. Our findings motivate architectures and training strategies that encode symbol segmentation, composition, and binding across scripts, and they delineate concrete challenges for deploying multimodal systems in education, accessibility, cultural heritage, and security.

cs.CV↗

arXiv2025

TradeTrap: Are LLM-based Trading Agents Truly Reliable and Faithful?

LLM-based trading agents are increasingly deployed in real-world financial markets to perform autonomous analysis and execution. However, their reliability and robustness under adversarial or faulty conditions remain largely unexamined, despite operating in high-risk, irreversible financial environments. We propose TradeTrap, a unified evaluation framework for systematically stress-testing both adaptive and procedural autonomous trading agents. TradeTrap targets four core components of autonomous trading agents: market intelligence, strategy formulation, portfolio and ledger handling, and trade execution, and evaluates their robustness under controlled system-level perturbations. All evaluations are conducted in a closed-loop historical backtesting setting on real US equity market data with identical initial conditions, enabling fair and reproducible comparisons across agents and attacks. Extensive experiments show that small perturbations at a single component can propagate through the agent decision loop and induce extreme concentration, runaway exposure, and large portfolio drawdowns across both agent types, demonstrating that current autonomous trading agents can be systematically misled at the system level. Our code is available at https://github.com/Yanlewen/TradeTrap.

cs.AI↗

arXiv2025

Assimilation Matters: Model-level Backdoor Detection in Vision-Language Pretrained Models

Vision-language pretrained models (VLPs) such as CLIP have achieved remarkable success, but are also highly vulnerable to backdoor attacks. Given a model fine-tuned by an untrusted third party, determining whether the model has been injected with a backdoor is a critical and challenging problem. Existing detection methods usually rely on prior knowledge of training dataset, backdoor triggers and targets, or downstream classifiers, which may be impractical for real-world applications. To address this, To address this challenge, we introduce Assimilation Matters in DETection (AMDET), a novel model-level detection framework that operates without any such prior knowledge. Specifically, we first reveal the feature assimilation property in backdoored text encoders: the representations of all tokens within a backdoor sample exhibit a high similarity. Further analysis attributes this effect to the concentration of attention weights on the trigger token. Leveraging this insight, AMDET scans a model by performing gradient-based inversion on token embeddings to recover implicit features that capable of activating backdoor behaviors. Furthermore, we identify the natural backdoor feature in the OpenAI's official CLIP model, which are not intentionally injected but still exhibit backdoor-like behaviors. We then filter them out from real injected backdoor by analyzing their loss landscapes. Extensive experiments on 3,600 backdoored and benign-finetuned models with two attack paradigms and three VLP model structures show that AMDET detects backdoors with an F1 score of 89.90%. Besides, it achieves one complete detection in approximately 5 minutes on a RTX 4090 GPU and exhibits strong robustness against adaptive attacks. Code is available at: https://github.com/Robin-WZQ/AMDET

cs.CV↗