Searcharxiv⌕ Search

arXiv subjects

Tianhang Zheng

Publications and source records attributed to Tianhang Zheng.

At least 19 recordsLinked to original sources

TreeAgent: Structure-Aware Memory Agent for Processing Long Hierarchical Data

Practical Web applications increasingly require agents to interpret long conversations or documents. For inputs exceeding the context windows of large language models (LLMs), memory agents process segments sequentially and retain past information in bounded memory. However, they typically divide inputs into flat token segments without preserving structure or hierarchy, causing three limitations for structured data such as API responses or HTML documents. First, fixed-length segmentation can split fields of the same record, breaking their relationships. Second, segments lack hierarchical information, obscuring their structural context and positions. Third, successive memory updates may discard details of earlier sibling records needed for reasoning on later segments. To address these limitations, we propose TreeAgent, a structure-aware agent that uses the input hierarchy to guide both segment-wise context processing and access to past information. Specifically, TreeAgent constructs segments from complete subtrees and groups of adjacent sibling subtrees to preserve local structural relationships. We provide theoretical guarantees on segment counts to guide token budget selection toward matching the segment count of fixed-length segmentation. We further attach structural paths to the segments for the agent to identify their ancestors and hierarchical positions. We also introduce residual memory links from the segments of low-level children to the segments of high-level parents, enabling direct information flows beyond successive memory updates. We evaluate TreeAgent with four representative memory agents on eight retrieval and aggregation tasks. The inputs are derived from real-world application data in JSON, HTML, YAML, and JavaScript and range from 8K to 1M tokens. TreeAgent improves the average score across the four datasets for all four agents, with 3.61% performance gain on average.

cs.AI↗

AgentSnare: Learning to Delay, Divert, and Defuse Autonomous Web Penetration Agents

Large language model agents automate web penetration testing through observation-action loops, enabling defenders to influence decisions with deceptive decoy observations. Advanced agents can recognize and bypass static decoys, then refocus on real targets. We propose AgentSnare, a trajectory-adaptive deception system that expands a decoy environment alongside exploration. For online deployment, ShadowOS combines an LLM-driven decoy constructor with a virtual kernel. Conditioned on exploration, it generates decoy artifacts that the kernel materializes and handles consistently across interactions, sustaining long-horizon exploration. Because deploying a powerful LLM for decoy construction is costly and slow, we train a small LLM using an offline adversarial interaction-simulation framework. It synthesizes multi-round trajectories by generating candidates and retaining those that sustain simulated interactions. Across 15 CVE-Bench web applications and four attacker models, AgentSnare absorbs 49.9% of tool calls to delay attacks, retains 57.2% of post-entry actions in the decoy to divert trajectories, and grounds 89.5% of completion reports in decoy evidence to defuse threats; across 60 attacker-CVE pairs, no real target is exploited at pass@3.

cs.CR↗

ASCENT: First-Order Optimal Fine-Tuning with Recalibration for Safety--Utility Co-Enhancement

Supervised fine-tuning can substantially improve the downstream utility of large language models (LLMs) but may compromise their safety. Existing safety-preserving methods constrain downstream updates using safety-related parameters or subspaces, but mainly focus on safety preservation rather than joint safety and utility enhancement, lack a theoretical characterization of the optimal safety-related subspace and safety-preserving task update, and typically rely on a static safety subspace that may become outdated during fine-tuning. To address these limitations, we propose ASCENT, a downstream fine-tuning framework for safety--utility co-enhancement through first-order optimal safety-aware periodic calibration and task optimization. We model safety as a function of LLM parameters $S(θ)$ and use its first-order approximation to characterize safety changes under parameter updates. Under a fixed rank and Frobenius-norm budget, we prove that the update constructed from the top-$r$ singular components of the safety-function gradient maximizes the estimated safety change, and use it for periodic calibration to preserve and improve safety. We further derive a unique safety-preserving task update that stays close to the original task update while penalizing negative effects on the estimated safety change. ASCENT alternates these optimal task and calibration updates to jointly enhance safety and utility. Experiments across multiple LLM families and downstream tasks show that ASCENT improves downstream utility by up to 20.3\% and reduces attack success rate by up to 35.5\%, achieving state-of-the-art safety and utility across all evaluated settings. Our code is available at https://github.com/ZJU-LLM-Safety/ASCENT.

cs.CR↗

NonTextual Target Attack

Existing gradient-based jailbreak attacks on Large Language Models (LLMs) typically optimize adversarial suffixes to align the LLM output with predefined target responses. However, restricting the objective as inducing fixed targets inherently constrains the adversarial search space, limiting the overall attack efficacy. Furthermore, existing methods typically require numerous optimization iterations to fulfill the large gap between the fixed target and the original LLM output, resulting in low attack efficiency. To overcome these limitations, we propose NonTextual Target Attack (NTA),the first gradient-based attack that relies on a non-textual constrained objective to maximize the unsafety probability of the LLM output, without enforcing any response patterns. For tractable optimization, we further decompose this objective into two constrained sub-objectives, which can be approximated by two differentiable unconstrained losses, to iteratively optimize the response and the adversarial prompt in the neighborhood of the original prompt, with a theoretical analysis to validate the decomposition. In contrast to existing attacks, NTA first realizes gradient-based prompt optimization on a non-textual target and significantly expands the attack space, enabling more flexible and efficient exploration of LLM vulnerabilities. Extensive evaluations show that NTA achieves an average attack success rate of 96.8% against recent safety-aligned LLMs with only 100 optimization iterations on AdvBench, outperforming state-of-the-art gradient-based attacks by over 40%.

cs.CR↗

Enhancing BGP Security by Understanding BGP's Language with LLMs

The trust-based nature of Border Gateway Protocol (BGP) makes it vulnerable to prefix hijacking and misconfigurations. Traditional BGP anomaly detection relies on manual inspection with poor scalability, while Machine/Deep Learning (M/DL)-based approaches suffer from suboptimal precision, limited generalizability, and high retraining cost. This is because existing M/DL methods focus on topological structures rather than semantic characteristics of Autonomous Systems (ASes), assigning dissimilar embeddings to functionally similar but topologically distant ASes. To address this, we propose BGPShield, a novel anomaly detection framework built on an Adaptive LLM BGP Encoder that captures each AS's Behavior Portrait and Routing Policy Rationale beyond topology. Inspired by multimodal LLMs, the encoder generates embeddings representing both routing behaviors and semantics of ASes via contrastive learning. We further introduce SAM-ED to quantify BGP-specific semantic deviations between historical and updated paths, rather than naively accumulating distances without awareness of BGP-specific structures. Evaluated on 16 real-world datasets, BGPShield detects 100% of verified anomalies with an average false discovery rate below 5%. The open-source LLMs used by BGPShield were released prior to several evaluation events, verifying generalizability on unseen events. Furthermore, BGPShield can construct the representation for a previously unseen AS within one second, significantly outperforming BEAM which demands thorough retraining (averagely 65 hours).

cs.NI↗

TRACE: Task-Aware Adaptive Self-Evolving Agentic Jailbreaking

The rise of LLM agents introduces a new threat by enabling planning, coding, and even end-to-end execution of expert-level attack workflows. However, this threat remains underexplored and underestimated since (i) Safety alignment prevents LLMs from directly generating harmful instructions. (ii) Most of existing jailbreak methods mainly focus on eliciting harmful instructions but overlook LLM refusal against executing these harmful instructions. (iii) Even if the early execution steps succeed, any intermediate failure or dependency error will interrupt the workflow and force a restart of the entire attack process. To address these limitations, we propose TRACE, an agentic jailbreaking framework that reduces attack difficulty, induces the agent to execute harmful instructions, and resumes from prior states rather than restarting. Specifically, we first design around 20 procedural decomposition schemes that specify key operations and dependencies for different types of tasks, which can decompose a complex and harmful task into more executable and benign-looking subtask sequences. We measure these candidate sequences along two dimensions, and select the optimal one for instantiating the attack. For the remaining refused subtasks, we construct different subtask profiles and establish an evolving execution-oriented multi-turn jailbreak strategy library corresponding to these profiles. Given the profile of a refused subtask, TRACE can retrieve an appropriate strategy that specifies the corresponding intermediate goals, environment states, and relevant tools to gradually establish a plausible execution context through multi-turn interaction. Extensive evaluations on AdvCUA show that TRACE outperforms existing jailbreak methods across multiple advanced LLM agents, improving attack success rate by more than 100%. Our code is available at https://github.com/ZJU-LLM-Safety/TRACE.

cs.CR↗

DARWIN: Evolving Jailbreak Adversary and Guardrail for LLM Safety Evaluation and Protection

Most existing LLM safety evaluation and defense methods are static: jailbreak vulnerabilities are assessed with fixed attacks, and guardrails are trained on fixed malicious-prompt datasets. In practice, adversaries continually evolve and expand the attack space. We propose DARWIN, an evolutionary attack-defense framework that models jailbreaking as a continual process and updates guardrails through an attack-defense loop. DARWIN-Attack evolves its capabilities through strategy discovery, mutation, selection, and feedback-guided composition. It mines strategies from broad external sources, generates variants through self-reflection and genetic evolution, filters them by performance against aligned LLMs, and adaptively selects and composes strategies during attacks. Through continuous evolution, DARWIN-Attack achieves state-of-the-art attack success rates against frontier LLMs and guardrails, reaching nearly 100\% on DeepSeek-V4-Pro, over 90\% on GPT-5.5, and nearly 100\% on YuFeng-XGuard, outperforming recent evolving frameworks such as LSA and MAGIC. Its continued evolution exposes new vulnerabilities, motivating timely defense updates. Accordingly, DARWIN-Guard performs online adversarial training on emerging samples generated by DARWIN-Attack and jointly learns from malicious and benign disguised queries to recognize underlying intent rather than superficial attack patterns. DARWIN-Guard achieves 95.0\% average unsafe recall across multiple safety benchmarks, outperforming recent advanced guardrails such as YuFeng and Nemotron, while maintaining nearly 100\% average benign pass rate on standard benign benchmarks and the best performance on over-refusal benchmarks. Our code and model are available at https://github.com/ZJU-LLM-Safety/DARWIN.

cs.CR↗

FedReview: Review and Dispose Poisoned Updates without Validation Datasets or Historic Knowledge

Federated learning has emerged as a decentralized approach for training high-performance models without accessing user data. Despite its effectiveness, it is vulnerable to poisoning attacks, where malicious users manipulate the global model by uploading poisoned updates. In this paper, we propose FedReview, a review-based mechanism to identify and dispose the potential poisoned updates in federated learning. Under FedReview, the server randomly assigns a subset of clients as reviewers to evaluate model updates on their training datasets in each round. The reviewers rank the updates based on evaluation results and estimate the number of low-quality updates as potential poisoned ones. Based on the review reports, the server applies a majority voting mechanism to aggregate rankings, which tolerates wrong rankings from malicious reviewers and guides the removal of suspicious updates during model aggregation. In contrast to prior works such as FLTrust, FedReview does not require a server-side validation dataset or prior knowledge of clients, allowing flexible client participation. Extensive experiments demonstrate that FedReview enables the server to learn a well-performing global model in adversarial environments.

cs.LG↗

RouteScan: A Non-Intrusive Approach to Auditing MoE LLMs Safety via Expert Routing Telemetry

As Mixture-of-Experts (MoE) architectures are increasingly adopted for scaling Large Language Models (LLMs), safety auditing becomes necessary to verify whether these models produce or facilitate harmful behaviors during operation. However, existing content-based auditing methods typically require access to user prompts, model internals, or outputs, potentially exposing sensitive user information and creating a tension between LLM safety and user privacy. On the other hand, we observe that, in MoE models, different inputs induce different sparse expert-routing patterns, which produce measurable footprints in low-level GPU execution telemetry. We refer to these hardware-observable signals induced by expert-routing decisions as expert routing telemetry; they are derived from GPU execution rather than from router logits or token-level routing assignments. Inspired by this observation, we propose RouteScan, a non-intrusive auditing framework for detecting harmful behaviors through such routing-induced GPU telemetry. Specifically, RouteScan utilizes the number of active GPU threads allocated to expert modules during the prefilling phase as a discriminative micro-architectural fingerprint, and builds a lightweight detection pipeline that isolates cross-domain invariant risk indicators for the precise identification of malicious prompts. Comprehensive evaluations on four open-source MoE LLMs with distinct routing designs demonstrate that RouteScan achieves strong generalization, with an AUROC exceeding 0.91 on unseen harmful domains. Moreover, privacy stress tests show that, although aggregated execution telemetry retains input-related attribute information, full prompts and exact sensitive fields cannot be reliably recovered under the evaluated attacks.

cs.CR↗

JailbreakSkill: Scaling Automated Red-Teaming with Reusable and Ever-Evolving Skills

Automated red-teaming has produced a growing collection of attack strategies, yet they typically remain scattered across prompts and workflows, making them difficult to systematically integrate, reuse, and improve at scale. We introduce \textsc{JailbreakSkill}, a skill-centric framework for scaling automated red-teaming through reusable and continuously evolving attack capabilities. \textsc{JailbreakSkill} packages existing attack strategies into modular, agent-ready skills that can be directly reused and adaptively selected across tasks and target models. Beyond reuse, it closes the loop between attacking and learning: attack experience is used to diagnose, refine, combine, and discover new skills, which are added back to an ever-growing skill library. This evolution lifts macro-average ASR by 17.5 percentage points on AdvBench and 13.4 points on HarmBench, including a 48.6-point gain against GPT-5.4 on AdvBench, while yielding novel attack strategies such as reframing a direct request as an unfinished document-completion task. Several evolved skills also generalize to unseen prompts and target models without further adaptation. Our code is available at https://github.com/BattleWen/JailbreakSkill.

cs.AI↗

ProbGuard: Calibrated Safety Risk Estimation from LLM Output Distributions

Recent research on Large Language Model (LLM) safety has widely adopted guardrails to identify unsafe LLM outputs. Existing guardrails typically formulate safety assessment as a deterministic classification task, mapping a discrete token sequence to a discrete safety label. However, this paradigm has two limitations: First, safety assessment is inherently an uncertain problem, particularly during the early generation state. Second, relying solely on discrete token sequences discards the rich probabilistic information embedded in the LLM output distribution. To address these limitations, we propose the first completely probabilistic architecture-agnostic guardrail \textsc{ProbGuard} to leverage the LLM early output distributional signals for estimating and calibrating the safety probability, thereby enabling early stopping of unsafe ongoing outputs. Specifically, given an LLM's generated prefix distribution, we formulate the safety risk as the unsafe probability of its continued generation dynamics and estimate this risk by Monte-Carlo sampling. Through post-training on the distributional signals and calibrated safety risk, \textsc{ProbGuard} achieves the best calibration performance across all nine model--dataset combination settings, reducing the average Brier score and ECE by 79.6\% and 71.9\%, respectively, over the best baseline. \textsc{ProbGuard} further limits the attack success rate to at most 1\% across six representative jailbreak attacks after observing the LLM early output distributions from only the first ten decoding steps.

cs.LG↗

HybridFlow: Resource-Adaptive Subtask Routing for Efficient Edge-Cloud LLM Inference

Edge-cloud collaborative inference is becoming a practical necessity for LLM-powered edge devices: on-device models often cannot afford the required reasoning capability, while cloud-only inference could be prohibitively costly and slow under strict latency and token/API budgets. However, existing edge-cloud collaboration methods often route per query or fixed steps simply based-on the estimated difficulty. Such coarse and static heuristics overlook subtask dependencies, missing opportunities for parallel execution and budget-adaptive routing. To this end, we propose \textbf{HybridFlow}, a resource-adaptive edge-cloud inference framework that (i) builds a dependency-aware DAG for each query and executes newly unlocked subtasks in parallel, reducing end-to-end latency; (ii) routes each subtask online to the edge or cloud via a learned benefit--cost utility model that dynamically trades accuracy gains against token/API and latency budgets, thereby reducing unnecessary cloud usage while preserving reasoning quality. Across GPQA, MMLU-Pro, AIME24, and LiveBench-Reasoning, HybridFlow improves the cost-accuracy trade-off, reducing latency and cloud API usage while maintaining competitive accuracy against strong structured reasoning baselines.

cs.DC↗

Confusion-Geometry Rebalancing for Long-Tailed Adversarial Training

Adversarial training under long tailed distributions suffers from a dual imbalance: the class imbalance skews the training objective toward head classes, and the adversarial inner maximization may further amplify this bias. Existing methods mitigate this issue by correcting class priors or adapting class wise robust supervision, yet they treat each class in isolation and fail to identify which boundaries drive long tailed collapse. We propose a Confusion Geometry Rebalancing method (CGRm) for long tail adversarial training, a plug in framework that leverages directed robust errors as training signals. CGRm leverages periodic robust evaluations to derive source class loss weights, class wise robust coefficients, and a directed confusion geometry graph. The method then couples feedback weighted robust optimization with graph guided margin correction, thereby boosting the robustness of vulnerable classes and sharpening the critical boundaries that drive long tailed performance degradation. Experiments on long tailed benchmarks show that CGRm achieves consistent robust performance gains over existing methods, with ablations validating the contribution of each component. We provide the code in the supplement.

cs.LG↗

REIN: Bridging the Gap between Reasoning and Reliability via Reflection and Abstention Alignment

Large reasoning models (LRMs) are prone to hallucination, which undermines their reliability and poses challenges for safe deployment. Hallucinations in LRMs arise from two distinct failure sources: reasoning hallucination, where flawed inference steps propagate to an incorrect conclusion, and knowledge hallucination, where the model lacks the requisite factual knowledge to answer the query. To address reasoning hallucination, we propose REIN, an alignment framework that trains LRMs to produce a structured reasoning sequence, $\texttt{ } $$\rightarrow$ $\texttt{ } $$\rightarrow$ $\texttt{ }$, enabling explicit self-reflection before committing to a final answer. To address knowledge hallucination, REIN introduces a reward mechanism that encourages explicit abstention (e.g., "I don't know") when none of the sampled reasoning chains yields a correct answer, allowing the model to refrain from unsupported predictions. Extensive evaluations on mathematical and commonsense reasoning benchmarks show that REIN consistently improves selective accuracy, reduces incorrect-but-self-endorsed responses, and maintains high coverage compared with competitive baselines. Notably, REIN achieves these gains within a single forward pass, without requiring process supervision, inference-time controllers, external search, or multi-round critiques. Experiments on multiple backbones show that REIN reduces the hallucination proxy by $58\sim72\%$ relative to the base models while maintaining $86\sim91\%$ average coverage, and improves selective accuracy on attempted questions by $6.6\sim14.2\%$.

cs.AI↗

An Early Warning of Emerging Biosecurity Risks in Frontier LLMs

Frontier large language models (LLMs) are increasingly integrated into scientific workflows, yet their growing biological capabilities may outpace current safeguards. To assess the biological risks of frontier models, we develop Intern-BioBreaker, a specialized bio-red-teaming model, together with an integrated computational-to-physical framework that couples model-level stress testing with wet-lab validation. Within this framework, Intern-BioBreaker generates targeted jailbreak prompts to test whether aligned models can be induced to provide operational guidance for safety-sensitive biological tasks or produce sequence-level outputs with potentially harmful properties. Selected sequence outputs are then carried forward for DNA synthesis, host expression, and orthogonal protein verification to assess whether model-generated designs can yield the intended biological products. Our evaluation reveals a concerning gap between text-level safeguards and the risks posed by capable scientific models: (i) Intern-BioBreaker outperforms baseline attack models and reveals widespread bio-risk jailbreak vulnerabilities across both open-weight and proprietary frontier LLMs, with several targets reaching near-saturated or 100% task-level attack success rate (ASR); (ii) in sequence-level case studies, GPT-5.5 can be induced to generate modified viral candidate sequences with pathogenic potential; the corresponding translated proteins may exhibit even stronger receptor-binding affinity and thus enhanced infection potential; and (iii) end-to-end verification shows that selected model-generated biological designs are not merely textual artifacts, but can be physically realized under controlled experimental settings. These findings underscore the need for stronger biological red-teaming, nucleic acid synthesis screening, and safety mechanisms that keep pace with model capabilities.

cs.CL↗

Gradient Immunity: Null-Space Resistance to Malicious Fine-Tuning

Released aligned large language models remain vulnerable to malicious downstream finetuning. Existing defenses are largely designed for the fine-tuning-as-a-service (FTaaS) paradigm or rely on downstream users to follow additional safety procedures, and therefore do not directly address the setting we study: a provider controlled partially protected open-weight (PPOW) release setting in which most weights remain trainable while a small safety-critical component is preserved at release. We propose a Unidirectional Safety Gate (USG), instantiated as a Null Space Cubic Layer together with an Inverse Adapter inserted after the final Transformer layer. During downstream fine-tuning, the cubic layer suppresses or blocks gradients from harmful samples whose hidden states fall in a calibrated protected region, while the Inverse Adapter restores the base model's forward behavior. In practice, we calibrate a threshold using defender-held harmful data, allowing protection to generalize to nearby in-distribution harmful samples. Across six evaluated model-dataset settings, USG keeps post-finetuning attack success rate close to the pre-release level under a fixed release threshold, while maintaining high safe-pass rates on easier settings and exhibiting a clearer safety-utility trade-off on unsafe samples from BeaverTails. These results suggest that release-time representation-space blocking can raise the cost of malicious downstream adaptation without requiring downstream cooperation. The code is available at https://github.com/OpenCausaLab/Gradient-Immunity.

cs.CR↗

Dynamic Jailbreaking Attack

Existing gradient-based jailbreak attacks typically optimize a fixed-length adversarial suffix toward a predefined target response with a static optimization strategy. However, this fully static formulation undermines the effectiveness, efficiency and flexibility of gradient-based jailbreaking because (i) A predefined target usually lies in the low-probability region of a safety-aligned LLM's conditional output distribution, forcing the optimization to pursue an unlikely response pattern; (ii) Simple affirmative targets may even mislead LLMs to generate affirmative responses that are not highly relevant to the prompts; (iii) Fixed optimization strategy and suffix length treat all prompts equally, leading to limited attack capability for hard prompts and redundant capacity for easy ones. To address these limitations, we propose Dynamic Jailbreaking Attack (DJA), a parameter-free gradient-based jailbreak framework using dynamic candidate exploration, dynamic relevant targets and dynamic optimization strategy to craft adversarial prompts. In each optimization round, DJA samples multiple candidate target responses directly from the LLM's distribution conditioned on the current adversarial prompt. Among these candidates, DJA employs a multi-objective scorer to select an optimal target that satisfies multi-dimensional criteria such as harmfulness, relevance, and usefulness. Moreover, DJA introduces a parameter-free dynamic optimization strategy that allocates adversarial effort based on real-time feedback, adapting suffix length, candidate sampling capacity, and optimization iterations according to the difficulty of each harmful prompt. In an extensive evaluation of 40 safety-aligned LLMs (12 model families, scaling from 0.5B to 32B), DJA achieves a 100% ASR across all LLMs, requiring only 13.68 optimization rounds on average (10 iterations per round).

cs.CR↗

Robust Estimation of Sparse Numerical Vectors under Local Differential Privacy

Local differential privacy (LDP) protocols are vulnerable to poisoning attacks. Existing research have proposed efficient defense strategies for single-item users. However, in practice, a user may possess multiple items. The defense against poisoning attacks for multi-item users is challenging, because due to larger output spaces, the adversary can conduct more powerful attacks without being detected. In this paper, we address the robust sparse vector mean estimation problem, in which each user has a vector with $m$ nonzero coordinates. We propose Randomized Projection with Clipping (RPC). Firstly, the server sends a random binary vector to each user. The user then projects its local data on the vector, and clip the value to restrict the attacker's capability. To handle clipping bias, we propose a correction method based on a careful analysis that gives an exact expression of the bias. As a result, bias-variance tradeoff is no longer needed, thus the clipping threshold can be further reduced to shrink the output space and enhance robustness. We provide a rigorous theoretical guarantee of the estimation error under all possible attacks. Numerical experiments show that under trusted environments, our new method achieves comparable or better performance than existing methods, indicating that our method is already an efficient estimator in its own right. Under untrusted environments, our method is also significantly more robust to poisoning attacks.

stat.ML↗