Searcharxiv⌕ Search

arXiv subjects

Hao Peng

Publications and source records attributed to Hao Peng.

At least 37 records · Page 2Linked to original sources
arXiv2026

Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios?

Rubric-based scoring has become a widely used paradigm in model evaluation, typically with LLM-as-a-Judge (LaaJ) for rubric scoring. However, the reliability of LaaJ for rubric scoring remains underexplored. This concern is especially pronounced in agentic scenarios, where long, complex outputs further challenge reliable scoring. To address this, we conduct a systematic meta-evaluation of LaaJ reliability for rubric verification. We introduce RuVerBench, the first benchmark for assessing LaaJ reliability in rubric verification for agentic scenarios. RuVerBench covers two prevalent agentic domains, deep research and agentic coding, with 2,458 instances, each containing a model-generated output, a rubric, and a human-annotated label indicating whether the output satisfies the rubric. Using RuVerBench, we evaluate numerous frontier LLMs and find that even the most advanced models achieve strong performance but still exhibit substantial noise. We further analyze the impact of key LaaJ strategies, including prompt design, batching, and majority voting, on rubric verification. We find that weaker models are more sensitive to prompt variations, batched verification presents a trade-off between accuracy and efficiency, and majority voting yields effective but diminishing returns. We have released our dataset and code to facilitate future research: https://github.com/THU-KEG/RuVerBench.

cs.CL↗
arXiv2026

Hindsight Memory-PRM: Supervising Memory Management with Auditable Hindsight Credit

Memory operations of long-horizon LLM agents are hard to supervise: an operation's value is unobservable when it is taken. But they are special -- they leave machine-readable evidence in the trajectory: retrieval hits and answer-time citations. Hindsight Memory-PRM exploits this audit trail twice: offline to train an operation-conditioned memory-utility critic, and online, where retrievals, citations, and one controlled deletion-and-reanswer per probe settle an intervention-calibrated entry-level presence credit, propagated along version chains as an action-level proxy reward -- no per-operation human labels, no Monte-Carlo replay of continuations. On held-out LoCoMo a local 8B policy reaches 77.5% under a fixed shared reader, surpassing its API teacher (65.1%) and all reproduced external systems, at one eighth the context of Mem0's official operating point; on LongMemEval, 79.0%. Ablations attribute the gain to causal calibration rather than signal density, and the policy converges to a multi-version memory organization whose gains no tested open-loop baseline reproduces.

cs.CL↗
arXiv2026

Useful Memories Become Faulty When Continuously Updated by LLMs

Learning from past experience benefits from two complementary forms of memory: episodic traces -- raw trajectories of what happened -- and consolidated abstractions distilled across many episodes into reusable, schema-like lessons. Recent agentic-memory systems pursue the consolidated form: an LLM rewrites past trajectories into a textual memory bank that it continuously updates with new interactions, promising self-improving agents without parameter updates. Yet we find that such consolidated memories produced by today's LLMs are often faulty even when derived from useful experiences. As consolidation proceeds, memory utility first rises, then degrades, and can fall below the no-memory baseline. More surprisingly, even when consolidating from ground-truth solutions, GPT-5.4 fails on 54% of a set of ARC-AGI problems it had previously solved without memory. We trace the regression to the consolidation step rather than the underlying experience: the same trajectories yield qualitatively different memories under different update schedules, and an episodic-only control that simply retains those trajectories remains competitive with the consolidators we test. In a controlled ARC-AGI Stream environment that exposes Retain, Delete, and Consolidate actions, agents preserve raw episodes by default and double the accuracy of their forced-consolidation counterparts; disabling consolidation entirely (episodic management only) matches this auto regime. Practically, robust agent memory should treat raw episodes as first-class evidence and gate consolidation explicitly rather than firing it after every interaction. Looking forward, reliable agentic memory will require LLMs that can consolidate without overwriting the evidence they depend on.

cs.AI↗
arXiv2026

GSPR: Aligning LLM Safeguards as Generalizable Safety Policy Reasoners

As large language models (LLMs) are integrated into numerous applications, LLMs' safety becomes critical for both application developers and intended users. Currently, great efforts have been made to develop safety benchmarks with fine-grained taxonomies. However, these benchmarks' taxonomies are disparate with different safety policies. Thus, existing safeguards trained on these benchmarks are either coarse-grained to only distinguish between "safe'' and "unsafe,'' or constrained by the specified narrow risk taxonomies. To leverage these fine-grained safety policies across multiple safety taxonomies, we propose GSPR, a Generalizable Safety Policy Reasoner to identify unsafe inputs and outputs with violated safety taxonomies and concise explanations. Unlike prior safeguards which only cover a fixed set of risk factors, GSPR incentivizes its reasoning capability with varied safety taxonomies through reinforcement learning. Our GSPR can be trained across multiple safety benchmarks with distinct taxonomies and naturally exhibits powerful generalization ability. We conduct extensive experiments to show that GSPR significantly improves existing safety guardrails' reasoning capabilities for both safety and category prediction tasks. Moreover, GSPR also achieves the least inference token costs with explanations.

cs.CR↗
arXiv2026

Role-Aware Morgan Fingerprints for Reaction Yield Prediction

Predicting reaction yield from molecular structure and reaction context can cut experimental trial-and-error and speed up condition screening in synthetic chemistry. Recent methods for this task use learned representations such as graph neural networks or Transformer encoders over reaction SMILES (Simplified Molecular Input Line Entry System), but these approaches carry heavy preprocessing overhead and can break when input formatting is inconsistent. We propose MFP, a reaction yield prediction method built on role-aware Morgan fingerprints where count-based circular fingerprints are computed for each reaction component, aggregated by chemical role (reactant, reagent, product), and combined with transformation-sensitive difference features into a fixed-length reaction descriptor fed to a feed-forward neural regressor. We test MFP against state of the art methods such as YieldBERT (with and without data augmentation) and GNAN (graph neural network) on the Suzuki-Miyaura and Buchwald-Hartwig benchmarks using a shared preprocessing and evaluation protocol. MFP reaches R2 = 0.878 on Suzuki-Miyaura and R2 = 0.969 on Buchwald-Hartwig while training an order of magnitude faster than graph- or Transformer-based alternatives. A formal complexity analysis confirms that MFP folds all representation cost into a one-time preprocessing step, removing the per-epoch message-passing overhead that graph methods carry. An ablation over fingerprint radius and folded vector length shows that radius-2 representations at nBits =2048 give the best balance of accuracy, speed, and cross-split stability on both datasets. These results establish MFP as an effective, reproducible, and efficient baseline for reaction yield prediction.

cs.LG↗
arXiv2026

Improving Influence-based Instruction Tuning Data Selection for Balanced Learning of Diverse Capabilities

Selecting appropriate training data is crucial for instruction fine-tuning of large language models (LLMs), which aims to (1) elicit strong capabilities, and (2) achieve balanced performance across different tasks. Influence-based methods show promise in achieving (1), by estimating the contribution of each training example to the model's predictions, but often struggle with (2). Our systematic investigation reveals that this underperformance can be attributed to an inherent bias, where some tasks intrinsically have greater influence than others. As a result, data selection is often biased towards these tasks, not only hurting the model's performance on others but also, counterintuitively, harming performance on these high-influence tasks themselves. To address this, we propose BIDS, a Balanced and Influential Data Selection algorithm. BIDS first normalizes influence scores of the training data, and then iteratively chooses the training example with the highest influence on the most underrepresented task. Experiments with both Llama-3 and Mistral-v0.3 on seven benchmarks spanning five diverse capabilities show that BIDS consistently outperforms both state-of-the-art influence-based algorithms and other non-influence-based frameworks. Surprisingly, training on a 15% subset selected by BIDS can even outperform full-dataset training with a much more balanced performance. Our analysis highlights the importance of both instance-level normalization and iterative optimization of selected data for balanced learning of diverse capabilities.

cs.CL↗
arXiv2026

Co-leading Teams Drive Scientific Novelty in Large-scale Research Infrastructures

Large-scale research infrastructures (LSRIs) have become the engine of modern scientific discovery. While these big machines predominantly operate under a user-oriented model where external teams conduct research with support from in-house researchers, the structural integration of staff scientists into user teams and its association with scientific novelty remains unclear. By leveraging a dataset of 273,109 publications across 76 global LSRIs and applying a hybrid machine-learning framework to classify papers into three collaboration patterns: external user only, staff participating, and staff co-leading, we find a distinct novelty premium for external teams that formally integrate staff as co-authors, especially when staff scientists play co-leading rather than participating roles. Further, the premium peaks at a relatively balanced user-staff team composition, potentially due to an "epistemic lock-in" by either party. Crucially, we find that the ideal collaboration architecture evolves with user experience: while newcomers can obtain a large novelty premium from mere staff participation, experienced users only benefit from staff co-leading teams. This result suggests a "knowledge saturation effect" for which a deeper intellectual partnership is needed to sustain novelty. By revealing how user-staff collaboration structure drives scientific creativity, our study offers practical policy implications for the strategic management and intervention of LSRIs in the era of human-machine collaboration.

cs.DL↗
arXiv2026

ProtoHGF-Net: Prototype HyperGraph Fusion with Intra-modal Calibration for RGBT Object Detection

RGB-Thermal (RGBT) object detection enables robust perception in complex scenes by leveraging the complementary strengths of visible textures and thermal cues. However, existing methods mainly rely on dense cross-modal interactions over full-resolution features, which inevitably introduce background interference and hinder the learning of target-relevant representations. In this paper, we propose the Prototype HyperGraph Fusion Network (ProtoHGF-Net), a novel framework that redefines cross-modal fusion as prototype-level semantic interaction rather than the dense cross-modal interaction paradigm. Specifically, we design Prototype HyperGraph Fusion to perform cross-modal interaction in a compact prototype-level semantic space. This design enables more selective fusion among target-relevant prototypes. To support this prototype-level fusion, we propose Teacher-Mask Calibration Distillation, which calibrates modality features before fusion using modality-specific teachers and target-aware masks. This strategy suppresses backgrou- nd-dominant responses and produces more target-focused features. Extensive experiments on DroneVehicle, DVTOD, and FLIR demonstrate that ProtoHGF-Net achieves state-of-the-art performance with 85.9\% $mAP_{50}$, 88.2\% $mAP_{50}$, and 79.1\% $mAP_{50}$, respectively. Our code is available at \href{https://github.com/ZiMo-Chen/ProtoHGF}{GitHub}.

cs.CV↗
arXiv2026

TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories

LLM-based agentic systems have shown remarkable capabilities in complex domains, while suffering from cascading errors and difficulty in debugging. Critical error detection aims to locate the earliest error step in a failed trajectory that is responsible for the final failure. However, progress faces two main challenges. First, long trajectories make it difficult to identify individual errors, since the evidence for judging a step may be scattered across distant instructions, observations, and prior context. Second, failed trajectories often contain multiple local errors with different downstream effects, only some of which remain responsible for the final failure. In this work, we propose TrajDebug, an error-lifecycle tracing framework that addresses long-trajectory error discovery with multi-granularity history compression and evidence-based error identification, and supports critical attribution by tracing each error's resolution status and terminal impact. We further construct TrajErrBench, a benchmark of 486 manually annotated failed trajectories from Tau2Bench and SWE-Bench Pro, covering realistic tool-use and coding scenarios. Experiments across diverse agent benchmarks show that TrajDebug achieves the best overall performance over existing baselines, and application studies further demonstrate that its diagnoses provide actionable feedback for improving downstream agent success. We will release the codes and data to facilitate further research.

cs.AI↗
arXiv2026

CAPMix: Robust KPI Anomaly Detection for AIOps in Noisy and Dynamic Environments

Time-series anomaly detection is crucial in AIOps for maintaining large-scale service reliability. In production, streams of Key Performance Indicators (KPI) are high-dimensional, non-stationary, and affected by noise, deployment changes, and latent anomalies, making real failures hard to distinguish from benign variation. Most existing methods assume either normality (learning from "normal" history) or rely on injected anomalies for training. Yet injected patterns often misalign with real failure modes, skewing decision boundaries -- aka. Anomaly Shift. We propose CAPMix, a controllable anomaly augmentation framework with prior-guided injection for realistic temporal behaviors. CAPMix combines label revision and dual-space mixup to enhance robustness under contaminated and mixed data. CAPMix consistently outperforms state-of-the-art methods on public AIOps and time-series benchmarks. It has been deployed in Kuaishou's large-scale production system, reducing false alarms and improving monitoring reliability. A real-world dataset is also released to enrich the research on robust KPI anomaly detection.

cs.LG↗
arXiv2026

EgoSafe: A First-Person Mobile-Captured Benchmark for Visual Safety Understanding

Reliable visual safety understanding in real-world scenarios demands more than just object recognition; it requires causal reasoning under epistemic uncertainty. While Large Vision-Language Models (LVLMs) demonstrate impressive semantic alignment on standard benchmarks, they often struggle to distinguish between superficial correlation and genuine forensic logic when grounded in the dynamic, partially observable nature of first-person experiences. Existing evaluations, dominated by third-person surveillance footage and binary classification metrics, fail to expose this cognitive gap. To address this, we introduce EgoSafe-Bench, a benchmark specifically designed to probe forensic reasoning in egocentric safety scenarios. It comprises 12,000 unique evaluation samples, generated by pairing each of the 3,000 video clips with a QA chain governed by our proposed Hierarchical Reasoning Evaluation (HRE) protocol. Unlike standard benchmarks, HRE mandates a rigorous reasoning trajectory from initial feature anchoring to blind-spot deduction and intent inference, thereby enforcing logical consistency and penalizing shortcut-based predictions. Extensive evaluations of state-of-the-art LVLMs (e.g., Qwen3-VL, Gemini, VideoLLaMA 3) reveal a significant perception-reasoning decoupling: models often achieve high descriptive scores but exhibit notable fragility in causal reasoning and logical closure. Our work provides both a challenging dataset and a systematic evaluation framework to foster the development of logically robust video understanding systems.

cs.CV↗
arXiv2026

Paying for Honesty Without Knowing the Truth: Reputation-Penalty Design for LLM Marketplace Agents

LLM agents increasingly act as autonomous merchants that write their own product listings, and under competitive pressure, they fabricate attributes to win sales. Even under instructions to be honest, they fabricate attributes in a majority of listings across models. A platform's obvious remedy---verifying each claim against the truth---is unavailable, because it observes only a noisy, biased complaint signal, never the ground truth. We design CARP, a reputation-penalty mechanism with a deadband that forgives complaint noise and a state-dependent severity that counters reputation-driven detection erosion. CARP requires no product-level ground truth and is robust to strategic gaming. CARP protects consumers by suppressing the sales volume of low-rated liars while sparing honest sellers. Paired with SPARC, it closes most of the consumer-welfare gap relative to a perfect-information oracle, without ever accessing the truth. It also achieves the best welfare of the policies we compare. We further show that this felt penalty becomes behaviorally binding through SPARC, a byte-clean code-gated reflection mechanism: LLM merchants fabricate when lying is free but restrain themselves when fabrication costs them sales, a self-interested response rather than compliance. We trace this distinction to penalty-gated self-correction reasoning, and observe the binding across models, with supporting confidence intervals.

cs.AI↗
arXiv2026

GuideSkill: Evolving Executable LLM Agent Skills for Guideline-Grounded Clinical Reasoning

Clinical practice guidelines (CPGs) encode diagnostic criteria, but LLM systems typically retrieve guideline text or absorb it through training rather than execute its rules. We introduce GuideSkill, an external reasoning layer that compiles disease-specific criteria into executable functions returning ordinal diagnostic-support scores. GuideSkill-Zero is initialized from guidelines, while GuideSkill-Evo uses case--diagnosis pairs to refine covered skills and add missing diagnoses. At inference, an LLM proposes a differential diagnosis, grounds the features required by each matched skill, and fuses its ranking with the executed skill scores. Across four benchmarks and four backbones, GuideSkill-Zero improves macro-average accuracy over guideline RAG by 13.45% on average. GuideSkill-Evo achieves the highest macro-average for every backbone, improves over direct inference by 18.49% relatively, and increases gold-label skill coverage from 56.5% to 99.5%. On Qwen3.5-9B, it also exceeds the strongest parameter-update baseline by 11.16% without updating the backbone. Expert evaluation further indicates that GuideSkill produces clinically sound and broadly acceptable skills, suggesting that its initialized and evolved rules are reliable and practically meaningful. These results support executable skills as a model-agnostic mechanism for combining guideline-derived procedures with case-derived diagnostic patterns.

cs.AI↗
arXiv2026

Characterizing Rhetorical Misalignment in Decision-Making with Language Models

Human decision-making is often shaped by a range of well-documented cognitive biases. As large language models (LLMs) become increasingly integrated into high-stakes human-AI decision-making, it is important to understand whether their outputs can amplify potential biases, how this influences human decisions, and crucially, whether it can lead to harmful consequences. In this work, we develop a decision-theoretic framework to study rhetorical misalignment, a failure mode where an LLM uses rhetorically inappropriate forms of presentation for a given decision context, thereby inducing suboptimal human decisions. We empirically investigate this phenomenon through a human-subject experiment in realistic clinical decision-making using a dataset curated from the United States Medical Licensing Examination. By measuring how LLM-generated information affects decisions, we observe that LLMs induce an average 2.81% rate of harmful decision flips across different models, where clinician participants change from a correct to an incorrect answer. Rationales reported by participants provide evidence that these revisions are closely related to the language used by LLMs that may induce different types of cognitive biases, including anchoring, authority bias, and loss aversion. To enable scalable evaluation, we instantiate our theoretical framework using decision-makers simulated by LLMs to computationally measure rhetorical misalignment. Our findings reveal a safety concern previously unrecognized in high-stakes domains: a model can be factually aligned yet still induce harm through its rhetorical presentation.

cs.CL↗
arXiv2026

SafeHarbor: Defining Precise Decision Boundaries via Hierarchical Memory-Augmented Guardrail for LLM Agent Safety

Recent advances in foundation models have transformed LLMs from passive conversational systems into autonomous agents capable of reasoning and tool execution. While these capabilities unlock substantial practical value, they also introduce new security risks, as adversaries can manipulate agents into performing harmful actions in real-world environments. Existing defense strategies mitigate such threats but frequently struggle to balance safety and utility, resulting in over-refusal of benign user requests. To mitigate this trade-off, we propose SafeHarbor, a novel framework designed to establish precise decision boundaries for LLM agents. Unlike static guidelines, SafeHarbor extracts context-aware defense rules through enhanced adversarial generation. We design a local hierarchical memory system for dynamic rule injection, offering a training-free, efficient, and plug-and-play solution. Furthermore, we introduce an information entropy-based self-evolution mechanism that continuously optimizes the memory structure through dynamic node splitting and merging. Extensive experiments demonstrate that SafeHarbor achieves state-of-the-art performance on both ambiguous benign tasks and explicit malicious attacks, notably attaining a peak benign utility of 63.6\% on GPT-4o while maintaining a robust refusal rate exceeding 93\% against harmful requests. The source code is publicly available at https://github.com/ljj-cyber/SafeHarbor.

cs.CR↗
arXiv2026

Personalized Recommendation Tool Learning via Autonomous Language Agents

Although large language models (LLMs) have recently gained traction in recommender systems due to their strong reasoning capabilities and extensive world knowledge, previous LLM-based agents suffer from hallucination and context-length limitations, and thus are not suitable for full-ranking recommendation tasks. To circumvent these limitations through architectural design rather than modifying the LLM itself, we propose an agent-based recommendation framework, memory-based $\textbf{P}$ersonalized $\textbf{R}$ecommendation $\textbf{T}$ool learning via autonomous language $\textbf{A}$gents (PRTA), in which an LLM acts as a central planner interacting with multiple recommendation models as tools. The LLM-based agent is responsible for high-level reasoning and personalized tool selection, while traditional recommendation models perform full-ranking scoring, leveraging their scalability in modeling behavioral patterns. To support personalized tool selection, we design reflection mechanisms that enable the agent to evaluate and compare tools for each user based on user profiles and candidate ranked lists. Extensive experiments across three public datasets demonstrate the superiority of \modelname over traditional recommendation and LLM-based baselines in improving full-ranking recommendation performance.

cs.IR↗
arXiv2026

Comprehensive, Efficient Large-Scale Community Detection via Structural Entropy Game

Community detection is a critical task in graph theory, social network analysis, and bioinformatics, where communities are defined as clusters of densely interconnected nodes. However, detecting communities in large-scale networks with millions of nodes and billions of edges remains challenging due to the inefficiency and unreliability of existing methods. Moreover, many existing methods are limited to specific types of graph structures (such as unweighted or undirected graphs) or are designed solely for detecting static communities, reducing their broader applicability. To address these issues, we propose a novel heuristic community detection algorithm, termed CoDeSEG, which identifies communities by minimizing the network's two-dimensional (2D) structural entropy within a potential game framework. In the game, nodes decide to stay in the current community or move to another based on a strategy that maximizes the 2D structural entropy utility function. Additionally, we introduce a structural entropy-based node overlapping heuristic for detecting overlapping communities, with a near-linear time complexity. Furthermore, we design a cascading influence propagation-based adaptive community update strategy, which dynamically identifies and processes nodes whose community affiliations may change during graph evolution, thereby effectively extending CoDeSEG to dynamic community detection scenarios. Experimental results on fourteen large-scale networks demonstrate that CoDeSEG achieves state-of-the-art performance across three community detection tasks (overlapping, non-overlapping, dynamic), while also delivering substantial improvements in detection efficiency.

cs.SI↗
arXiv2026

Fargues-Scholze correspondence and endoscopic classification for special orthogonal and unitary groups

Let $p$ be odd and let $K/\mathbb Q_p$ be unramified. For a special orthogonal group or a unitary group $G$ over $K$ that splits over an unramified extension, we prove that the Fargues-Scholze local Langlands correspondence agrees with the semisimplification of the classical correspondence for $G$ constructed in the work of Arthur and others. As applications, we construct an unambiguous local Langlands correspondence for even special orthogonal groups, deduce the strong Kottwitz conjecture and the eigensheaf conjecture of Fargues, and establish new torsion vanishing results for orthogonal and unitary Shimura varieties.

math.NT↗