arXiv subjects

Zhuoer Feng

Publications and source records attributed to Zhuoer Feng.

6 recordsLinked to original sources

arXiv2026

TraceSIR: A Multi-Agent Framework for Structured Analysis and Reporting of Agentic Execution Traces

Agentic systems augment large language models with external tools and iterative decision making, enabling complex tasks such as deep research, function calling, and coding. However, their long and intricate execution traces make failure diagnosis and root cause analysis extremely challenging. Manual inspection does not scale, while directly applying LLMs to raw traces is hindered by input length limits and unreliable reasoning. Focusing solely on final task outcomes further discards critical behavioral information required for accurate issue localization. To address these issues, we propose TraceSIR, a multi-agent framework for structured analysis and reporting of agentic execution traces. TraceSIR coordinates three specialized agents: (1) StructureAgent, which introduces a novel abstraction format, TraceFormat, to compress execution traces while preserving essential behavioral information; (2) InsightAgent, which performs fine-grained diagnosis including issue localization, root cause analysis, and optimization suggestions; (3) ReportAgent, which aggregates insights across task instances and generates comprehensive analysis reports. To evaluate TraceSIR, we construct TraceBench, covering three real-world agentic scenarios, and introduce ReportEval, an evaluation protocol for assessing the quality and usability of analysis reports aligned with industry needs. Experiments show that TraceSIR consistently produces coherent, informative, and actionable reports, significantly outperforming existing approaches across all evaluation dimensions. Our project and video are publicly available at https://github.com/SHU-XUN/TraceSIR.

cs.AI↗

arXiv2023

CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation

Since the natural language processing (NLP) community started to make large language models (LLMs) act as a critic to evaluate the quality of generated texts, most of the existing works train a critique generation model on the evaluation data labeled by GPT-4's direct prompting. We observe that these models lack the ability to generate informative critiques in both pointwise grading and pairwise comparison especially without references. As a result, their generated critiques cannot provide fine-grained distinguishability on generated texts, causing unsatisfactory evaluation performance. In this paper, we propose a simple yet effective method called Eval-Instruct, which can first acquire pointwise grading critiques with pseudo references and then revise these critiques via multi-path prompting to obtain informative evaluation data in different tasks and settings, including pointwise grading and pairwise comparison with / without references. After fine-tuning on these data, the resulting model CritiqueLLM is empirically shown to outperform ChatGPT and all the open-source baselines and even achieve comparable evaluation performance to GPT-4 in system-level correlations of pointwise grading. We also demonstrate that our generated critiques can act as scalable feedback to further improve the generation quality of strong LLMs like ChatGPT.

cs.CL↗

arXiv2023

AlignBench: Benchmarking Chinese Alignment of Large Language Models

Alignment has become a critical step for instruction-tuned Large Language Models (LLMs) to become helpful assistants. However, the effective evaluation of alignment for emerging Chinese LLMs is still largely unexplored. To fill in this gap, we introduce AlignBench, a comprehensive multi-dimensional benchmark for evaluating LLMs' alignment in Chinese. We design a human-in-the-loop data curation pipeline, containing eight main categories, 683 real-scenario rooted queries and corresponding human verified references. To ensure the correctness of references, each knowledge-intensive query is accompanied with evidences collected from reliable web sources (including URLs and quotations) by our annotators. For automatic evaluation, our benchmark employs a rule-calibrated multi-dimensional LLM-as-Judge~\cite{zheng2023judging} approach with Chain-of-Thought to generate explanations and final ratings, ensuring high reliability and interpretability. All evaluation code, data, and LLM generations are available at \url{https://github.com/THUDM/AlignBench}. Since its release, AlignBench has been adopted by top (Chinese) LLMs for evaluating their alignment capabilities in Chinese, including ChatGLM, Qwen, DeepSeek, Yi, Baichuan, and Abab.

cs.CL↗

arXiv2022

Cataloguing MoSi$_2$N$_4$ and WSi$_2$N$_4$ van der Waals Heterostructures: An Exceptional Material Platform for Excitonic Solar Cell Applications

Two-dimensional (2D) materials van der Waals heterostructures (vdWHs) provides a revolutionary route towards high-performance solar energy conversion devices beyond the conventional silicon-based pn junction solar cells. Despite tremendous research progress accomplished in recent years, the searches of vdWHs with exceptional excitonic solar cell conversion efficiency and optical properties remain an open theoretical and experimental quest. Here we show that the vdWH family composed of MoSi$_2$N$_4$ and WSi$_2$N$_4$ monolayers provides a compelling material platform for developing high-performance ultrathin excitonic solar cells and photonics devices. Using first-principle calculations, we construct and classify 51 types of MoSi$_2$N$_4$ and WSi$_2$N$_4$-based [(Mo,W)Si$_2$N$_4$] vdWHs composed of various metallic, semimetallic, semiconducting, insulating and topological 2D materials. Intriguingly, MoSi$_2$N$_4$/(InSe, WSe$_2$) are identified as Type-II vdWHs with exceptional excitonic solar cell power conversion efficiency reaching well over 20%, which are competitive to state-of-art silicon solar cells. The (Mo,W)Si$_2$N$_4$ vdWH family exhibits strong optical absorption in both the visible and ultraviolet regimes. Exceedingly large peak ultraviolet absorptions over 40%, approaching the maximum absorption limit of a free-standing 2D material, can be achieved in (Mo,W)Si$_2$N$_4$/$\alpha_2$-(Mo,W)Ge$_2$P$_4$ vdWHs. Our findings unravel the enormous potential of (Mo,W)Si$_2$N$_4$ vdWHs in designing ultimately compact excitonic solar cell device technology.

cond-mat.mtrl-sci↗

arXiv2021

LOT: A Story-Centric Benchmark for Evaluating Chinese Long Text Understanding and Generation

Standard multi-task benchmarks are essential for developing pretraining models that can generalize to various downstream tasks. Existing benchmarks for natural language processing (NLP) usually focus only on understanding or generating short texts. However, long text modeling requires many distinct abilities in contrast to short texts, such as the modeling of long-range discourse and commonsense relations, and the coherence and controllability of generation. The lack of standardized benchmarks makes it difficult to assess these abilities of a model and fairly compare different models, especially Chinese models. Therefore, we propose a story-centric benchmark named LOT for evaluating Chinese long text modeling, which aggregates two understanding tasks and two generation tasks. We construct new datasets for these tasks based on human-written Chinese stories with hundreds of words. Furthermore, we release an encoder-decoder-based Chinese long text pretraining model named LongLM with up to 1 billion parameters. We pretrain LongLM on 120G Chinese novels with two generative tasks including text infilling and conditional continuation. Extensive experiments show that LongLM outperforms similar-sized pretraining models substantially on both the understanding and generation tasks in LOT.

cs.CL↗

arXiv2021

OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics

Automatic metrics are essential for developing natural language generation (NLG) models, particularly for open-ended language generation tasks such as story generation. However, existing automatic metrics are observed to correlate poorly with human evaluation. The lack of standardized benchmark datasets makes it difficult to fully evaluate the capabilities of a metric and fairly compare different metrics. Therefore, we propose OpenMEVA, a benchmark for evaluating open-ended story generation metrics. OpenMEVA provides a comprehensive test suite to assess the capabilities of metrics, including (a) the correlation with human judgments, (b) the generalization to different model outputs and datasets, (c) the ability to judge story coherence, and (d) the robustness to perturbations. To this end, OpenMEVA includes both manually annotated stories and auto-constructed test examples. We evaluate existing metrics on OpenMEVA and observe that they have poor correlation with human judgments, fail to recognize discourse-level incoherence, and lack inferential knowledge (e.g., causal order between events), the generalization ability and robustness. Our study presents insights for developing NLG models and metrics in further research.

cs.CL↗