arXiv subjects

Zhuohan Xie

Publications and source records attributed to Zhuohan Xie.

At least 19 recordsLinked to original sources

arXiv2026

PRIME: Predictive Routing of Interventions by Modeling Effects for Scientific Ideation

Feedback-driven iteration is widely used to improve LLM-generated scientific ideas, but existing systems either leave revision decisions implicit in generation, align revisions with feedback-localized weaknesses, or restrict action selection to system-specific operations. This is problematic because problems, hypotheses, and solutions are interdependent: the location where a weakness manifests does not itself determine the revision scope that will produce a better successor. We term this phenomenon diagnosis-intervention mismatch. We introduce Predictive Routing of Interventions by Modeling Effects (PRIME), a plug-and-play feedback-to-revision decision layer for existing scientific ideators. PRIME operates in three stages: (1) Structural Attribution interprets feedback over the problem-hypothesis-solution structure without fixing a revision target; (2) Counterfactual Routing constructs alternative scope-conditioned interventions and compares their projected successor utility and risks through lightweight rollouts; and (3) Intervention Execution compiles the selected intervention into an ordered generator-facing plan with semantic actions, preservation constraints, and dependency requirements for successor generation. We validate diagnosis intervention mismatch on 112 human-annotated idea-feedback pairs. Across four benchmarks, two ideators, and fixed- and open-problem settings, PRIME improves successor quality and routing reliability over strong baselines; removing counterfactual routing reduces quality from 3.463 to 1.960 and increases route regret from 0.426 to 1.929, confirming cross-scope effect modeling as the primary source of its gains.

cs.AI↗

arXiv2026

Do Transaction Prices Reflect Visual Preference? A Diagnostic Study with Multimodal Language Models

Existing visual preference evaluation relies on costly human annotations or preference reward models, whereas online marketplace transaction prices are readily available and reflect real purchasing behavior, offering potential low-cost supervision. However, transaction prices conflate visual preferences with non-visual factors such as provenance, demand, and competition, making aggregate accuracy insufficient to distinguish model errors, unreliable price signals, and protocol confounds. To address this identifiability challenge, we introduce MarketImage, a diagnostic framework that identifies when transaction prices reflect visual preferences, where they fail, and whether multimodal language models can recover human-supported visual preference signals. MarketImage comprises three components: (i) MarketImageArt, 266 six-image comparison sets with transaction prices and price-blind human rankings; (ii) MarketImageLens, a tie-preserving protocol that partitions 3,990 within-set image pairs into price-human agreement, conflict, and price-tie regions while separately auditing presentation-order effects; and (iii) MarketValuator, six elicitation methods spanning direct comparison, independent scoring, budget constraints, and multi-agent aggregation to evaluate signal recovery by static visual scorers and three multimodal language models. Experiments show that only 24.2% of image pairs have a non-tied price direction, of which 34.7% agree with the human majority. Within this region, HPSv2 leads at 66.4% Recoverable Accuracy; Qwen's budgeted and aggregated variants trail baselines, while GPT interface gains remain statistically unresolved. Thus, transaction prices encode recoverable visual preferences only under limited conditions and should be treated as uncertain proxies rather than ground truth.

cs.AI↗

arXiv2026

All the Facts, Still the Wrong Answer: Integration Gaps in Vision-Language Models

Multimodal reasoning requires models to combine multiple pieces of evidence into decisions that satisfy all relevant constraints. Evaluations centered on final-answer accuracy leave unclear whether failures arise from reading individual facts or using them together. We study this question through the multi-evidence integration gap, linking local fact recovery to global decision validity on controlled subset-selection and path-finding tasks. Our evaluation combines conditional performance given correct local answers with same-grid local queries, text-oracle inputs, retrieval controls, and constraint ablations. We further examine hidden-state probes and activation replacement across network depth, token positions, and model families. Across the evaluated models, accurate local recovery coexists with substantial global failure. For example, Gemma-3-27B recovers all evaluated isolated item facts correctly but achieves only 19.75% global validity on eight-item visual instances. Probes decode information about solution membership from global-prompt states, while corresponding clean-state replacement recovers failed decisions over model-dependent depth ranges. These findings show that recovering facts and representing solution-related information are insufficient for reliable joint decisions. They motivate evaluations and model improvements that explicitly target how available evidence is combined, alongside advances in perception and model scale.

cs.AI↗

arXiv2026

EngTrace: A Symbolic Benchmark for Verifiable Process Evaluation of Engineering Reasoning

Most engineering problems involve complex multi-step derivations through checkable intermediate quantities. Grading only the final answer misjudges LLMs, crediting correct answers reached through flawed steps and penalizing wrong answers built on mostly sound ones. Yet, existing benchmarks such as SuperGPQA and EEE-Bench score only final answers, and step-level ones such as TPS-CalcBench rely on LLM judges or human graders, or cover a single domain. We therefore introduce EngTrace, a symbolic benchmark for verifiable process evaluation of engineering reasoning. Its 150 expert-certified templates, spanning five branches, 15 domains, and 42 areas, generate each problem with a gold reasoning trace, so that every intermediate quantity is known exactly. We design and validate an evaluator that checks final answers, coverage of the gold trace's intermediate quantities (milestones), and displayed arithmetic, mostly without an LLM judge. Across 11 LLMs, final answers barely separate the strongest models, while their derivations differ. Wrong answers still state about half of the gold milestones, and domain experts trace most wrong answers to calculation slips, with conceptual errors rarer on easy problems than on harder ones. Overall, as LLMs near the final-answer ceiling, checking derivations, as EngTrace does, reveals differences that final answers miss.

cs.CL↗

arXiv2026

Herculean: An Agentic Benchmark for Financial Intelligence

As AI agents advance, the central question is no longer whether they can solve isolated financial tasks, but whether their capabilities generalize across complete professional workflows. Existing benchmarks cannot answer this question because they evaluate either static competencies or agents within a single interaction structure. We introduce Herculean, a benchmark of end-to-end agentic financial intelligence that crosses agent architecture, backbone model, and four structurally distinct workflows: Trading, Hedging, Market Insights, and Auditing. Each workflow is implemented as an MCP skill that preserves its native state, actions, constraints, and outcome criteria, allowing intact agent systems to be evaluated under matched external conditions. Across 30 architecture--backbone configurations, we find that agent capability is not globally ordered. Workflow rankings are weakly aligned, changing the architecture frequently reverses performance under the same backbone, and each architecture concentrates failure in different forms of workflow control. These results establish that agent performance is an interaction between architecture and workflow demand, rather than an intrinsic property of the underlying model. Consequently, success on a single financial workflow provides little evidence of dependable, general financial intelligence.

cs.AI↗

arXiv2026

Aligning Cultural Behaviors in Arabic Language Models via Preferences

Arabic LLMs have improved rapidly, but aligning them with country-specific cultural behavioral preferences remains underexplored. Previous work has focused primarily on instruction-tuning for factual knowledge, overlooking nuanced cultural behaviors. We propose ArabPref, which to our knowledge is the first large-scale pan-Arab preference data set for the alignment of cultural behavior conditioned on the country. Our contribution includes two resources in English and Modern Standard Arabic: a culturally grounded preference dataset for training and evaluation, and a multiple-choice benchmark designed to assess culturally aligned behavior across nations. All test data are validated by native speakers to ensure authenticity. To optimize cultural alignment, we experiment with post-training methods across multilingual and Arabic-centric language models, evaluating performance on both generation tasks and our cultural multiple-choice benchmark. By releasing an evaluation benchmark, we provide a foundation for systematically studying and improving culturally aligned behavior in language models.

cs.CL↗

arXiv2026

The Bottleneck Is Retrieval, Not Reading: Diagnosing Evidence-Based Generation with RAG for Long Context Question Answering

Retrieval-augmented generation (RAG) is the dominant paradigm for grounding large language models in external corpora, but its similarity-driven, chunked pipeline has structural blind spots: it cannot anticipate which passages a question requires and discards document-level context that resolves cross-passage references. We introduce evidence-based generation (EBG), requiring models to return an answer with line-level citations, and a joint Ans+Evid metric that scores correctness and grounding simultaneously. We reformulate two corpora into evidence-grounded multi-document QA benchmarks- MultiHop-RAG and QASPER-MP, a multi-paper adaptation of QASPER with realistic short paper cues, and pair them with a controlled synthetic two-hop haystack and an evidence-augmented NoLiMa spanning 10K-1M characters. Across ten proprietary frontier and three open-weight LLMs, EBG isolates two RAG limitations that long-context models avoid: similarity retrieval surfaces any relevant chunk far more reliably than all required chunks for multi-fact queries, and chunking destroys document meta-information needed to resolve identity-bearing queries. EBG also exposes grounding failures invisible to answer-only evaluation, including incomplete citations in two-needle settings and a widening gap between Ans and Ans+Evid when context contradicts parametric knowledge. Data and code are available at https://github.com/debobanerjee/LC-EBG.

cs.CL↗

arXiv2026

LLM-BabyBench: Can Language Models Plan in Worlds They Can Simulate?

Interactive benchmarks for LLMs usually report a single metric, even though succeeding in them requires models to show several different skills. While those benchmarks are useful for measuring the overall capability of models to perform the downstream task, it is harder to identify which skill was lacking and caused failure solely from the aggregate score. For example, in benchmarks where models are supposed to navigate a simulated world to perform a task, failures can come from perception, an ambiguous instruction, retrieval, missing commonsense about what actions do, unknown dynamics, or planning. \benchmark recasts the procedurally generated BabyAI gridworld as a fully observable, purely textual environment designed to evaluate planning separately from those factors. The nature of the environment and the structure of the prompt control for the other factors cited above. To evaluate the models, we define the PPD suite: \textbf{Predict} asks for the state that follows an action sequence, \textbf{Plan} asks for an action sequence that reaches a goal, and \textbf{Decompose} asks for a subgoal sequence that achieves a mission. Across seven frontier and open models, queried zero-shot, we find simulation far ahead of planning for every model, close to saturated for the strongest. The length of the required solution governs planning failure, and each of the four strongest models has a horizon beyond which single-attempt success collapses. The responses of claude-opus-4-6 and claude-sonnet-4-6, which show their working step by step, reveal that they plan by proposing L-shaped corridors, which is consistent with their successful plans staying near-optimal while success falls. At the subgoal level, decomposition precision likewise falls to zero on long missions, while comprehension of those missions persists.

cs.AI↗

arXiv2026

Same Scores, Different Decisions: Evaluating JEV and Language Models for Legal Document Understanding

Aggregate accuracy summarizes how often a model is correct, but not which judgments remain correct when requests change. We evaluate Jev and nine language models on ContractNLI, with repeated requests that preserve the contract and target while varying visible hypotheses, requested answers, and output order. A four-model follow-up adds an equal-call repetition control and coverage of the remaining test contracts. Corrections and regressions can cancel, leaving unchanged accuracy despite changed decisions; consistent answers can also preserve errors. Majority voting over recorded responses yields small, uncertain gains, with repeated wrong answers accounting for much of the observed error. Gold-class diagnostics further reveal different error profiles behind aggregate model rankings. Resource comparisons add distinct trade-offs: Jev is cheapest and fastest for the joint baseline, while hosted language models achieve higher accuracy. Together, these findings connect decision-level evaluation to whether additional calls improve correctness and what the resulting decisions cost.

cs.CL↗

arXiv2026

SQLStructEval: Structural Evaluation of LLM Text-to-SQL Generation

In Text-to-SQL tasks, large language models can generate structurally different SQL queries that return correct answers for the same intent. We call this phenomenon execution-correct structural divergence (ECSD). We introduce SQLStructEval, a framework that analyzes this behavior through canonical abstract syntax tree representations. We quantify structural diversity and agreement across generations. Experiments with different LLMs on multiple Text-to-SQL datasets, including Spider, document ECSD across models and datasets. Furthermore, our experiments demonstrate that generated queries are sensitive to question paraphrases and schema presentation. To address ECSD, we adopt a pipeline that first generates structured intermediate representations and then deterministically compiles them into SQL, improving execution accuracy and structural agreement among correct outputs. Structural analysis thus provides an additional diagnostic perspective that complements execution-based evaluation. Code is available at https://xanderzhou2022.github.io/AACL2026-SQLSTRUCTEVAL/.

cs.CL↗

arXiv2026

Prediction-Market Seed Capital Recovery from Noise-Dominant Flow

Automated prediction markets require sponsors to prefund liquidity before observing order flow, creating a financing challenge at launch. We study whether nonnegative charges conditioned on observable payoff direction can improve recovery of this prefunded capital while limiting their effect on informed participation. We develop Seed Capital Flow (SCF), a direction-conditioned levy, in a stylized binary cost-function market with informed and liquidity-motivated traders. When order composition differs across directions, SCF concentrates the permitted fee burden on the direction with relatively more liquidity-motivated flow, whereas a uniform fee spreads it across both directions. Under a sufficiently tight common retention constraint, this allocation yields higher expected recovery capacity and can make additional liquidity choices financially viable. Synthetic numerical audits examine robustness to alternative flow patterns, stochastic arrivals, and label misspecification. The results characterize a mechanism-design tradeoff rather than an empirical prediction: the market remains prefunded, recovery is expected rather than guaranteed, and the analysis is limited to an opening-cohort setting.

cs.CE↗

arXiv2026

TextAlign: Preference Alignment for Text Rendering with Hierarchical Rewards

Faithful text rendering remains a persistent weakness of large text-to-image generative models, as it requires both semantic instruction following and fine-grained glyph-level structure. Prior methods often improve this ability through architecture-specific modules or encoder modifications, which complicate deployment across foundation models. We study text rendering as a post-training preference-alignment problem and propose TextAlign, a non-invasive framework that keeps the generator architecture unchanged. The key component is a hierarchical vision-language model (VLM)-based reward that decomposes rendering errors into global, word, and glyph levels, then converts binary defect judgments into a scalar preference signal. The resulting signal supports both Group Relative Policy Optimization (GRPO) and Direct Preference Optimization (DPO). Experiments on FLUX.1-dev and Z-Image-Turbo show consistent gains in OCR-based text accuracy without degrading general generation quality. Compared with strong foundation and text-rendering baselines, including SD3.5, Qwen-Image, AnyText, and TextDiffuser, these results indicate that reward design offers a scalable alternative to model redesign for improving text rendering.

cs.CV↗

arXiv2026

FinCUABuild: Can Agents Build Reliable Benchmarks for Dynamic Financial Computer Use?

Financial scenarios are diverse and complex, spanning varying data conditions, tool configurations, and workflows. Yet existing CUA, Computer-Using Agent, evaluation tasks remain largely manually constructed, limiting scalable coverage of real-world financial scenarios. Then, can agents autonomously construct diverse CUA evaluation tasks for financial scenarios? Evaluating this capability poses three key challenges: scenario coverage of construction requests, fair comparison across construction methods, and reliable assessment of generated task quality. To solve these, we introduce FinCUABuildBench, a benchmark for evaluating financial CUA task construction, featuring: (i) 576 construction requests covering 24 financial workflows and three types of runtime variation; (ii) standardized input, budget, and output specifications; and (iii) a task qualification mechanism based on execution tests and quality checks. We further introduce FinCUABuildAgent, a multi-agent system for automatically constructing dynamic financial CUA evaluation tasks. It consists of three modules that jointly construct tasks, environments, and validators. On FinCUABuildBench, under the same model backbone, existing agent-based construction methods achieve strict qualification rates of only 1.3-8.3%, while FinCUABuildAgent reaches 31.3%. Downstream evaluations further show that the constructed tasks can effectively differentiate CUA task-execution capabilities. These results demonstrate that agents can autonomously construct financial CUA tasks with meaningful evaluation value, offering a practical path toward broader evaluation coverage in financial scenarios. Code: https://github.com/FengxianJi/FinCUABuild

cs.AI↗

arXiv2026

Right Frame, Wrong Rule: Cultural Cues Expose the Financial Knowledge Gap They Were Meant to Close

When a question has valid answers under different normative frameworks, a language model must decide which framework to use and whether it can answer correctly within it. We call this setting normative pluralism and study it in Islamic finance using a four-choice taxonomy that separates framework selection from within-framework correctness. This separation reveals the stereotype trap: a cultural cue steers a model toward one framework, but the model selects an incorrect answer within that framework. Across twelve models, two languages, and fifty demographic signals, cultural cues change framework selection and reveal substantial differences in accuracy, especially among non-frontier models. Under the strongest signal, large open-weight models select the Islamic framework 97% of the time. A two-choice evaluation would report near-perfect alignment, although 57--66% of those selections are incorrect. These findings motivate, but do not directly test, the competence-conditioned routing hypothesis: models may favor frameworks where they are more accurate, while cultural cues may expose framework-specific competence gaps.

cs.CL↗

arXiv2026

FinExam-10K: When Retrieval Helps Financial Reasoning?

Professional financial examinations require models to combine domain knowledge, calculation, and judgment, yet no benchmark covers the full CFA and FRM structure under one protocol. We introduce FinExam-10K, to our knowledge the largest reported English benchmark for this setting, with 10,198 expert-reannotated questions spanning CFA Levels I-III and FRM Parts I-II. We release 5,110 questions and sequester 5,088 for a quarterly maintained leaderboard. To separate coverage from local answerability, we report a 10,198-item Full-Coverage Track and a 7,625-item Context-Complete Reasoning Track, which is the primary basis for claims about reasoning from the supplied record. Across 17 models, the best accuracy is 85.29% overall. On the frozen Hard band, the best score is 34.68% on the Full-Coverage Track and 54.57% on the 372 context-complete items. All 17 models share 47 context-complete failures. Function-RAG and FunctionGraph-RAG rescue hundreds of errors but also overturn many correct answers, producing little or negative net gain. A gate trained only on public data decides from the question and initial response when FunctionGraph-RAG should run. On the 5,088 held-out items, the gate invokes FunctionGraph-RAG for 7.9% of questions and improves accuracy from 70.83% to 71.23% (p = .0446).

cs.CL↗

arXiv2026

Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation

However, whether these judges truly evaluate the scientific substance of ideas or are influenced by superficial stylistic presentation remains an open question. To address this question, we propose SciStyleBench, a unified three-component benchmark for diagnosing and mitigating stylistic bias in LLM-based idea evaluation: (i) First, SciStyleStage, a three-stage evaluation environment that applies controlled stylistic perturbations to fixed scientific content across three settings no context, fixed-domain context, and open-domain retrieval context, covering 600 scientific ideas and 15 style variants, with 9,000 evaluation instances per setting; (ii) Second, SciStyleMetrics, a set of quantitative measures, including Style Bias Index (SBI), Substance Recognition Rate (SRR), and Adversarial Win Rate (AWR), to characterize how stylistic variation affects scoring stability, substance discrimination, and ranking robustness; (iii) Third, SciStyleExtractor, a plug-and-play evaluation module that separates presentation style from scientific content by predicting style type and deviation before style-conditioned evaluation, enabling us to assess whether style awareness reduces stylistic bias. Experiments on SciStyleBench show that direct LLM judges remain sensitive to writing style and struggle to distinguish scientific substance. In contrast, SciStyleExtractor reduces SBI from 0.566 to 0.501 while increasing SRR and AWR from 0.504 and 0.554 to 0.759 and 0.899, respectively. These results suggest that robust idea evaluation requires invariance to stylistic variation without sacrificing sensitivity to scientific substance. Overall, SciStyleBench provides a systematic framework for identifying, quantifying, and mitigating stylistic bias in scientific idea evaluation.

cs.CL↗

arXiv2026

Think with Structured Grounding: Perceptual Reinforcement Learning for Chart and Visual-Tabular Understanding

Multimodal Large Language Models (MLLMs) capable of thinking with images often rely on external tools for fine-grained perception. However, this reliance introduces significant inference latency and fails to effectively resolve the spatial-structural gap-a fundamental challenge in text-dense and structurally relational visuals (e.g., charts and visual tables) where strict relative spatial arrangements bind textual elements. Without external tools, standard MLLMs struggle with such fine-grained visual reasoning tasks. To address these issues, we propose Think with Structured Grounding (TwSG), a novel fine-grained image perception framework designed to internalize complex images's tool-use capabilities within the model. TwSG distills the benefits of multi-step reasoning and micro-cropping into a single efficient forward pass during inference. Specifically, we use an MLLM to identify key regions guided by ground-truth answers, and then prompt a teacher model to generate high-quality visual question-answering (VQA) data. These fine-grained, region-based supervisory signals are subsequently distilled back into the full-image representation. Our training pipeline consists of two stages: (1) a cold-start supervised fine-tuning (SFT) phase using multi-turn data with focused area descriptions to foster complex reasoning and error recovery; and (2) a reinforcement fine-tuning (RFT) phase driven by a novel process reward mechanism, TL-GRPO, which encourages strategic reasoning. Extensive experiments across various MLLM architectures demonstrate that TwSG reduces inference latency while substantially improving accuracy and robustness, endowing models with native fine-grained region description and flexible reasoning capabilities.

cs.AI↗

arXiv2026

Seeing Is Not Deciding: Can Multimodal LLMs Act as Effective CEOs?

Large language models are increasingly applied as autonomous decision-making agents. However, in executive business decisions, existing benchmarks are limited to textonly settings. This makes it unclear whether models can perceive visual business evidence and effectively integrate it to improve decision quality. We introduce C-SUITEBENCH, a controlled multimodal benchmark that includes five decision tasks under paired text-only and multimodal conditions across 50 scenarios. We place nine frontier models in the role of a chief executive officer and evaluate their decision-making ability. Multimodal inputs consistently improve evidence-centric reasoning, with the largest and most reliable gains appearing in risk forecasting and board-facing justification. However, we uncover a multimodal integration paradox: adding visual business information degrades constrained resource allocation for all nine models, even as visual grounding itself improves. Ablation experiments reveal that this failure emerges from signal crowding, although each visual channel helps individually, their combination disrupts constraint satisfaction during decoding. These findings demonstrate that visual perception and constrained action are separable bottlenecks in multimodal agents, and that indiscriminate visual augmentation can harm high-stakes decision making, motivating selective grounding strategies for future executive AI systems.

cs.AI↗