arXiv subjects

Shuai Fu

Publications and source records attributed to Shuai Fu.

At least 19 recordsLinked to original sources

arXiv2026

Automating the Design of Embodied Agent Architectures

Embodied agents are typically built as hand-designed compositions of perception, memory, planning, and action modules. This modularity exposes a large architectural design space, but current systems still rely on researcher intuition to choose where information is stored, how observations are processed, and how model calls are connected. Agent Architecture Search (AAS) automates such design for text-domain agents, but has not been systematically evaluated on perceptual embodied agents through simulator rollouts. We study this transfer. We introduce AgentCanvas, a typed-graph runtime that hosts embodied executors as editable node-and-wire programs with simulator-aware execution and episode-level logs, and KDLoop, a coding-agent search procedure that cycles through proposal, critique, experiment, and distillation, with triggered reflection after stalls. We evaluate three AAS variants across four embodied executors spanning vision-language navigation, embodied question answering, and language-conditioned manipulation. The resulting 3x4 matrix shows that architecture-level search can produce deployable and directional success-rate gains on embodied tasks, while one apparent high-scoring candidate is rejected as leak-bearing. At the same time, the experiments expose constraints that are muted in text-domain AAS: optimization signals can be masked by rollout noise, search can become trapped in local edit basins, and episode-level credit assignment only partially emerges even when detailed logs are available. These results characterize both the promise and the current limits of automated architecture search for embodied agents.

cs.RO↗

arXiv2026

Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence

While Large Language Models (LLMs) have substantially advanced text-to-code generation, many real programming tasks specify intent through visual artifacts such as screenshots, charts, and videos. These tasks require models to connect visual perception to executable programs, as correctness depends not only on syntax but also on layout, data semantics, and domain-specific constraints that apply after execution. This survey reviews Multimodal Code Intelligence, covering systems that generate, edit, refine, or reason with code under visually grounded inputs and outputs. We first formulate the field by the role that code plays in each task, distinguishing code as a rendered artifact, an editable structure, an intermediate reasoning trace, or an executable tool interface. Then we organize benchmarks and methods into four domains: Graphical User Interface, Scientific Visualization, Structured Graphics, and Frontier Tasks and Frameworks. This taxonomy connects artifact-generation problems to agentic and unified settings and allows us to compare how different tasks treat evidence of correctness. Across the literature, we argue that reliable evaluation requires evidence about semantics and interaction beyond visual fidelity. Looking ahead, future research may benefit from four verification-centered directions. Multi-signal validation can combine complementary evidence of correctness, multi-state verification can test behavior across execution trajectories, cross-task transfer testing can probe reusable visual-code skills, and verifiable agent traces can reveal whether agent actions are grounded in visual evidence. Together, these directions may move this field from single-output imitation toward evidence-grounded executable systems. An ongoing project and resources are available on \href{https://github.com/xjywhu/Awesome-Multimodal-LLM-for-Code}{GitHub}.

cs.CL↗

arXiv2026

Probing Solar Wind Structures with Solar Energetic Particle Observations from Solar Orbiter

The propagation of solar energetic particles (SEPs) through the heliosphere is primarily guided by the interplanetary magnetic field (IMF) which is embedded in the solar wind plasma. Large-scale IMF structures can drive transient variations in SEP intensities. Using Solar Orbiter observations, we identify a distinct class of SEP variations: SEP flux deflections (SFDs), which are commonly detected in SEP events and frequently recur multiple times within a single event. SFDs are characterized by a sudden change in SEP flux directions where the intensities drop in one direction and increase in another direction, without a significant net change in total flux magnitude. These deflections occur dispersionlessly across a broad energy range-from tens of keV to over 100 MeV-and exhibit steep intensity gradients. SFDs are typically associated with magnetic flux tubes with boundary features consistent with tangential discontinuities. We further show that the solar wind inside these structures exhibits distinct plasma properties, and that the SEP streaming direction within SFDs aligns closely to the flux-tube axis. These observations suggest that magnetic flux tubes are a prevalent structural element of the solar wind, and demonstrate that SEPs can serve as an effective diagnostic tool for probing the topology and dynamics of solar wind structures.

astro-ph.SR↗

arXiv2026

Suppressed excitonic effects enable high mobility, high-yield photoconductivity in a two-dimensional polymer crystal with axial pyridine coordination

Two-dimensional polymers (2DPs) and their layer-stacked covalent organic frameworks (2D COFs) offer modular, atomically precise platforms for organic optoelectronics, yet their photoconductive responses remain fundamentally constrained by strong excitonic effects and localized charge transport. Here, we demonstrate that a diyne-linked 2DP crystal with axial pyridine coordination overcomes this limitation, enabling simultaneous efficient free-carrier generation and band-like transport. Introducing pyridine ligands that axially coordinate to Cu-porphyrin nodes transforms weak van der Waals stacking into a pyridine-bridged architecture with pronounced interlayer band dispersion and substantially reduced carrier effective masses. The resulting strong interlayer electronic coupling suppresses the exciton binding energy to well below thermal energy, such that optical excitation directly populates delocalized electronic states. Time-resolved terahertz spectroscopy reveals Drude-type photoconductivity with room-temperature mobilities approaching 500 cm^2 V^-1 s^-1 and a photon-to-free-carrier conversion ratio of ~0.4, yielding a photoconductive response that exceeds that of state-of-the-art organic and many inorganic photoactive materials. These results establish interlayer coordination as a powerful strategy for mitigating excitonic effects and accessing inorganic-like charge transport in organic 2D crystals, opening a pathway toward highly efficient photo-to-electricity conversion in organic-based systems.

cond-mat.mtrl-sci↗

arXiv2025

Decoupled Action Expert: Confining Task Knowledge to the Conditioning Pathway

Many recent Vision-Language-Action models employ diffusion or flow-matching backbones with hundreds of millions of parameters for action generation. However, unlike image synthesis where the output spans millions of diverse pixels, a manipulation policy generates only short sequences of low-dimensional, physically correlated action values, a far simpler target that should not demand such capacity. We confirm this intuition and show that task-specific knowledge in these policies can be fully confined to the conditioning pathway, leaving the action backbone task-agnostic. To establish this, we introduce a decoupled training recipe: a general-purpose action head is first pretrained on observation-free forward-kinematics data, then frozen while only the conditioning pathway is trained for downstream tasks. Using Diffusion Policy as a testbed, we show that on both MimicGen and LIBERO, a single frozen backbone shared across all tasks matches normally trained counterparts. This confirms that the action expert encodes little task-specific knowledge. Ablations show that the specific pretraining signal (joint positions, end-effector poses, or no conditioning at all) has no effect on downstream performance, indicating that the backbone learns only general trajectory structure. Pushing this finding further, we replace the 244M U-Net in Diffusion Policy with a 5M-parameter MLP backbone that matches or exceeds its performance, calling into question the large capacity budgets allocated to action generation in current VLA designs.

cs.RO↗

arXiv2025

Counting Hallucinations in Diffusion Models

Diffusion probabilistic models (DPMs) have demonstrated remarkable progress in generative tasks, such as image and video synthesis. However, they still often produce hallucinated samples (hallucinations) that conflict with real-world knowledge, such as generating an implausible duplicate cup floating beside another cup. Despite their prevalence, the lack of feasible methodologies for systematically quantifying such hallucinations hinders progress in addressing this challenge and obscures potential pathways for designing next-generation generative models under factual constraints. In this work, we bridge this gap by focusing on a specific form of hallucination, which we term counting hallucination, referring to the generation of an incorrect number of instances or structured objects, such as a hand image with six fingers, despite such patterns being absent from the training data. To this end, we construct a dataset suite CountHalluSet, with well-defined counting criteria, comprising ToyShape, SimObject, and RealHand. Using these datasets, we develop a standardized evaluation protocol for quantifying counting hallucinations, and systematically examine how different sampling conditions in DPMs, including solver type, ODE solver order, sampling steps, and initial noise, affect counting hallucination levels. Furthermore, we analyze their correlation with common evaluation metrics such as FID, revealing that this widely used image quality metric fails to capture counting hallucinations consistently. This work aims to take the first step toward systematically quantifying hallucinations in diffusion models and offer new insights into the investigation of hallucination phenomena in image generation.

cs.CV↗

arXiv2025

ScaleTrack: Scaling and back-tracking Automated GUI Agents

Automated GUI agents aims to facilitate user interaction by automatically performing complex tasks in digital environments, such as web, mobile, desktop devices. It receives textual task instruction and GUI description to generate executable actions (\emph{e.g.}, click) and operation boxes step by step. Training a GUI agent mainly involves grounding and planning stages, in which the GUI grounding focuses on finding the execution coordinates according to the task, while the planning stage aims to predict the next action based on historical actions. However, previous work suffers from the limitations of insufficient training data for GUI grounding, as well as the ignorance of backtracking historical behaviors for GUI planning. To handle the above challenges, we propose ScaleTrack, a training framework by scaling grounding and backtracking planning for automated GUI agents. We carefully collected GUI samples of different synthesis criterions from a wide range of sources, and unified them into the same template for training GUI grounding models. Moreover, we design a novel training strategy that predicts the next action from the current GUI image, while also backtracking the historical actions that led to the GUI image. In this way, ScaleTrack explains the correspondence between GUI images and actions, which effectively describes the evolution rules of the GUI environment. Extensive experimental results demonstrate the effectiveness of ScaleTrack. Data and code will be available at url.

cs.AI↗

arXiv2025

Beyond the band edge: Unveiling high-mobility hot carriers in a two-dimensional conjugated coordination polymer

Hot carriers, inheriting excess kinetic energy from high-energy photons, underpin numerous optoelectronic applications involving non-equilibrium transport processes. Current research on hot carriers has predominantly focused on inorganic materials, with little attention paid to organic-based systems due to their ultrafast energy relaxation and inefficient charge transport. Here, we overturn this paradigm by demonstrating highly mobile hot carriers in solution-processable, highly crystalline two-dimensional conjugated coordination polymer (2D c-CP) Cu3BHT (BHT = benzenehexathiol) films. Leveraging a suite of ultrafast spectroscopic and imaging techniques, we unravel the microscopic charge transport landscape in Cu3BHT films following non-equilibrium photoexcitation across temporal, spatial, and frequency domains, revealing two distinct high-mobility transport regimes. In the non-equilibrium transport regime, hot carriers achieve ultrahigh mobility of ~2,000 cm2 V-1 s-1, traversing grain boundaries up to 300 nm within a picosecond. In the quasi-equilibrium transport regime, free carriers exhibit Drude-type band-like transport with a remarkable mobility of ~400 cm2 V-1 s-1 and an intrinsic diffusion length exceeding 1 micrometer. These findings establish 2D c-CPs as versatile platforms for exploring high-mobility non-equilibrium transport, unlocking new opportunities for organic-based hot carrier applications.

cond-mat.mtrl-sci↗

arXiv2024

MoPD: Mixture-of-Prompts Distillation for Vision-Language Models

Soft prompt learning methods are effective for adapting vision-language models (VLMs) to downstream tasks. Nevertheless, empirical evidence reveals a tendency of existing methods that they overfit seen classes and exhibit degraded performance on unseen classes. This limitation is due to the inherent bias in the training data towards the seen classes. To address this issue, we propose a novel soft prompt learning method, named Mixture-of-Prompts Distillation (MoPD), which can effectively transfer useful knowledge from hard prompts manually hand-crafted (a.k.a. teacher prompts) to the learnable soft prompt (a.k.a. student prompt), thereby enhancing the generalization ability of soft prompts on unseen classes. Moreover, the proposed MoPD method utilizes a gating network that learns to select hard prompts used for prompt distillation. Extensive experiments demonstrate that the proposed MoPD method outperforms state-of-the-art baselines especially on on unseen classes.

cs.CV↗

arXiv2024

Nemesis: Normalizing the Soft-prompt Vectors of Vision-Language Models

With the prevalence of large-scale pretrained vision-language models (VLMs), such as CLIP, soft-prompt tuning has become a popular method for adapting these models to various downstream tasks. However, few works delve into the inherent properties of learnable soft-prompt vectors, specifically the impact of their norms to the performance of VLMs. This motivates us to pose an unexplored research question: ``Do we need to normalize the soft prompts in VLMs?'' To fill this research gap, we first uncover a phenomenon, called the \textbf{Low-Norm Effect} by performing extensive corruption experiments, suggesting that reducing the norms of certain learned prompts occasionally enhances the performance of VLMs, while increasing them often degrades it. To harness this effect, we propose a novel method named \textbf{N}ormalizing th\textbf{e} soft-pro\textbf{m}pt v\textbf{e}ctors of vi\textbf{si}on-language model\textbf{s} (\textbf{Nemesis}) to normalize soft-prompt vectors in VLMs. To the best of our knowledge, our work is the first to systematically investigate the role of norms of soft-prompt vector in VLMs, offering valuable insights for future research in soft-prompt tuning. The code is available at \texttt{\href{https://github.com/ShyFoo/Nemesis}{https://github.com/ShyFoo/Nemesis}}.

cs.CV↗

arXiv2024

Enhancing Sharpness-Aware Minimization by Learning Perturbation Radius

Sharpness-aware minimization (SAM) is to improve model generalization by searching for flat minima in the loss landscape. The SAM update consists of one step for computing the perturbation and the other for computing the update gradient. Within the two steps, the choice of the perturbation radius is crucial to the performance of SAM, but finding an appropriate perturbation radius is challenging. In this paper, we propose a bilevel optimization framework called LEarning the perTurbation radiuS (LETS) to learn the perturbation radius for sharpness-aware minimization algorithms. Specifically, in the proposed LETS method, the upper-level problem aims at seeking a good perturbation radius by minimizing the squared generalization gap between the training and validation losses, while the lower-level problem is the SAM optimization problem. Moreover, the LETS method can be combined with any variant of SAM. Experimental results on various architectures and benchmark datasets in computer vision and natural language processing demonstrate the effectiveness of the proposed LETS method in improving the performance of SAM.

cs.LG↗

arXiv2024

Learning Retrieval Augmentation for Personalized Dialogue Generation

Personalized dialogue generation, focusing on generating highly tailored responses by leveraging persona profiles and dialogue context, has gained significant attention in conversational AI applications. However, persona profiles, a prevalent setting in current personalized dialogue datasets, typically composed of merely four to five sentences, may not offer comprehensive descriptions of the persona about the agent, posing a challenge to generate truly personalized dialogues. To handle this problem, we propose $\textbf{L}$earning Retrieval $\textbf{A}$ugmentation for $\textbf{P}$ersonalized $\textbf{D}$ial$\textbf{O}$gue $\textbf{G}$eneration ($\textbf{LAPDOG}$), which studies the potential of leveraging external knowledge for persona dialogue generation. Specifically, the proposed LAPDOG model consists of a story retriever and a dialogue generator. The story retriever uses a given persona profile as queries to retrieve relevant information from the story document, which serves as a supplementary context to augment the persona profile. The dialogue generator utilizes both the dialogue history and the augmented persona profile to generate personalized responses. For optimization, we adopt a joint training framework that collaboratively learns the story retriever and dialogue generator, where the story retriever is optimized towards desired ultimate metrics (e.g., BLEU) to retrieve content for the dialogue generator to generate personalized responses. Experiments conducted on the CONVAI2 dataset with ROCStory as a supplementary data source show that the proposed LAPDOG method substantially outperforms the baselines, indicating the effectiveness of the proposed method. The LAPDOG model code is publicly available for further exploration. https://github.com/hqsiswiliam/LAPDOG

cs.CL↗

arXiv2024

Goanna: Resolving Haskell Type Errors With Minimal Correction Subsets

Statically typed languages offer significant advantages, such as bug prevention, enhanced code quality, and reduced maintenance costs. However, these benefits often come at the expense of a steep learning curve and a slower development pace. Haskell, known for its expressive and strict type system, poses challenges for inexperienced programmers in learning and using its type system, especially in debugging type errors. We introduce Goanna, a novel tool that serves as a type checker and an interactive type error debugging tool for Haskell. When encountering type errors, Goanna identifies a comprehensive list of potential causes and resolutions based on the minimum correction subsets (MCS) enumeration. We evaluated Goanna's effectiveness using 86 diverse Haskell programs from online discourse, demonstrating its ability to accurately identify and resolve type errors. Additionally, we present a collection of techniques and heuristics to enhance Goanna's suggestion-based error diagnosis and show their effectiveness from our evaluation.

cs.HC↗

arXiv2024

GeckoGraph: A Visual Language for Polymorphic Types

Polymorphic types are an important feature in most strongly typed programming languages. They allow functions to be written in a way that can be used with different data types, while still enforcing the relationship and constraints between the values. However, programmers often find polymorphic types difficult to use and understand and tend to reason using concrete types. We propose GeckoGraph, a graphical notation for types. GeckoGraph aims to accompany traditional text-based type notation and to make reading, understanding, and comparing types easier. We conducted a large-scale human study using GeckoGraph compared to text-based type notation. To our knowledge, this is the largest controlled user study on functional programming ever conducted. The results of the study show that GeckoGraph helps improve programmers' ability to succeed in the programming tasks we designed, especially for novice programmers.

cs.PL↗

arXiv2024

Anisotropic electron-phonon interactions in 2D lead-halide perovskites

Two-dimensional hybrid organic-inorganic metal halide perovskites offer enhanced stability for perovskite-based applications. Their crystal structure's soft and ionic nature gives rise to strong interactions between charge carriers and ionic rearrangements. Here, we investigate the interaction of photo-generated electrons and ionic polarizations in single-crystal 2D perovskite butylammonium lead iodide, varying the inorganic lammelae thickness in the 2D single crystals. We determined the directionality of the transition dipole moments of the relevant phonon modes (in the 0.3-3 THz range) by angle-and-polarization dependent THz transmission measurements. We find a clear anisotropy of the in-plane photoconductivity, with a 10% reduction along the axis parallel with the transition dipole moment of the most strongly coupled phonon. Detailed calculations, based on Feynman polaron theory, indicate that the anisotropy originates from directional electron-phonon interactions.

cond-mat.mtrl-sci↗

arXiv2024

Identifying the Origin of Thermal Modulation of Exchange Bias in MnPS3/Fe3GeTe2 van der Waals Heterostructures

The exchange bias phenomenon, inherent in exchange-coupled ferromagnetic and antiferromagnetic systems, has intrigued researchers for decades. Van der Waals materials, with their layered structure, provide an optimal platform for probing such physical phenomena. However, achieving a facile and effective means to manipulate exchange bias in van der Waals heterostructures remains challenging. In this study, we investigate the origin of exchange bias in MnPS3/Fe3GeTe2 van der Waals heterostructures. Our work demonstrates a method to modulate unidirectional exchange anisotropy, achieving an unprecedented nearly 1000% variation through simple thermal cycling. Despite the compensated interfacial spin configuration of MnPS3, magneto-transport measurements reveal a huge 170 mT exchange bias at 5 K, one of the largest observed in van der Waals antiferromagnet-ferromagnet interfaces. This substantial magnitude of the exchange bias is linked to an anomalous weak ferromagnetic ordering in MnPS3 below 40 K. On the other hand, the tunability of exchange bias during thermal cycling is ascribed to the modified arrangement of interfacial atoms and changes in the vdW gap during field cooling. Our findings highlight a robust and easily adjustable exchange bias in van der Waals antiferromagnetic/ferromagnetic heterostructures, presenting a straightforward approach to enhance other interface-related spintronic phenomena for practical applications. A detailed study of the interface reveals migration of atoms between the layers, leading to the formation of amorphous region on either side of the van der Waals gap, underscoring the importance of precise characterization of interfaces in van der Waals heterostructures, which are often presumed to have pristine interfaces.

cond-mat.mtrl-sci↗

arXiv2024

GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoning

Large Language Models (LLMs) are increasingly used for various tasks with graph structures. Though LLMs can process graph information in a textual format, they overlook the rich vision modality, which is an intuitive way for humans to comprehend structural information and conduct general graph reasoning. The potential benefits and capabilities of representing graph structures as visual images (i.e., $\textit{visual graph}$) are still unexplored. To fill the gap, we innovatively propose an end-to-end framework, called $\textbf{G}$raph to v$\textbf{I}$sual and $\textbf{T}$extual Integr$\textbf{A}$tion (GITA), which firstly incorporates visual graphs into general graph reasoning. Besides, we establish $\textbf{G}$raph-based $\textbf{V}$ision-$\textbf{L}$anguage $\textbf{Q}$uestion $\textbf{A}$nswering (GVLQA) dataset from existing graph data, which is the first vision-language dataset for general graph reasoning purposes. Extensive experiments on the GVLQA dataset and five real-world datasets show that GITA outperforms mainstream LLMs in terms of general graph reasoning capabilities. Moreover, We highlight the effectiveness of the layout augmentation on visual graphs and pretraining on the GVLQA dataset.

cs.CL↗

arXiv2023

Does Misclassifying Non-confounding Covariates as Confounders Affect the Causal Inference within the Potential Outcomes Framework?

The Potential Outcome Framework (POF) plays a prominent role in the field of causal inference. Most causal inference models based on the POF (CIMs-POF) are designed for eliminating confounding bias and default to an underlying assumption of Confounding Covariates. This assumption posits that the covariates consist solely of confounders. However, the assumption of Confounding Covariates is challenging to maintain in practice, particularly when dealing with high-dimensional covariates. While certain methods have been proposed to differentiate the distinct components of covariates prior to conducting causal inference, the consequences of treating non-confounding covariates as confounders remain unclear. This ambiguity poses a potential risk when conducting causal inference in practical scenarios. In this paper, we present a unified graphical framework for the CIMs-POF, which greatly enhances the comprehension of these models' underlying principles. Using this graphical framework, we quantitatively analyze the extent to which the inference performance of CIMs-POF is influenced when incorporating various types of non-confounding covariates, such as instrumental variables, mediators, colliders, and adjustment variables. The key findings are: in the task of eliminating confounding bias, the optimal scenario is for the covariates to exclusively encompass confounders; in the subsequent task of inferring counterfactual outcomes, the adjustment variables contribute to more accurate inferences. Furthermore, extensive experiments conducted on synthetic datasets consistently validate these theoretical conclusions.

stat.ME↗