arXiv subjects

Andrew Zhu

Publications and source records attributed to Andrew Zhu.

At least 19 recordsLinked to original sources

arXiv2025

Surface adsorption at the thermodynamic limit using periodic DLPNO-MP2 theory: A study of CO on MgO at dilute and dense coverages

We apply periodic domain-based local pair natural orbital second-order M{\o}ller--Plesset perturbation theory (DLPNO-MP2) to probe the adsorption energy of CO on MgO(001), the consensus toy model system for surface adsorption. A number of robust correlated wavefunction methods now achieve excellent agreement with experiment for the adsorption of a single CO molecule onto the MgO surface. However, studies probing denser coverage ratios are scarce because of the increased computational expense and the larger configuration space to optimize. We leverage the computational efficiency of periodic DLPNO-MP2 to perform simulations beyond a single unit cell. By using large supercells, we highlight the importance of accurately representing the thermodynamic limit of the surface, and demonstrate in turn that different coverage ratios can be consistently probed. In the dilute regime, we show that adsorption energies obtained from periodic DLPNO-MP2 agree with existing benchmarks. We then obtain adsorption energies at increasing densities approaching full monolayer coverage. Our results show a reduction in binding strength at full coverage, agreeing with experimental observations, which is explained by the increasing lateral repulsions between the COs. This study demonstrates the efficacy of periodic DLPNO-MP2 for probing increasingly sophisticated adsorption systems at the thermodynamic limit.

physics.chem-ph↗

arXiv2025

BabyVLM-V2: Toward Developmentally Grounded Pretraining and Benchmarking of Vision Foundation Models

Early children's developmental trajectories set up a natural goal for sample-efficient pretraining of vision foundation models. We introduce BabyVLM-V2, a developmentally grounded framework for infant-inspired vision-language modeling that extensively improves upon BabyVLM-V1 through a longitudinal, multifaceted pretraining set, a versatile model, and, most importantly, DevCV Toolbox for cognitive evaluation. The pretraining set maximizes coverage while minimizing curation of a longitudinal, infant-centric audiovisual corpus, yielding video-utterance, image-utterance, and multi-turn conversational data that mirror infant experiences. DevCV Toolbox adapts all vision-related measures of the recently released NIH Baby Toolbox into a benchmark suite of ten multimodal tasks, covering spatial reasoning, memory, and vocabulary understanding aligned with early children's capabilities. Experimental results show that a compact model pretrained from scratch can achieve competitive performance on DevCV Toolbox, outperforming GPT-4o on some tasks. We hope the principled, unified BabyVLM-V2 framework will accelerate research in developmentally plausible pretraining of vision foundation models.

cs.CV↗

arXiv2025

World Simulation with Video Foundation Models for Physical AI

We introduce [Cosmos-Predict2.5], the latest generation of the Cosmos World Foundation Models for Physical AI. Built on a flow-based architecture, [Cosmos-Predict2.5] unifies Text2World, Image2World, and Video2World generation in a single model and leverages [Cosmos-Reason1], a Physical AI vision-language model, to provide richer text grounding and finer control of world simulation. Trained on 200M curated video clips and refined with reinforcement learning-based post-training, [Cosmos-Predict2.5] achieves substantial improvements over [Cosmos-Predict1] in video quality and instruction alignment, with models released at 2B and 14B scales. These capabilities enable more reliable synthetic data generation, policy evaluation, and closed-loop simulation for robotics and autonomous systems. We further extend the family with [Cosmos-Transfer2.5], a control-net style framework for Sim2Real and Real2Real world translation. Despite being 3.5$\times$ smaller than [Cosmos-Transfer1], it delivers higher fidelity and robust long-horizon video generation. Together, these advances establish [Cosmos-Predict2.5] and [Cosmos-Transfer2.5] as versatile tools for scaling embodied intelligence. To accelerate research and deployment in Physical AI, we release source code, pretrained checkpoints, and curated benchmarks under the NVIDIA Open Model License at https://github.com/nvidia-cosmos/cosmos-predict2.5 and https://github.com/nvidia-cosmos/cosmos-transfer2.5. We hope these open resources lower the barrier to adoption and foster innovation in building the next generation of embodied intelligence.

cs.CV↗

arXiv2025

Overhearing LLM Agents: A Survey, Taxonomy, and Roadmap

Imagine AI assistants that enhance conversations without interrupting them: quietly providing relevant information during a medical consultation, seamlessly preparing materials as teachers discuss lesson plans, or unobtrusively scheduling meetings as colleagues debate calendars. While modern conversational LLM agents directly assist human users with tasks through a chat interface, we study this alternative paradigm for interacting with LLM agents, which we call "overhearing agents." Rather than demanding the user's attention, overhearing agents continuously monitor ambient activity and intervene only when they can provide contextual assistance. In this paper, we present the first analysis of overhearing LLM agents as a distinct paradigm in human-AI interaction and establish a taxonomy of overhearing agent interactions and tasks grounded in a survey of works on prior LLM-powered agents and exploratory HCI studies. Based on this taxonomy, we create a list of best practices for researchers and developers building overhearing agent systems. Finally, we outline the remaining research gaps and reveal opportunities for future research in the overhearing paradigm.

cs.CL↗

arXiv2025

DLPNO-MP2 with Periodic Boundary Conditions

We present domain-based local pair natural orbital M{\o}ller--Plesset second order perturbation theory (DLPNO-MP2) with Born--von K{\'a}rm{\'a}n boundary (BvK) conditions. The approach is based on well-localised Wannier functions in a LCAO formalism and extends the molecular DLPNO-MP2 implementation Tubromole program package to periodic systems. The PNOs are formed through a PAO-OSV-PNO cascade, using BvK projected atomic orbitals and orbital specific virtuals as intermediaries in an analogous manner to the molecular scheme. Our chargeless and surface-dipole corrected local density fitting approach is shown to be numerically stable and to ensure convergent lattice summations over the periodic images for the two- and three-index Coulomb integrals. Through careful benchmarking, we show that the DLPNO approximations in the BvK-DLPNO-MP2 methods are entirely consistent with those of molecular DLPNO-MP2 calculations, and with an alternative periodic approach Megacell-DLPNO-MP2, reported in Paper II of this series. Smooth convergence to the canonical correlation energy with tightening PNO threshold is observed. Reference MP2 correlation energies are provided for a set of 2D and 3D periodic systems using a triple-zeta basis and supercell sizes up to 11$\times$11 and 7$\times$7$\times$7.

physics.chem-ph↗

arXiv2025

DLPNO-MP2 for Periodic Systems using Megacell Embedding

We present a domain-based local pair natural orbital M{\o}ller--Plesset second order perturbation theory (DLPNO-MP2) for periodic systems, working within an LCAO formalism within the Tubromole program package. This approach, Megacell-DLPNO-MP2, embeds a supercell correlation treatment within a megacell and does not involve periodic image summation for the Coulomb integrals. Working in a basis of well-localised Wannier functions, periodicity is instead imposed through rigorous translational symmetry of Hamiltonian integrals and wavefunction parameters. The accuracy of the method is validated through comparison with a complementary periodic DLPNO-MP2 method that employs Born--von K{\'a}rm{\'a}n boundary conditions, described in paper I of this series. The PNO approximations are shown to be equivalent in the two approaches and entirely consistent with molecular DLPNO-MP2 calculations. The Megacell-DLPNO-MP2 method displays sub-linear scaling with respect to supercell size at the asymptotic limit and example calculations are presented with up to 15000 basis functions in the correlation treatment.

physics.chem-ph↗

arXiv2025

PAL: Designing Conversational Agents as Scalable, Cooperative Patient Simulators for Palliative-Care Training

Effective communication in serious illness and palliative care is essential but often under-taught due to limited access to training resources like standardized patients. We present PAL (Palliative Assisted Learning-bot), a conversational system that simulates emotionally nuanced patient interactions and delivers structured feedback grounded in an existing empathy-based framework. PAL supports text and voice modalities and is designed to scaffold clinical skill-building through repeated, low-cost practice. Through a mixed-methods study with 17 U.S. medical trainees and clinicians, we explore user engagement with PAL, evaluate usability, and examine design tensions around modalities, emotional realism, and feedback delivery. Participants found PAL helpful for reflection and skill refinement, though some noted limitations in emotional authenticity and the adaptability of feedback. We contribute: (1) empirical evidence that large language models can support palliative communication training; (2) design insights for modality-aware, emotionally sensitive simulation tools; and (3) implications for systems that support emotional labor, cooperative learning, and AI-augmented training in high-stakes care settings.

cs.HC↗

arXiv2025

Improved Accuracy in Pelvic Tumor Resections Using a Real-Time Vision-Guided Surgical System

Pelvic bone tumor resections remain significantly challenging due to complex three-dimensional anatomy and limited surgical visualization. Current navigation systems and patient-specific instruments, while accurate, present limitations including high costs, radiation exposure, workflow disruption, long production time, and lack of reusability. This study evaluates a real-time vision-guided surgical system combined with modular jigs to improve accuracy in pelvic bone tumor resections. A vision-guided surgical system combined with modular cutting jigs and real-time optical tracking was developed and validated. Five female pelvis sawbones were used, with each hemipelvis randomly assigned to either the vision-guided and modular jig system or traditional freehand method. A total of twenty resection planes were analyzed for each method. Accuracy was assessed by measuring distance and angular deviations from the planned resection planes. The vision-guided and modular jig system significantly improved resection accuracy compared to the freehand method, reducing the mean distance deviation from 2.07 $\pm$ 1.71 mm to 1.01 $\pm$ 0.78 mm (p=0.0193). In particular, all specimens resected using the vision-guided system exhibited errors of less than 3 mm. Angular deviations also showed significant improvements with roll angle deviation reduced from 15.36 $\pm$ 17.57$^\circ$ to 4.21 $\pm$ 3.46$^\circ$ (p=0.0275), and pitch angle deviation decreased from 6.17 $\pm$ 4.58$^\circ$ to 1.84 $\pm$ 1.48$^\circ$ (p<0.001). The proposed vision-guided and modular jig system significantly improves the accuracy of pelvic bone tumor resections while maintaining workflow efficiency. This cost-effective solution provides real-time guidance without the need for referencing external monitors, potentially improving surgical outcomes in complex pelvic bone tumor cases.

eess.IV↗

arXiv2025

First Steps Towards Overhearing LLM Agents: A Case Study With Dungeons & Dragons Gameplay

Much work has been done on conversational LLM agents which directly assist human users with tasks. We present an alternative paradigm for interacting with LLM agents, which we call "overhearing agents". These overhearing agents do not actively participate in conversation -- instead, they "listen in" on human-to-human conversations and perform background tasks or provide suggestions to assist the user. In this work, we explore the overhearing agents paradigm through the lens of Dungeons & Dragons gameplay. We present an in-depth study using large multimodal audio-language models as overhearing agents to assist a Dungeon Master. We perform a human evaluation to examine the helpfulness of such agents and find that some large audio-language models have the emergent ability to perform overhearing agent tasks using implicit audio cues. Finally, we release Python libraries and our project code to support further research into the overhearing agents paradigm at https://github.com/zhudotexe/overhearing_agents.

cs.CL↗

arXiv2025

Unifying Inference-Time Planning Language Generation

A line of work in planning uses LLM not to generate a plan, but to generate a formal representation in some planning language, which can be input into a symbolic solver to deterministically find a plan. While showing improved trust and promising performance, dozens of recent publications have proposed scattered methods on a variety of benchmarks under different experimental settings. We attempt to unify the inference-time LLM-as-formalizer methodology for classical planning by proposing a unifying framework based on intermediate representations. We thus systematically evaluate more than a dozen pipelines that subsume most existing work, while proposing novel ones that involve syntactically similar but high resource intermediate languages (such as a Python wrapper of PDDL). We provide recipes for planning language generation pipelines, draw a series of conclusions showing the efficacy of their various components, and evidence their robustness against problem complexity.

cs.CL↗

arXiv2025

GenAI Content Detection Task 3: Cross-Domain Machine-Generated Text Detection Challenge

Recently there have been many shared tasks targeting the detection of generated text from Large Language Models (LLMs). However, these shared tasks tend to focus either on cases where text is limited to one particular domain or cases where text can be from many domains, some of which may not be seen during test time. In this shared task, using the newly released RAID benchmark, we aim to answer whether or not models can detect generated text from a large, yet fixed, number of domains and LLMs, all of which are seen during training. Over the course of three months, our task was attempted by 9 teams with 23 detector submissions. We find that multiple participants were able to obtain accuracies of over 99% on machine-generated text from RAID while maintaining a 5% False Positive Rate -- suggesting that detectors are able to robustly detect text from many domains and models simultaneously. We discuss potential interpretations of this result and provide directions for future research.

cs.CL↗

arXiv2024

You Have Thirteen Hours in Which to Solve the Labyrinth: Enhancing AI Game Masters with Function Calling

Developing a consistent and reliable AI game master for text-based games is a challenging task due to the limitations of large language models (LLMs) and the complexity of the game master's role. This paper presents a novel approach to enhance AI game masters by leveraging function calling in the context of the table-top role-playing game "Jim Henson's Labyrinth: The Adventure Game." Our methodology involves integrating game-specific controls through functions, which we show improves the narrative quality and state update consistency of the AI game master. The experimental results, based on human evaluations and unit tests, demonstrate the effectiveness of our approach in enhancing gameplay experience and maintaining coherence with the game state. This work contributes to the advancement of game AI and interactive storytelling, offering insights into the design of more engaging and consistent AI-driven game masters.

cs.CL↗

arXiv2024

ReDel: A Toolkit for LLM-Powered Recursive Multi-Agent Systems

Recently, there has been increasing interest in using Large Language Models (LLMs) to construct complex multi-agent systems to perform tasks such as compiling literature reviews, drafting consumer reports, and planning vacations. Many tools and libraries exist for helping create such systems, however none support recursive multi-agent systems -- where the models themselves flexibly decide when to delegate tasks and how to organize their delegation structure. In this work, we introduce ReDel: a toolkit for recursive multi-agent systems that supports custom tool-use, delegation schemes, event-based logging, and interactive replay in an easy-to-use web interface. We show that, using ReDel, we are able to easily identify potential areas of improvements through the visualization and debugging tools. Our code, documentation, and PyPI package are open-source and free to use under the MIT license at https://github.com/zhudotexe/redel.

cs.CL↗

arXiv2024

Wannier function localisation using Bloch intrinsic atomic orbitals

We extend the Intrinsic Atomic Orbital (IAO) method for localisation of molecular orbitals to calculate well-localised generalised Wannier functions in crystals using the Pipek--Mezey locality metric. We furthermore present a one-shot diabatic Wannierisation procedure that aligns the phases of the Bloch functions, providing immediate Wannier localisation, which serves as an excellent initial guess for optimisation. We test our Wannier localisation implementation on a number of solid state systems, highlighting the effectiveness of the diabatic preparation, especially for localising core bands. Partial charges of Wannier functions generated using Bloch IAOs align well with chemical intuition, which we demonstrate through the example of adsorption of CO on a MgO surface.

cond-mat.mtrl-sci↗

arXiv2024

RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors

Many commercial and open-source models claim to detect machine-generated text with extremely high accuracy (99% or more). However, very few of these detectors are evaluated on shared benchmark datasets and even when they are, the datasets used for evaluation are insufficiently challenging-lacking variations in sampling strategy, adversarial attacks, and open-source generative models. In this work we present RAID: the largest and most challenging benchmark dataset for machine-generated text detection. RAID includes over 6 million generations spanning 11 models, 8 domains, 11 adversarial attacks and 4 decoding strategies. Using RAID, we evaluate the out-of-domain and adversarial robustness of 8 open- and 4 closed-source detectors and find that current detectors are easily fooled by adversarial attacks, variations in sampling strategies, repetition penalties, and unseen generative models. We release our data along with a leaderboard to encourage future research.

cs.CL↗

arXiv2024

FanOutQA: A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Models

One type of question that is commonly found in day-to-day scenarios is ``fan-out'' questions, complex multi-hop, multi-document reasoning questions that require finding information about a large number of entities. However, there exist few resources to evaluate this type of question-answering capability among large language models. To evaluate complex reasoning in LLMs more fully, we present FanOutQA, a high-quality dataset of fan-out question-answer pairs and human-annotated decompositions with English Wikipedia as the knowledge base. We formulate three benchmark settings across our dataset and benchmark 7 LLMs, including GPT-4, LLaMA 2, Claude-2.1, and Mixtral-8x7B, finding that contemporary models still have room to improve reasoning over inter-document dependencies in a long context. We provide our dataset and open-source tools to run models to encourage evaluation at https://fanoutqa.com

cs.CL↗

arXiv2023

Kani: A Lightweight and Highly Hackable Framework for Building Language Model Applications

Language model applications are becoming increasingly popular and complex, often including features like tool usage and retrieval augmentation. However, existing frameworks for such applications are often opinionated, deciding for developers how their prompts ought to be formatted and imposing limitations on customizability and reproducibility. To solve this we present Kani: a lightweight, flexible, and model-agnostic open-source framework for building language model applications. Kani helps developers implement a variety of complex features by supporting the core building blocks of chat interaction: model interfacing, chat management, and robust function calling. All Kani core functions are easily overridable and well documented to empower developers to customize functionality for their own needs. Kani thus serves as a useful tool for researchers, hobbyists, and industry professionals alike to accelerate their development while retaining interoperability and fine-grained control.

cs.SE↗

arXiv2023

CALYPSO: LLMs as Dungeon Masters' Assistants

The role of a Dungeon Master, or DM, in the game Dungeons & Dragons is to perform multiple tasks simultaneously. The DM must digest information about the game setting and monsters, synthesize scenes to present to other players, and respond to the players' interactions with the scene. Doing all of these tasks while maintaining consistency within the narrative and story world is no small feat of human cognition, making the task tiring and unapproachable to new players. Large language models (LLMs) like GPT-3 and ChatGPT have shown remarkable abilities to generate coherent natural language text. In this paper, we conduct a formative evaluation with DMs to establish the use cases of LLMs in D&D and tabletop gaming generally. We introduce CALYPSO, a system of LLM-powered interfaces that support DMs with information and inspiration specific to their own scenario. CALYPSO distills game context into bite-sized prose and helps brainstorm ideas without distracting the DM from the game. When given access to CALYPSO, DMs reported that it generated high-fidelity text suitable for direct presentation to players, and low-fidelity ideas that the DM could develop further while maintaining their creative agency. We see CALYPSO as exemplifying a paradigm of AI-augmented tools that provide synchronous creative assistance within established game worlds, and tabletop gaming more broadly.

cs.CL↗