arXiv subjects

Harsh Trivedi

Publications and source records attributed to Harsh Trivedi.

At least 19 recordsLinked to original sources

arXiv2026

AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use

Tool-use agents that address day-to-day digital tasks such as ordering groceries must not only operate applications, but also interact with the user, e.g., to ask clarification questions, prompt for confirmation, and inform the user when the instruction is infeasible. However, current benchmarks for evaluating agent-user interactions do not capture the diversity of such interactions. Further, they operate in small environments with few, often non-state-changing, APIs. To address this gap, we introduce AppWorld-UL, a ``user-in-the-loop'' benchmark of 516 challenging tasks requiring diverse agent-user interactions. Building upon the AppWorld framework with 9 popular simulated apps like Amazon and Spotify, we systematically modify original tasks to introduce ambiguities and constraints that necessitate various types of agent-user interaction. User behavior is simulated by an LLM prompted to respond with carefully designed knowledge boundaries, offering more reliable simulation than the unconstrained or overly rigid alternatives used in prior work. Our evaluation reveals that a state-of-the-art LLM, Claude Opus 4.7, achieves only 48.6% success on AppWorld-UL, and only 35.7% on the harder, compositional subset. On the stricter, scenario-level metric, compositional task performance drops to only 21.3%. Our analysis reveals that correct user-interaction is crucial for success. This demonstrates the benchmark's difficulty and its potential to advance research on user-in-the-loop tool-use agents.

cs.AI↗

arXiv2026

MolmoWeb: Open Visual Web Agent and Open Data for the Open Web

Web agents--autonomous systems that navigate and execute tasks on the web on behalf of users--have the potential to transform how people interact with the digital world. However, the most capable web agents today rely on proprietary models with undisclosed training data and recipes, limiting scientific understanding, reproducibility, and community-driven progress. We believe agents for the open web should be built in the open. To this end, we introduce (1) MolmoWebMix, a large and diverse mixture of browser task demonstrations and web-GUI perception data and (2) MolmoWeb, a family of fully open multimodal web agents. Specifically, MolmoWebMix combines over 100K synthetic task trajectories from multiple complementary generation pipelines with 30K+ human demonstrations, atomic web-skill trajectories, and GUI perception data, including referring expression grounding and screenshot question answering. MolmoWeb agents operate as instruction-conditioned visual-language action policies: given a task instruction and a webpage screenshot, they predict the next browser action, requiring no access to HTML, accessibility trees, or specialized APIs. Available in 4B and 8B size, on browser-use benchmarks like WebVoyager, Online-Mind2Web, and DeepShop, MolmoWeb agents achieve state-of-the-art results outperforming similar scale open-weight-only models such as Fara-7B, UI-Tars-1.5-7B, and Holo1-7B. MolmoWeb-8B also surpasses set-of-marks (SoM) agents built on much larger closed frontier models like GPT-4o. We further demonstrate consistent gains through test-time scaling via parallel rollouts with best-of-N selection, achieving 94.7% and 60.5% pass@4 (compared to 78.2% and 35.3% pass@1) on WebVoyager and Online-Mind2Web respectively. We will release model checkpoints, training data, code, and a unified evaluation harness to enable reproducibility and accelerate open research on web agents.

cs.CV↗

arXiv2026

Wold-type decomposition for doubly twisted left-invertible covariant representations

In this article, we have introduced the notion of a near-isometric covariant representation of a $C^*$-correspondence. The other objective is to provide a unified approach to several known results for a large class of left-invertible covariant representations of a product system and prove Wold-type decomposition for the case of doubly twisted left-invertible covariant representations and study some applications.

math.OA↗

arXiv2026

Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces

AI agents may soon become capable of autonomously completing valuable, long-horizon tasks in diverse domains. Current benchmarks either do not measure real-world tasks, or are not sufficiently difficult to meaningfully measure frontier models. To this end, we present Terminal-Bench 2.0: a carefully curated hard benchmark composed of 89 tasks in computer terminal environments inspired by problems from real workflows. Each task features a unique environment, human-written solution, and comprehensive tests for verification. We show that frontier models and agents score less than 65\% on the benchmark and conduct an error analysis to identify areas for model and agent improvement. We publish the dataset and evaluation harness to assist developers and researchers in future work at https://www.tbench.ai/ .

cs.SE↗

arXiv2025

Olmo 3

We introduce Olmo 3, a family of state-of-the-art, fully-open language models at the 7B and 32B parameter scales. Olmo 3 model construction targets long-context reasoning, function calling, coding, instruction following, general chat, and knowledge recall. This release includes the entire model flow, i.e., the full lifecycle of the family of models, including every stage, checkpoint, data point, and dependency used to build it. Our flagship model, Olmo 3 Think 32B, is the strongest fully-open thinking model released to-date.

cs.CL↗

arXiv2025

Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation

AI agents have been developed for complex real-world tasks from coding to customer service. But AI agent evaluations suffer from many challenges that undermine our understanding of how well agents really work. We introduce the Holistic Agent Leaderboard (HAL) to address these challenges. We make three main contributions. First, we provide a standardized evaluation harness that orchestrates parallel evaluations across hundreds of VMs, reducing evaluation time from weeks to hours while eliminating common implementation bugs. Second, we conduct three-dimensional analysis spanning models, scaffolds, and benchmarks. We validate the harness by conducting 21,730 agent rollouts across 9 models and 9 benchmarks in coding, web navigation, science, and customer service with a total cost of about $40,000. Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs. Third, we use LLM-aided log inspection to uncover previously unreported behaviors, such as searching for the benchmark on HuggingFace instead of solving a task, or misusing credit cards in flight booking tasks. We share all agent logs, comprising 2.5B tokens of language model calls, to incentivize further research into agent behavior. By standardizing how the field evaluates agents and addressing common pitfalls in agent evaluation, we hope to shift the focus from agents that ace benchmarks to agents that work reliably in the real world.

cs.AI↗

arXiv2025

MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documents

Automated agents, powered by Large language models (LLMs), are emerging as the go-to tool for querying information. However, evaluation benchmarks for LLM agents rarely feature natural questions that are both information-seeking and genuinely time-consuming for humans. To address this gap we introduce MoNaCo, a benchmark of 1,315 natural and time-consuming questions that require dozens, and at times hundreds, of intermediate steps to solve -- far more than any existing QA benchmark. To build MoNaCo, we developed a decomposed annotation pipeline to elicit and manually answer real-world time-consuming questions at scale. Frontier LLMs evaluated on MoNaCo achieve at most 61.2% F1, hampered by low recall and hallucinations. Our results underscore the limitations of LLM-powered agents in handling the complexity and sheer breadth of real-world information-seeking tasks -- with MoNaCo providing an effective resource for tracking such progress. The MoNaCo benchmark, codebase, prompts and models predictions are all publicly available at: https://tomerwolgithub.github.io/monaco

cs.CL↗

arXiv2024

AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents

Autonomous agents that address day-to-day digital tasks (e.g., ordering groceries for a household), must not only operate multiple apps (e.g., notes, messaging, shopping app) via APIs, but also generate rich code with complex control flow in an iterative manner based on their interaction with the environment. However, existing benchmarks for tool use are inadequate, as they only cover tasks that require a simple sequence of API calls. To remedy this gap, we built $\textbf{AppWorld Engine}$, a high-quality execution environment (60K lines of code) of 9 day-to-day apps operable via 457 APIs and populated with realistic digital activities simulating the lives of ~100 fictitious users. We then created $\textbf{AppWorld Benchmark}$ (40K lines of code), a suite of 750 natural, diverse, and challenging autonomous agent tasks requiring rich and interactive code generation. It supports robust programmatic evaluation with state-based unit tests, allowing for different ways of completing a task while also checking for unexpected changes, i.e., collateral damage. The state-of-the-art LLM, GPT-4o, solves only ~49% of our 'normal' tasks and ~30% of 'challenge' tasks, while other models solve at least 16% fewer. This highlights the benchmark's difficulty and AppWorld's potential to push the frontiers of interactive coding agents. The project website is available at https://appworld.dev/.

cs.SE↗

arXiv2024

Quantum $U$-channels on $S$-spaces

If the symmetry, (an operator $J$ satisfying $J=J^*=J^{-1}$) which defines the Krein space, is replaced by a (not necessarily self-adjoint) unitary, then we have the notion of an $S$-space which was introduced by Szafraniec. In this paper, we consider $S$-spaces and study the structure of completely $U$-positive maps between the algebras of bounded linear operators. We first give a Stinespring-type representation for a completely $U$-positive map. On the other hand, we introduce Choi $U$-matrix of a linear map and establish the equivalence of the Kraus $U$-decompositions and Choi $U$-matrices. Then we study properties of nilpotent completely $U$-positive maps. We develop the $U$-PPT criterion for separability of quantum $U$-states and discuss the entanglement breaking condition of quantum $U$-channels and explore $U$-PPT squared conjecture. Finally, we give concrete examples of completely $U$-positive maps and examples of $3 \otimes 3$ quantum $U$-states which are $U$-entangled and $U$-separable.

math.FA↗

arXiv2024

Powers and roots of partial isometric covariant representations

Isometric covariant representations play an important role in the study of Cuntz-Pimsner algebras. In this article, we study partial isometric covariant representations and explore under what conditions powers and roots of partial isometric covariant representations are also partial isometric covariant representations.

math.OA↗

arXiv2024

Block quantum dynamical semigroups of completely positive definite kernels

Kolmogorov decomposition for a given completely positive definite kernel is a generalization of Paschke's GNS construction for the completely positive map. Using Kolmogorov decomposition, to every quantum dynamical semigroup (QDS) for completely positive definite kernels over a set $S$ on given $C^*$-algebra $\mathcal{A},$ we shall assign an inclusion system $F = (F_s)_{s\ge 0}$ of Hilbert bimodules over $\mathcal{A}$ with a generating unit $\xi^{\sigma}=(\xi^{\sigma}_s)_{s\ge 0}.$ Consider a von Neumann algebra $\mathcal{B}$, and let $\mathfrak{T}=(\mathfrak{T}_s)_{s\ge 0}$ be a QDS over a set $S$ on the algebra $M_2(\mathcal{B})$ with $\mathfrak{T}_s=\begin{pmatrix}\mathfrak{K}_{s,1} & \mathfrak{L}_s\\\mathfrak{L}_s^*& \mathfrak{K}_{s,2} \end{pmatrix}$ which acts block-wise. Further, suppose that $(F^i_s )_{s\ge 0}$ is the inclusion system affiliated to the diagonal QDS $(\mathfrak{K}_{s,i})_{s\ge 0}$ along with the generating unit $(\xi^{\sigma}_{s,i} )_{s\ge 0},$ $\sigma\in S,i\in \{1,2\}$, then we prove that there exists a unique contractive (weak) morphism $V = (V_s)_{s\ge 0}:F^2_s \to F^1_s$ such that $\mathfrak{L}_s^{\sigma,\sigma'}(b)=\langle \xi_{s,1}^{\sigma},V_s b\xi_{s,2}^{\sigma'}\rangle$ for every $\sigma',\sigma\in S$ and $b\in \mathcal{B}.$ We also study the semigroup version of a factorization theorem for $\mathfrak{K}$-families.

math.OA↗

arXiv2023

A characterization of invariant subspaces for isometric representations of product system over $\mathbb{N}_0^{k}$

Using the Wold-von Neumann decomposition for the isometric covariant representations due to Muhly and Solel, we prove an explicit representation of the commutant of a doubly commuting pure isometric representation of the product system over $\mathbb{N}_0^{k}.$ As an application, we study a complete characterization of invariant subspaces for a doubly commuting pure isometric representation of the product system. This provides us a complete set of isomorphic invariants. Finally, we classify a large class of commuting isometric representations of the product system.

math.OA↗

arXiv2023

Beurling quotient subspaces for covariant representations of product systems

We characterize Beurling quotient subspaces for pure doubly commuting isometric representations of product systems. As a consequence, we derive a concrete regular dilation theorem for a pure completely contractive covariant representation which satisfies Brehmer-Solel condition and using it and the above characterization, we provide a necessary and sufficient condition that when a completely contractive covariant representation is unitarily equivalent to the compression of the induced representation on the Beurling quotient subspace. Further, we study the relation between Sz.Nagy-Foias type factorization of isometric multi-analytic operators and joint invariant subspaces.

math.OA↗

arXiv2022

Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions

Prompting-based large language models (LLMs) are surprisingly powerful at generating natural language reasoning steps or Chains-of-Thoughts (CoT) for multi-step question answering (QA). They struggle, however, when the necessary knowledge is either unavailable to the LLM or not up-to-date within its parameters. While using the question to retrieve relevant text from an external knowledge source helps LLMs, we observe that this one-step retrieve-and-read approach is insufficient for multi-step QA. Here, \textit{what to retrieve} depends on \textit{what has already been derived}, which in turn may depend on \textit{what was previously retrieved}. To address this, we propose IRCoT, a new approach for multi-step QA that interleaves retrieval with steps (sentences) in a CoT, guiding the retrieval with CoT and in turn using retrieved results to improve CoT. Using IRCoT with GPT3 substantially improves retrieval (up to 21 points) as well as downstream QA (up to 15 points) on four datasets: HotpotQA, 2WikiMultihopQA, MuSiQue, and IIRC. We observe similar substantial gains in out-of-distribution (OOD) settings as well as with much smaller models such as Flan-T5-large without additional training. IRCoT reduces model hallucination, resulting in factually more accurate CoT reasoning. Code, data, and prompts are available at \url{https://github.com/stonybrooknlp/ircot}

cs.CL↗

arXiv2022

Two-Turn Debate Doesn't Help Humans Answer Hard Reading Comprehension Questions

The use of language-model-based question-answering systems to aid humans in completing difficult tasks is limited, in part, by the unreliability of the text these systems generate. Using hard multiple-choice reading comprehension questions as a testbed, we assess whether presenting humans with arguments for two competing answer options, where one is correct and the other is incorrect, allows human judges to perform more accurately, even when one of the arguments is unreliable and deceptive. If this is helpful, we may be able to increase our justified trust in language-model-based systems by asking them to produce these arguments where needed. Previous research has shown that just a single turn of arguments in this format is not helpful to humans. However, as debate settings are characterized by a back-and-forth dialogue, we follow up on previous results to test whether adding a second round of counter-arguments is helpful to humans. We find that, regardless of whether they have access to arguments or not, humans perform similarly on our task. These findings suggest that, in the case of answering reading comprehension questions, debate is not a helpful format.

cs.CL↗

arXiv2022

Decomposed Prompting: A Modular Approach for Solving Complex Tasks

Few-shot prompting is a surprisingly powerful way to use Large Language Models (LLMs) to solve various tasks. However, this approach struggles as the task complexity increases or when the individual reasoning steps of the task themselves are hard to learn, especially when embedded in more complex tasks. To address this, we propose Decomposed Prompting, a new approach to solve complex tasks by decomposing them (via prompting) into simpler sub-tasks that can be delegated to a library of prompting-based LLMs dedicated to these sub-tasks. This modular structure allows each prompt to be optimized for its specific sub-task, further decomposed if necessary, and even easily replaced with more effective prompts, trained models, or symbolic functions if desired. We show that the flexibility and modularity of Decomposed Prompting allows it to outperform prior work on few-shot prompting using GPT3. On symbolic reasoning tasks, we can further decompose sub-tasks that are hard for LLMs into even simpler solvable sub-tasks. When the complexity comes from the input length, we can recursively decompose the task into the same task but with smaller inputs. We also evaluate our approach on textual multi-step reasoning tasks: on long-context multi-hop QA task, we can more effectively teach the sub-tasks via our separate sub-tasks prompts; and on open-domain multi-hop QA, we can incorporate a symbolic information retrieval within our decomposition framework, leading to improved performance on both tasks. Datasets, Code and Prompts available at https://github.com/allenai/DecomP.

cs.CL↗

arXiv2022

Regular covariant representations and their Wold-type decomposition

Olofsson introduced a growth condition regarding elements of an orbit for an expansive operator and generalized Richter's wandering subspace theorem. Later on, using the Moore-Penrose inverse, Ezzahraoui, Mbekhta, and Zerouali extended the growth condition and obtained a Shimorin-Wold-type decomposition. Shimorin-Wold-type decomposition for completely bounded covariant representations, which are close to isometric representations, is obtained in \cite{HV19}. This paper extends this decomposition for regular, completely bounded covariant representation having reduced minimum modulus $\geq 1$ that satisfies the growth condition. To prove the decomposition, we introduce the terms regular, algebraic core, and reduced minimum modulus in the completely bounded covariant representation setting and work out several fundamental results. Consequently, we shall analyze the weighted unilateral shift introduced by Muhly and Solel and introduce and explore a non-commutative weighted bilateral shift.

math.OA↗

arXiv2022

Berger-Coburn-Lebow representation for pure isometric representations of product system over $\mathbb N^2_0$

We obtain Berger-Coburn-Lebow (BCL)-representation for pure isometric covariant representation of product system over $\mathbb{N}_0^2$. Then the corresponding complete set of (joint) unitary invariants is studied, and the BCL- representations are compared with other canonical multi-analytic descriptions of the pure isometric covariant representation. We characterize the invariant subspaces for the pure isometric covariant representation. Also, we study the connection between the joint defect operators and Fringe operators, and the Fredholm index is introduced in this case. Finally, we introduce the notion of congruence relation to classify the isometric covariant representations of the product system over $\mathbb{N}_0^2$.

math.OA↗