arXiv subjects

Omar Attia

Publications and source records attributed to Omar Attia.

13 recordsLinked to original sources

arXiv2026

ASTRA-bench: Evaluating Tool-Use Agent Reasoning and Action Planning with Personal User Context

Next-generation AI must manage vast personal data, diverse tools, and multi-step reasoning, yet most benchmarks remain context-free and single-turn. We present ASTRA-bench (Assistant Skills in Tool-use, Reasoning \& Action-planning), a benchmark that uniquely unifies time-evolving personal context with an interactive toolbox and complex user intents. Our event-driven pipeline generates 2,413 scenarios across four protagonists, grounded in longitudinal life events and annotated by referential, functional, and informational complexity. Evaluation of state-of-the-art models (e.g., Claude-4.5-Opus, DeepSeek-V3.2) reveals significant performance degradation under high-complexity conditions, with argument generation emerging as the primary bottleneck. These findings expose critical limitations in current agents' ability to ground reasoning within messy personal context and orchestrate reliable multi-step plans. We release ASTRA-bench with a full execution environment and evaluation scripts to provide a diagnostic testbed for developing truly context-aware AI assistants.

cs.AI↗

arXiv2026

Expanding LLM Agent Boundaries with Strategy-Guided Exploration

Reinforcement learning (RL) has demonstrated notable success in post-training large language models (LLMs) as agents for tasks such as computer use, tool calling, and coding. However, exploration remains a central challenge in RL for LLM agents, especially as they operate in language-action spaces with complex observations and sparse outcome rewards. In this work, we address exploration for LLM agents by leveraging the ability of LLMs to plan and reason in language about the environment to shift exploration from low-level actions to higher-level language strategies. We thus propose Strategy-Guided Exploration (SGE), which first generates a concise natural-language strategy that describes what to do to make progress toward the goal, and then generates environment actions conditioned on that strategy. By exploring in the space of strategies rather than the space of actions, SGE induces structured and diverse exploration that targets different environment outcomes. To increase strategy diversity during RL, SGE introduces mixed-temperature sampling, which explores diverse strategies in parallel, along with a strategy reflection process that grounds strategy generation on the outcomes of previous strategies in the environment. Across UI interaction, tool-calling, coding, and embodied agent environments, SGE consistently outperforms exploration-focused RL baselines, improving both learning efficiency and final performance. We show that SGE enables the agent to learn to solve tasks too difficult for the base model.

cs.LG↗

arXiv2025

UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action

Computer-use agents face a fundamental limitation. They rely exclusively on primitive GUI actions (click, type, scroll), creating brittle execution chains prone to cascading failures. While API-driven agents harness rich capabilities through structured interfaces and tools, computer-use agents remain constrained to low-level visual interactions. We present UltraCUA, a foundation model that transcends this limitation through hybrid action-seamlessly unifying primitive GUI operations with high-level tool execution. Our innovation rests on four critical advances. First, an automated pipeline extracts and scales tool capabilities from software documentation and code repositories. Second, a synthetic data engine produces 17,000+ verifiable tasks capturing real-world computer-use complexity. Third, comprehensive hybrid action trajectory collection incorporates both GUI primitives and strategic tool calls. Fourth, a two-stage training methodology combines supervised fine-tuning with online reinforcement learning, enabling intelligent action selection between GUI and API. Evaluation with our 7B and 32B UltraCUA models reveals transformative performance gains. On OSWorld, UltraCUA achieves 22% relative improvement while executing 11% faster than existing approaches, averagely. Cross-domain validation on WindowsAgentArena demonstrates robust generalization with 21.7% success rate, surpassing Windows-trained baselines. The hybrid action paradigm proves essential, reducing error propagation while improving execution efficiency. This work establishes a scalable paradigm bridging primitive GUI interactions and high-level tool intelligence, enabling more resilient and adaptable computer use agents for diverse environments and complex real-world tasks.

cs.CV↗

arXiv2025

GRACE: A Language Model Framework for Explainable Inverse Reinforcement Learning

Inverse Reinforcement Learning aims to recover reward models from expert demonstrations, but traditional methods yield black-box models that are difficult to interpret and debug. In this work, we introduce GRACE (Generating Rewards As CodE), a method for using Large Language Models within an evolutionary search to reverse-engineer an interpretable, code-based reward function directly from expert trajectories. The resulting reward function is executable code that can be inspected and verified. We empirically validate GRACE on the MuJoCo, BabyAI and AndroidWorld benchmarks, where it efficiently learns highly accurate rewards, even in complex, multi-task settings. Further, we demonstrate that the resulting reward leads to strong policies, compared to both competitive Imitation Learning and online RL approaches with ground-truth rewards. Finally, we show that GRACE is able to build complex reward APIs in multi-task setups.

cs.LG↗

arXiv2025

Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents

Developing autonomous agents that effectively interact with Graphic User Interfaces (GUIs) remains a challenging open problem, especially for small on-device models. In this paper, we present Ferret-UI Lite, a compact, end-to-end GUI agent that operates across diverse platforms, including mobile, web, and desktop. Utilizing techniques optimized for developing small models, we build our 3B Ferret-UI Lite agent through curating a diverse GUI data mixture from real and synthetic sources, strengthening inference-time performance through chain-of-thought reasoning and visual tool-use, and reinforcement learning with designed rewards. Ferret-UI Lite achieves competitive performance with other small-scale GUI agents. In GUI grounding, Ferret-UI Lite attains scores of $91.6\%$, $53.3\%$, and $61.2\%$ on the ScreenSpot-V2, ScreenSpot-Pro, and OSWorld-G benchmarks, respectively. For GUI navigation, Ferret-UI Lite achieves success rates of $28.0\%$ on AndroidWorld and $19.8\%$ on OSWorld. We share our methods and lessons learned from developing compact, on-device GUI agents.

cs.CV↗

arXiv2025

Scaling Synthetic Task Generation for Agents via Exploration

Post-Training Multimodal Large Language Models (MLLMs) to build interactive agents holds promise across domains such as computer-use, web navigation, and robotics. A key challenge in scaling such post-training is lack of high-quality downstream agentic task datasets with tasks that are diverse, feasible, and verifiable. Existing approaches for task generation rely heavily on human annotation or prompting MLLM with limited downstream environment information, which is either costly or poorly scalable as it yield tasks with limited coverage. To remedy this, we present AutoPlay, a scalable pipeline for task generation that explicitly explores interactive environments to discover possible interactions and current state information to synthesize environment-grounded tasks. AutoPlay operates in two stages: (i) an exploration phase, where an MLLM explorer agent systematically uncovers novel environment states and functionalities, and (ii) a task generation phase, where a task generator leverages exploration trajectories and a set of task guideline prompts as context to synthesize diverse, executable, and verifiable tasks. We show AutoPlay generates 20k tasks across 20 Android applications and 10k tasks across 13 applications Ubuntu applications to train mobile-use and computer-use agents. AutoPlay generated tasks enable large-scale task demonstration synthesis without human annotation by employing an MLLM task executor and verifier. This data enables training MLLM-based UI agents that improve success rates up to $20.0\%$ on mobile-use and $10.9\%$ on computer-use scenarios. In addition, AutoPlay generated tasks combined with MLLM verifier-based rewards enable scaling reinforcement learning training of UI agents, leading to an additional $5.7\%$ gain. coverage. These results establish AutoPlay as a scalable approach for post-training capable MLLM agents reducing reliance on human annotation.

cs.AI↗

arXiv2025

Comprehensive Evaluation for a Large Scale Knowledge Graph Question Answering Service

Question answering systems for knowledge graph (KGQA), answer factoid questions based on the data in the knowledge graph. KGQA systems are complex because the system has to understand the relations and entities in the knowledge-seeking natural language queries and map them to structured queries against the KG to answer them. In this paper, we introduce Chronos, a comprehensive evaluation framework for KGQA at industry scale. It is designed to evaluate such a multi-component system comprehensively, focusing on (1) end-to-end and component-level metrics, (2) scalable to diverse datasets and (3) a scalable approach to measure the performance of the system prior to release. In this paper, we discuss the unique challenges associated with evaluating KGQA systems at industry scale, review the design of Chronos, and how it addresses these challenges. We will demonstrate how it provides a base for data-driven decisions and discuss the challenges of using it to measure and improve a real-world KGQA system.

cs.CL↗

arXiv2024

From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons

We examine the capability of Multimodal Large Language Models (MLLMs) to tackle diverse domains that extend beyond the traditional language and vision tasks these models are typically trained on. Specifically, our focus lies in areas such as Embodied AI, Games, UI Control, and Planning. To this end, we introduce a process of adapting an MLLM to a Generalist Embodied Agent (GEA). GEA is a single unified model capable of grounding itself across these varied domains through a multi-embodiment action tokenizer. GEA is trained with supervised learning on a large dataset of embodied experiences and with online RL in interactive simulators. We explore the data and algorithmic choices necessary to develop such a model. Our findings reveal the importance of training with cross-domain data and online RL for building generalist agents. The final GEA model achieves strong generalization performance to unseen tasks across diverse benchmarks compared to other generalist models and benchmark-specific approaches.

cs.LG↗

arXiv2024

AGRaME: Any-Granularity Ranking with Multi-Vector Embeddings

Ranking is a fundamental and popular problem in search. However, existing ranking algorithms usually restrict the granularity of ranking to full passages or require a specific dense index for each desired level of granularity. Such lack of flexibility in granularity negatively affects many applications that can benefit from more granular ranking, such as sentence-level ranking for open-domain question-answering, or proposition-level ranking for attribution. In this work, we introduce the idea of any-granularity ranking, which leverages multi-vector embeddings to rank at varying levels of granularity while maintaining encoding at a single (coarser) level of granularity. We propose a multi-granular contrastive loss for training multi-vector approaches, and validate its utility with both sentences and propositions as ranking units. Finally, we demonstrate the application of proposition-level ranking to post-hoc citation addition in retrieval-augmented generation, surpassing the performance of prompt-driven citation generation.

cs.CL↗

arXiv2024

Entity Disambiguation via Fusion Entity Decoding

Entity disambiguation (ED), which links the mentions of ambiguous entities to their referent entities in a knowledge base, serves as a core component in entity linking (EL). Existing generative approaches demonstrate improved accuracy compared to classification approaches under the standardized ZELDA benchmark. Nevertheless, generative approaches suffer from the need for large-scale pre-training and inefficient generation. Most importantly, entity descriptions, which could contain crucial information to distinguish similar entities from each other, are often overlooked. We propose an encoder-decoder model to disambiguate entities with more detailed entity descriptions. Given text and candidate entities, the encoder learns interactions between the text and each candidate entity, producing representations for each entity candidate. The decoder then fuses the representations of entity candidates together and selects the correct entity. Our experiments, conducted on various entity disambiguation benchmarks, demonstrate the strong and robust performance of this model, particularly +1.5% in the ZELDA benchmark compared with GENRE. Furthermore, we integrate this approach into the retrieval/reader framework and observe +1.5% improvements in end-to-end entity linking in the GERBIL benchmark compared with EntQA.

cs.CL↗

arXiv2024

Discovery of two warm mini-Neptunes with contrasting densities orbiting the young K3V star TOI-815

We present the discovery and characterization of two warm mini-Neptunes transiting the K3V star TOI-815 in a K-M binary system. Analysis of the spectra and rotation period reveal it to be a young star with an age of $200^{+400}_{-200}$Myr. TOI-815b has a 11.2-day period and a radius of 2.94$\pm$0.05$\it{R_{\rm\mathrm{\oplus}}}$ with transits observed by TESS, CHEOPS, ASTEP, and LCOGT. The outer planet, TOI-815c, has a radius of 2.62$\pm$0.10$\it{R_{\rm\mathrm{\oplus}}}$, based on observations of three non-consecutive transits with TESS, while targeted CHEOPS photometry and radial velocity follow-up with ESPRESSO were required to confirm the 35-day period. ESPRESSO confirmed the planetary nature of both planets and measured masses of 7.6$\pm$1.5 $\it{M_{\rm \mathrm{\oplus}}}$ ($ρ_\mathrm{P}$=1.64$^{+0.33}_{-0.31}$gcm$^{-3}$) and 23.5$\pm$2.4$\it{M_{\rm\mathrm{\oplus}}}$ ($ρ_\mathrm{P}$=7.2$^{+1.1}_{-1.0}$gcm$^{-3}$) respectively. Thus, the planets have very different masses, unlike the usual similarity of masses in compact multi-planet systems. Moreover, our statistical analysis of mini-Neptunes orbiting FGK stars suggests that weakly irradiated planets tend to have higher bulk densities compared to those suffering strong irradiation. This could be ascribed to their cooler atmospheres, which are more compressed and denser. Internal structure modeling of TOI-815b suggests it likely has a H-He atmosphere constituting a few percent of the total planet mass, or higher if the planet is assumed to have no water. In contrast, the measured mass and radius of TOI-815c can be explained without invoking any atmosphere, challenging planetary formation theories. Finally, we infer from our measurements that the star is viewed close to pole-on, which implies a spin-orbit misalignment at the 3$σ$ level.

astro-ph.EP↗

arXiv2023

Three Saturn-mass planets transiting F-type stars revealed with TESS and HARPS

While the sample of confirmed exoplanets continues to increase, the population of transiting exoplanets around early-type stars is still limited. These planets allow us to investigate the planet properties and formation pathways over a wide range of stellar masses and study the impact of high irradiation on hot Jupiters orbiting such stars. We report the discovery of TOI-615b, TOI-622b, and TOI-2641b, three Saturn-mass planets transiting main sequence, F-type stars. The planets were identified by the Transiting Exoplanet Survey Satellite (TESS) and confirmed with complementary ground-based and radial velocity observations. TOI-615b is a highly irradiated ($\sim$1277 $F_{\oplus}$) and bloated Saturn-mass planet (1.69$^{+0.05}_{-0.06}$$R_{Jup}$ and 0.43$^{+0.09}_{-0.08}$$M_{Jup}$) in a 4.66 day orbit transiting a 6850 K star. TOI-622b has a radius of 0.82$^{+0.03}_{-0.03}$$R_{Jup}$ and a mass of 0.30$^{+0.07}_{-0.08}$~$M_{Jup}$ in a 6.40 day orbit. Despite its high insolation flux ($\sim$600 $F_{\oplus}$), TOI-622b does not show any evidence of radius inflation. TOI-2641b is a 0.39$^{+0.02}_{-0.04}$$M_{Jup}$ planet in a 4.88 day orbit with a grazing transit (b = 1.04$^{+0.05}_{-0.06 }$) that results in a poorly constrained radius of 1.61$^{+0.46}_{-0.64}$$R_{Jup}$. Additionally, TOI-615b is considered attractive for atmospheric studies via transmission spectroscopy with ground-based spectrographs and $\textit{JWST}$. Future atmospheric and spin-orbit alignment observations are essential since they can provide information on the atmospheric composition, formation and migration of exoplanets across various stellar types.

astro-ph.EP↗

arXiv2021

Introducing SPHINX-MHD: The Impact of Primordial Magnetic Fields on the First Galaxies, Reionization, and the Global 21cm Signal

We present the first results from SPHINX-MHD, a suite of cosmological radiation-magnetohydrodynamics simulations designed to study the impact of primordial magnetic fields (PMFs) on galaxy formation and the evolution of the intergalactic medium during the epoch of reionization. The simulations are among the first to employ multi-frequency, on-the-fly radiation transfer and constrained transport ideal MHD in a cosmological context to simultaneously model the inhomogeneous process of reionization as well as the growth of PMFs. We run a series of $(5\,\text{cMpc})^3$ cosmological volumes, varying both the strength of the seed magnetic field ($B_0$) and its spectral index ($n_B$). We find that PMFs that have $n_B > -0.562\log_{10}\left(\frac{B_0}{1{\rm n}G}\right) - 3.35$ produce electron optical depths ($τ_e$) that are inconsistent with CMB constraints due to the unrealistically early collapse of low-mass dwarf galaxies. For $n_B\geq-2.9$, our constraints are considerably tighter than the $\sim{\rm n}G$ constraints from Planck. PMFs that do not satisfy our constraints have little impact on the reionization history or the shape of the UV luminosity function. Likewise, detecting changes in the Lya forest due to PMFs will be challenging because photoionisation and photoheating efficiently smooth the density field. However, we find that the first absorption feature in the global 21cm signal is a sensitive indicator of the properties of the PMFs, even for those that satisfy our $τ_e$ constraint. Furthermore, strong PMFs can marginally increase the escape of LyC photons by up to 25\% and shrink the effective radii of galaxies by $\sim44\%$ which could increase the completeness fraction of galaxy surveys. Finally, our simulations show that surveys with a magnitude limit of ${\rm M_{UV,1500}=-13}$ can probe the sources that provide the majority of photons for reionization out to $z=12$.

astro-ph.CO↗