Searcharxiv⌕ Search

arXiv subjects

Ada Chen

Publications and source records attributed to Ada Chen.

3 recordsLinked to original sources

RealGUINoise: An Interactive Cross-Platform Benchmark for GUI Agent Robustness under Real-World Interface Noise

Graphical User Interface (GUI) agents and Computer-Using Agents (CUAs) are rapidly becoming practical tools. However, real-world deployment increasingly exposes performance failures and safety risks, while a major yet underexplored source of these problems lies in the complex and noisy conditions of everyday interfaces. Existing benchmarks largely assume clean environments or focus narrowly on security-specific settings, and lack a unified framework for consistent, automated end-to-end evaluation across diverse agents and platforms. Hence, we introduce RealGUINoise, an interactive cross-platform, extensible benchmark for systematically evaluating GUI Agents under common realistic interface noise in fully interactive environments. Specifically, RealGUINoise comprises 42 noise types spanning web, desktop, and mobile tasks and integrates 7 representative agent frameworks. It evaluates these agents on real-world daily tasks through real-time interaction, comparing their performance against task-specific golden rubrics and clean-environment trajectories in terms of reliability, safety, and trajectory-level behavior. Our experiments show that these noises not only degrade task performance but also substantially redirect agents' action trajectories and increase their propensity for unsafe behavior. These findings expose a critical gap between capability in clean environments and dependable operation in real-world settings, establishing RealGUINoise as a testbed for developing more robust and trustworthy GUI agents.

cs.HC↗

A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?

Recently, AI-driven interactions with computing devices have advanced from basic prototype tools to sophisticated, LLM-based systems that emulate human-like operations in graphical user interfaces. We are now witnessing the emergence of \emph{Computer-Using Agents} (CUAs), capable of autonomously performing tasks such as navigating desktop applications, web pages, and mobile apps. However, as these agents grow in capability, they also introduce novel safety and security risks. Vulnerabilities in LLM-driven reasoning, with the added complexity of integrating multiple software components and multimodal inputs, further complicate the security landscape. In this paper, we present a systematization of knowledge on the safety and security threats of CUAs. We conduct a comprehensive literature review and distill our findings along four research objectives: \textit{\textbf{(i)}} define the CUA that suits safety analysis; \textit{\textbf{(ii)} } categorize current safety threats among CUAs; \textit{\textbf{(iii)}} propose a comprehensive taxonomy of existing defensive strategies; \textit{\textbf{(iv)}} summarize prevailing benchmarks, datasets, and evaluation metrics used to assess the safety and performance of CUAs. Building on these insights, our work provides future researchers with a structured foundation for exploring unexplored vulnerabilities and offers practitioners actionable guidance in designing and deploying secure Computer-Using Agents.

cs.CL↗

Prompt-MII: Meta-Learning Instruction Induction for LLMs

A popular method to adapt large language models (LLMs) to new tasks is in-context learning (ICL), which is effective but incurs high inference costs as context length grows. In this paper we propose a method to perform instruction induction, where we take training examples and reduce them to a compact but descriptive prompt that can achieve performance comparable to ICL over the full training set. Specifically, we propose PROMPT-MII, a reinforcement learning (RL) based framework to meta-learn an instruction induction model that can generate compact instructions on the fly for an arbitrary new dataset. We train on over 3,000 diverse classification datasets from the HuggingFace hub, and evaluate on 90 unseen tasks. PROMPT-MII improves downstream model quality by 4-9 F1 points (10-20% relative), matching ICL performance while requiring 3-13x fewer tokens.

cs.CL↗