arXiv subjects

Misha Sra

Publications and source records attributed to Misha Sra.

At least 37 records · Page 2Linked to original sources

arXiv2025

CoDA: Agentic Systems for Collaborative Data Visualization

Deep research has revolutionized data analysis, yet data scientists still devote substantial time to manually crafting visualizations, highlighting the need for robust automation from natural language queries. However, current systems struggle with complex datasets containing multiple files and iterative refinement. Existing approaches, including simple single- or multi-agent systems, often oversimplify the task, focusing on initial query parsing while failing to robustly manage data complexity, code errors, or final visualization quality. In this paper, we reframe this challenge as a collaborative multi-agent problem. We introduce CoDA, a multi-agent system that employs specialized LLM agents for metadata analysis, task planning, code generation, and self-reflection. We formalize this pipeline, demonstrating how metadata-focused analysis bypasses token limits and quality-driven refinement ensures robustness. Extensive evaluations show CoDA achieves substantial gains in the overall score, outperforming competitive baselines by up to 41.5%. This work demonstrates that the future of visualization automation lies not in isolated code generation but in integrated, collaborative agentic workflows.

cs.AI↗

arXiv2025

GraspR: A Computational Model of Spatial User Preferences for Adaptive Grasp UI Design

Grasp User Interfaces (grasp UIs) enable dual-tasking in XR by allowing interaction with digital content while holding physical objects. However, current grasp UI design practices face a fundamental challenge: existing approaches either capture user preferences through labor-intensive elicitation studies that are difficult to scale or rely on biomechanical models that overlook subjective factors. We introduce GraspR, the first computational model that predicts user preferences for single-finger microgestures in grasp UIs. Our data-driven approach combines the scalability of computational methods with human preference modeling, trained on 1,520 preferences collected via a two-alternative forced choice paradigm across eight participants and four frequently used grasp variations. We demonstrate GraspR's effectiveness through a working prototype that dynamically adjusts interface layouts across four everyday tasks. We release both the dataset and code to support future research in adaptive grasp UIs.

cs.HC↗

arXiv2025

TR-LLM: Integrating Trajectory Data for Scene-Aware LLM-Based Human Action Prediction

Accurate prediction of human behavior is crucial for AI systems to effectively support real-world applications, such as autonomous robots anticipating and assisting with human tasks. Real-world scenarios frequently present challenges such as occlusions and incomplete scene observations, which can compromise predictive accuracy. Thus, traditional video-based methods often struggle due to limited temporal and spatial perspectives. Large Language Models (LLMs) offer a promising alternative. Having been trained on a large text corpus describing human behaviors, LLMs likely encode plausible sequences of human actions in a home environment. However, LLMs, trained primarily on text data, lack inherent spatial awareness and real-time environmental perception. They struggle with understanding physical constraints and spatial geometry. Therefore, to be effective in a real-world spatial scenario, we propose a multimodal prediction framework that enhances LLM-based action prediction by integrating physical constraints derived from human trajectories. Our experiments demonstrate that combining LLM predictions with trajectory data significantly improves overall prediction performance. This enhancement is particularly notable in situations where the LLM receives limited scene information, highlighting the complementary nature of linguistic knowledge and physical constraints in understanding and anticipating human behavior.

cs.HC↗

arXiv2025

Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Risk

Standard benchmarks fixate on how well large language model (LLM) agents perform in finance, yet say little about whether they are safe to deploy. We argue that accuracy metrics and return-based scores provide an illusion of reliability, overlooking vulnerabilities such as hallucinated facts, stale data, and adversarial prompt manipulation. We take a firm position: financial LLM agents should be evaluated first and foremost on their risk profile, not on their point-estimate performance. Drawing on risk-engineering principles, we outline a three-level agenda: model, workflow, and system, for stress-testing LLM agents under realistic failure modes. To illustrate why this shift is urgent, we audit six API-based and open-weights LLM agents on three high-impact tasks and uncover hidden weaknesses that conventional benchmarks miss. We conclude with actionable recommendations for researchers, practitioners, and regulators: audit risk-aware metrics in future studies, publish stress scenarios alongside datasets, and treat ``safety budget'' as a primary success criterion. Only by redefining what ``good'' looks like can the community responsibly advance AI-driven finance.

q-fin.GN↗

arXiv2025

SiCo: An Interactive Size-Controllable Virtual Try-On Approach for Informed Decision-Making

Virtual try-on (VTO) applications aim to replicate the in-store shopping experience and enhance online shopping by enabling users to interact with garments. However, many existing tools adopt a one-size-fits-all approach when visualizing clothing items. This approach limits user interaction with garments, particularly regarding size and fit adjustments, and fails to provide direct insights for size recommendations. As a result, these limitations contribute to high return rates in online shopping. To address this, we introduce SiCo, a new online VTO system that allows users to upload images of themselves and interact with garments by visualizing how different sizes would fit their bodies. Our user study demonstrates that our approach significantly improves users' ability to assess how outfits will appear on their bodies and increases their confidence in selecting clothing sizes that align with their preferences. Based on our evaluation, we believe that SiCo has the potential to reduce return rates and transform the online clothing shopping experience.

cs.HC↗

arXiv2025

Designing Scaffolded Interfaces for Enhanced Learning and Performance in Professional Software

Professional software offers immense power but also presents significant learning challenges. Its complex interfaces, as well as insufficient built-in structured guidance and unfamiliar terminology, often make newcomers struggle with task completion. To address these challenges, we introduce ScaffoldUI, a method for scaffolded interface design to reduce interface complexity, provide structured guidance, and enhance software learnability. The scaffolded interface presents task-relevant tools, progressively discloses tool complexity, and organizes tools based on domain concepts, aiming to assist task performance and software learning. To evaluate the feasibility of our interface design method, we present a technical pipeline for scaffolded interface implementation in professional 3D software, i.e., Blender, and conduct user studies with beginners (N=32) and experts (N=8). Study results demonstrate that our scaffolded interfaces significantly reduce perceived task load caused by interface complexity, support task performance through structured guidance, and augment learning by clearly connecting concepts and tools within the taskflow context. Based on a discussion of the user study findings, we offer insights for future research on designing scaffolded interfaces to support instruction, productivity, creativity, and cross-software workflows.

cs.HC↗

arXiv2025

CrowdGenUI: Aligning LLM-Based UI Generation with Crowdsourced User Preferences

Large Language Models (LLMs) have demonstrated remarkable potential across various design domains, including user interface (UI) generation. However, current LLMs for UI generation tend to offer generic solutions that lack a nuanced understanding of task context and user preferences. We present CrowdGenUI, a framework that enhances LLM-based UI generation with crowdsourced user preferences. This framework addresses the limitations by guiding LLM reasoning with real user preferences, enabling the generation of UI widgets that reflect user needs and task-specific requirements. We evaluate our framework in the image editing domain by collecting a library of 720 user preferences from 50 participants, covering preferences such as predictability, efficiency, and explorability of various UI widgets. A user study (N=78) demonstrates that UIs generated with our preference-guided framework can better match user intentions compared to those generated by LLMs alone, highlighting the effectiveness of our proposed framework. We further discuss the study findings and present insights for future research on LLM-based user-centered UI generation.

cs.HC↗

arXiv2025

Instruct-CLIP: Improving Instruction-Guided Image Editing with Automated Data Refinement Using Contrastive Learning

Although natural language instructions offer an intuitive way to guide automated image editing, deep-learning models often struggle to achieve high-quality results, largely due to the difficulty of creating large, high-quality training datasets. To do this, previous approaches have typically relied on text-to-image (T2I) generative models to produce pairs of original and edited images that simulate the input/output of an instruction-guided image-editing model. However, these image pairs often fail to align with the specified edit instructions due to the limitations of T2I models, which negatively impacts models trained on such datasets. To address this, we present Instruct-CLIP (I-CLIP), a selfsupervised method that learns the semantic changes between original and edited images to refine and better align the instructions in existing datasets. Furthermore, we adapt Instruct-CLIP to handle noisy latent images and diffusion timesteps so that it can be used to train latent diffusion models (LDMs) and efficiently enforce alignment between the edit instruction and the image changes in latent space at any step of the diffusion pipeline. We use Instruct-CLIP to correct the InstructPix2Pix dataset and get over 120K refined samples we then use to fine-tune their model, guided by our novel I-CLIP-based loss function. The resulting model can produce edits that are more aligned with the given instructions. Our code and dataset are available at https://github.com/SherryXTChen/Instruct-CLIP.git.

cs.CV↗

arXiv2025

LocoVR: Multiuser Indoor Locomotion Dataset in Virtual Reality

Understanding human locomotion is crucial for AI agents such as robots, particularly in complex indoor home environments. Modeling human trajectories in these spaces requires insight into how individuals maneuver around physical obstacles and manage social navigation dynamics. These dynamics include subtle behaviors influenced by proxemics - the social use of space, such as stepping aside to allow others to pass or choosing longer routes to avoid collisions. Previous research has developed datasets of human motion in indoor scenes, but these are often limited in scale and lack the nuanced social navigation dynamics common in home environments. To address this, we present LocoVR, a dataset of 7000+ two-person trajectories captured in virtual reality from over 130 different indoor home environments. LocoVR provides accurate trajectory data and precise spatial information, along with rich examples of socially-motivated movement behaviors. For example, the dataset captures instances of individuals navigating around each other in narrow spaces, adjusting paths to respect personal boundaries in living areas, and coordinating movements in high-traffic zones like entryways and kitchens. Our evaluation shows that LocoVR significantly enhances model performance in three practical indoor tasks utilizing human trajectories, and demonstrates predicting socially-aware navigation patterns in home environments.

cs.RO↗

arXiv2025

GraphEval36K: Benchmarking Coding and Reasoning Capabilities of Large Language Models on Graph Datasets

Large language models (LLMs) have achieved remarkable success in natural language processing (NLP), demonstrating significant capabilities in processing and understanding text data. However, recent studies have identified limitations in LLMs' ability to manipulate, program, and reason about structured data, especially graphs. We introduce GraphEval36K, the first comprehensive graph dataset, comprising 40 graph coding problems and 36,900 test cases to evaluate the ability of LLMs on graph problem-solving. Our dataset is categorized into eight primary and four sub-categories to ensure a thorough evaluation across different types of graphs. We benchmark ten LLMs, finding that private models outperform open-source ones, though the gap is narrowing. We also analyze the performance of LLMs across directed vs undirected graphs, different kinds of graph concepts, and network models. Furthermore, to improve the usability of our evaluation framework, we propose Structured Symbolic Decomposition (SSD), an instruction-based method designed to enhance LLM performance on complex graph tasks. Results show that SSD improves the average passing rate of GPT-4, GPT-4o, Gemini-Pro and Claude-3-Sonnet by 8.38%, 6.78%, 29.28% and 25.28%, respectively.

cs.AI↗

arXiv2025

Agentic Workflows for Conversational Human-AI Interaction Design

Conversational human-AI interaction (CHAI) have recently driven mainstream adoption of AI. However, CHAI poses two key challenges for designers and researchers: users frequently have ambiguous goals and an incomplete understanding of AI functionalities, and the interactions are brief and transient, limiting opportunities for sustained engagement with users. AI agents can help address these challenges by suggesting contextually relevant prompts, by standing in for users during early design testing, and by helping users better articulate their goals. Guided by research-through-design, we explored agentic AI workflows through the development and testing of a probe over four iterations with 10 users. We present our findings through an annotated portfolio of design artifacts, and through thematic analysis of user experiences, offering solutions to the problems of ambiguity and transient in CHAI. Furthermore, we examine the limitations and possibilities of these AI agent workflows, suggesting that similar collaborative approaches between humans and AI could benefit other areas of design.

cs.HC↗

arXiv2025

Engaging with AI: How Interface Design Shapes Human-AI Collaboration in High-Stakes Decision-Making

As reliance on AI systems for decision-making grows, it becomes critical to ensure that human users can appropriately balance trust in AI suggestions with their own judgment, especially in high-stakes domains like healthcare. However, human + AI teams have been shown to perform worse than AI alone, with evidence indicating automation bias as the reason for poorer performance, particularly because humans tend to follow AI's recommendations even when they are incorrect. In many existing human + AI systems, decision-making support is typically provided in the form of text explanations (XAI) to help users understand the AI's reasoning. Since human decision-making often relies on System 1 thinking, users may ignore or insufficiently engage with the explanations, leading to poor decision-making. Previous research suggests that there is a need for new approaches that encourage users to engage with the explanations and one proposed method is the use of cognitive forcing functions (CFFs). In this work, we examine how various decision-support mechanisms impact user engagement, trust, and human-AI collaborative task performance in a diabetes management decision-making scenario. In a controlled experiment with 108 participants, we evaluated the effects of six decision-support mechanisms split into two categories of explanations (text, visual) and four CFFs. Our findings reveal that mechanisms like AI confidence levels, text explanations, and performance visualizations enhanced human-AI collaborative task performance, and improved trust when AI reasoning clues were provided. Mechanisms like human feedback and AI-driven questions encouraged deeper reflection but often reduced task performance by increasing cognitive effort, which in turn affected trust. Simple mechanisms like visual explanations had little effect on trust, highlighting the importance of striking a balance in CFF and XAI design.

cs.HC↗

arXiv2024

An Interaction Design Toolkit for Physical Task Guidance with Artificial Intelligence and Mixed Reality

Physical skill acquisition, from sports techniques to surgical procedures, requires instruction and feedback. In the absence of a human expert, Physical Task Guidance (PTG) systems can offer a promising alternative. These systems integrate Artificial Intelligence (AI) and Mixed Reality (MR) to provide realtime feedback and guidance as users practice and learn skills using physical tools and objects. However, designing PTG systems presents challenges beyond engineering complexities. The intricate interplay between users, AI, MR interfaces, and the physical environment creates unique interaction design hurdles. To address these challenges, we present an interaction design toolkit derived from our analysis of PTG prototypes developed by eight student teams during a 10-week-long graduate course. The toolkit comprises Design Considerations, Design Patterns, and an Interaction Canvas. Our evaluation suggests that the toolkit can serve as a valuable resource for practitioners designing PTG systems and researchers developing new tools for human-AI interaction design.

cs.HC↗

arXiv2024

Everyday AR through AI-in-the-Loop

This workshop brings together experts and practitioners from augmented reality (AR) and artificial intelligence (AI) to shape the future of AI-in-the-loop everyday AR experiences. With recent advancements in both AR hardware and AI capabilities, we envision that everyday AR -- always-available and seamlessly integrated into users' daily environments -- is becoming increasingly feasible. This workshop will explore how AI can drive such everyday AR experiences. We discuss a range of topics, including adaptive and context-aware AR, generative AR content creation, always-on AI assistants, AI-driven accessible design, and real-world-oriented AI agents. Our goal is to identify the opportunities and challenges in AI-enabled AR, focusing on creating novel AR experiences that seamlessly blend the digital and physical worlds. Through the workshop, we aim to foster collaboration, inspire future research, and build a community to advance the research field of AI-enhanced AR.

cs.HC↗

arXiv2024

Virtual Steps: The Experience of Walking for a Lifelong Wheelchair User in Virtual Reality

Many people often take walking for granted, but for individuals with mobility disabilities, this seemingly simple act can feel out of reach. This reality can foster a sense of disconnect from the world since walking is a fundamental way in which people interact with each other and the environment. Advances in virtual reality and its immersive capabilities have made it possible to enable those who have never walked in their life to virtually experience walking. We co-designed a VR walking experience with a person with Spinal Muscular Atrophy who has been a lifelong wheelchair user. Over 9 days, we collected data on this person's experience through a diary study and analyzed this data to better understand the design elements required. Given that they had only ever seen others walking and had not experienced it first-hand, determining which design parameters must be considered in order to match the virtual experience to their idea of walking was challenging. Generally, we found the experience of walking to be quite positive, providing a perspective from a higher vantage point than what was available in a wheelchair. Our findings provide insights into the emotional complexities and evolving sense of agency accompanying virtual walking. These findings have implications for designing more inclusive and emotionally engaging virtual reality experiences.

cs.HC↗

arXiv2024

Virtual Buddy: Redefining Conversational AI Interactions for Individuals with Hand Motor Disabilities

Advances in artificial intelligence have transformed the paradigm of human-computer interaction, with the development of conversational AI systems playing a pivotal role. These systems employ technologies such as natural language processing and machine learning to simulate intelligent and human-like conversations. Driven by the personal experience of an individual with a neuromuscular disease who faces challenges with leaving home and contends with limited hand-motor control when operating digital systems, including conversational AI platforms, we propose a method aimed at enriching their interaction with conversational AI. Our prototype allows the creation of multiple agent personas based on hobbies and interests, to support topic-based conversations. In contrast with existing systems, such as Replika, that offer a 1:1 relation with a virtual agent, our design enables one-to-many relationships, easing the process of interaction for this individual by reducing the need for constant data input. We can imagine our prototype potentially helping others who are in a similar situation with reduced typing/input ability.

cs.HC↗

arXiv2024

ARfy: A Pipeline for Adapting 3D Scenes to Augmented Reality

Virtual content placement in physical scenes is a crucial aspect of augmented reality (AR). This task is particularly challenging when the virtual elements must adapt to multiple target physical environments that are unknown during development. AR authors use strategies such as manual placement performed by end-users, automated placement powered by author-defined constraints, and procedural content generation to adapt virtual content to physical spaces. Although effective, these options require human effort or annotated virtual assets. As an alternative, we present ARfy, a pipeline to support the adaptive placement of virtual content from pre-existing 3D scenes in arbitrary physical spaces. ARfy does not require intervention by end-users or asset annotation by AR authors. We demonstrate the pipeline capabilities using simulations on a publicly available indoor space dataset. ARfy automatically makes any generic 3D scene AR-ready and provides evaluation tools to facilitate future research on adaptive virtual content placement.

cs.HC↗

arXiv2024

ARLang: An Outdoor Augmented Reality Application for Portuguese Vocabulary Learning

With recent computer vision techniques and user-generated content, we can augment the physical world with metadata that describes attributes, such as names, geo-locations, and visual features of physical objects. To assess the benefits of these potentially ubiquitous labels for foreign vocabulary learning, we built a proof-of-concept system that displays bilingual text and sound labels on physical objects outdoors using augmented reality. Established tools for language learning have focused on effective content delivery methods such as books and flashcards. However, recent research and consumer learning tools have begun to focus on how learning can become more mobile, ubiquitous, and desirable. To test whether our system supports vocabulary learning, we conducted a preliminary between-subjects (N=44) study. Our results indicate that participants preferred learning with virtual labels on real-world objects outdoors over learning with flashcards. Our findings motivate further investigation into mobile AR-based learning systems in outdoor settings.

cs.HC↗