Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.24640

Say It or Grab It? Conversational Control of UI Adaptations in Mixed Reality

Sina Elahimanesh·Joao Marcelo Evangelista Belo·Isidora Jeknic·Alexander Koller·Anna Maria Feit
View original source ↗Read source document ↗Export metadata ↓

Abstract

Mixed reality interfaces must continuously adapt to a user's changing task and environment, yet adaptation policies are fixed in advance, either preset by the system or configured by the user. Natural language offers a way to state changing needs mid-task, but whether users can express interface behavior this way, and what it gains over direct manipulation, is unknown. We first elicited how users verbalize adaptation needs (N=19). We then built ConverseMR, which translates a spoken request into adaptation objectives and constraints and satisfies them continuously as the user's context changes. In a within-subjects study (N=24), conversational and direct control proved complementary: direct manipulation for precision, ConverseMR for reach and expressive breadth, with their combination rated highest. Spoken commands were rarely final and users frequently refined and replaced them upon experiencing the result, characteristic of, what we call, user-led adaptation, where users communicate adaption needs and the system identifies a solution.

Explore related subjects

cs.HC

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Sina Elahimanesh, Joao Marcelo Evangelista Belo, Isidora Jeknic, Alexander Koller, Anna Maria Feit. 2026-10-17. Say It or Grab It? Conversational Control of UI Adaptations in Mixed Reality. https://arxiv.org/abs/2610.24640

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

JumpStarter: Human-AI Planning with Task-Structured Context Curation

Human-AI collaboration on complex planning goals is bottlenecked by how LLM interfaces handle context: users must manually curate and re-surface relevant information across long and unstructured chat histories. Despite advances in long-context prompting and memory-augmented retrieval, this burden remains unresolved: users still have to identify and supply the right context at each decision point, regardless of how much the model can store or surface. We propose JumpStarter, a system that enables LLMs to collaborate with humans on complex goals by dynamically decomposing tasks to help users manage context. We specifically introduce task-structured context curation, a framework that breaks down a user's goal into a hierarchy of actionable subtasks and scopes context to localized decision points, enabling finer-grained personalization and reuse. The framework is realized through three core mechanisms: context elicitation, selection, and reuse. In a within-subjects user study, plans produced with JumpStarter were rated substantially higher in quality than those produced with ChatGPT. A complementary automatic simulation study shows that JumpStarter consistently outperforms ChatGPT baselines, planning and memory agents, and workflow ablations. These findings show that effective human-AI planning depends not on the volume of context provided, but on attaching the right context to the right subtask at the right time.

cs.HC↗
arXiv2026

What Drives Students' Use of AI Chatbots? Technology Acceptance in Conversational AI

Conversational AI tools have been rapidly adopted by students and are becoming part of their learning routines. This widespread use comes amid ongoing concerns about the reliability of AI-generated responses and the appropriate role of AI in education. Drawing on the Technology Acceptance Model, we examine how students' perceptions of conversational AI, including its usefulness, ease of use, trustworthiness, enjoyment, and social influence, relate to their intentions to use these tools for learning. We investigate both the direct and indirect relationships among these perceptions to understand how they are associated with students' behavioral intentions. Using survey data from 229 students and partial least squares structural equation modeling, we found that perceived usefulness was the strongest direct predictor of intention, followed by perceived enjoyment. Perceived ease of use, trust, and subjective norms did not have significant direct associations with intention. However, trust and subjective norms had significant total indirect associations, while perceived enjoyment had both significant direct and indirect associations. The model explained 70.2% of the variance in behavioral intention. These findings suggest that students' intentions to use conversational AI are associated less directly with effort-related considerations and more with perceived usefulness and affective experience. They also highlight the importance of understanding what students perceive as valuable in conversational AI interactions, particularly when designing tools intended to encourage engagement while supporting learning.

cs.HC↗
arXiv2026

GazeAgent: VLM-Guided Implicit Gaze Calibration from Egocentric Multimodal Behavior for AI Glasses

AI glasses typically require explicit, user-specific eye-tracker calibration, and accuracy can drift when the device shifts during use. We present GazeAgent, a VLM-guided framework that continuously and adaptively improves gaze-calibration accuracy from naturally occurring behavior without asking users to interrupt their activities. GazeAgent identifies precise visual attention (PVA) events from multimodal streams of inertial measurement unit (IMU), gaze, and wearer-hand signals, validates their task relevance, and localizes functional targets as pseudo-calibration targets. In a 10-participant scripted study, the system detected all 820 predefined PVA opportunities and, under device disturbance, reduced median gaze error by 44.1% compared with explicit calibration. In a five-participant case study, autonomous calibration produced the lowest workload and interruption ratings, and all participants preferred it for routine use. By deriving task-critical targets from natural multimodal behavior, GazeAgent demonstrates the potential for continuous, adaptive, and low-burden gaze interaction on egocentric AI glasses.

cs.HC↗