arXiv subjects

Jens Grubert

Publications and source records attributed to Jens Grubert.

At least 19 recordsLinked to original sources

arXiv2026

Extended Reality as a Mediation Layer for Situated Human Control in Human-Robot Teaming

Extended Reality (XR) is increasingly used in human-robot interaction to communicate robot intent, planned motion, reachability, and state. We argue that XR should also be understood as a mediation layer for situated human control in human-robot teaming. Situated human control denotes the human collaborator's ability to understand, shape, authorize, and interrupt robot action within the concrete physical, social, and temporal context in which that action unfolds. We ground this perspective in scenarios from robot-assisted bedside nursing, multi-arm supervisory control, and collaborative assembly under divided attention. Across these scenarios, robot autonomy must remain inspectable and adjustable as people move, goals change, sensing is incomplete, control roles shift, and plans become invalid. We identify four mediation functions connecting human intent and robot autonomy, robot plans and human judgment, levels of shared control, and team roles, handover, and recovery. Building on these functions, we derive six design dimensions: joint action possibilities, socio-physical constraints, uncertainty and plan validity, multimodal control and correction, roles, handover, and accountability, and anticipatory recovery. The paper outlines a research agenda for XR systems that make robot autonomy more actionable and accountable in dynamic shared environments.

cs.HC↗

arXiv2026

Beyond Conversations: Spatially-Anchored Previews for Intent Disambiguation in LLM-Assisted Geometry Editing in Virtual Reality

User intent disambiguation remains a key challenge in intelligent interactive systems. While they have been widely studied in dialogue systems in 2D interfaces, research on how intent disambiguation could be incorporated within Large Language Model (LLM) assisted editing workflows in immersive environments remains limited. Recent advances in LLMs create opportunities to leverage the immersive nature of virtual and augmented reality (VR/AR) environments to provide better disambiguation support. In this paper, we evaluate how traditional dialogue-based disambiguation can be augmented with spatially-anchored graphical previews to resolve ambiguous user commands in LLM-assisted parameter-driven editing workflows. A within-subjects study in which 24 participants completed complex geometry editing tasks in VR simulate scenarios where VR scenes are controlled by numerical parameters. Compared with the condition where disambiguation is not available, quantitative metrics and qualitative feedback indicate that a hybrid approach which combines clarification questions and graphical previews can support better interaction stability with fewer conversation rounds while improving user experience. These findings provide empirical evidence on the effectiveness of disambiguation methods in LLM-assisted editing of parameter-driven immersive scenes and inform design guidelines for future integration of LLMs in advanced VR/AR systems.

cs.HC↗

arXiv2026

An AI-Driven Virtual Patient for Breaking Bad News: An Expert Formative Study on Facial Expression Intensity

Interactive virtual patients driven by large language models (LLMs) offer scalable solutions for medical communication training, such as breaking bad news. However, designing their emotional expressiveness remains a challenge. This paper presents an AI-driven virtual patient framework combining LLM dialogue with real-time facial animation in virtual reality (VR). We conducted an exploratory, formative evaluation with seven medical experts to gather early feedback and elicit design requirements. The evaluation focused on how variations in facial expression intensity affect perceived realism and the virtual patient's emotion intelligibility. While descriptive quantitative ratings remained baseline across conditions, qualitative interviews provided deep insights into how experts perceive virtual emotional cues. The findings suggest that experts evaluate emotional realism holistically through multiple verbal and non-verbal channels; isolated facial adjustments are easily overshadowed by dialogue nuances and vocal prosody. Based on these insights, we present a roadmap for future medical training simulators, highlighting the need for synchronized, multi-modal pipelines incorporating body gestures, conversational pauses, and vocal dynamics.

cs.HC↗

arXiv2025

Towards Embodied Conversational Agents for Reducing Oral Exam Anxiety in Extended Reality

Oral examinations are a prevalent but psychologically demanding form of assessment in higher education. Many students experience intense anxiety, which can impair cognitive performance and hinder academic success. This position paper explores the potential of embodied conversational agents (ECAs) in extended reality (XR) environments to support students preparing for oral exams. We propose a system concept that integrates photorealistic ECAs with real-time capable large language models (LLMs) to enable psychologically safe, adaptive, and repeatable rehearsal of oral examination scenarios. We also discuss the potential benefits and challenges of such an envisioned system.

cs.HC↗

arXiv2025

ReachVox: Clutter-free Reachability Visualization for Robot Motion Planning in Virtual Reality

Human-Robot-Collaboration can enhance workflows by leveraging the mutual strengths of human operators and robots. Planning and understanding robot movements remain major challenges in this domain. This problem is prevalent in dynamic environments that might need constant robot motion path adaptation. In this paper, we investigate whether a minimalistic encoding of the reachability of a point near an object of interest, which we call ReachVox, can aid the collaboration between a remote operator and a robotic arm in VR. Through a user study (n=20), we indicate the strength of the visualization relative to a point-based reachability check-up.

cs.HC↗

arXiv2025

Behavioral and Symbolic Fillers as Delay Mitigation for Embodied Conversational Agents in Virtual Reality

When communicating with embodied conversational agents (ECAs) in virtual reality, there might be delays in the responses of the agents lasting several seconds, for example, due to more extensive computations of the answers when large language models are used. Such delays might lead to unnatural or frustrating interactions. In this paper, we investigate filler types to mitigate these effects and lead to a more positive experience and perception of the agent. In a within-subject study, we asked 24 participants to communicate with ECAs in virtual reality, comparing four strategies displayed during the delays: a multimodal behavioral filler consisting of conversational and gestural fillers, a base condition with only idle motions, and two symbolic indicators with progress bars, one embedded as a badge on the agent, the other one external and visualized as a thinking bubble. Our results indicate that the behavioral filler improved perceived response time, three subscales of presence, humanlikeness, and naturalness. Participants looked away from the face more often when symbolic indicators were displayed, but the visualizations did not lead to a more positive impression of the agent or to increased presence. The majority of participants preferred the behavioral fillers, only 12.5% and 4.2% favored the symbolic embedded and external conditions, respectively.

cs.HC↗

arXiv2025

Text-to-Image Generation for Vocabulary Learning Using the Keyword Method

The 'keyword method' is an effective technique for learning vocabulary of a foreign language. It involves creating a memorable visual link between what a word means and what its pronunciation in a foreign language sounds like in the learner's native language. However, these memorable visual links remain implicit in the people's mind and are not easy to remember for a large set of words. To enhance the memorisation and recall of the vocabulary, we developed an application that combines the keyword method with text-to-image generators to externalise the memorable visual links into visuals. These visuals represent additional stimuli during the memorisation process. To explore the effectiveness of this approach we first run a pilot study to investigate how difficult it is to externalise the descriptions of mental visualisations of memorable links, by asking participants to write them down. We used these descriptions as prompts for text-to-image generator (DALL-E2) to convert them into images and asked participants to select their favourites. Next, we compared different text-to-image generators (DALL-E2, Midjourney, Stable and Latent Diffusion) to evaluate the perceived quality of the generated images by each. Despite heterogeneous results, participants mostly preferred images generated by DALL-E2, which was used also for the final study. In this study, we investigated whether providing such images enhances the retention of vocabulary being learned, compared to the keyword method only. Our results indicate that people did not encounter difficulties describing their visualisations of memorable links and that providing corresponding images significantly improves memory retention.

cs.HC↗

arXiv2025

Spatial Bar: Exploring Window Switching Techniques for Large Virtual Displays

Virtual displays provided through head-worn displays (HWDs) offer users large screen space for productivity, but managing this space effectively presents challenges. This paper explores how to enhance window-switching strategies for virtual displays by leveraging eye tracking provided by HWDs and underutilized spaces around the main display area. We investigate the efficiency and usability of different cursor behaviors and selection modes in a Spatial Bar interface for window-switching tasks in augmented reality environments. Results show gaze coupled with teleport led to the quickest window-switching times, particularly in tasks where the original cursor position or the target window was far from the Spatial Bar.

cs.HC↗

arXiv2024

Generative AI for Accessible and Inclusive Extended Reality

Artificial Intelligence-Generated Content (AIGC) has the potential to transform how people build and interact with virtual environments. Within this paper, we discuss potential benefits but also challenges that AIGC has for the creation of inclusive and accessible virtual environments. Specifically, we touch upon the decreased need for 3D modeling expertise, benefits of symbolic-only as well as multimodal input, 3D content editing, and 3D model accessibility as well as foundation model-specific challenges.

cs.HC↗

arXiv2024

Analyzing Multimodal Interaction Strategies for LLM-Assisted Manipulation of 3D Scenes

As more applications of large language models (LLMs) for 3D content for immersive environments emerge, it is crucial to study user behaviour to identify interaction patterns and potential barriers to guide the future design of immersive content creation and editing systems which involve LLMs. In an empirical user study with 12 participants, we combine quantitative usage data with post-experience questionnaire feedback to reveal common interaction patterns and key barriers in LLM-assisted 3D scene editing systems. We identify opportunities for improving natural language interfaces in 3D design tools and propose design recommendations for future LLM-integrated 3D content creation systems. Through an empirical study, we demonstrate that LLM-assisted interactive systems can be used productively in immersive environments.

cs.HC↗

arXiv2024

Large Language Model-assisted Speech and Pointing Benefits Multiple 3D Object Selection in Virtual Reality

Selection of occluded objects is a challenging problem in virtual reality, even more so if multiple objects are involved. With the advent of new artificial intelligence technologies, we explore the possibility of leveraging large language models to assist multi-object selection tasks in virtual reality via a multimodal speech and raycast interaction technique. We validate the findings in a comparative user study (n=24), where participants selected target objects in a virtual reality scene with different levels of scene perplexity. The performance metrics and user experience metrics are compared against a mini-map based occluded object selection technique that serves as the baseline. Results indicate that the introduced technique, AssistVR, outperforms the baseline technique when there are multiple target objects. Contrary to the common belief for speech interfaces, AssistVR was able to outperform the baseline even when the target objects were difficult to reference verbally. This work demonstrates the viability and interaction potential of an intelligent multimodal interactive system powered by large laguage models. Based on the results, we discuss the implications for design of future intelligent multimodal interactive systems in immersive environments.

cs.HC↗

arXiv2024

Working with Mixed Reality in Public: Effects of Virtual Display Layouts on Productivity, Feeling of Safety, and Social Acceptability

Nowadays, Mixed Reality (MR) headsets are a game-changer for knowledge work. Unlike stationary monitors, MR headsets allow users to work with large virtual displays anywhere they wear the headset, whether in a professional office, a public setting like a cafe, or a quiet space like a library. This study compares four different layouts (eye level-close, eye level-far, below eye level-close, below eye level-far) of virtual displays regarding feelings of safety, perceived productivity, and social acceptability when working with MR in public. We test which layout is most preferred by users and seek to understand which factors affect users' layout preferences. The aim is to derive useful insights for designing better MR layouts. A field study in a public library was conducted using a within-subject design. While the participants interact with a layout, they are asked to work on a planning task. The results from a repeated measure ANOVA show a statistically significant effect on productivity but not on safety and social acceptability. Additionally, we report preferences expressed by the users regarding the layouts and using MR in public.

cs.HC↗

arXiv2024

Accented Character Entry Using Physical Keyboards in Virtual Reality

Research on text entry in Virtual Reality (VR) has gained popularity but the efficient entry of accented characters, characters with diacritical marks, in VR remains underexplored. Entering accented characters is supported on most capacitive touch keyboards through a long press on a base character and a subsequent selection of the accented character. However, entering those characters on physical keyboards is still challenging, as they require a recall and an entry of respective numeric codes. To address this issue this paper investigates three techniques to support accented character entry on physical keyboards in VR. Specifically, we compare a context-aware numeric code technique that does not require users to recall a code, a key-press-only condition in which the accented characters are dynamically remapped to physical keys next to a base character, and a multimodal technique, in which eye gaze is used to select the accented version of a base character previously selected by key-press on the keyboard. The results from our user study (n=18) reveal that both the key-press-only and the multimodal technique outperform the baseline technique in terms of text entry speed.

cs.HC↗

arXiv2024

Working in Extended Reality in the Wild: Worker and Bystander Experiences of XR Virtual Displays in Real-World Settings

Although access to sufficient screen space is crucial to knowledge work, workers often find themselves with limited access to display infrastructure in remote or public settings. While virtual displays can be used to extend the available screen space through extended reality (XR) head-worn displays (HWD), we must better understand the implications of working with them in public settings from both users' and bystanders' viewpoints. To this end, we conducted two user studies. We first explored the usage of a hybrid AR display across real-world settings and tasks. We focused on how users take advantage of virtual displays and what social and environmental factors impact their usage of the system. A second study investigated the differences between working with a laptop, an AR system, or a VR system in public. We focused on a single location and participants performed a predefined task to enable direct comparisons between the conditions while also gathering data from bystanders. The combined results suggest a positive acceptance of XR technology in public settings and show that virtual displays can be used to accompany existing devices. We highlighted some environmental and social factors. We saw that previous XR experience and personality can influence how people perceive the use of XR in public. In addition, we confirmed that using XR in public still makes users stand out and that bystanders are curious about the devices, yet have no clear understanding of how they can be used.

cs.HC↗

arXiv2024

Hold Tight: Identifying Behavioral Patterns During Prolonged Work in VR through Video Analysis

VR devices have recently been actively promoted as tools for knowledge workers and prior work has demonstrated that VR can support some knowledge worker tasks. However, only a few studies have explored the effects of prolonged use of VR such as a study observing 16 participant working in VR and a physical environment for one work-week each and reporting mainly on subjective feedback. As a nuanced understanding of participants' behavior in VR and how it evolves over time is still missing, we report on the results from an analysis of 559 hours of video material obtained in this prior study. Among other findings, we report that (1) the frequency of actions related to adjusting the headset reduced by 46% and the frequency of actions related to supporting the headset reduced by 42% over the five days; (2) the HMD was removed 31% less frequently over the five days but for 41% longer periods; (3) wearing an HMD is disruptive to normal patterns of eating and drinking, but not to social interactions, such as talking. The combined findings in this work demonstrate the value of long-term studies of deployed VR systems and can be used to inform the design of better, more ergonomic VR systems as tools for knowledge workers.

cs.HC↗

arXiv2023

Working with XR in Public: Effects on Users and Bystanders

Recent commercial off-the-shelf virtual and augmented reality devices have been promoted as tools for knowledge work and research findings show how this kind of work can benefit from the affordances of extended reality (XR). One major advantage that XR can provide is the enlarged display space that can be used to display virtual screens which is a feature already readily available in many commercial devices. This could be especially helpful in mobile contexts, in which users might not have access to their optimal physical work setup. Such situations often occur in a public setting, for example when working on a train while traveling to a business meeting. At the same time, the use of XR devices is still uncommon in public, which might impact both users and bystanders. Hence, there is a need to better understand the implications of using XR devices for work in public both on the user itself, as well as on bystanders. We report the results of a study in a university cafeteria in which participants used three different systems. In one setup they only used a laptop with a single screen, in a second setup, they combined the laptop with an optical see-through AR headset, and in the third, they combined the laptop with an immersive VR headset. In addition, we also collected 231 responses from bystanders through a questionnaire. The combined results indicate that (1) users feel safer if they can see their physical surroundings; (2) current use of XR in public makes users stand out; and (3) prior XR experience can influence how users feel when using XR in public.

cs.HC↗

arXiv2023

Text Entry Performance and Situation Awareness of a Joint Optical See-Through Head-Mounted Display and Smartphone System

Optical see-through head-mounted displays (OST HMDs) are a popular output medium for mobile Augmented Reality (AR) applications. To date, they lack efficient text entry techniques. Smartphones are a major text entry medium in mobile contexts but attentional demands can contribute to accidents while typing on the go. Mobile multi-display ecologies, such as combined OST HMD-smartphone systems, promise performance and situation awareness benefits over single-device use. We study the joint performance of text entry on mobile phones with text output on optical see-through head-mounted displays. A series of five experiments with a total of 86 participants indicate that, as of today, the challenges in such a joint interactive system outweigh the potential benefits.

cs.HC↗

arXiv2023

The Effect of an Exergame on the Shadow Play Skill Based on Muscle Memory for Young Female Participants: The Case of Forehand Drive in Table Tennis

Learning and practicing table tennis with traditional methods is a long, tedious process and may even lead to the internalization of incorrect techniques if not supervised by a coach. To overcome these issues, the presented study proposes an exergame with the aim of enhancing young female novice players' performance by boosting muscle memory, making practice more interesting, and decreasing the probability of faulty training. Specifically, we propose an exergame based on skeleton tracking and a virtual avatar to support correct shadow practice to learn forehand drive technique without the presence of a coach. We recruited 44 schoolgirls aged between 8 and 12 years without a background in playing table tennis and divided them into control and experimental groups. We examined their stroke skills (via the Mott-Lockhart test) and the error coefficient of their forehand drives (using a ball machine) in the pretest, post-test, and follow-up tests (10 days after the post-test). Our results showed that the experimental group had progress in the short and long term, while the control group had an improvement only in the short term. Further, the scale of improvement in the experimental group was significantly higher than in the control group. Given that the early stages of learning, particularly in girls children, are important in the internalization of individual skills in would-be athletes, this method could support promoting correct training for young females.

cs.HC↗