arXiv subjects

Jörg Müller

Publications and source records attributed to Jörg Müller.

At least 19 recordsLinked to original sources

arXiv2025

From Brick to Click: Comparing LEGO Building in Virtual Reality and the Physical World

We present a comparative study of building with LEGO in three environments: the physical world, a Virtual Reality (VR) counterpart, and a VR setting enhanced with "superpowers". The study aims to understand how traditional creative hands-on activities translate to virtual environments, with potential benefits for educational, training, entertainment, and therapeutic uses. 22 participants engaged in both structured assembly and creative free-building tasks across these environments. We investigated differences in user performance, engagement, and creativity, with a focus on how the additional VR functionalities influenced the building experience. The findings reveal that while the physical environment offers a familiar tactile experience, VR, particularly with added superpowers, was clearly favoured by participants in the creative free-building scenario. Our recommendations for VR design include balancing automation with user control to enhance task efficiency while maintaining engagement, and implementing intuitive systems that manage complexity to prevent user overwhelm and support creative freedom.

cs.HC↗

arXiv2024

Working in Extended Reality in the Wild: Worker and Bystander Experiences of XR Virtual Displays in Real-World Settings

Although access to sufficient screen space is crucial to knowledge work, workers often find themselves with limited access to display infrastructure in remote or public settings. While virtual displays can be used to extend the available screen space through extended reality (XR) head-worn displays (HWD), we must better understand the implications of working with them in public settings from both users' and bystanders' viewpoints. To this end, we conducted two user studies. We first explored the usage of a hybrid AR display across real-world settings and tasks. We focused on how users take advantage of virtual displays and what social and environmental factors impact their usage of the system. A second study investigated the differences between working with a laptop, an AR system, or a VR system in public. We focused on a single location and participants performed a predefined task to enable direct comparisons between the conditions while also gathering data from bystanders. The combined results suggest a positive acceptance of XR technology in public settings and show that virtual displays can be used to accompany existing devices. We highlighted some environmental and social factors. We saw that previous XR experience and personality can influence how people perceive the use of XR in public. In addition, we confirmed that using XR in public still makes users stand out and that bystanders are curious about the devices, yet have no clear understanding of how they can be used.

cs.HC↗

arXiv2024

SIM2VR: Towards Automated Biomechanical Testing in VR

Automated biomechanical testing has great potential for the development of VR applications, as initial insights into user behaviour can be gained in silico early in the design process. In particular, it allows prediction of user movements and ergonomic variables, such as fatigue, prior to conducting user studies. However, there is a fundamental disconnect between simulators hosting state-of-the-art biomechanical user models and simulators used to develop and run VR applications. Existing user simulators often struggle to capture the intricacies of real-world VR applications, reducing ecological validity of user predictions. In this paper, we introduce SIM2VR, a system that aligns user simulation with a given VR application by establishing a continuous closed loop between the two processes. This, for the first time, enables training simulated users directly in the same VR application that real users interact with. We demonstrate that SIM2VR can predict differences in user performance, ergonomics and strategies in a fast-paced, dynamic arcade game. In order to expand the scope of automated biomechanical testing beyond simple visuomotor tasks, advances in cognitive models and reward function design will be needed.

cs.HC↗

arXiv2024

Efficient and Interpretable Traffic Destination Prediction using Explainable Boosting Machines

Developing accurate models for traffic trajectory predictions is crucial for achieving fully autonomous driving. Various deep neural network models have been employed to address this challenge, but their black-box nature hinders transparency and debugging capabilities in a deployed system. Glass-box models offer a solution by providing full interpretability through methods like \ac{GAM}. In this study, we evaluate an efficient additive model called \ac{EBM} for traffic prediction on three popular mixed traffic datasets: \ac{SDD}, \ac{InD}, and Argoverse. Our results show that the \ac{EBM} models perform competitively in predicting pedestrian destinations within \ac{SDD} and \ac{InD} while providing modest predictions for vehicle-dominant Argoverse dataset. Additionally, our transparent trained models allow us to analyse feature importance and interactions, as well as provide qualitative examples of predictions explanation. The full training code will be made public upon publication.

cs.LG↗

arXiv2024

Hold Tight: Identifying Behavioral Patterns During Prolonged Work in VR through Video Analysis

VR devices have recently been actively promoted as tools for knowledge workers and prior work has demonstrated that VR can support some knowledge worker tasks. However, only a few studies have explored the effects of prolonged use of VR such as a study observing 16 participant working in VR and a physical environment for one work-week each and reporting mainly on subjective feedback. As a nuanced understanding of participants' behavior in VR and how it evolves over time is still missing, we report on the results from an analysis of 559 hours of video material obtained in this prior study. Among other findings, we report that (1) the frequency of actions related to adjusting the headset reduced by 46% and the frequency of actions related to supporting the headset reduced by 42% over the five days; (2) the HMD was removed 31% less frequently over the five days but for 41% longer periods; (3) wearing an HMD is disruptive to normal patterns of eating and drinking, but not to social interactions, such as talking. The combined findings in this work demonstrate the value of long-term studies of deployed VR systems and can be used to inform the design of better, more ergonomic VR systems as tools for knowledge workers.

cs.HC↗

arXiv2023

Working with XR in Public: Effects on Users and Bystanders

Recent commercial off-the-shelf virtual and augmented reality devices have been promoted as tools for knowledge work and research findings show how this kind of work can benefit from the affordances of extended reality (XR). One major advantage that XR can provide is the enlarged display space that can be used to display virtual screens which is a feature already readily available in many commercial devices. This could be especially helpful in mobile contexts, in which users might not have access to their optimal physical work setup. Such situations often occur in a public setting, for example when working on a train while traveling to a business meeting. At the same time, the use of XR devices is still uncommon in public, which might impact both users and bystanders. Hence, there is a need to better understand the implications of using XR devices for work in public both on the user itself, as well as on bystanders. We report the results of a study in a university cafeteria in which participants used three different systems. In one setup they only used a laptop with a single screen, in a second setup, they combined the laptop with an optical see-through AR headset, and in the third, they combined the laptop with an immersive VR headset. In addition, we also collected 231 responses from bystanders through a questionnaire. The combined results indicate that (1) users feel safer if they can see their physical surroundings; (2) current use of XR in public makes users stand out; and (3) prior XR experience can influence how users feel when using XR in public.

cs.HC↗

arXiv2023

Video Analysis of Behavioral Patterns During Prolonged Work in VR

VR has recently been promoted as a tool for knowledge workers and studies have shown that it has the potential to improve knowledge work. However, studies on its prolonged use have been scarce. A prior study compared working in VR for one week to working in a physical environment, focusing on performance measures and subjective feedback. However, a nuanced understanding and comparison of participants' behavior in VR and the physical environment is still missing. To this end, we analyzed video material made available from this previously conducted experiment, carried out over a working week, and present our findings on comparing the behavior of participants while working in VR and in a physical environment.

cs.HC↗

arXiv2023

Feel the Breeze: Promoting Relaxation in Virtual Reality using Mid-Air Haptics

Mid-air haptic interfaces employ focused ultrasound waves to generate touchless haptic sensations on the skin. Prior studies have demonstrated the potential positive impact of mid-air haptic feedback on virtual experiences, enhancing aspects such as enjoyment, immersion, and sense of agency. As a highly immersive environment, Virtual Reality (VR) is being explored as a tool for stress management and relaxation in current research. However, the impact of incorporating mid-air haptic stimuli into relaxing experiences in VR has not been studied thus far. In this paper, for the first time, we design a mid-air haptic stimulation that is congruent with a relaxing scene in VR, and conduct a user study investigating the effectiveness of this experience. Our user study encompasses three different conditions: a control group with no relaxation intervention, a VR-only relaxation experience, and a VR+Haptics relaxation experience that includes the mid-air haptic feedback. While we did not find any significant differences between the conditions, a trend suggesting that the VR+Haptics condition might be associated with greater pleasure emerged, requiring further validation with a larger sample size. These initial findings set the foundation for future investigations into leveraging multimodal interventions in VR, utilising mid-air haptics to potentially enhance relaxation experiences.

cs.HC↗

arXiv2023

The Impact of Different Virtual Work Environments on Flow, Performance, User Emotions, and Preferences

This research explores how different virtual work environments, differing in the type and amount of elements they include, impact users' flow, performance, emotional state, and preferences. Pre-study interviews were conducted to inform the design of three VR work environments: the Dark Room, the Empty Room, and the Furnished Room. Fifteen participants took part in a user study where they engaged in a logic-based task simulating deep work while experiencing each environment. The findings suggest that while objective performance measures did not differ significantly, subjective experiences and perceptions varied across the environments. Participants reported feeling less distracted and more focused in the Dark Room and the Empty Room compared to the Furnished Room. The Empty Room was associated with the highest levels of relaxation and calmness, while the Furnished Room was perceived as visually appealing yet more distracting. These findings highlight the variability of user preferences and emphasise the importance of considering user comfort and well-being in the design of virtual work environments. The study contributes to the better understanding of virtual workspaces and provides insights for designing environments that promote flow, productivity, and user well-being.

cs.HC↗

arXiv2023

Towards a Deep(er) Understanding of Interaction through Modeling, Simulation, and Optimization

The traditional user-centered design process can hardly keep up with the ever faster technical development and increasingly diverse user preferences. As a solution, we propose to augment the tried-and-tested approach of conducting user studies with simulation and optimization of the entire human-computer interaction loop. This approach allows to better understand phenomena through explicit modeling, build virtual prototypes through simulation, and improve interaction techniques through optimization. Building predictive user models also supports the creation and validation of HCI theories, and constitutes a decisive step towards new, intelligent, and adaptive user interfaces. We report our experience in virtually developing new interaction techniques on the example of acoustic levitation, and present our optimization-based framework for HCI. With this, we strive to gain a better understanding of interaction and at the same time feed the discussion on questions such as which tools and tutorials are necessary to make virtual prototyping more accessible to different audiences.

cs.HC↗

arXiv2022

Reinforcement Learning Control of a Biomechanical Model of the Upper Extremity

Among the infinite number of possible movements that can be produced, humans are commonly assumed to choose those that optimize criteria such as minimizing movement time, subject to certain movement constraints like signal-dependent and constant motor noise. While so far these assumptions have only been evaluated for simplified point-mass or planar models, we address the question of whether they can predict reaching movements in a full skeletal model of the human upper extremity. We learn a control policy using a motor babbling approach as implemented in reinforcement learning, using aimed movements of the tip of the right index finger towards randomly placed 3D targets of varying size. We use a state-of-the-art biomechanical model, which includes seven actuated degrees of freedom. To deal with the curse of dimensionality, we use a simplified second-order muscle model, acting at each degree of freedom instead of individual muscles. The results confirm that the assumptions of signal-dependent and constant motor noise, together with the objective of movement time minimization, are sufficient for a state-of-the-art skeletal model of the human upper extremity to reproduce complex phenomena of human movement, in particular Fitts' Law and the 2/3 Power Law. This result supports the notion that control of the complex human biomechanical system can plausibly be determined by a set of simple assumptions and can easily be learned.

q-bio.QM↗

arXiv2022

Optimal Feedback Control for Modeling Human-Computer Interaction

Optimal feedback control (OFC) is a theory from the motor control literature that explains how humans move their body to achieve a certain goal, e.g., pointing with the finger. OFC is based on the assumption that humans aim to control their body optimally, within the constraints imposed by body, environment, and task. In this paper, we explain how this theory can be applied to understanding Human-Computer Interaction (HCI) in the case of pointing. We propose that the human body and computer dynamics can be interpreted as a single dynamical system. The system state is controlled by the user via muscle control signals, and estimated from observations. Between-trial variability arises from signal-dependent control noise and observation noise. We compare four different models from optimal control theory and evaluate to what degree these models can replicate movements in the case of mouse pointing. We introduce a procedure to identify parameters that best explain observed user behavior. To support HCI researchers in simulating, analyzing, and optimizing interaction movements, we provide the Python toolbox OFC4HCI. We conclude that OFC presents a powerful framework for HCI to understand and simulate motion of the human body and of the interface on a moment by moment basis.

cs.HC↗

arXiv2022

Simulating Interaction Movements via Model Predictive Control

We present a method to simulate movement in interaction with computers, using Model Predictive Control (MPC). The method starts from understanding interaction from an Optimal Feedback Control (OFC) perspective. We assume that users aim to minimize an internalized cost function, subject to the constraints imposed by the human body and the interactive system. In contrast to previous linear approaches used in HCI, MPC can compute optimal controls for nonlinear systems. This allows us to use state-of-the-art biomechanical models and handle nonlinearities that occur in almost any interactive system. Instead of torque actuation, our model employs second-order muscles acting directly at the joints. We compare three different cost functions and evaluate the simulated trajectories against user movements in a Fitts' Law type pointing study with four different interaction techniques. Our results show that the combination of distance, control, and joint acceleration cost matches individual users' movements best, and predicts movements with an accuracy that is within the between-user variance. To aid HCI researchers and designers, we introduce CFAT, a novel method to identify maximum voluntary torques in joint-actuated models based on experimental data, and give practical advice on how to simulate human movement for different users, interaction techniques, and tasks.

cs.HC↗

arXiv2022

OptiTrap: Optimal Trap Trajectories for Acoustic Levitation Displays

Acoustic levitation has recently demonstrated the ability to create volumetric content by trapping and quickly moving particles along reference paths to reveal shapes in mid-air. However, the problem of specifying physically feasible trap trajectories to display desired shapes remains unsolved. Even if only the final shape is of interest to the content creator, the trap trajectories need to determine where and when the traps need to be, for the particle to reveal the intended shape. We propose OptiTrap, the first structured numerical approach to compute trap trajectories for acoustic levitation displays. Our approach generates trap trajectories that are physically feasible and nearly time-optimal, and reveal generic mid-air shapes, given only a reference path (i.e., a shape with no time information). We provide a multi-dimensional model of the acoustic forces around a trap to model the trap-particle system dynamics and compute optimal trap trajectories by formulating and solving a non-linear path following problem. We formulate our approach and evaluate it, demonstrating how OptiTrap consistently produces feasible and nearly optimal paths, with increases in size, frequency, and accuracy of the shapes rendered, allowing us to demonstrate larger and more complex shapes than ever shown to date.

cs.HC↗

arXiv2022

Evoking realistic affective touch experiences in virtual reality

This study aims to better understand the emotional and physiological correlates of being caressed in VR depending on the type of multisensory feedback provided and the animate or inanimate nature of the virtual representation that touches an embodied virtual body. We evaluated how pleasure, arousal, embodiment, and the illusion of being touched in VR were influenced by the inclusion of only visual feedback compared to visuotactile stimulation conditions, where participants, in addition to seeing an avatar or feather caressing their virtual bodies, also perceived congruent mid-air ultrasonic tactile stimulation or real interpersonal touch. We found that visuotactile feedback, either based on ultrasound or real interpersonal touch, boosts the illusion of being affectively touched and embodied in a virtual body compared to conditions only based on visual feedback. However, real interpersonal touch led to the strongest behavioral and emotional responses compared to the other conditions. Moreover, arousal and the desire to withdraw the caressed hand was highest when being touched by a female avatar compared to a virtual feather. Female participants reported a stronger illusion of being caressed in VR compared to males. Overall, this study advances knowledge of the emotional and physiological impact of affective touch in VR.

cs.HC↗

arXiv2021

Intermittent control as a model of mouse movements

We present Intermittent Control (IC) models as a candidate framework for modelling human input movements in Human--Computer Interaction (HCI). IC differs from continuous control in that users are not assumed to use feedback to adjust their movements continuously, but only when the difference between the observed pointer position and predicted pointer positions become large. We use a parameter optimisation approach to identify the parameters of an intermittent controller from experimental data, where users performed one-dimensional mouse movements in a reciprocal pointing task. Compared to previous published work with continuous control models, based on the Kullback-Leibler divergence from the experimental observations, IC is better able to generatively reproduce the distinctive dynamical features and variability of the pointing task across participants and over repeated tasks. IC is compatible with current physiological and psychological theory and provides insight into the source of variability in HCI tasks.

cs.HC↗

arXiv2020

Impact of Visuomotor Feedback on the Embodiment of Virtual Hands Detached from the Body

It has been shown that mere observation of body discontinuity leads to diminished body ownership. However, the impact of body discontinuity has mainly been investigated in conditions where participants observe a collocated static virtual body from a first-person perspective. This study explores the influence of body discountinuity on the sense of embodiment, when rich visuomotor correlations between a real and an artificial virtual body are established. In two experiments, we evaluated body ownership and motor performance, when participants interacted in virtual reality either using virtual hands connected or disconnected from a body. We found that even under the presence of congruent visuomotor feedback, mere observation of body discontinuity resulted in diminished embodiment. Contradictory evidence was found in relation to motor performance, where further research is needed to understand the role of visual body discontinuity in motor tasks. Preliminary findings on physiological reactions to a threat were also assessed, indicating that body visual discontinuity does not differently impact threat-related skin conductance responses. The present results are in accordance with past evidence showing that body discontinuity negatively impacts embodiment. However, further research is needed to understand the influence of visuomotor feedback and body morphological congruency on motor performance and threat-related physiological reactions.

cs.HC↗

arXiv2020

Impact of Information Placement and User Representations in VR on Performance and Embodiment

Human sensory processing is sensitive to the proximity of stimuli to the body. It is therefore plausible that these perceptual mechanisms also modulate the detectability of content in VR, depending on its location. We evaluate this in a user study and further explore the impact of the user's representation during interaction. We also analyze how embodiment and motor performance are influenced by these factors. In a dual-task paradigm, participants executed a motor task, either through virtual hands, virtual controllers, or a keyboard. Simultaneously, they detected visual stimuli appearing in different locations. We found that, while actively performing a motor task in the virtual environment, performance in detecting additional visual stimuli is higher when presented near the user's body. This effect is independent of how the user is represented and only occurs when the user is also engaged in a secondary task. We further found improved motor performance and increased embodiment when interacting through virtual tools and hands in VR, compared to interacting with a keyboard. This study contributes to better understanding the detectability of visual content in VR, depending on its location in the virtual environment, as well as the impact of different user representations on information processing, embodiment, and motor performance.

cs.HC↗