arXiv subjects

Giulio Jacucci

Publications and source records attributed to Giulio Jacucci.

13 recordsLinked to original sources

arXiv2026

Lexical Coupling in GUI Element Grounding: Sentence Embeddings Track Labels across Mobile and Web

GUI grounding evaluations that expose UI elements as text metadata often treat high instruction-element embedding similarity as evidence of semantic grounding. Across three mobile and web benchmarks, we show that this interpretation is frequently confounded by visible-label recovery. Lexical baselines remain competitive at top-1, label-poor targets remain weak for text-only methods, and encoder top-1 hits are predictable from lexical rank, candidate-pool size, and label type. We evaluate each action as a same-screen ranking task, comparing five off-the-shelf single-vector encoders with lexical baselines. Encoders recover some lexical misses, but deployable fusion gains are much smaller than target-aware oracle gains. These findings show that embedding-based evaluations can conflate visible-label recovery with semantic GUI grounding. Embedding-based evaluations should therefore report lexical baselines, label-type stratification, and deployable-fusion diagnostics. Our released repository provides analysis scripts and detexted per-step panels: https://github.com/qijia123/lexical-coupling-release.

cs.CL↗

arXiv2026

SkillDroid: Compile Once, Reuse Forever

LLM-based mobile GUI agents treat every task invocation as an independent reasoning episode, requiring a full LLM inference call at each action step. This per-step dependence makes them stateless: a task completed successfully yesterday is re-derived from scratch today, with no improvement in reliability or speed. We present SkillDroid, a three-layer skill agent that compiles successful LLM-guided GUI trajectories into parameterized skill templates (sequences of UI actions with weighted element locators and typed parameter slots) and replays them on future invocations without any LLM calls. A matching cascade (regex patterns, embedding similarity, and app filtering) routes incoming instructions to stored skills, while a failure-learning layer triggers recompilation when skill reliability degrades. Over a 150-round longitudinal evaluation with systematic instruction variation and controlled perturbations, SkillDroid achieves an 85.3% success rate (23 percentage points above a stateless LLM baseline) while using 49% fewer LLM calls. The skill replay mechanism achieves a perfect 1000% success rate across 79 replay rounds at 2.4 times the speed of full LLM execution. Most critically, the system improves with use: its success rate converges upward from 87% to 91%, while the baseline degrades from 80% to 44%.

cs.HC↗

arXiv2026

Understanding Newcomer Persistence in Social VR: A Case Study of VRChat

Newcomers are crucial for the growth of online communities, yet their successful integration into these spaces requires overcoming significant initial hurdles. Social Virtual Reality (VR) platforms are novel avenues that offer unprecedented online interaction experiences. Unlike well-studied two-dimensional online environments, the pathways to successful newcomer integration in online VR spaces are underexplored. Our research addresses this gap by examining the strategies used by newcomers to navigate early challenges in social VR and how they adapt. By focusing on active participants (ranging from newcomers currently navigating these hurdles to veterans who have successfully integrated) we isolate the specific strategies necessary for retention. We interviewed 24 active social VR users and conducted a reflexive thematic analysis. While participants identified barriers such as unfamiliar user interfaces, social norms, and overwhelming sensory input, our analysis reveals the adaptation strategies required to overcome them. Our findings expand on understanding newcomer persistence beyond traditional 2D environments, emphasizing how social dynamics influence the management of VR-specific issues like VR sickness during onboarding. Additionally, we highlight how successful newcomers overcome the lack of clear objectives in social VR by proactively constructing social meaning. We propose design suggestions to scaffold these successful integration pathways.

cs.HC↗

arXiv2026

Social, Spatial, and Self-Presence as Predictors of Basic Psychological Need Satisfaction in Social Virtual Reality

Extensive research has examined presence and basic psychological needs (drawing on Self-Determination Theory) in digital media. While prior work offers hints of potential connections, we lack a systematic account of whether and how distinct presence dimensions map onto the basic needs of autonomy, competence, and relatedness. We surveyed 301 social VR users and analyzed using Structural Equation Modeling. Results show that social presence predicts all three needs, while self-presence predicts competence and relatedness, and spatial presence shows no direct or moderating effects. Gender and age moderated these relationships: women benefited more from social presence for autonomy and relatedness, men from self- and spatial presence for competence and autonomy, and younger users showed stronger associations between social presence and relatedness, and between self-presence and autonomy. These findings position presence as a motivational mechanism shaped by demographic factors. The results offer theoretical insights and practical implications for designing inclusive, need-supportive multiuser VR environments.

cs.HC↗

arXiv2026

Usage Matters: The Role of Frequency, Duration, and Experience in Presence Formation in Social Virtual Reality

The sense of presence is central to immersive experiences in Virtual Reality (VR), and particularly salient in socially rich platforms like social VR. While prior studies have explored various aspects related to presence, less is known about how ongoing usage behaviors shape presence in everyday engagement. To address this gap, we examine whether usage intensity, captured through frequency of use, session duration, and years of VR experience, predicts presence in social VR. A survey of 295 users assessed overall, social, spatial, and self-presence using validated scales. Results show that both frequency and duration consistently predict higher presence across all dimensions, with interaction effects indicating that frequent and extended sessions synergistically amplify the experience of "being there." These effects were stable across age and gender. Our findings extend presence research beyond the laboratory by identifying behavioral predictors in social VR and offer insights for building inclusive environments that reliably foster presence.

cs.HC↗

arXiv2023

Supporting Management of Gestational Diabetes with Comprehensive Self-Tracking: Mixed-Method Study of Wearable Sensors

Gestational diabetes (GDM) poses a growing health risk to both pregnant women and their offspring. While telehealth interventions for GDM management have proven effective, they have traditionally relied on healthcare professionals for guidance and feedback. Our aim was to explore self-tracking in GDM with wearable sensors from self-discovery (i.e., learning associations between glucose levels and lifestyle) and user experience perspectives. We conducted a mixed-methods study with women diagnosed with GDM, utilizing continuous glucose monitor and three types of physical activity sensors (activity bracelet, hip-worn sensor, and electrocardiography sensor) for a week. Data from the sensors was collected, and participants were later interviewed about their experience with the wearable sensors. Additionally, we gathered maternal nutrition data through a 3-day food diary and recorded self-reported physical activity using a logbook. We discovered that continuous glucose monitors were especially valuable for self-discovery, particularly when establishing links between glucose levels and nutritional intake. Challenges associated with using wearable sensors data for self-discovery in GDM included: (1) Separation of glucose and physical activity data in different applications, (2) Missing key trackable features, such as light physical activity and non-walking activities, (3) Discrepancies in data, and (4) Differences in perceived versus measured physical activity. The placement of sensors on the body emerged as a critical factor influencing data quality and personal preferences. To conclude, an app where glucose, nutrition, and physical activity data are combined is needed to support self-discovery. This app should enable tracking of essential features for women with GDM, including light physical activity, with data originating from a single sensor to ensure consistency and eliminate redundancy.

cs.HC↗

arXiv2023

Revisiting Embodiment for Brain-Computer Interfaces

Researchers increasingly explore deploying brain-computer interfaces (BCIs) for able-bodied users, with the motivation of accessing mental states more directly than allowed by existing body-mediated interaction. This motivation seems to contradict the long-standing HCI emphasis on embodiment, namely the general claim that the body is crucial for cognition. This paper addresses this apparent contradiction through a review of insights from embodied cognition and interaction. We first critically examine the recent interest in BCIs and identify the extent cognition in the brain is integrated with the wider body as a central concern for research. We then define the implications of an integrated view of cognition for interface design and evaluation. A counterintuitive conclusion we draw is that embodiment per se should not imply a preference for body movements over brain signals. Yet it can nevertheless guide research by 1) providing body-grounded explanations for BCI performance, 2) proposing evaluation considerations that are neglected in modular views of cognition, and 3) through the direct transfer of its design insights to BCIs. We finally reflect on HCI's understanding of embodiment and identify the neural dimension of embodiment as hitherto overlooked.

cs.HC↗

arXiv2022

Characterizing Visualization Insights through Entity-Based Interaction: An Exploratory Study

One of the primary purposes of visualization is to assist users in discovering insights. While there has been much research in information visualization aiming at complex data transformation and novel presentation techniques, relatively little has been done to understand how users derive insights through interactive visualization of data. This paper presents a crowdsourced study with 158 participants investigating the relation between entity-based interaction (an action + its target entity) and the resulting insight. To this end, we generalized the interaction with an existing CO2 Explorer as entity-based interaction and enabled users to input notes and refer to relevant entities to assist their narratives. We logged interactions of users freely exploring the visualization and characterized their externalized insights about the data. Using entity-based interactions and references to infer insight characteristics (category, overview versus detail, and prior knowledge), we found evidence that compared with interactions, entity references improved insight characterization from slight/fair to fair/moderate agreements. To interpret prediction outcomes, feature importance and correlation analysis indicated that, e.g., detailed insights tended to have more mouse-overs in the chart area and cite the vertical reference lines in the line chart as evidence. We discuss study limitations and implications on knowledge-assisted visualization, e.g., insight recommendations based on user exploration.

cs.HC↗

arXiv2021

Interactive Visual Facets to Support Fluid Exploratory Search

Exploratory search starts with ill-defined goals and involves browsing, learning, and formulating new targets for search. To fluidly support such dynamic search behaviours, we focus on devising interactive visual facets (IVF), visualising information facets to support user comprehension and control of the information space. To do this, we reviewed existing faceted search interfaces and derived two design requirements (DR) that have not been fully addressed to support fluid interactions in exploratory search. We then exemplified the requirements through devising an IVF tool, which coordinates a linear and a categorical facet representing the distribution and summarisation of items, respectively, and providing context for faceted exploration (DR1). To support rapid transitions between search criteria (DR2), the tool introduces a novel design concept of using facets to select items without filtering the item space. Particularly, we propose a filter-swipe technique that enables users to drag a categorical facet value sequentially over linear facet bars to view the items in the intersection of the two facets along with the categorical facet dynamically summarizing the items in the interaction. Three applications demonstrate how the features support information discovery with ease. A user study of 11 participants with realistic email search tasks shows that dynamic suggestions through the timeline navigation can help discover useful suggestions for search; the novel design concept was favoured over using facet values as filters. Based on these practices, we derive IVF design implications for fluid, exploratory searches.

cs.HC↗

arXiv2020

Characterizing the Quality of Insight by Interactions: A Case Study

Understanding the quality of insight has become increasingly important with the trend of allowing users to post comments during visual exploration, yet approaches for qualifying insight are rare. This paper presents a case study to investigate the possibility of characterizing the quality of insight via the interactions performed. To do this, we devised the interaction of a visualization tool-MediSyn-for insight generation. MediSyn supports five types of interactions: selecting, connecting, elaborating, exploring, and sharing. We evaluated MediSyn with 14 participants by allowing them to freely explore the data and generate insights. We then extracted seven interaction patterns from their interaction logs and correlated the patterns to four aspects of insight quality. The results show the possibility of qualifying insights via interactions. Among other findings, exploration actions can lead to unexpected insights; the drill-down pattern tends to increase the domain values of insights. A qualitative analysis shows that using domain knowledge to guide exploration can positively affect the domain value of derived insights. We discuss the study's implications, lessons learned, and future research opportunities.

cs.HC↗

arXiv2019

PeyeDF: an Eye-Tracking Application for Reading and Self-Indexing Research

PeyeDF is a Portable Document Format (PDF) reader with eye tracking support, available as free and open source software. It is especially useful to researchers investigating reading and learning phenomena, as it integrates PDF reading-related behavioural data with gaze-related data. It is suitable for short and long-term research and supports multiple eye tracking systems. We utilised it to conduct an experiment which demonstrated that features obtained from both gaze and reading data collected in the past can predict reading comprehension which takes place in the future. PeyeDF also provides an integrated means for data collection and indexing using the DiMe personal data storage system. It is designed to collect data in the background without interfering with the reading experience, behaving like a modern lightweight PDF reader. Moreover, it supports annotations, tagging and collaborative work. A modular design allows the application to be easily modified in order to support additional eye tracking protocols and run controlled experiments. We discuss the implementation of the software and report on the results of the experiment which we conducted with it.

cs.HC↗

arXiv2017

Interactive Elicitation of Knowledge on Feature Relevance Improves Predictions in Small Data Sets

Providing accurate predictions is challenging for machine learning algorithms when the number of features is larger than the number of samples in the data. Prior knowledge can improve machine learning models by indicating relevant variables and parameter values. Yet, this prior knowledge is often tacit and only available from domain experts. We present a novel approach that uses interactive visualization to elicit the tacit prior knowledge and uses it to improve the accuracy of prediction models. The main component of our approach is a user model that models the domain expert's knowledge of the relevance of different features for a prediction task. In particular, based on the expert's earlier input, the user model guides the selection of the features on which to elicit user's knowledge next. The results of a controlled user study show that the user model significantly improves prior knowledge elicitation and prediction accuracy, when predicting the relative citation counts of scientific documents in a specific domain.

cs.AI↗

arXiv2016

Natural brain-information interfaces: Recommending information by relevance inferred from human brain signals

Finding relevant information from large document collections such as the World Wide Web is a common task in our daily lives. Estimation of a user's interest or search intention is necessary to recommend and retrieve relevant information from these collections. We introduce a brain-information interface used for recommending information by relevance inferred directly from brain signals. In experiments, participants were asked to read Wikipedia documents about a selection of topics while their EEG was recorded. Based on the prediction of word relevance, the individual's search intent was modeled and successfully used for retrieving new, relevant documents from the whole English Wikipedia corpus. The results show that the users' interests towards digital content can be modeled from the brain signals evoked by reading. The introduced brain-relevance paradigm enables the recommendation of information without any explicit user interaction, and may be applied across diverse information-intensive applications.

cs.IR↗