arXiv subjects

Julien Gori

Publications and source records attributed to Julien Gori.

3 recordsLinked to original sources

arXiv2026

TargetFinder: Detecting Widgets from Pixels on Desktop Interfaces

''Target-aware'' pointing techniques, like Bubble Cursor or Semantic Pointing, outperform traditional pointing by leveraging knowledge of target locations. Yet the lack of application-agnostic widget geometry information limits their adoption across the desktop. We present TargetFinder, a computer vision-based system for real-time detection of GUI widgets. TargetFinder leverages several fine-tuned YOLO networks trained on a new dataset of 520 annotated desktop screenshots (~38,000 annotations) spanning Windows, macOS, Ubuntu, and web interfaces. TargetFinder uses lightweight screen monitoring and low-latency detection, achieving millisecond responsiveness suitable for interactive use. Evaluations show that TargetFinder outperforms the baseline methods (OmniParser and REMAUI), while system-wide implementations of Bubble Cursor and Semantic Pointing demonstrate the feasibility of deploying universal target-aware techniques that work across applications. We release the dataset, models, annotation tool, and an open-source library for research and applications.

cs.HC↗

arXiv2022

A Novel Mixture Model for Characterizing Human Aiming Performance Data

Fitts' law is often employed as a predictive model for human movement, especially in the field of human-computer interaction. Models with an assumed Gaussian error structure are usually adequate when applied to data collected from controlled studies. However, observational data (often referred to as data gathered "in the wild") typically display noticeable positive skewness relative to a mean trend as users do not routinely try to minimize their task completion time. As such, the exponentially-modified Gaussian (EMG) regression model has been applied to aimed movements data. However, it is also of interest to reasonably characterize those regions where a user likely was not trying to minimize their task completion time. In this paper, we propose a novel model with a two-component mixture structure -- one Gaussian and one exponential -- on the errors to identify such a region. An expectation-conditional-maximization (ECM) algorithm is developed for estimation of such a model and some properties of the algorithm are established. The efficacy of the proposed model, as well as its ability to inform model-based clustering, are addressed in this work through extensive simulations and an insightful analysis of a human aiming performance study.

stat.AP↗

arXiv2020

A Feedback Information-Theoretic Transmission Scheme (FITTS) for Modeling Trajectory Variability in Aimed Movements

Trajectories in human aimed movements are inherently variable. Using the concept of positional variance profiles, such trajectories are shown to be decomposable into two phases: In a first phase, the variance of the limb position over many trajectories increases rapidly; in a second phase, it then decreases steadily. A new theoretical model, where the aiming task is seen as a Shannon-like communication problem, is developed to describe the second phase: Information is transmitted from a source (determined by the position at the end of the first phase), to a destination (the movement's end-point) over a channel perturbed by Gaussian noise, with the presence of a noiseless feedback link. Information-theoretic considerations show that the positional variance decreases exponentially with a rate equal to the channel capacity C. Two existing datasets for simple pointing tasks are re-analyzed and observations on real data confirm our model. The first phase has constant duration and C is found constant across instructions and task parameters, which thus characterizes the participant's performance. Our model provides a clear understanding of the speed-accuracy tradeoff in aimed movements: Since the participant's capacity is fixed, a higher prescribed accuracy necessarily requires a longer second phase resulting in an increased overall movement time. The well-known Fitts' law is also recovered using this approach.

cs.HC↗