Your Eyes Controlled the Game: Real-Time Cognitive Training Adaptation based on Eye-Tracking and Physiological Data in Virtual Reality

Abstract

Cognitive training for sustained attention and working memory is vital across domains relying on robust mental capacity such as education or rehabilitation. Adaptive systems are essential, dynamically matching difficulty to user ability to maintain engagement and accelerate learning. Current adaptive systems often rely on simple performance heuristics or predict visual complexity and affect instead of cognitive load. This study presents the first implementation of real-time adaptive cognitive load control in Virtual Reality cognitive training based on eye-tracking and physiological data. We developed a bidirectional LSTM model with a self-attention mechanism, trained on eye-tracking and physiological (PPG, GSR) data from 74 participants. We deployed it in real-time with 54 participants across single-task (sustained attention) and dual-task (sustained attention + mental arithmetic) paradigms. Difficulty was adjusted dynamically based on participant self-assessment or model's real-time cognitive load predictions. Participants showed a tendency to estimate the task as too difficult, even though they were objectively performing at their best. Over the course of a 10-minute session, both adaptation methods converged at equivalent difficulty in single-task scenarios, with no significant differences in subjective workload or game performance. However, in the dual-task conditions, the model successfully pushed users to higher difficulty levels without performance penalties or increased frustration, highlighting a user tendency to underestimate capacity under high cognitive load. Findings indicate that machine learning models may provide more objective cognitive capacity assessments than self-directed approaches, mitigating subjective performance biases and enabling more effective training by pushing users beyond subjective comfort zones toward physiologically-determined optimal challenge levels.