arXiv subjects

Suranga Nanayakkara

Publications and source records attributed to Suranga Nanayakkara.

At least 19 recordsLinked to original sources

arXiv2026

HeadRoom: Lightweight, Edge-deployable Pipeline for Adaptive Notification Routing

Emerging wearables, such as smart glasses, can deliver notifications through multiple sensory channels, but there is still a limited understanding of how to choose the right channel at the right moment. We propose HeadRoom, a lightweight, edge-deployable pipeline that estimates the availability of visual and auditory channels in real time from egocentric video and audio. Our controlled user study (N=25) shows that, under high perceptual load, routing notifications to the more available channel reduces response time relative to routing them to the less available channel. This work opens up a new possibility for adaptive routing of notifications in wearable and immersive systems.

cs.HC↗

arXiv2026

DRONEAUDIONET: Noise Suppression for Drone Audition-based Search and Rescue

Microphones mounted on UAVs enable aerial acoustic scene analysis applications such as search-and-rescue, wildlife monitoring, and industrial inspection. However, drone rotor noise often dominates the mixture signal at SNRs well below -10 dB, making source recovery extremely challenging. Existing enhancement and source separation methods are typically designed for near-balanced mixtures and degrade substantially in drone audition settings. In this work, we propose DRONEAUDIONET, a drone noise suppression method that reframes a source separation model as a drone noise estimator. To better model drone-dominant mixtures, we introduce a learnable mask-scaling mechanism that allows mask magnitudes beyond unity, together with an additive residual correction term for improved drone estimation and source recovery. We train and evaluate our model on a publicly available drone audition dataset and test generalizability on an out-of-domain dataset with unseen drone hardware and flight modes. Results show that DRONEAUDIONET consistently improves downstream sound classification performance, with the largest gains observed for human vocal sounds. Our findings demonstrate the importance of drone-specific modeling for robust aerial acoustic perception and highlight the potential of source separation methods for real-world drone-assisted search-and-rescue.

cs.SD↗

arXiv2026

Two-Stage Multimodal Framework for Emotion Mimicry Intensity Prediction

We present our submission to the Hume-ABAW10 Emotional Mimicry Intensity (EMI) Challenge, which aims to predict six continuous emotion intensity dimensions: Admiration, Amusement, Determination, Empathic Pain, Excitement, and Joy, from in-the-wild multimodal video clips. We propose a staged multimodal framework that combines textual, acoustic, and visual representations, with an optional motion branch. Our approach first trains modality-specific encoders independently and then fuses their learned representations through a lightweight regressor with modality dropout and controlled encoder adaptation. Across our submitted systems, the best validation performance is obtained by the text--audio--vision--motion fusion model under the expanded 4:1 split, achieving an average Pearson correlation of 0.4722. Although the motion branch yields only very slight gains, its behavior can be interesting to study. Our team was placed third in the EMI challenge, achieving an average Pearson correlation of 0.57 for the test set. Overall, we provide a practical and reproducible baseline for EMI prediction.

cs.CV↗

arXiv2026

Beyond Descriptions: A Generative Scene2Audio Framework for Blind and Low-Vision Users to Experience Vista Landscapes

Current scene perception tools for Blind and Low Vision (BLV) individuals rely on spoken descriptions but lack engaging representations of visually pleasing distant environmental landscapes (Vista spaces). Our proposed Scene2Audio framework generates comprehensible and enjoyable nonverbal audio using generative models informed by psychoacoustics, and principles of scene audio composition. Through a user study with 11 BLV participants, we found that combining the Scene2Audio sounds with speech creates a better experience than speech alone, as the sound effects complement the speech making the scene easier to imagine. A mobile app "in-the-wild" study with 7 BLV users for more than a week further showed the potential of Scene2Audio in enhancing outdoor scene experiences. Our work bridges the gap between visual and auditory scene perception by moving beyond purely descriptive aids, addressing the aesthetic needs of BLV users.

cs.HC↗

arXiv2026

VisceroHaptics: Investigating the Effects of Gut-based Audio-Haptic Feedback on Gastric Feelings and Gastric Interoceptive Behavior

Gastric interoception influences eating behavior and emotions, making its modulation valuable for healthcare and human-computer-interaction applications. However, whether gastric interoception can be modulated noninvasively in humans remains unclear. While previous research indicates that abdominal-sound-driven haptic feedback resembles gut sensations, its impact on feelings and gastric interoceptive behavior is unknown. We conducted three experiments totalling 55 participants to investigate how gut-sound-driven audio-haptic feedback applied to the stomach (1) affects user's feelings (2) influences perception of hunger and satiety levels and (3) influences gastric interoceptive behavior, quantified with Water Load Test-II. Results revealed that audio-haptic feedback patterns (a) induced the feelings of hunger, fullness, thirst, stomach upset, (b) increased hunger level, and (c) significantly increased volumes of ingested water. This work provides the first evidence showing that audio-haptic stimulation can alter gastric interoceptive behavior, motivating the use of noninvasive methods to influence users' feelings and behaviors in future applications.

cs.HC↗

arXiv2026

VisGuardian: A Lightweight Group-based Privacy Control Technique For Front Camera Data From AR Glasses in Home Environments

Always-on sensing of AI applications on AR glasses makes traditional permission techniques ill-suited for context-dependent visual data, especially within home environments. The home presents a highly challenging privacy context due to the high density of sensitive objects, and the frequent presence of non-consenting family members, and the intimate nature of daily routines, making it a critical focus area for scalable privacy control mechanisms. Existing fine-grained controls, while offering nuanced choices, are inefficient for managing multiple private objects. We propose VisGuardian, a fine-grained content-based visual permission technique for AR glasses. VisGuardian features a group-based control mechanism that enables users to efficiently manage permissions for multiple private objects. VisGuardian detects objects using YOLO and adopts a pre-classified schema to group them. By selecting a single object, users can efficiently obscure groups of related objects based on criteria including privacy sensitivity, object category, or spatial proximity. A technical evaluation shows VisGuardian achieves mAP50 of 0.6704 with only 14.0 ms latency and a 1.7% increase in battery consumption per hour. Furthermore, a user study (N=24) comparing VisGuardian to slider-based and object-based baselines found it to be significantly faster for setting permissions and was preferred by users for its efficiency, effectiveness, and ease of use.

cs.HC↗

arXiv2025

DroneAudioset: An Audio Dataset for Drone-based Search and Rescue

Unmanned Aerial Vehicles (UAVs) or drones, are increasingly used in search and rescue missions to detect human presence. Existing systems primarily leverage vision-based methods which are prone to fail under low-visibility or occlusion. Drone-based audio perception offers promise but suffers from extreme ego-noise that masks sounds indicating human presence. Existing datasets are either limited in diversity or synthetic, lacking real acoustic interactions, and there are no standardized setups for drone audition. To this end, we present DroneAudioset (The dataset is publicly available at https://huggingface.co/datasets/ahlab-drone-project/DroneAudioSet/ under the MIT license), a comprehensive drone audition dataset featuring 23.5 hours of annotated recordings, covering a wide range of signal-to-noise ratios (SNRs) from -57.2 dB to -2.5 dB, across various drone types, throttles, microphone configurations as well as environments. The dataset enables development and systematic evaluation of noise suppression and classification methods for human-presence detection under challenging conditions, while also informing practical design considerations for drone audition systems, such as microphone placement trade-offs, and development of drone noise-aware audio processing. This dataset is an important step towards enabling design and deployment of drone-audition systems.

eess.AS↗

arXiv2025

Towards Temporally Explainable Dysarthric Speech Clarity Assessment

Dysarthria, a motor speech disorder, affects intelligibility and requires targeted interventions for effective communication. In this work, we investigate automated mispronunciation feedback by collecting a dysarthric speech dataset from six speakers reading two passages, annotated by a speech therapist with temporal markers and mispronunciation descriptions. We design a three-stage framework for explainable mispronunciation evaluation: (1) overall clarity scoring, (2) mispronunciation localization, and (3) mispronunciation type classification. We systematically analyze pretrained Automatic Speech Recognition (ASR) models in each stage, assessing their effectiveness in dysarthric speech evaluation (Code available at: https://github.com/augmented-human-lab/interspeech25_speechtherapy, Supplementary webpage: https://apps.ahlab.org/interspeech25_speechtherapy/). Our findings offer clinically relevant insights for automating actionable feedback for pronunciation assessment, which could enable independent practice for patients and help therapists deliver more effective interventions.

eess.AS↗

arXiv2025

EmoSign: A Multimodal Dataset for Understanding Emotions in American Sign Language

Unlike spoken languages where the use of prosodic features to convey emotion is well studied, indicators of emotion in sign language remain poorly understood, creating communication barriers in critical settings. Sign languages present unique challenges as facial expressions and hand movements simultaneously serve both grammatical and emotional functions. To address this gap, we introduce EmoSign, the first sign video dataset containing sentiment and emotion labels for 200 American Sign Language (ASL) videos. We also collect open-ended descriptions of emotion cues. Annotations were done by 3 Deaf ASL signers with professional interpretation experience. Alongside the annotations, we include baseline models for sentiment and emotion classification. This dataset not only addresses a critical gap in existing sign language research but also establishes a new benchmark for understanding model capabilities in multimodal emotion recognition for sign languages. The dataset is made available at https://huggingface.co/datasets/catfang/emosign.

cs.CV↗

arXiv2025

Perspectives on Capturing Emotional Expressiveness in Sign Language

Significant advances have been made in our ability to understand and generate emotionally expressive content such as text and speech, yet comparable progress in sign language technologies remain limited. While computational approaches to sign language translation have focused on capturing lexical content, the emotional dimensions of sign language communication remain largely unexplored. Through semi-structured interviews with eight sign language users across Singapore, Sri Lanka and the United States, including both Deaf and Hard of hearing (DHH) and hearing signers, we investigate how emotions are expressed and perceived in sign languages. Our findings highlight the role of both manual and non-manual elements in emotional expression, revealing universal patterns as well as individual and cultural variations in how signers communicate emotions. We identify key challenges in capturing emotional nuance for sign language translation, and propose design considerations for developing more emotionally-aware sign language technologies. This work contributes to both theoretical understanding of emotional expression in sign language and practical development of interfaces to better serve diverse signing communities.

cs.HC↗

arXiv2025

Factually: Exploring Wearable Fact-Checking for Augmented Truth Discernment

Wearable devices are transforming human capabilities by seamlessly augmenting cognitive functions. In this position paper, we propose a voice-based, interactive learning companion designed to amplify and extend cognitive abilities through informal learning. Our vision is threefold: (1) to enable users to discover new knowledge on-the-go through contextual interactive quizzes, fostering critical thinking and mindfulness, (2) to proactively detect misinformation, empowering users to critically assess information in real time, and (3) to provide spoken language correction and prompting hints for second language learning and effective communication. As an initial step toward this vision, we present Factually - a proactive, wearable fact-checking system integrated into devices like smartwatches or rings. Factually discreetly alerts users to potential falsehoods via vibrotactile feedback, helping them assess information critically. We demonstrate its utility through three illustrative scenarios, highlighting its potential to extend cognitive abilities for real-time misinformation detection. Early qualitative feedback suggests that Factually can enhance users' fact-checking capabilities, offering both practical and experiential benefits.

cs.HC↗

arXiv2025

Navigating the State of Cognitive Flow: Context-Aware AI Interventions for Effective Reasoning Support

Flow theory describes an optimal cognitive state where individuals experience deep focus and intrinsic motivation when a task's difficulty aligns with their skill level. In AI-augmented reasoning, interventions that disrupt the state of cognitive flow can hinder rather than enhance decision-making. This paper proposes a context-aware cognitive augmentation framework that adapts interventions based on three key contextual factors: type, timing, and scale. By leveraging multimodal behavioral cues (e.g., gaze behavior, typing hesitation, interaction speed), AI can dynamically adjust cognitive support to maintain or restore flow. We introduce the concept of cognitive flow, an extension of flow theory in AI-augmented reasoning, where interventions are personalized, adaptive, and minimally intrusive. By shifting from static interventions to context-aware augmentation, our approach ensures that AI systems support deep engagement in complex decision-making and reasoning without disrupting cognitive immersion.

cs.HC↗

arXiv2025

Broadening Participation through Physical Computing: Replicating Sensor-Based Programming Workshops for Rural Students in Sri Lanka

In today's digital world, computing education offers critical opportunities, yet systemic inequities exclude under-represented communities, especially in rural, under-resourced regions. Early engagement is vital for building interest in computing careers and achieving equitable participation. Recent work has shown that the use of sensor-enabled tools and block-based programming can improve engagement and self-efficacy for students from under-represented groups, but these findings lack replication in diverse, resource-constrained settings. This study addresses this gap by implementing sensor-based programming workshops with rural students in Sri Lanka. Replicating methods from the literature, we conduct a between-group study (sensor vs. non-sensor) using Scratch and real-time environmental sensors. We found that students in both groups reported significantly higher confidence in programming in Scratch after the workshop. In addition, average changes in both self-efficacy and outcome expectancy were higher in the experimental (sensor) group than in the control (non-sensor) group, mirroring trends observed in the original study being replicated. We also found that using the sensors helped to enhance creativity and inspired some students to express an interest in information and communications technology (ICT) careers, supporting the value of such hands-on activities in building programming confidence among under-represented groups.

cs.CY↗

arXiv2025

The Goldilocks Time Window for Proactive Interventions in Wearable AI Systems

As AI systems become increasingly integrated into our daily lives and into wearable form factors, there's a fundamental tension between their potential to proactively assist us and the risk of creating intrusive, dependency-forming experiences. This work proposes the concept of a Goldilocks Time Window -- a contextually adaptive time window for proactive AI systems to deliver effective interventions. We discuss the critical factors that determine the time window, and the need of a framework for designing and evaluating proactive AI systems that can navigate this tension successfully.

cs.HC↗

arXiv2025

Mirai: A Wearable Proactive AI "Inner-Voice" for Contextual Nudging

People often find it difficult to turn their intentions into real actions -- a challenge that affects both personal growth and mental well-being. While established methods like cognitive-behavioral therapy and mindfulness training help people become more aware of their behaviors and set clear goals, these approaches cannot provide immediate guidance when people fall into automatic reactions or habits. We introduce Mirai, a novel wearable AI system with an integrated camera, real-time speech processing, and personalized voice-cloning to provide proactive and contextual nudges for positive behavior change. Mirai continuously monitors and analyzes the user's environment to anticipate their intentions, generating contextually-appropriate responses delivered in the user's own cloned voice. We demonstrate the application of Mirai through three scenarios focusing on dietary choices, work productivity, and communication skills. We also discuss future work on improving the proactive agent via human feedback and the need for a longitudinal study in naturalistic settings.

cs.HC↗

arXiv2025

Tailored Real-time AR Captioning Interface for Enhancing Learning Experience of Deaf and Hard-of-Hearing (DHH) Students

Deaf and hard-of-hearing (DHH) students face significant challenges in specialized educational settings, such as limited exposure to written and spoken language, a lack of tailored educational tools, and restricted access to resources, impacting their language literacy development and overall educational experience. We, therefore, employed a User-Centered Design (UCD) process, collaborating with 8 DHH students and 2 Teachers of the Deaf (ToDs) from a School of Deaf to effectively develop and utilize a real-time captioning augmented reality (AR) system to their school settings, aiming to enhance their learning experience. User study with 24 DHH participants revealed a strong preference (87.5\%) for our system, underscoring its potential to enhance learning experience. We present a comprehensive needs analysis, the UCD process, system implementation, and user feedback, showcasing the effectiveness of tailored AR caption interfaces for DHH students. We also discuss the implications for future development of educational technologies for DHH students.

cs.HC↗

arXiv2024

Investigating the Use of Productive Failure as a Design Paradigm for Learning Introductory Python Programming

Productive Failure (PF) is a learning approach where students initially tackle novel problems targeting concepts they have not yet learned, followed by a consolidation phase where these concepts are taught. Recent application in STEM disciplines suggests that PF can help learners develop more robust conceptual knowledge. However, empirical validation of PF for programming education remains under-explored. In this paper, we investigate the use of PF to teach Python lists to undergraduate students with limited prior programming experience. We designed a novel PF-based learning activity that incorporated the unobtrusive collection of real-time heart-rate data from consumer-grade wearable sensors. This sensor data was used both to make the learning activity engaging and to infer cognitive load. We evaluated our approach with 20 participants, half of whom were taught Python concepts using Direct Instruction (DI), and the other half with PF. We found that although there was no difference in initial learning outcomes between the groups, students who followed the PF approach showed better knowledge retention and performance on delayed but similar tasks. In addition, physiological measurements indicated that these students also exhibited a larger decrease in cognitive load during their tasks after instruction. Our findings suggest that PF-based approaches may lead to more robust learning, and that future work should investigate similar activities at scale across a range of concepts.

cs.CY↗

arXiv2024

Motion as Emotion: Detecting Affect and Cognitive Load from Free-Hand Gestures in VR

Affect and cognitive load influence many user behaviors. In this paper, we propose Motion as Emotion, a novel method that utilizes fine differences in hand motion to recognise affect and cognitive load in virtual reality (VR). We conducted a study with 22 participants who used common free-hand gesture interactions to carry out tasks of varying difficulty in VR environments. We find that the affect and cognitive load induced by tasks are associated with significant differences in gesture features such as speed, distance and hand tension. Standard support vector classification (SVC) models could accurately predict two levels (low, high) of valence, arousal and cognitive load from these features. Our results demonstrate the potential of Motion as Emotion as an accurate and reliable method of inferring user affect and cognitive load from free-hand gestures, without needing any additional wearable sensors or modifications to a standard VR headset.

cs.HC↗