arXiv subjects

Wilton W. T. Fok

Publications and source records attributed to Wilton W. T. Fok.

3 recordsLinked to original sources

arXiv2026

Learning to See through Illumination Extremes with Event Streaming in Multimodal Large Language Models

Multimodal Large Language Models (MLLMs) perform strong vision-language reasoning under standard conditions but fail in extreme illumination, where RGB inputs lose irrevocable structure and semantics. We propose Event-MLLM, an event-enhanced model that performs all-light visual reasoning by dynamically fusing event streams with RGB frames. Two key components drive our approach: an Illumination Indicator - a learnable signal derived from a DINOv2 branch that represents exposure degradation and adaptively modulates event-RGB fusion - and an Illumination Correction Loss that aligns fused features with non-degraded (normal-light) semantics in the latent space, compensating for information lost in extreme lighting. We curate the first multi-illumination event-instruction corpus for MLLMs, with 2,241 event-RGB samples (around 6 QA pairs each) across diverse scenes and 17 brightness rates (0.05x - 20x), plus an instruct-following benchmark for reasoning, counting, and fine-grained recognition under extreme lighting. Experiments show that Event-MLLM markedly outperforms general-purpose, illumination-adaptive, and event-only baselines, setting a new state of the art in robust multimodal perception and reasoning under challenging illumination.

cs.CV↗

arXiv2025

Aligning Effective Tokens with Video Anomaly in Large Language Models

Understanding abnormal events in videos is a vital and challenging task that has garnered significant attention in a wide range of applications. Although current video understanding Multi-modal Large Language Models (MLLMs) are capable of analyzing general videos, they often struggle to handle anomalies due to the spatial and temporal sparsity of abnormal events, where the redundant information always leads to suboptimal outcomes. To address these challenges, exploiting the representation and generalization capabilities of Vison Language Models (VLMs) and Large Language Models (LLMs), we propose VA-GPT, a novel MLLM designed for summarizing and localizing abnormal events in various videos. Our approach efficiently aligns effective tokens between visual encoders and LLMs through two key proposed modules: Spatial Effective Token Selection (SETS) and Temporal Effective Token Generation (TETG). These modules enable our model to effectively capture and analyze both spatial and temporal information associated with abnormal events, resulting in more accurate responses and interactions. Furthermore, we construct an instruction-following dataset specifically for fine-tuning video-anomaly-aware MLLMs, and introduce a cross-domain evaluation benchmark based on XD-Violence dataset. Our proposed method outperforms existing state-of-the-art methods on various benchmarks.

cs.CV↗

arXiv2024

Validation of Consumer-grade Digital Camera-based Human Activity Evaluation for Upper Limb Exercises and Development of a Therapist-guided, Automated Telerehabilitation Framework and Platform for Stroke Rehabilitation

Timely and adequate rehabilitation is critical in facilitating post-stroke recovery. However, the organization and delivery of rehabilitation are resource-demanding, and are only available to approximately 25% of stroke survivors in low-to-middle-income countries. Improving access to stroke rehabilitation services through innovative solutions is therefore urgently required. Tele-rehabilitation, which transits care to home- and community settings, has emerged as a promising solution. However, current approaches using video tutorial, teleconference, or other specialized devices face inherent shortfalls that limit their uptake. In this study, we proposed and validated the use of an open-source, markerless motion capture model with consumer-grade devices to overcome these challenges. Our solution enables reliable measurement of the end range of motion during upper limb exercises with near-perfect waveform similarity and intraclass correlation to that of the gold standard Kinect approach. Our multidisciplinary team developed an automated telerehabilitation framework incorporating the validated markerless technique to facilitate a seamless telerehabilitation process. It enables personalized rehabilitation plans with real-time feedback, and individual progress reports using objective quantitative and qualitative features to improve patient monitoring and management, and home-based rehabilitation service uptake and compliance. This study serves as a proof-of-concept in preparation for the future development of a detailed model of care, and feasibility, usability, and cost-effectiveness studies of an automated telerehabilitation platform and framework in improving the state of post-stroke rehabilitation and functional outcome.

cs.HC↗