arXiv subjects

Yujiang Wang

Publications and source records attributed to Yujiang Wang.

At least 73 records · Page 4Linked to original sources

arXiv2022

Effects of anterior temporal lobe resection on cortical morphology

Anterior temporal lobe resection (ATLR) is a surgical procedure to treat drug-resistant temporal lobe epilepsy (TLE). Resection may involve large amounts of cortical tissue. Here, we examine the effects of this surgery on cortical morphology measured in independent variables both near the resection and remotely. We studied 101 individuals with TLE (55 left, 46 right onset) who underwent ATLR. For each individual we considered one pre-surgical MRI and one follow-up MRI 2 to 13 months after surgery. We used our newly developed surface-based method to locally compute traditional morphological variables (average cortical thickness, exposed surface area, and total surface area), and the independent measures $K$, $I$, and $S$, where $K$ measures white matter tension, $I$ captures isometric scaling, and $S$ contains the remaining information about cortical shape. Data from 924 healthy controls was included to account for healthy ageing effects occurring during scans. A SurfStat random field theory clustering approach assessed changes across the cortex caused by ATLR. Compared to preoperative data, surgery had marked effects on all morphological measures. Ipsilateral effects were located in the orbitofrontal and inferior frontal gyri, the pre- and postcentral gyri and supramarginal gyrus, and the lateral occipital gyrus and lingual cortex. Contralateral effects were in the lateral occipital gyrus, and inferior frontal gyrus and frontal pole. The restructuring following ATLR is reflected in widespread morphological changes, mainly in regions near the resection, but also remotely in regions that are structurally connected to the anterior temporal lobe. The causes could include mechanical effects, Wallerian degeneration, or compensatory plasticity. The study of independent measures revealed additional effects compared to traditional measures.

q-bio.NC↗

arXiv2022

Training Strategies for Improved Lip-reading

Several training strategies and temporal models have been recently proposed for isolated word lip-reading in a series of independent works. However, the potential of combining the best strategies and investigating the impact of each of them has not been explored. In this paper, we systematically investigate the performance of state-of-the-art data augmentation approaches, temporal models and other training strategies, like self-distillation and using word boundary indicators. Our results show that Time Masking (TM) is the most important augmentation followed by mixup and Densely-Connected Temporal Convolutional Networks (DC-TCN) are the best temporal model for lip-reading of isolated words. Using self-distillation and word boundary indicators is also beneficial but to a lesser extent. A combination of all the above methods results in a classification accuracy of 93.4%, which is an absolute improvement of 4.6% over the current state-of-the-art performance on the LRW dataset. The performance can be further improved to 94.1% by pre-training on additional datasets. An error analysis of the various training strategies reveals that the performance improves by increasing the classification accuracy of hard-to-recognise words.

cs.CV↗

arXiv2022

Lip-reading with Densely Connected Temporal Convolutional Networks

In this work, we present the Densely Connected Temporal Convolutional Network (DC-TCN) for lip-reading of isolated words. Although Temporal Convolutional Networks (TCN) have recently demonstrated great potential in many vision tasks, its receptive fields are not dense enough to model the complex temporal dynamics in lip-reading scenarios. To address this problem, we introduce dense connections into the network to capture more robust temporal features. Moreover, our approach utilises the Squeeze-and-Excitation block, a light-weight attention mechanism, to further enhance the model's classification power. Without bells and whistles, our DC-TCN method has achieved 88.36% accuracy on the Lip Reading in the Wild (LRW) dataset and 43.65% on the LRW-1000 dataset, which has surpassed all the baseline methods and is the new state-of-the-art on both datasets.

cs.CV↗

arXiv2022

Do Smart Glasses Dream of Sentimental Visions? Deep Emotionship Analysis for Eyewear Devices

Emotion recognition in smart eyewear devices is highly valuable but challenging. One key limitation of previous works is that the expression-related information like facial or eye images is considered as the only emotional evidence. However, emotional status is not isolated; it is tightly associated with people's visual perceptions, especially those sentimental ones. However, little work has examined such associations to better illustrate the cause of different emotions. In this paper, we study the emotionship analysis problem in eyewear systems, an ambitious task that requires not only classifying the user's emotions but also semantically understanding the potential cause of such emotions. To this end, we devise EMOShip, a deep-learning-based eyewear system that can automatically detect the wearer's emotional status and simultaneously analyze its associations with semantic-level visual perceptions. Experimental studies with 20 participants demonstrate that, thanks to the emotionship awareness, EMOShip not only achieves superior emotion recognition accuracy over existing methods (80.2% vs. 69.4%), but also provides a valuable understanding of the cause of emotions. Pilot studies with 20 participants further motivate the potential use of EMOShip to empower emotion-aware applications, such as emotionship self-reflection and emotionship life-logging.

cs.CV↗

arXiv2022

Intracranial EEG structure-function coupling predicts surgical outcomes in focal epilepsy

Alterations to structural and functional brain networks have been reported across many neurological conditions. However, the relationship between structure and function -- their coupling -- is relatively unexplored, particularly in the context of an intervention. Epilepsy surgery alters the brain structure and networks to control the functional abnormality of seizures. Given that surgery is a structural modification aiming to alter the function, we hypothesized that stronger structure-function coupling preoperatively is associated with a greater chance of post-operative seizure control. We constructed structural and functional brain networks in 39 subjects with medication-resistant focal epilepsy using data from intracranial EEG (pre-surgery), structural MRI (pre-and post-surgery), and diffusion MRI (pre-surgery). We investigated pre-operative structure-function coupling at two spatial scales a) at the global iEEG network level and b) at the resolution of individual iEEG electrode contacts using virtual surgeries. At global network level, seizure-free individuals had stronger structure-function coupling pre-operatively than those that were not seizure-free regardless of the choice of interictal segment or frequency band. At the resolution of individual iEEG contacts, the virtual surgery approach provided complementary information to localize epileptogenic tissues. In predicting seizure outcomes, structure-function coupling measures were more important than clinical attributes, and together they predicted seizure outcomes with an accuracy of 85% and sensitivity of 87%. The underlying assumption that the structural changes induced by surgery translate to the functional level to control seizures is valid when the structure-functional coupling is strong. Mapping the regions that contribute to structure-functional coupling using virtual surgeries may help aid surgical planning.

q-bio.NC↗

arXiv2022

A real-time and unsupervised face Re-Identification system for Human-Robot Interaction

In the context of Human-Robot Interaction (HRI), face Re-Identification (face Re-ID) aims to verify if certain detected faces have already been observed by robots. The ability of distinguishing between different users is crucial in social robots as it will enable the robot to tailor the interaction strategy toward the users' individual preferences. So far face recognition research has achieved great success, however little attention has been paid to the realistic applications of Face Re-ID in social robots. In this paper, we present an effective and unsupervised face Re-ID system which simultaneously re-identifies multiple faces for HRI. This Re-ID system employs Deep Convolutional Neural Networks to extract features, and an online clustering algorithm to determine the face's ID. Its performance is evaluated on two datasets: the TERESA video dataset collected by the TERESA robot, and the YouTube Face Dataset (YTF Dataset). We demonstrate that the optimised combination of techniques achieves an overall 93.55% accuracy on TERESA dataset and an overall 90.41% accuracy on YTF dataset. We have implemented the proposed method into a software module in the HCI^2 Framework for it to be further integrated into the TERESA robot, and has achieved real-time performance at 10~26 Frames per second.

cs.CV↗

arXiv2022

FP-Age: Leveraging Face Parsing Attention for Facial Age Estimation in the Wild

Image-based age estimation aims to predict a person's age from facial images. It is used in a variety of real-world applications. Although end-to-end deep models have achieved impressive results for age estimation on benchmark datasets, their performance in-the-wild still leaves much room for improvement due to the challenges caused by large variations in head pose, facial expressions, and occlusions. To address this issue, we propose a simple yet effective method to explicitly incorporate facial semantics into age estimation, so that the model would learn to correctly focus on the most informative facial components from unaligned facial images regardless of head pose and non-rigid deformation. To this end, we design a face parsing-based network to learn semantic information at different scales and a novel face parsing attention module to leverage these semantic features for age estimation. To evaluate our method on in-the-wild data, we also introduce a new challenging large-scale benchmark called IMDB-Clean. This dataset is created by semi-automatically cleaning the noisy IMDB-WIKI dataset using a constrained clustering method. Through comprehensive experiment on IMDB-Clean and other benchmark datasets, under both intra-dataset and cross-dataset evaluation protocols, we show that our method consistently outperforms all existing age estimation methods and achieves a new state-of-the-art performance. To the best of our knowledge, our work presents the first attempt of leveraging face parsing attention to achieve semantic-aware age estimation, which may be inspiring to other high level facial analysis tasks. Code and data are available on \url{https://github.com/ibug-group/fpage}.

cs.CV↗

arXiv2022

Volumetric and structural connectivity abnormalities co-localise in TLE

Patients with temporal lobe epilepsy (TLE) exhibit both volumetric and structural connectivity abnormalities relative to healthy controls. How these abnormalities inter-relate and their mechanisms are unclear. We computed grey matter volumetric changes and white matter structural connectivity abnormalities in 144 patients with unilateral TLE and 96 healthy controls. Regional volumes were calculated using T1-weighted MRI, while structural connectivity was derived using white matter fibre tractography from diffusion-weighted MRI. For each regional volume and each connection strength, we calculated the effect size between patient and control groups in a group-level analysis. We then applied hierarchical regression to investigate the relationship between volumetric and structural connectivity abnormalities in individuals. Additionally, we quantified whether abnormalities co-localised within individual patients by computing Dice similarity scores. In TLE, white matter connectivity abnormalities were greater when joining two grey matter regions with abnormal volumes. Similarly, grey matter volumetric abnormalities were greater when joined by abnormal white matter connections. The extent of volumetric and connectivity abnormalities related to epilepsy duration, but co-localisation did not. Co-localisation was primarily driven by neighbouring abnormalities in the ipsilateral hemisphere. Overall, volumetric and structural connectivity abnormalities were related in TLE. Our results suggest that shared mechanisms may underlie changes in both volume and connectivity alterations in patients with TLE.

q-bio.NC↗

arXiv2022

Dilated Convolutions with Lateral Inhibitions for Semantic Image Segmentation

Dilated convolutions are widely used in deep semantic segmentation models as they can enlarge the filters' receptive field without adding additional weights nor sacrificing spatial resolution. However, as dilated convolutional filters do not possess positional knowledge about the pixels on semantically meaningful contours, they could lead to ambiguous predictions on object boundaries. In addition, although dilating the filter can expand its receptive field, the total number of sampled pixels remains unchanged, which usually comprises a small fraction of the receptive field's total area. Inspired by the Lateral Inhibition (LI) mechanisms in human visual systems, we propose the dilated convolution with lateral inhibitions (LI-Convs) to overcome these limitations. Introducing LI mechanisms improves the convolutional filter's sensitivity to semantic object boundaries. Moreover, since LI-Convs also implicitly take the pixels from the laterally inhibited zones into consideration, they can also extract features at a denser scale. By integrating LI-Convs into the Deeplabv3+ architecture, we propose the Lateral Inhibited Atrous Spatial Pyramid Pooling (LI-ASPP), the Lateral Inhibited MobileNet-V2 (LI-MNV2) and the Lateral Inhibited ResNet (LI-ResNet). Experimental results on three benchmark datasets (PASCAL VOC 2012, CelebAMask-HQ and ADE20K) show that our LI-based segmentation models outperform the baseline on all of them, thus verify the effectiveness and generality of the proposed LI-Convs.

cs.CV↗

arXiv2022

Normative brain mapping of interictal intracranial EEG to localise epileptogenic tissue

The identification of abnormal electrographic activity is important in a wide range of neurological disorders, including epilepsy for localising epileptogenic tissue. However, this identification may be challenging during non-seizure (interictal) periods, especially if abnormalities are subtle compared to the repertoire of possible healthy brain dynamics. Here, we investigate if such interictal abnormalities become more salient by quantitatively accounting for the range of healthy brain dynamics in a location-specific manner. To this end, we constructed a normative map of brain dynamics, in terms of relative band power, from interictal intracranial recordings from 234 subjects (21,598 electrode contacts). We then compared interictal recordings from 62 patients with epilepsy to the normative map to identify abnormal regions. We hypothesised that if the most abnormal regions were spared by surgery, then patients would be more likely to experience continued seizures post-operatively. We first confirmed that the spatial variations of band power in the normative map across brain regions were consistent with healthy variations reported in the literature. Second, when accounting for the normative variations, regions which were spared by surgery were more abnormal than those resected only in patients with persistent post-operative seizures (t=-3.6, p=0.0003), confirming our hypothesis. Third, we found that this effect discriminated patient outcomes (AUC=0.75 p=0.0003). Normative mapping is a well-established practice in neuroscientific research. Our study suggests that this approach is feasible to detect interictal abnormalities in intracranial EEG, and of potential clinical value to identify pathological tissue in epilepsy. Finally, we make our normative intracranial map publicly available to facilitate future investigations in epilepsy and beyond.

q-bio.NC↗

arXiv2021

Domain Generalisation for Apparent Emotional Facial Expression Recognition across Age-Groups

Apparent emotional facial expression recognition has attracted a lot of research attention recently. However, the majority of approaches ignore age differences and train a generic model for all ages. In this work, we study the effect of using different age-groups for training apparent emotional facial expression recognition models. To this end, we study Domain Generalisation in the context of apparent emotional facial expression recognition from facial imagery across different age groups. We first compare several domain generalisation algorithms on the basis of out-of-domain-generalisation, and observe that the Class-Conditional Domain-Adversarial Neural Networks (CDANN) algorithm has the best performance. We then study the effect of variety and number of age-groups used during training on generalisation to unseen age-groups and observe that an increase in the number of training age-groups tends to increase the apparent emotional facial expression recognition performance on unseen age-groups. We also show that exclusion of an age-group during training tends to affect more the performance of the neighbouring age groups.

cs.CV↗

arXiv2021

MemX: An Attention-Aware Smart Eyewear System for Personalized Moment Auto-capture

This work presents MemX: a biologically-inspired attention-aware eyewear system developed with the goal of pursuing the long-awaited vision of a personalized visual Memex. MemX captures human visual attention on the fly, analyzes the salient visual content, and records moments of personal interest in the form of compact video snippets. Accurate attentive scene detection and analysis on resource-constrained platforms is challenging because these tasks are computation and energy intensive. We propose a new temporal visual attention network that unifies human visual attention tracking and salient visual content analysis. Attention tracking focuses computation-intensive video analysis on salient regions, while video analysis makes human attention detection and tracking more accurate. Using the YouTube-VIS dataset and 30 participants, we experimentally show that MemX significantly improves the attention tracking accuracy over the eye-tracking-alone method, while maintaining high system energy efficiency. We have also conducted 11 in-field pilot studies across a range of daily usage scenarios, which demonstrate the feasibility and potential benefits of MemX.

cs.CV↗

arXiv2021

Seizure pathways and seizure durations can vary independently within individual patients with focal epilepsy

A seizure's electrographic dynamics are characterised by its spatiotemporal evolution, also termed dynamical "pathway" and the time it takes to complete that pathway, which results in the seizure's duration. Both seizure pathways and durations can vary within the same patient, producing seizures with different dynamics, severity, and clinical implications. However, it is unclear whether seizures following the same pathway will have the same duration or if these features can vary independently. We compared within-subject variability in these seizure features using 1) epilepsy monitoring unit intracranial EEG (iEEG) recordings of 31 patients (mean 6.7 days, 16.5 seizures/subject), 2) NeuroVista chronic iEEG recordings of 10 patients (mean 521.2 days, 252.6 seizures/subject), and 3) chronic iEEG recordings of 3 dogs with focal-onset seizures (mean 324.4 days, 62.3 seizures/subject). While the strength of the relationship between seizure pathways and durations was highly subject-specific, in most subjects, changes in seizure pathways were only weakly to moderately associated with differences in seizure durations. The relationship between seizure pathways and durations was weakened by seizures that 1) had a common pathway, but different durations ("elastic pathways"), or 2) had similar durations, but followed different pathways ("duplicate durations"). Even in subjects with distinct populations of short and long seizures, seizure durations were not a reliable indicator of different seizure pathways. These findings suggest that seizure pathways and durations are modulated by different processes. Uncovering such modulators may reveal novel therapeutic targets for reducing seizure duration and severity.

q-bio.NC↗

arXiv2021

Fluctuations in EEG band power at subject-specific timescales over minutes to days explain changes in seizure evolutions

Epilepsy is recognised as a dynamic disease, where both seizure susceptibility and seizure characteristics themselves change over time. Specifically, we recently quantified the variable electrographic spatio-temporal seizure evolutions that exist within individual patients. This variability appears to follow subject-specific circadian, or longer, timescale modulations. It is therefore important to know whether continuously-recorded interictal iEEG features can capture signatures of these modulations over different timescales. In this work, we analyse continuous intracranial electroencephalographic (iEEG) recordings from video-telemetry units and find fluctuations in iEEG band power over timescales ranging from minutes up to twelve days. As expected and in agreement with previous studies, we find that all subjects show a circadian fluctuation in their iEEG band power. We additionally find other fluctuations of similar magnitude on subject-specific timescales. Importantly, we find that a combination of these fluctuations on different timescales can explain changes in seizure evolutions in most subjects above chance level. These results suggest that subject-specific fluctuations in iEEG band power over timescales of minutes to days may serve as markers of seizure modulating processes. We hope that future work can link these detected fluctuations to their biological driver(s). There is a critical need to better understand seizure modulating processes, as this will enable the development of novel treatment strategies that could minimise the seizure spread, duration, or severity and therefore the clinical impact of seizures.

q-bio.NC↗

arXiv2021

Long-term changes in functional connectivity predict responses to intracranial stimulation of the human brain

Targeted electrical stimulation of the brain perturbs neural networks and modulates their rhythmic activity both at the site of stimulation and at remote brain regions. Understanding, or even predicting, this neuromodulatory effect is crucial for any therapeutic use of brain stimulation. To this end, we analyzed the stimulation responses in 131 stimulation sessions across 66 patients with focal epilepsy recorded through intracranial EEG (iEEG). We considered functional and structural connectivity features as predictors of the response at every iEEG contact. Taking advantage of multiple recordings over days, we also investigated how slow changes in interictal functional connectivity (FC) ahead of the stimulation relate to stimulation responses. The results reveal that, indeed, this long-term variability of FC exhibits strong association with the stimulation-induced increases in delta and theta band power. Furthermore, we show through cross-validation that long-term variability of FC improves prediction of responses above the performance of spatial predictors alone. These findings can enhance the patient-specific design of effective neuromodulatory protocols for therapeutic interventions.

q-bio.NC↗

arXiv2021

RoI Tanh-polar Transformer Network for Face Parsing in the Wild

Face parsing aims to predict pixel-wise labels for facial components of a target face in an image. Existing approaches usually crop the target face from the input image with respect to a bounding box calculated during pre-processing, and thus can only parse inner facial Regions of Interest~(RoIs). Peripheral regions like hair are ignored and nearby faces that are partially included in the bounding box can cause distractions. Moreover, these methods are only trained and evaluated on near-frontal portrait images and thus their performance for in-the-wild cases has been unexplored. To address these issues, this paper makes three contributions. First, we introduce iBugMask dataset for face parsing in the wild, which consists of 21,866 training images and 1,000 testing images. The training images are obtained by augmenting an existing dataset with large face poses. The testing images are manually annotated with $11$ facial regions and there are large variations in sizes, poses, expressions and background. Second, we propose RoI Tanh-polar transform that warps the whole image to a Tanh-polar representation with a fixed ratio between the face area and the context, guided by the target bounding box. The new representation contains all information in the original image, and allows for rotation equivariance in the convolutional neural networks~(CNNs). Third, we propose a hybrid residual representation learning block, coined HybridBlock, that contains convolutional layers in both the Tanh-polar space and the Tanh-Cartesian space, allowing for receptive fields of different shapes in CNNs. Through extensive experiments, we show that the proposed method improves the state-of-the-art for face parsing in the wild and does not require facial landmarks for alignment.

cs.CV↗

arXiv2021

A Reinforcement-Learning-Based Energy-Efficient Framework for Multi-Task Video Analytics Pipeline

Deep-learning-based video processing has yielded transformative results in recent years. However, the video analytics pipeline is energy-intensive due to high data rates and reliance on complex inference algorithms, which limits its adoption in energy-constrained applications. Motivated by the observation of high and variable spatial redundancy and temporal dynamics in video data streams, we design and evaluate an adaptive-resolution optimization framework to minimize the energy use of multi-task video analytics pipelines. Instead of heuristically tuning the input data resolution of individual tasks, our framework utilizes deep reinforcement learning to dynamically govern the input resolution and computation of the entire video analytics pipeline. By monitoring the impact of varying resolution on the quality of high-dimensional video analytics features, hence the accuracy of video analytics results, the proposed end-to-end optimization framework learns the best non-myopic policy for dynamically controlling the resolution of input video streams to globally optimize energy efficiency. Governed by reinforcement learning, optical flow is incorporated into the framework to minimize unnecessary spatio-temporal redundancy that leads to re-computation, while preserving accuracy. The proposed framework is applied to video instance segmentation which is one of the most challenging computer vision tasks, and achieves better energy efficiency than all baseline methods of similar accuracy on the YouTube-VIS dataset.

cs.CV↗

arXiv2021

Local Morphological Measures Confirm that Folding within Small Partitions of the Human Cortex Follows Universal Scaling Law

The universal scaling law of cortical morphology describes cortical folding as the covariance of average grey matter thickness, pial surface area, and exposed surface area. It applies for mammalian species, humans, and across lobes, however it remains to be shown that local cortical folding obeys the same rules. Here, we develop a method to obtain morphological measures for small regions across the cortex and correct surface areas by curvature to account for differences in patch size, resulting in a map of local morphology. It enables a near-pointwise analysis of morphological variables and their regional changes due to processes such as healthy ageing. We confirm empirically that the theorised covariance of morphological measures still holds at this level of local partition sizes as predicted, justifying the use of independent variables derived from the scaling law to identify regional differences in folding, subject-specific abnormalities, and local effects of ageing.

q-bio.NC↗