arXiv subjects

Yujiang Wang

Publications and source records attributed to Yujiang Wang.

94 records · Page 6Linked to original sources

arXiv2017

End-to-End Audiovisual Fusion with LSTMs

Several end-to-end deep learning approaches have been recently presented which simultaneously extract visual features from the input images and perform visual speech classification. However, research on jointly extracting audio and visual features and performing classification is very limited. In this work, we present an end-to-end audiovisual model based on Bidirectional Long Short-Term Memory (BLSTM) networks. To the best of our knowledge, this is the first audiovisual fusion model which simultaneously learns to extract features directly from the pixels and spectrograms and perform classification of speech and nonlinguistic vocalisations. The model consists of multiple identical streams, one for each modality, which extract features directly from mouth regions and spectrograms. The temporal dynamics in each stream/modality are modeled by a BLSTM and the fusion of multiple streams/modalities takes place via another BLSTM. An absolute improvement of 1.9% in the mean F1 of 4 nonlingusitic vocalisations over audio-only classification is reported on the AVIC database. At the same time, the proposed end-to-end audiovisual fusion system improves the state-of-the-art performance on the AVIC database leading to a 9.7% absolute increase in the mean F1 measure. We also perform audiovisual speech recognition experiments on the OuluVS2 database using different views of the mouth, frontal to profile. The proposed audiovisual system significantly outperforms the audio-only model for all views when the acoustic noise is high.

cs.CV↗

arXiv2017

End-to-End Multi-View Lipreading

Non-frontal lip views contain useful information which can be used to enhance the performance of frontal view lipreading. However, the vast majority of recent lipreading works, including the deep learning approaches which significantly outperform traditional approaches, have focused on frontal mouth images. As a consequence, research on joint learning of visual features and speech classification from multiple views is limited. In this work, we present an end-to-end multi-view lipreading system based on Bidirectional Long-Short Memory (BLSTM) networks. To the best of our knowledge, this is the first model which simultaneously learns to extract features directly from the pixels and performs visual speech classification from multiple views and also achieves state-of-the-art performance. The model consists of multiple identical streams, one for each view, which extract features directly from different poses of mouth images. The temporal dynamics in each stream/view are modelled by a BLSTM and the fusion of multiple streams/views takes place via another BLSTM. An absolute average improvement of 3% and 3.8% over the frontal view performance is reported on the OuluVS2 database when the best two (frontal and profile) and three views (frontal, profile, 45) are combined, respectively. The best three-view model results in a 10.5% absolute improvement over the current multi-view state-of-the-art performance on OuluVS2, without using external databases for training, achieving a maximum classification accuracy of 96.9%.

cs.CV↗

arXiv2017

Mechanisms underlying different onset patterns of focal seizures

Focal seizures are episodes of pathological brain activity that appear to arise from a localised area of the brain. The onset patterns of focal seizure activity have been studied intensively, and they have largely been distinguished into two types - low amplitude fast oscillations (LAF), or high amplitude spikes (HAS). Here we explore whether these two patterns arise from fundamentally different mechanisms. Here, we use a previously established computational model of neocortical tissue, and validate it as an adequate model using clinical recordings of focal seizures. We then reproduce the two onset patterns in their most defining properties and investigate the possible mechanisms underlying the different focal seizure onset patterns in the model. We show that the two patterns are associated with different mechanisms at the spatial scale of a single ECoG electrode. The LAF onset is initiated by independent patches of localised activity, which slowly invade the surrounding tissue and coalesce over time. In contrast, the HAS onset is a global, systemic transition to a coexisting seizure state triggered by a local event. We find that such a global transition is enabled by an increase in the excitability of the "healthy" surrounding tissue, which by itself does not generate seizures, but can support seizure activity when incited. In our simulations, the difference in surrounding tissue excitability also offers a simple explanation of the clinically reported difference in surgical outcomes. Finally, we demonstrate in the model how changes in tissue excitability could be elucidated, in principle, using active stimulation. Taken together, our modelling results suggest that the excitability of the tissue surrounding the seizure core may play a determining role in the seizure onset pattern, as well as in the surgical outcome.

q-bio.NC↗

arXiv2017

Understanding Epileptiform After-Discharges as Rhythmic Oscillatory Transients

Electro-cortical activity in patients with epilepsy may show abnormal rhythmic transients in response to stimulation. Even when using the same stimulation parameters in the same patient, wide variability in the duration of transient response has been reported. These transients have long been considered important for the mapping of the excitability levels in the epileptic brain but their dynamic mechanism is still not well understood. To understand the occurrence of abnormal transients dynamically, we use a thalamo-cortical neural population model of epileptic spike-wave activity and study the interaction between slow and fast subsystems. In a reduced version of the thalamo-cortical model, slow wave oscillations arise from a fold of cycles (FoC) bifurcation. This marks the onset of a region of bistability between a high amplitude oscillatory rhythm and the background state. In vicinity of the bistability in parameter space, the model has excitable dynamics, showing prolonged rhythmic transients in response to suprathreshold pulse stimulation. We analyse the state space geometry of the bistable and excitable states, and find that the rhythmic transient arises when the impending FoC bifurcation deforms the state space and creates an area of locally reduced attraction to the fixed point. This area essentially allows trajectories to dwell there before escaping to the stable steady state, thus creating rhythmic transients. In the full thalamo-cortical model, we find a similar FoC bifurcation structure. Based on the analysis, we propose an explanation of why stimulation induced epileptiform activity may vary between trials, and predict how the variability could be related to ongoing oscillatory background activity.

q-bio.NC↗