Searcharxiv⌕ Search

arXiv subjects

Yuanhang Zhang

Publications and source records attributed to Yuanhang Zhang.

47 records · Page 3Linked to original sources
arXiv2021

UniCon: Unified Context Network for Robust Active Speaker Detection

We introduce a new efficient framework, the Unified Context Network (UniCon), for robust active speaker detection (ASD). Traditional methods for ASD usually operate on each candidate's pre-cropped face track separately and do not sufficiently consider the relationships among the candidates. This potentially limits performance, especially in challenging scenarios with low-resolution faces, multiple candidates, etc. Our solution is a novel, unified framework that focuses on jointly modeling multiple types of contextual information: spatial context to indicate the position and scale of each candidate's face, relational context to capture the visual relationships among the candidates and contrast audio-visual affinities with each other, and temporal context to aggregate long-term information and smooth out local uncertainties. Based on such information, our model optimizes all candidates in a unified process for robust and reliable ASD. A thorough ablation study is performed on several challenging ASD benchmarks under different settings. In particular, our method outperforms the state-of-the-art by a large margin of about 15% mean Average Precision (mAP) absolute on two challenging subsets: one with three candidate speakers, and the other with faces smaller than 64 pixels. Together, our UniCon achieves 92.0% mAP on the AVA-ActiveSpeaker validation set, surpassing 90% for the first time on this challenging dataset at the time of submission. Project website: https://unicon-asd.github.io/.

cs.CV↗
arXiv2020

Simultaneous Sorting of Wavelengths and Spatial Modes using Multi-Plane Light Conversion

We propose a wavelength-mode sorter realized by multi-plane light conversion (MPLC). For the first time, to our best knowledge, wavelengths and spatial modes can be sorted simultaneously. We first demonstrate pure wavelength sorting by a series of phase masks, which could find applications in high-power wavelength beam combining (WBC) or coarse wavelength-division multiplexing (CWDM), for example. We then present a design of a 4-wavelength, 3-mode sorter using only 5 phase masks. Insertion loss (IL) and mode dependent loss (MDL) as low as 1.27 dB and 2.45 dB can be achieved, respectively.

physics.optics↗
arXiv2020

Deformation Flow Based Two-Stream Network for Lip Reading

Lip reading is the task of recognizing the speech content by analyzing movements in the lip region when people are speaking. Observing on the continuity in adjacent frames in the speaking process, and the consistency of the motion patterns among different speakers when they pronounce the same phoneme, we model the lip movements in the speaking process as a sequence of apparent deformations in the lip region. Specifically, we introduce a Deformation Flow Network (DFN) to learn the deformation flow between adjacent frames, which directly captures the motion information within the lip region. The learned deformation flow is then combined with the original grayscale frames with a two-stream network to perform lip reading. Different from previous two-stream networks, we make the two streams learn from each other in the learning process by introducing a bidirectional knowledge distillation loss to train the two branches jointly. Owing to the complementary cues provided by different branches, the two-stream network shows a substantial improvement over using either single branch. A thorough experimental evaluation on two large-scale lip reading benchmarks is presented with detailed analysis. The results accord with our motivation, and show that our method achieves state-of-the-art or comparable performance on these two challenging datasets.

cs.CV↗
arXiv2020

Can We Read Speech Beyond the Lips? Rethinking RoI Selection for Deep Visual Speech Recognition

Recent advances in deep learning have heightened interest among researchers in the field of visual speech recognition (VSR). Currently, most existing methods equate VSR with automatic lip reading, which attempts to recognise speech by analysing lip motion. However, human experience and psychological studies suggest that we do not always fix our gaze at each other's lips during a face-to-face conversation, but rather scan the whole face repetitively. This inspires us to revisit a fundamental yet somehow overlooked problem: can VSR models benefit from reading extraoral facial regions, i.e. beyond the lips? In this paper, we perform a comprehensive study to evaluate the effects of different facial regions with state-of-the-art VSR models, including the mouth, the whole face, the upper face, and even the cheeks. Experiments are conducted on both word-level and sentence-level benchmarks with different characteristics. We find that despite the complex variations of the data, incorporating information from extraoral facial regions, even the upper face, consistently benefits VSR performance. Furthermore, we introduce a simple yet effective method based on Cutout to learn more discriminative features for face-based VSR, hoping to maximise the utility of information encoded in different facial regions. Our experiments show obvious improvements over existing state-of-the-art methods that use only the lip region as inputs, a result we believe would probably provide the VSR community with some new and exciting insights.

cs.CV↗
arXiv2019

Stable homotopy, 1-dimensional NCCW complexes, and Property (H)

In this paper, we show that the homomorphisms between two unital one-dimensional NCCW complexes with the same KK-class are stably homotopic, i.e., with adding on a common homomorphism (with finite dimensional image), they are homotopic. As a consequence, any one-dimensional NCCW complex has the Property (H).

math.OA↗
arXiv2019

Operators which are polynomially isometric to a normal operator

Let $\mathcal{H}$ be a complex, separable Hilbert space and $\mathcal{B}(\mathcal{H})$ denote the algebra of all bounded linear operators acting on $\mathcal{H}$. Given a unitarily-invariant norm $\| \cdot \|_u$ on $\mathcal{B}(\mathcal{H})$ and two linear operators $A$ and $B$ in $\mathcal{B}(\mathcal{H})$, we shall say that $A$ and $B$ are \emph{polynomially isometric relative to} $\| \cdot \|_u$ if $\| p(A) \|_u = \| p(B) \|_u$ for all polynomials $p$. In this paper, we examine to what extent an operator $A$ being polynomially isometric to a normal operator $N$ implies that $A$ is itself normal. More explicitly, we first show that if $\| \cdot \|_u$ is any unitarily-invariant norm on $\mathbb{M}_n(\mathbb{C})$, if $A, N \in \mathbb{M}_n(\mathbb{C})$ are polynomially isometric and $N$ is normal, then $A$ is normal. We then extend this result to the infinite-dimensional setting by showing that if $A, N \in \mathcal{B}(\mathcal{H})$ are polynomially isometric relative to the operator norm and $N$ is a normal operator whose spectrum neither disconnects the plane nor has interior, then $A$ is normal, while if the spectrum of $N$ is not of this form, then there always exists a non-normal operator $B$ such that $B$ and $N$ are polynomially isometric. Finally, we show that if $A$ and $N$ are compact operators with $N$ normal, and if $A$ and $N$ are polynomially isometric with respect to the $(c,p)$-norm studied by Chan, Li and Tu, then $A$ is again normal.

math.FA↗
arXiv2019

Normal operators with highly incompatible off-diagonal corners

Let $\mathcal{H}$ be a complex, separable Hilbert space, and $\mathcal{B}(\mathcal{H})$ denote the set of all bounded linear operators on $\mathcal{H}$. Given an orthogonal projection $P \in \mathcal{B}(\mathcal{H})$ and an operator $D \in \mathcal{B}(\mathcal{H})$, we may write $D=\begin{bmatrix} D_1& D_2 D_3 & D_4 \end{bmatrix}$ relative to the decomposition $\mathcal{H} = \mathrm{ran}\, P \oplus \mathrm{ran}\, (I-P)$. In this paper we study the question: for which non-negative integers $j, k$ can we find a normal operator $D$ and an orthogonal projection $P$ such that $\mathrm{rank}\, D_2 = j$ and $\mathrm{rank}\, D_3 = k$? Complete results are obtained in the case where $\mathrm{dim}\, \mathcal{H} < \infty$, and partial results are obtained in the infinite-dimensional setting.

math.FA↗
arXiv2019

Dilated FCN: Listening Longer to Hear Better

Deep neural network solutions have emerged as a new and powerful paradigm for speech enhancement (SE). The capabilities to capture long context and extract multi-scale patterns are crucial to design effective SE networks. Such capabilities, however, are often in conflict with the goal of maintaining compact networks to ensure good system generalization. In this paper, we explore dilation operations and apply them to fully convolutional networks (FCNs) to address this issue. Dilations equip the networks with greatly expanded receptive fields, without increasing the number of parameters. Different strategies to fuse multi-scale dilations, as well as to install the dilation modules are explored in this work. Using Noisy VCTK and AzBio sentences datasets, we demonstrate that the proposed dilation models significantly improve over the baseline FCN and outperform the state-of-the-art SE solutions.

cs.SD↗
arXiv2019

LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wild

Large-scale datasets have successively proven their fundamental importance in several research fields, especially for early progress in some emerging topics. In this paper, we focus on the problem of visual speech recognition, also known as lipreading, which has received increasing interest in recent years. We present a naturally-distributed large-scale benchmark for lip reading in the wild, named LRW-1000, which contains 1,000 classes with 718,018 samples from more than 2,000 individual speakers. Each class corresponds to the syllables of a Mandarin word composed of one or several Chinese characters. To the best of our knowledge, it is currently the largest word-level lipreading dataset and also the only public large-scale Mandarin lipreading dataset. This dataset aims at covering a "natural" variability over different speech modes and imaging conditions to incorporate challenges encountered in practical applications. It has shown a large variation in this benchmark in several aspects, including the number of samples in each class, video resolution, lighting conditions, and speakers' attributes such as pose, age, gender, and make-up. Besides providing a detailed description of the dataset and its collection pipeline, we evaluate several typical popular lipreading methods and perform a thorough analysis of the results from several aspects. The results demonstrate the consistency and challenges of our dataset, which may open up some new promising directions for future work.

cs.CV↗