arXiv subjects

Jianyuan Zhong

Publications and source records attributed to Jianyuan Zhong.

21 records · Page 2Linked to original sources

arXiv2020

Multi-task self-supervised learning for Robust Speech Recognition

Despite the growing interest in unsupervised learning, extracting meaningful knowledge from unlabelled audio remains an open challenge. To take a step in this direction, we recently proposed a problem-agnostic speech encoder (PASE), that combines a convolutional encoder followed by multiple neural networks, called workers, tasked to solve self-supervised problems (i.e., ones that do not require manual annotations as ground truth). PASE was shown to capture relevant speech information, including speaker voice-print and phonemes. This paper proposes PASE+, an improved version of PASE for robust speech recognition in noisy and reverberant environments. To this end, we employ an online speech distortion module, that contaminates the input signals with a variety of random disturbances. We then propose a revised encoder that better learns short- and long-term speech dynamics with an efficient combination of recurrent and convolutional networks. Finally, we refine the set of workers used in self-supervision to encourage better cooperation. Results on TIMIT, DIRHA and CHiME-5 show that PASE+ significantly outperforms both the previous version of PASE as well as common acoustic features. Interestingly, PASE+ learns transferable representations suitable for highly mismatched acoustic conditions.

eess.AS↗

arXiv2019

UR-FUNNY: A Multimodal Language Dataset for Understanding Humor

Humor is a unique and creative communicative behavior displayed during social interactions. It is produced in a multimodal manner, through the usage of words (text), gestures (vision) and prosodic cues (acoustic). Understanding humor from these three modalities falls within boundaries of multimodal language; a recent research trend in natural language processing that models natural language as it happens in face-to-face communication. Although humor detection is an established research area in NLP, in a multimodal context it is an understudied area. This paper presents a diverse multimodal dataset, called UR-FUNNY, to open the door to understanding multimodal language used in expressing humor. The dataset and accompanying studies, present a framework in multimodal humor detection for the natural language processing community. UR-FUNNY is publicly available for research.

cs.LG↗

arXiv2000

On The Homflypt Skein Module of S^1 x S^2

Let $k$ be a subring of the field of rational functions in $x, v, s$ which contains $x^{\pm 1}, v^{\pm 1}, s^{\pm 1}$. If $M$ is an oriented 3-manifold, let $S(M)$ denote the Homflypt skein module of $M$ over $k$. This is the free $k$-module generated by isotopy classes of framed oriented links in $M$ quotiented by the Homflypt skein relations: (1) $x^{-1}L_{+}-xL_{-}=(s-s^{-1})L_{0}$; (2) $L$ with a positive twist $=(xv^{-1})L$; (3) $L\sqcup O=(\frac{v-v^{-1}}{s-s^{-1}})L$ where $O$ is the unknot. We give two bases for the relative Homflypt skein module of the solid torus with 2 points in the boundary. The first basis is related to the basis of $S(S^1\times D^2)$ given by V. Turaev and also J. Hoste and M. Kidwell; the second basis is related to a Young idempotent basis for $S(S^1\times D^2)$ based on the work of A. Aiston, H. Morton and C. Blanchet. We prove that if the elements $s^{2n}-1$, for $n$ a nonzero integer, and the elements $s^{2m}-v^{2}$, for any integer $m$, are invertible in $k$, then $S(S^{1} \times S^2)=k$-torsion module $\oplus k$. Here the free part is generated by the empty link $ϕ$. In addition, if the elements $s^{2m}-v^{4}$, for $m$ an integer, are invertible in $k$, then $S(S^{1} \times S^2)$ has no torsion. We also obtain some results for more general $k$.

math.GT↗