arXiv subjects

Haoyu Li

Publications and source records attributed to Haoyu Li.

116 records · Page 7Linked to original sources

arXiv2020

Improved Prosody from Learned F0 Codebook Representations for VQ-VAE Speech Waveform Reconstruction

Vector Quantized Variational AutoEncoders (VQ-VAE) are a powerful representation learning framework that can discover discrete groups of features from a speech signal without supervision. Until now, the VQ-VAE architecture has previously modeled individual types of speech features, such as only phones or only F0. This paper introduces an important extension to VQ-VAE for learning F0-related suprasegmental information simultaneously along with traditional phone features.The proposed framework uses two encoders such that the F0 trajectory and speech waveform are both input to the system, therefore two separate codebooks are learned. We used a WaveRNN vocoder as the decoder component of VQ-VAE. Our speaker-independent VQ-VAE was trained with raw speech waveforms from multi-speaker Japanese speech databases. Experimental results show that the proposed extension reduces F0 distortion of reconstructed speech for all unseen test speakers, and results in significantly higher preference scores from a listening test. We additionally conducted experiments using single-speaker Mandarin speech to demonstrate advantages of our architecture in another language which relies heavily on F0.

eess.AS↗

arXiv2020

Noise Tokens: Learning Neural Noise Templates for Environment-Aware Speech Enhancement

In recent years, speech enhancement (SE) has achieved impressive progress with the success of deep neural networks (DNNs). However, the DNN approach usually fails to generalize well to unseen environmental noise that is not included in the training. To address this problem, we propose "noise tokens" (NTs), which are a set of neural noise templates that are jointly trained with the SE system. NTs dynamically capture the environment variability and thus enable the DNN model to handle various environments to produce STFT magnitude with higher quality. Experimental results show that using NTs is an effective strategy that consistently improves the generalization ability of SE systems across different DNN architectures. Furthermore, we investigate applying a state-of-the-art neural vocoder to generate waveform instead of traditional inverse STFT (ISTFT). Subjective listening tests show the residual noise can be significantly suppressed through mel-spectrogram correction and vocoder-based waveform synthesis.

eess.AS↗

arXiv2020

iMetricGAN: Intelligibility Enhancement for Speech-in-Noise using Generative Adversarial Network-based Metric Learning

The intelligibility of natural speech is seriously degraded when exposed to adverse noisy environments. In this work, we propose a deep learning-based speech modification method to compensate for the intelligibility loss, with the constraint that the root mean square (RMS) level and duration of the speech signal are maintained before and after modifications. Specifically, we utilize an iMetricGAN approach to optimize the speech intelligibility metrics with generative adversarial networks (GANs). Experimental results show that the proposed iMetricGAN outperforms conventional state-of-the-art algorithms in terms of objective measures, i.e., speech intelligibility in bits (SIIB) and extended short-time objective intelligibility (ESTOI), under a Cafeteria noise condition. In addition, formal listening tests reveal significant intelligibility gains when both noise and reverberation exist.

eess.AS↗

arXiv2019

Faster super-resolution imaging with auto-correlation two-step deconvolution

Despite super-resolution fluorescence blinking microscopes break the diffraction limit, the intense phototoxic illumination and long-term image sequences thus far still pose to major challenges in visualizing live-organisms. Here, we proposed a super-resolution method based on auto-correlation two-step deconvolution (SACD) to enhance the temporal resolution at lower signal intensity levels. Unlike conventional techniques, such as super-resolution optical fluctuation imaging (SOFI) or stochastic optical reconstruction microscopy (STORM), our model allows 16 frames to generate super-resolution images, without noticeable degradation in recording quality. We demonstrate SACD both in simulated predictions and experimental validations, with the resulting spatial resolution of 64 nm and 2-10 fold speed improvements. The use of low signal-to-noise ratio acquisition of image sequences, our SACD enables fast, background-free, super-resolution imaging that maybe advance implemented as a suitable tool for rapid live-cells imaging.

physics.optics↗

arXiv2019

NNVA: Neural Network Assisted Visual Analysis of Yeast Cell Polarization Simulation

Complex computational models are often designed to simulate real-world physical phenomena in many scientific disciplines. However, these simulation models tend to be computationally very expensive and involve a large number of simulation input parameters which need to be analyzed and properly calibrated before the models can be applied for real scientific studies. We propose a visual analysis system to facilitate interactive exploratory analysis of high-dimensional input parameter space for a complex yeast cell polarization simulation. The proposed system can assist the computational biologists, who designed the simulation model, to visually calibrate the input parameters by modifying the parameter values and immediately visualizing the predicted simulation outcome without having the need to run the original expensive simulation for every instance. Our proposed visual analysis system is driven by a trained neural network-based surrogate model as the backend analysis framework. Surrogate models are widely used in the field of simulation sciences to efficiently analyze computationally expensive simulation models. In this work, we demonstrate the advantage of using neural networks as surrogate models for visual analysis by incorporating some of the recent advances in the field of uncertainty quantification, interpretability and explainability of neural network-based models. We utilize the trained network to perform interactive parameter sensitivity analysis of the original simulation at multiple levels-of-detail as well as recommend optimal parameter configurations using the activation maximization framework of neural networks. We also facilitate detail analysis of the trained network to extract useful insights about the simulation model, learned by the network, during the training process.

cs.HC↗

arXiv2017

Espresso: Brewing Java For More Non-Volatility with Non-volatile Memory

Fast, byte-addressable non-volatile memory (NVM) embraces both near-DRAM latency and disk-like persistence, which has generated considerable interests to revolutionize system software stack and programming models. However, it is less understood how NVM can be combined with managed runtime like Java virtual machine (JVM) to ease persistence management. This paper proposes Espresso, a holistic extension to Java and its runtime, to enable Java programmers to exploit NVM for persistence management with high performance. Espresso first provides a general persistent heap design called Persistent Java Heap (PJH) to manage persistent data as normal Java objects. The heap is then strengthened with a recoverable mechanism to provide crash consistency for heap metadata. It then provides a new abstraction called Persistent Java Object (PJO) to provide an easy-to-use but safe persistent programming model for programmers to persist application data. The evaluation confirms that Espresso significantly outperforms state-of-art NVM support for Java (i.e., JPA and PCJ) while being compatible to existing data structures in Java programs.

cs.PL↗

arXiv2016

Volumetric Light-field Encryption at the Microscopic Scale

We report a light-field based method that allows the optical encryption of three-dimensional (3D) volumetric information at the microscopic scale in a single 2D light-field image. The system consists of a microlens array and an array of random phase/amplitude masks. The method utilizes a wave optics model to account for the dominant diffraction effect at this new scale, and the system point-spread function (PSF) serves as the key for encryption and decryption. We successfully developed and demonstrated a deconvolution algorithm to retrieve spatially multiplexed discrete and continuous volumetric data from 2D light-field images. Showing that the method is practical for data transmission and storage, we obtained a faithful reconstruction of the 3D volumetric information from a digital copy of the encrypted light-field image. The method represents a new level of optical encryption, paving the way for broad industrial and biomedical applications in processing and securing 3D data at the microscopic scale.

cs.CR↗

arXiv2013

Tap-Wave-Rub: Lightweight Malware Prevention for Smartphones Using Intuitive Human Gestures

In this paper, we introduce a lightweight permission enforcement approach - Tap-Wave-Rub (TWR) - for smartphone malware prevention. TWR is based on simple human gestures that are very quick and intuitive but less likely to be exhibited in users' daily activities. Presence or absence of such gestures, prior to accessing an application, can effectively inform the OS whether the access request is benign or malicious. Specifically, we present the design of two mechanisms: (1) accelerometer based phone tapping detection; and (2) proximity sensor based finger tapping, rubbing or hand waving detection. The first mechanism is geared for NFC applications, which usually require the user to tap her phone with another device. The second mechanism involves very simple gestures, i.e., tapping or rubbing a finger near the top of phone's screen or waving a hand close to the phone, and broadly appeals to many applications (e.g., SMS). In addition, we present the TWR-enhanced Android permission model, the prototypes implementing the underlying gesture recognition mechanisms, and a variety of novel experiments to evaluate these mechanisms. Our results suggest the proposed approach could be very effective for malware detection and prevention, with quite low false positives and false negatives, while imposing little to no additional burden on the users.

cs.CR↗