arXiv subjects

Pengfei Sun

Publications and source records attributed to Pengfei Sun.

30 records · Page 2Linked to original sources

arXiv2021

NTIRE 2021 Challenge on Perceptual Image Quality Assessment

This paper reports on the NTIRE 2021 challenge on perceptual image quality assessment (IQA), held in conjunction with the New Trends in Image Restoration and Enhancement workshop (NTIRE) workshop at CVPR 2021. As a new type of image processing technology, perceptual image processing algorithms based on Generative Adversarial Networks (GAN) have produced images with more realistic textures. These output images have completely different characteristics from traditional distortions, thus pose a new challenge for IQA methods to evaluate their visual quality. In comparison with previous IQA challenges, the training and testing datasets in this challenge include the outputs of perceptual image processing algorithms and the corresponding subjective scores. Thus they can be used to develop and evaluate IQA methods on GAN-based distortions. The challenge has 270 registered participants in total. In the final testing stage, 13 participating teams submitted their models and fact sheets. Almost all of them have achieved much better results than existing IQA methods, while the winning method can demonstrate state-of-the-art performance.

eess.IV↗

arXiv2019

GLMNet: Graph Learning-Matching Networks for Feature Matching

Recently, graph convolutional networks (GCNs) have shown great potential for the task of graph matching. It can integrate graph node feature embedding, node-wise affinity learning and matching optimization together in a unified end-to-end model. One important aspect of graph matching is the construction of two matching graphs. However, the matching graphs we feed to existing graph convolutional matching networks are generally fixed and independent of graph matching, which thus are not guaranteed to be optimal for the graph matching task. Also, existing GCN matching method employs several general smoothing-based graph convolutional layers to generate graph node embeddings, in which extensive smoothing convolution operation may dilute the desired discriminatory information of graph nodes. To overcome these issues, we propose a novel Graph Learning-Matching Network (GLMNet) for graph matching problem. GLMNet has three main aspects. (1) It integrates graph learning into graph matching which thus adaptively learn a pair of optimal graphs that best serve graph matching task. (2) It further employs a Laplacian sharpening convolutional module to generate more discriminative node embeddings for graph matching. (3) A new constraint regularized loss is designed for GLMNet training which can encode the desired one-to-one matching constraints in matching optimization. Experiments on two benchmarks demonstrate the effectiveness of GLMNet and advantages of its main modules.

cs.CV↗

arXiv2019

Selfie: User-defined Sensitive Memory Protection and Recovery

Different users always have different requirement for sensitive memory definition. It is not flexible for aborting program execution once detecting memory corruption. Because the users may loose some sensitive data. We presented Selfie, a hybrid solution to provide one flexible solution to protect the sensitive memory according to users' requirements in runtime. Finally, Selfie can provide one solution to decide whether execution needs to be recovered. If the memory corruption doesn't belong sensitive memory, Selfie provides symbolic solver that can help figure out whether the memory corruption can affect the sensitive memory in future.

cs.CR↗

arXiv2019

Brain2Char: A Deep Architecture for Decoding Text from Brain Recordings

Decoding language representations directly from the brain can enable new Brain-Computer Interfaces (BCI) for high bandwidth human-human and human-machine communication. Clinically, such technologies can restore communication in people with neurological conditions affecting their ability to speak. In this study, we propose a novel deep network architecture Brain2Char, for directly decoding text (specifically character sequences) from direct brain recordings (called Electrocorticography, ECoG). Brain2Char framework combines state-of-the-art deep learning modules --- 3D Inception layers for multiband spatiotemporal feature extraction from neural data and bidirectional recurrent layers, dilated convolution layers followed by language model weighted beam search to decode character sequences, optimizing a connectionist temporal classification (CTC) loss. Additionally, given the highly non-linear transformations that underlie the conversion of cortical function to character sequences, we perform regularizations on the network's latent representations motivated by insights into cortical encoding of speech production and artifactual aspects specific to ECoG data acquisition. To do this, we impose auxiliary losses on latent representations for articulatory movements, speech acoustics and session specific non-linearities. In 3 participants tested here, Brain2Char achieves 10.6\%, 8.5\% and 7.0\% Word Error Rates (WER) respectively on vocabulary sizes ranging from 1200 to 1900 words. Brain2Char also performs well when 2 participants silently mimed sentences. These results set a new state-of-the-art on decoding text from brain and demonstrate the potential of Brain2Char as a high-performance communication BCI.

cs.LG↗

arXiv2019

Modeling neural dynamics during speech production using a state space variational autoencoder

Characterizing the neural encoding of behavior remains a challenging task in many research areas due in part to complex and noisy spatiotemporal dynamics of evoked brain activity. An important aspect of modeling these neural encodings involves separation of robust, behaviorally relevant signals from background activity, which often contains signals from irrelevant brain processes and decaying information from previous behavioral events. To achieve this separation, we develop a two-branch State Space Variational AutoEncoder (SSVAE) model to individually describe the instantaneous evoked foreground signals and the context-dependent background signals. We modeled the spontaneous speech-evoked brain dynamics using smoothed Gaussian mixture models. By applying the proposed SSVAE model to track ECoG dynamics in one participant over multiple hours, we find that the model can predict speech-related dynamics more accurately than other latent factor inference algorithms. Our results demonstrate that separately modeling the instantaneous speech-evoked and slow context-dependent brain dynamics can enhance tracking performance, which has important implications for the development of advanced neural encoding and decoding models in various neuroscience sub-disciplines.

cs.NE↗

arXiv2017

Investigations of Auditory Filters Based Excitation Patterns for Assessment of Noise Induced Hearing Loss

Noise induced hearing loss (NIHL) as one of major avoidable occupational related health issues has been studied for decades. To assess NIHL, the excitation pattern (EP) has been considered as one of mechanisms to estimate movements of basilar membrane (BM) in cochlea. In this study, two auditory filters, dual resonance nonlinear (DRNL) filter and rounded-exponential (ROEX) filter, have been applied to create two EPs, referring as the velocity EP and the loudness EP, respectively. Two noise hazard metrics are also proposed based on the developed EPs to evaluate hazardous levels caused by different types of noise. Moreover, Gaussian noise and pure-tone noise have been simulated to evaluate performances of the developed EPs and noise metrics. The results show that both developed EPs can reflect the responses of BM to different types of noise. For Gaussian noise, there is a frequency shift between the velocity EP and the loudness EP. For pure-tone noise, both EPs can reflect the frequencies of input noise accurately. The results suggest that both EPs can be potentially used for assessment of NIHL.

physics.bio-ph↗

arXiv2017

Enhanced Factored Three-Way Restricted Boltzmann Machines for Speech Detection

In this letter, we propose enhanced factored three way restricted Boltzmann machines (EFTW-RBMs) for speech detection. The proposed model incorporates conditional feature learning by multiplying the dynamical state of the third unit, which allows a modulation over the visible-hidden node pairs. Instead of stacking previous frames of speech as the third unit in a recursive manner, the correlation related weighting coefficients are assigned to the contextual neighboring frames. Specifically, a threshold function is designed to capture the long-term features and blend the globally stored speech structure. A factored low rank approximation is introduced to reduce the parameters of the three-dimensional interaction tensor, on which non-negative constraint is imposed to address the sparsity characteristic. The validations through the area-under-ROC-curve (AUC) and signal distortion ratio (SDR) show that our approach outperforms several existing 1D and 2D (i.e., time and time-frequency domain) speech detection algorithms in various noisy environments.

cs.SD↗

arXiv2017

Neural networks based EEG-Speech Models

In this paper, we propose an end-to-end neural network (NN) based EEG-speech (NES) modeling framework, in which three network structures are developed to map imagined EEG signals to phonemes. The proposed NES models incorporate a language model based EEG feature extraction layer, an acoustic feature mapping layer, and a restricted Boltzmann machine (RBM) based the feature learning layer. The NES models can jointly realize the representation of multichannel EEG signals and the projection of acoustic speech signals. Among three proposed NES models, two augmented networks utilize spoken EEG signals as either bias or gate information to strengthen the feature learning and translation of imagined EEG signals. Experimental results show that all three proposed NES models outperform the baseline support vector machine (SVM) method on EEG-speech classification. With respect to binary classification, our approach achieves comparable results relative to deep believe network approach.

cs.SD↗

arXiv2017

Semi-supervised Speech Enhancement in Envelop and Details Subspaces

In this study, we propose a modulation decoupling based single channel speech enhancement subspace framework, in which the spectrogram of noisy speech is decoupled as the product of a spectral envelop subspace and a spectral details subspace. This decoupling approach provides a method to specifically work on elimination of those noises that greatly affect the intelligibility. Two supervised low-rank and sparse decomposition schemes are developed in the spectral envelop subspace to obtain a robust recovery of speech components. A Bayesian formulation of non-negative factorization is used to learn the speech dictionary from the spectral envelop subspace of clean speech samples. In the spectral details subspace, a standard robust principal component analysis is implemented to extract the speech components. The validation results show that compared with four speech enhancement algorithms, including MMSE-SPP, NMF-RPCA, RPCA, and LARC, the proposed MS based algorithms achieve satisfactory performance on improving perceptual quality, and especially speech intelligibility.

cs.SD↗

arXiv2016

Speech Enhancement via Two-Stage Dual Tree Complex Wavelet Packet Transform with a Speech Presence Probability Estimator

In this paper, a two-stage dual tree complex wavelet packet transform (DTCWPT) based speech enhancement algorithm has been proposed, in which a speech presence probability (SPP) estimator and a generalized minimum mean squared error (MMSE) estimator are developed. To overcome the drawback of signal distortions caused by down sampling of WPT, a two-stage analytic decomposition concatenating undecimated WPT (UWPT) and decimated WPT is employed. An SPP estimator in the DTCWPT domain is derived based on a generalized Gamma distribution of speech, and Gaussian noise assumption. The validation results show that the proposed algorithm can obtain enhanced perceptual evaluation of speech quality (PESQ), and segmental signal-to-noise ratio (SegSNR) at low SNR nonstationary noise, compared with other four state-of-the-art speech enhancement algorithms, including optimally modified LSA (OM-LSA), soft masking using a posteriori SNR uncertainty (SMPO), a posteriori SPP based MMSE estimation (MMSE-SPP), and adaptive Bayesian wavelet thresholding (BWT).

cs.SD↗

arXiv2016

Low Rank and Sparsity Analysis Applied to Speech Enhancement via Online Estimated Dictionary

We propose an online estimated dictionary based single channel speech enhancement algorithm, which focuses on low rank and sparse matrix decomposition. In this proposed algorithm, a noisy speech spectral matrix is considered as the summation of low rank background noise components and an activation of the online speech dictionary, on which both low rank and sparsity constraints are imposed. This decomposition takes the advantage of local estimated dictionary high expressiveness on speech components. The local dictionary can be obtained through estimating the speech presence probability by applying Expectation Maximal algorithm, in which a generalized Gamma prior for speech magnitude spectrum is used. The evaluation results show that the proposed algorithm achieves significant improvements when compared to four other speech enhancement algorithms.

cs.SD↗

arXiv2015

A Novel Scattered Pilot Design for FBMC/OQAM Systems

Filter bank multi-carrier with offset quadrature amplitude modulation (FBMC/OQAM) has been heavily studied as an alternative waveform for 5G systems. Its advantages of higher spectrum efficiency, localized frequency response and insensitivity to synchronization errors may enable promising performance when orthogonal frequency division multiplexing (OFDM) fails. However, performing channel estimation under the intrinsic interference has been a fundamental obstacle towards adopting FBMC/OQMA in a practical system. Several schemes are available but the performance is far from satisfaction. In this paper, we will show the existing methods are trapped by the paradigm that a clean pilot is mandatory so as to explicitly carry a reference symbol to the receiver for the purpose of channel estimation. By breaking this paradigm, a novel dual dependent pilot scheme is proposed, which gives up the independent pilot and derives dual pilots from the imposed interference. By doing this, the interference between pilots can be fully utilized. Consequentially, the new scheme significantly outperforms existing solutions and the simulation results show FBMC/OQAM can achieve close-to-OFDM performance in a practical system even with the presence of strong intrinsic interference.

cs.IT↗