arXiv subjects

Xin Jing

Publications and source records attributed to Xin Jing.

33 records · Page 2Linked to original sources

arXiv2024

STAA-Net: A Sparse and Transferable Adversarial Attack for Speech Emotion Recognition

Speech contains rich information on the emotions of humans, and Speech Emotion Recognition (SER) has been an important topic in the area of human-computer interaction. The robustness of SER models is crucial, particularly in privacy-sensitive and reliability-demanding domains like private healthcare. Recently, the vulnerability of deep neural networks in the audio domain to adversarial attacks has become a popular area of research. However, prior works on adversarial attacks in the audio domain primarily rely on iterative gradient-based techniques, which are time-consuming and prone to overfitting the specific threat model. Furthermore, the exploration of sparse perturbations, which have the potential for better stealthiness, remains limited in the audio domain. To address these challenges, we propose a generator-based attack method to generate sparse and transferable adversarial examples to deceive SER models in an end-to-end and efficient manner. We evaluate our method on two widely-used SER datasets, Database of Elicited Mood in Speech (DEMoS) and Interactive Emotional dyadic MOtion CAPture (IEMOCAP), and demonstrate its ability to generate successful sparse adversarial examples in an efficient manner. Moreover, our generated adversarial examples exhibit model-agnostic transferability, enabling effective adversarial attacks on advanced victim models.

cs.SD↗

arXiv2023

Kohn-Sham accuracy from orbital-free density functional theory via $Δ$-machine learning

We present a $Δ$-machine learning model for obtaining Kohn-Sham accuracy from orbital-free density functional theory (DFT) calculations. In particular, we employ a machine learned force field (MLFF) scheme based on the kernel method to capture the difference between Kohn-Sham and orbital-free DFT energies/forces. We implement this model in the context of on-the-fly molecular dynamics simulations, and study its accuracy, performance, and sensitivity to parameters for representative systems. We find that the formalism not only improves the accuracy of Thomas-Fermi-von Weizs{ä}cker (TFW) orbital-free energies and forces by more than two orders of magnitude, but is also more accurate than MLFFs based solely on Kohn-Sham DFT, while being more efficient and less sensitive to model parameters. We apply the framework to study the structure of molten Al$_{0.88}$Si$_{0.12}$, the results suggesting no aggregation of Si atoms, in agreement with a previous Kohn-Sham study performed at an order of magnitude smaller length and time scales.

physics.chem-ph↗

arXiv2023

Ab-initio investigation of finite size effects in rutile titania nanoparticles with semilocal and nonlocal density functionals

In this work, we employ hybrid and generalized gradient approximation (GGA) level density functional theory (DFT) calculations to investigate the convergence of surface properties and band structure of rutile titania (TiO$_2$) nanoparticles with particle size. The surface energies and band structures are calculated for cuboidal particles with minimum dimension ranging from 3.7 Å (24 atoms) to 10.3 Å (384 atoms) using a highly-parallel real-space DFT code to enable hybrid level DFT calculations of larger nanoparticles than are typically practical. We deconvolute the geometric and electronic finite size effects in surface energy, and evaluate the influence of defects on band structure and density of states (DOS). The electronic finite size effects in surface energy vanish when the minimum length scale of the nanoparticles becomes greater than 10 Å. We show that this length scale is consistent with a computationally efficient numerical analysis of the characteristic length scale of electronic interactions. The surface energy of nanoparticles having minimum dimension beyond this characteristic length can be approximated using slab calculations that account for the geometric defects. In contrast, the finite size effects on the band structure is highly dependent on the shape and size of these particles. The DOS for cuboidal particles and more realistic particles constructed using the Wulff algorithm reveal that defect states within the bandgap play a key role in determining the band structure of nanoparticles and the bandgap does not converge to the bulk limit for the particle sizes investigated.

cond-mat.mtrl-sci↗

arXiv2023

Assessing the source of error in the Thomas-Fermi-von Weizsäcker density functional

We investigate the source of error in the Thomas-Fermi-von Weizsäcker (TFW) density functional relative to Kohn-Sham density functional theory (DFT). In particular, through numerical studies on a range of materials, for a variety of crystal structures subject to strain and atomic displacements, we find that while the ground state electron density in TFW orbital-free DFT is close to the Kohn-Sham density, the corresponding energy deviates significantly from the Kohn-Sham value. We show that these differences are a consequence of the poor representation of the linear response within the TFW approximation for the electronic kinetic energy, confirming conjectures in the literature. In so doing, we find that the energy computed from a non-self-consistent Kohn-Sham calculation using the TFW electronic ground state density is in very good agreement with that obtained from the fully self-consistent Kohn-Sham solution.

cond-mat.mtrl-sci↗

arXiv2023

U-DiT TTS: U-Diffusion Vision Transformer for Text-to-Speech

Deep learning has led to considerable advances in text-to-speech synthesis. Most recently, the adoption of Score-based Generative Models (SGMs), also known as Diffusion Probabilistic Models (DPMs), has gained traction due to their ability to produce high-quality synthesized neural speech in neural speech synthesis systems. In SGMs, the U-Net architecture and its variants have long dominated as the backbone since its first successful adoption. In this research, we mainly focus on the neural network in diffusion-model-based Text-to-Speech (TTS) systems and propose the U-DiT architecture, exploring the potential of vision transformer architecture as the core component of the diffusion models in a TTS system. The modular design of the U-DiT architecture, inherited from the best parts of U-Net and ViT, allows for great scalability and versatility across different data scales. The proposed U-DiT TTS system is a mel spectrogram-based acoustic model and utilizes a pretrained HiFi-GAN as the vocoder. The objective (ie Frechet distance) and MOS results show that our DiT-TTS system achieves state-of-art performance on the single speaker dataset LJSpeech. Our demos are publicly available at: https://eihw.github.io/u-dit-tts/

cs.SD↗

arXiv2023

HEAR4Health: A blueprint for making computer audition a staple of modern healthcare

Recent years have seen a rapid increase in digital medicine research in an attempt to transform traditional healthcare systems to their modern, intelligent, and versatile equivalents that are adequately equipped to tackle contemporary challenges. This has led to a wave of applications that utilise AI technologies; first and foremost in the fields of medical imaging, but also in the use of wearables and other intelligent sensors. In comparison, computer audition can be seen to be lagging behind, at least in terms of commercial interest. Yet, audition has long been a staple assistant for medical practitioners, with the stethoscope being the quintessential sign of doctors around the world. Transforming this traditional technology with the use of AI entails a set of unique challenges. We categorise the advances needed in four key pillars: Hear, corresponding to the cornerstone technologies needed to analyse auditory signals in real-life conditions; Earlier, for the advances needed in computational and data efficiency; Attentively, for accounting to individual differences and handling the longitudinal nature of medical data; and, finally, Responsibly, for ensuring compliance to the ethical standards accorded to the field of medicine.

cs.SD↗

arXiv2022

Version 2.0.0 -- M-SPARC: Matlab-Simulation Package for Ab-initio Real-space Calculations

M-SPARC is a Matlab code for performing ab initio Kohn-Sham Density Functional Theory simulations. Version 2.0.0 of the software further extends its capability to include relativistic effects, dispersion interactions, and advanced semilocal/nonlocal exchange-correlation functionals. These features significantly increase the fidelity of first principles calculations that can be performed using M-SPARC.

cond-mat.mtrl-sci↗

arXiv2022

Redundancy Reduction Twins Network: A Training framework for Multi-output Emotion Regression

In this paper, we propose the Redundancy Reduction Twins Network (RRTN), a redundancy reduction training framework that minimizes redundancy by measuring the cross-correlation matrix between the outputs of the same network fed with distorted versions of a sample and bringing it as close to the identity matrix as possible. RRTN also applies a new loss function, the Barlow Twins loss function, to help maximize the similarity of representations obtained from different distorted versions of a sample. However, as the distribution of losses can cause performance fluctuations in the network, we also propose the use of a Restrained Uncertainty Weight Loss (RUWL) or joint training to identify the best weights for the loss function. Our best approach on CNN14 with the proposed methodology obtains a CCC over emotion regression of 0.678 on the ExVo Multi-task dev set, a 4.8% increase over a vanilla CNN 14 CCC of 0.647, which achieves a significant difference at the 95% confidence interval (2-tailed).

cs.SD↗

arXiv2022

Dynamic Restrained Uncertainty Weighting Loss for Multitask Learning of Vocal Expression

We propose a novel Dynamic Restrained Uncertainty Weighting Loss to experimentally handle the problem of balancing the contributions of multiple tasks on the ICML ExVo 2022 Challenge. The multitask aims to recognize expressed emotions and demographic traits from vocal bursts jointly. Our strategy combines the advantages of Uncertainty Weight and Dynamic Weight Average, by extending weights with a restraint term to make the learning process more explainable. We use a lightweight multi-exit CNN architecture to implement our proposed loss approach. The experimental H-Mean score (0.394) shows a substantial improvement over the baseline H-Mean score (0.335).

cs.SD↗

arXiv2022

Exploring speaker enrolment for few-shot personalisation in emotional vocalisation prediction

In this work, we explore a novel few-shot personalisation architecture for emotional vocalisation prediction. The core contribution is an `enrolment' encoder which utilises two unlabelled samples of the target speaker to adjust the output of the emotion encoder; the adjustment is based on dot-product attention, thus effectively functioning as a form of `soft' feature selection. The emotion and enrolment encoders are based on two standard audio architectures: CNN14 and CNN10. The two encoders are further guided to forget or learn auxiliary emotion and/or speaker information. Our best approach achieves a CCC of $.650$ on the ExVo Few-Shot dev set, a $2.5\%$ increase over our baseline CNN14 CCC of $.634$.

cs.SD↗

arXiv2022

A Temporal-oriented Broadcast ResNet for COVID-19 Detection

Detecting COVID-19 from audio signals, such as breathing and coughing, can be used as a fast and efficient pre-testing method to reduce the virus transmission. Due to the promising results of deep learning networks in modelling time sequences, and since applications to rapidly identify COVID in-the-wild should require low computational effort, we present a temporal-oriented broadcasting residual learning method that achieves efficient computation and high accuracy with a small model size. Based on the EfficientNet architecture, our novel network, named Temporal-oriented ResNet~(TorNet), constitutes of a broadcasting learning block, i.e. the Alternating Broadcast (AB) Block, which contains several Broadcast Residual Blocks (BC ResBlocks) and a convolution layer. With the AB Block, the network obtains useful audio-temporal features and higher level embeddings effectively with much less computation than Recurrent Neural Networks~(RNNs), typically used to model temporal information. TorNet achieves 72.2% Unweighted Average Recall (UAR) on the INTERPSEECH 2021 Computational Paralinguistics Challenge COVID-19 cough Sub-Challenge, by this showing competitive results with a higher computational efficiency than other state-of-the-art alternatives.

cs.SD↗

arXiv2022

An Overview & Analysis of Sequence-to-Sequence Emotional Voice Conversion

Emotional voice conversion (EVC) focuses on converting a speech utterance from a source to a target emotion; it can thus be a key enabling technology for human-computer interaction applications and beyond. However, EVC remains an unsolved research problem with several challenges. In particular, as speech rate and rhythm are two key factors of emotional conversion, models have to generate output sequences of differing length. Sequence-to-sequence modelling is recently emerging as a competitive paradigm for models that can overcome those challenges. In an attempt to stimulate further research in this promising new direction, recent sequence-to-sequence EVC papers were systematically investigated and reviewed from six perspectives: their motivation, training strategies, model architectures, datasets, model inputs, and evaluation methods. This information is organised to provide the research community with an easily digestible overview of the current state-of-the-art. Finally, we discuss existing challenges of sequence-to-sequence EVC.

cs.SD↗

arXiv2022

DXQ-Net: Differentiable LiDAR-Camera Extrinsic Calibration Using Quality-aware Flow

Accurate LiDAR-camera extrinsic calibration is a precondition for many multi-sensor systems in mobile robots. Most calibration methods rely on laborious manual operations and calibration targets. While working online, the calibration methods should be able to extract information from the environment to construct the cross-modal data association. Convolutional neural networks (CNNs) have powerful feature extraction ability and have been used for calibration. However, most of the past methods solve the extrinsic as a regression task, without considering the geometric constraints involved. In this paper, we propose a novel end-to-end extrinsic calibration method named DXQ-Net, using a differentiable pose estimation module for generalization. We formulate a probabilistic model for LiDAR-camera calibration flow, yielding a prediction of uncertainty to measure the quality of LiDAR-camera data association. Testing experiments illustrate that our method achieves a competitive with other methods for the translation component and state-of-the-art performance for the rotation component. Generalization experiments illustrate that the generalization performance of our method is significantly better than other deep learning-based methods.

cs.RO↗

arXiv2022

Audio Self-supervised Learning: A Survey

Inspired by the humans' cognitive ability to generalise knowledge and skills, Self-Supervised Learning (SSL) targets at discovering general representations from large-scale data without requiring human annotations, which is an expensive and time consuming task. Its success in the fields of computer vision and natural language processing have prompted its recent adoption into the field of audio and speech processing. Comprehensive reviews summarising the knowledge in audio SSL are currently missing. To fill this gap, in the present work, we provide an overview of the SSL methods used for audio and speech processing applications. Herein, we also summarise the empirical works that exploit the audio modality in multi-modal SSL frameworks, and the existing suitable benchmarks to evaluate the power of SSL in the computer audition domain. Finally, we discuss some open problems and point out the future directions on the development of audio SSL.

cs.SD↗

arXiv2021

Real-space density kernel method for Kohn-Sham density functional theory calculations at high temperature

Kohn-Sham density functional theory calculations using conventional diagonalization based methods become increasingly expensive as temperature increases due to the need to compute increasing numbers of partially occupied states. We present a density matrix based method for Kohn-Sham calculations at high temperature that eliminates the need for diagonalization entirely, thus reducing the cost of such calculations significantly. Specifically, we develop real-space expressions for the electron density, electronic free energy, Hellmann-Feynman forces, and Hellmann-Feynman stress tensor in terms of an orthonormal auxiliary orbital basis and its density kernel transform, the density kernel being the matrix representation of the density operator in the auxiliary basis. Using Chebyshev filtering to generate the auxiliary basis, we next develop an approach akin to Clenshaw-Curtis spectral quadrature to calculate the individual columns of the density kernel based on the Fermi operator expansion in Chebyshev polynomials; and employ a similar approach to evaluate band structure and entropic energy components. We implement the proposed formulation in the SPARC electronic structure code, using which we show systematic convergence of the aforementioned quantities to exact diagonalization results, and obtain significant speedups relative to conventional diagonalization based methods. Finally, we employ the new method to compute the self-diffusion coefficient and viscosity of aluminum at 116,045 K from Kohn-Sham quantum molecular dynamics, where we find agreement with previous more approximate orbital-free density functional methods.

physics.chem-ph↗