arXiv subjects

Ning Jiang

Publications and source records attributed to Ning Jiang.

At least 127 records · Page 7Linked to original sources

arXiv2024

Conversational Speech Recognition by Learning Audio-textual Cross-modal Contextual Representation

Automatic Speech Recognition (ASR) in conversational settings presents unique challenges, including extracting relevant contextual information from previous conversational turns. Due to irrelevant content, error propagation, and redundancy, existing methods struggle to extract longer and more effective contexts. To address this issue, we introduce a novel conversational ASR system, extending the Conformer encoder-decoder model with cross-modal conversational representation. Our approach leverages a cross-modal extractor that combines pre-trained speech and text models through a specialized encoder and a modal-level mask input. This enables the extraction of richer historical speech context without explicit error propagation. We also incorporate conditional latent variational modules to learn conversational level attributes such as role preference and topic coherence. By introducing both cross-modal and conversational representations into the decoder, our model retains context over longer sentences without information loss, achieving relative accuracy improvements of 8.8% and 23% on Mandarin conversation datasets HKUST and MagicData-RAMC, respectively, compared to the standard Conformer model.

cs.SD↗

arXiv2024

Boosting Multi-Speaker Expressive Speech Synthesis with Semi-supervised Contrastive Learning

This paper aims to build a multi-speaker expressive TTS system, synthesizing a target speaker's speech with multiple styles and emotions. To this end, we propose a novel contrastive learning-based TTS approach to transfer style and emotion across speakers. Specifically, contrastive learning from different levels, i.e. utterance and category level, is leveraged to extract the disentangled style, emotion, and speaker representations from speech for style and emotion transfer. Furthermore, a semi-supervised training strategy is introduced to improve the data utilization efficiency by involving multi-domain data, including style-labeled data, emotion-labeled data, and abundant unlabeled data. To achieve expressive speech with diverse styles and emotions for a target speaker, the learned disentangled representations are integrated into an improved VITS model. Experiments on multi-domain data demonstrate the effectiveness of the proposed method.

eess.AS↗

arXiv2024

Compressible Euler limit from Boltzmann equation with complete diffusive boundary condition in half-space

In this paper, we prove the compressible Euler limit from the Boltzmann equation with hard sphere collisional kernel and complete diffusive boundary condition in half-space by employing the Hilbert expansion which includes interior and Knudsen layers. This rigorously justifies the corresponding formal analysis in Sone's book \cite{Sone-2007-Book} in the context of short time smooth solutions, and also generalizes the classic Caflisch's result \cite{Caflish-1980-CPAM} to initial-boundary problem case.

math.AP↗

arXiv2024

AT2023lli: A Tidal Disruption Event with Prominent Optical Early Bump and Delayed Episodic X-ray Emission

High-cadence, multiwavelength observations have continuously revealed the diversity of tidal disruption events (TDEs), thus greatly advancing our knowledge and understanding of TDEs. In this work, we conducted an intensive optical-UV and X-ray follow-up campaign of TDE AT2023lli, and found a remarkable month-long bump in its UV/optical light curve nearly two months prior to maximum brightness. The bump represents the longest separation time from the main peak among known TDEs to date. The main UV/optical outburst declines as $t^{-4.10}$, making it one of the fastest decaying optically selected TDEs. Furthermore, we detected sporadic X-ray emission 30 days after the UV/optical peak, accompanied by a reduction in the period of inactivity. It is proposed that the UV/optical bump could be caused by the self-intersection of the stream debris, whereas the primary peak is generated by the reprocessed emission of the accretion process. In addition, our results suggest that episodic X-ray radiation during the initial phase of decline may be due to the patched obscurer surrounding the accretion disk, a phenomenon associated with the inhomogeneous reprocessing process. The double TDE scenario, in which two stars are disrupted in sequence, is also a possible explanation for producing the observed early bump and main peak. We anticipate that the multicolor light curves of TDEs, especially in the very early stages, and the underlying physics can be better understood in the near future with the assistance of dedicated surveys such as the deep high-cadence survey of the 2.5-meter Wide Field Survey Telescope (WFST).

astro-ph.HE↗

arXiv2024

Magnetically arrested disks in FR I radio galaxies

A sample of 17 FR I radio galaxies constructed from the 3CR catalog, which is characterized by edge-darkened radio structures, is studied. The optical core luminosities derived from Hubble Space Telescope observation are used to estimate the Eddington ratios which are found to be below $10^{-3.4}$ for this sample. This is supported by the Baldwin-Phillips-Terlevich optical diagnostic diagrams derived with the spectroscopic observation of Telescopio Nazionale Galileo, suggesting that these sources are of low ionization nuclear Emission-line Regions (LINERs). It implies that the accretion in these FR I sources can be modeled as advection-dominated accretion flows (ADAFs). Given the low accretion rate, the predicted jet power with a fast-spinning black hole (BH) $a=0.95$ in the Blandford-Znajek mechanics is lower than the estimated one for almost all the sources in our sample. Such powerful jets indicate the presence of magnetically arrested disks (MAD) in the inner region of the ADAF, in the sense that the magnetic fields in the inner accretion zone are strong. Moreover, we show that, even in the MAD scenario, the BH spins in the sample are most likely moderate and/or fast with $a\gtrsim0.5$.

astro-ph.CO↗

arXiv2024

Awakening of A Blazar at Redshift 2.7 Temporally Coincident with Arrival of Cospatial Neutrino Event IceCube-201221A

We report on multiwavelength studies of a blazar NVSS J171822+423948, which is identified as the low-energy counterpart of 4FGL J1718.5+4237, the unique $γ$-ray source known to be cospatial with the IceCube neutrino event IC-201221A. After a 12-year long quiescent period undetected by Fermi-LAT, $γ$-ray activities with a tenfold flux increase emerge soon (a few tens of days) after arrival of the neutrino. Associated optical flares in the ZTF $g$, $r$, and $i$ bands are observed together with elevated WISE infrared fluxes. Synchronized variations suggest that both the $γ$-ray emission and the neutrino event are connected to the blazar. Furthermore, the optical spectrum reveals emission lines at a redshift $z$ = 2.68 $\pm$ 0.01. Thus, it is the first candidate for a neutrino-emitting blazar at the redshift above 2. Discussions of theoretical constraints of neutrino production and comparisons with other candidates are presented.

astro-ph.HE↗

arXiv2024

Factorized Learning Assisted with Large Language Model for Gloss-free Sign Language Translation

Previous Sign Language Translation (SLT) methods achieve superior performance by relying on gloss annotations. However, labeling high-quality glosses is a labor-intensive task, which limits the further development of SLT. Although some approaches work towards gloss-free SLT through jointly training the visual encoder and translation network, these efforts still suffer from poor performance and inefficient use of the powerful Large Language Model (LLM). Most seriously, we find that directly introducing LLM into SLT will lead to insufficient learning of visual representations as LLM dominates the learning curve. To address these problems, we propose Factorized Learning assisted with Large Language Model (FLa-LLM) for gloss-free SLT. Concretely, we factorize the training process into two stages. In the visual initialing stage, we employ a lightweight translation model after the visual encoder to pre-train the visual encoder. In the LLM fine-tuning stage, we freeze the acquired knowledge in the visual encoder and integrate it with a pre-trained LLM to inspire the LLM's translation potential. This factorized training strategy proves to be highly effective as evidenced by significant improvements achieved across three SLT datasets which are all conducted under the gloss-free setting.

cs.CL↗

arXiv2024

ASASSN-18ap: A Dusty Tidal Disruption Event Candidate with an Early Bump in the Light Curve

We re-examined the classification of the optical transient ASASSN-18ap, which was initially identified as a supernova (SNe) upon its discovery. Based on newly emerged phenomena, such as a delayed luminous infrared outburst and the emergence of luminous coronal emission lines, we suggest that ASASSN-18ap is more likely a tidal disruption event (TDE) in a dusty environment, rather than a supernova. The total energy in the infrared outburst is $\rm 3.1\times10^{51}$ erg, which is an order of magnitude higher than the total energy in the optical-to-ultraviolet range, indicating a large dust extinction, an extra-EUV component, or anisotropic continuum emission. A bumpy feature appeared in the optical light curve at the start of brightening, which was reported in a couple of TDEs very recently. This early bump may have been overlooked in the past due to the lack of sufficient sampling of the light curves of most TDEs during their ascending phase, and it could provide insight into the origin of optical emission.

astro-ph.HE↗

arXiv2024

An audio-quality-based multi-strategy approach for target speaker extraction in the MISP 2023 Challenge

This paper describes our audio-quality-based multi-strategy approach for the audio-visual target speaker extraction (AVTSE) task in the Multi-modal Information based Speech Processing (MISP) 2023 Challenge. Specifically, our approach adopts different extraction strategies based on the audio quality, striking a balance between interference removal and speech preservation, which benifits the back-end automatic speech recognition (ASR) systems. Experiments show that our approach achieves a character error rate (CER) of 24.2% and 33.2% on the Dev and Eval set, respectively, obtaining the second place in the challenge.

cs.SD↗

arXiv2024

Asymptotic behavior of 3-D evolutionary model of Magnetoelasticity for small data

In this article, we consider the evolutionary model for magnetoelasticity with vanishing viscosity/damping, which is a nonlinear dispersive system. The global regularity and scattering of the evolutionary model for magnetoelasticity under small size of initial data is proved. Our proof relies on the idea of vector-field method due to the quasilinearity and the presence of convective term. A key observation is that we construct a suitable energy functional including the mass quantity, which enable us to provide a good decay estimates for Schrödinger flow. In particular, we establish the asymptotic behavior in both mass and energy spaces for Schrödinger map, not only for gauged equation.

math.AP↗

arXiv2024

SELM: Speech Enhancement Using Discrete Tokens and Language Models

Language models (LMs) have shown superior performances in various speech generation tasks recently, demonstrating their powerful ability for semantic context modeling. Given the intrinsic similarity between speech generation and speech enhancement, harnessing semantic information holds potential advantages for speech enhancement tasks. In light of this, we propose SELM, a novel paradigm for speech enhancement, which integrates discrete tokens and leverages language models. SELM comprises three stages: encoding, modeling, and decoding. We transform continuous waveform signals into discrete tokens using pre-trained self-supervised learning (SSL) models and a k-means tokenizer. Language models then capture comprehensive contextual information within these tokens. Finally, a detokenizer and HiFi-GAN restore them into enhanced speech. Experimental results demonstrate that SELM achieves comparable performance in objective metrics alongside superior results in subjective perception. Our demos are available https://honee-w.github.io/SELM/.

eess.AS↗

arXiv2023

An Explanation for Overrepresentation of Tidal Disruption Events in Post-starburst Galaxies

Tidal disruption events\,(TDEs) provide a valuable probe in studying the dynamics of stars in the nuclear environments of galaxies. Recent observations show that TDEs are strongly overrepresented in post-starburst or "green valley" galaxies, although the underlying physical mechanism remains unclear. Considering the possible interaction between stars and active galactic nucleus\,(AGN) disk, the TDE rates can be greatly changed compared to those in quiescent galactic nuclei. In this work, we revisit TDE rates by incorporating an evolving AGN disk within the framework of the "loss cone" theory. We numerically evolve the Fokker-Planck equations by considering the star-disk interactions, in-situ star formation in the unstable region of the outer AGN disk and the evolution of the accretion process for supermassive black holes\,(SMBHs). We find that the TDE rates are enhanced by about two orders of magnitude shortly after the AGN transitions into a non-active stage. During this phase, the accumulated stars are rapidly scattered into the loss cone due to the disappearance of the inner standard thin disk. Our results provide an explanation for the overrepresentation of TDEs in post-starburst galaxies.

astro-ph.HE↗

arXiv2023

VoxBlink: A Large Scale Speaker Verification Dataset on Camera

In this paper, we introduce a large-scale and high-quality audio-visual speaker verification dataset, named VoxBlink. We propose an innovative and robust automatic audio-visual data mining pipeline to curate this dataset, which contains 1.45M utterances from 38K speakers. Due to the inherent nature of automated data collection, introducing noisy data is inevitable. Therefore, we also utilize a multi-modal purification step to generate a cleaner version of the VoxBlink, named VoxBlink-clean, comprising 18K identities and 1.02M utterances. In contrast to the VoxCeleb, the VoxBlink sources from short videos of ordinary users, and the covered scenarios can better align with real-life situations. To our best knowledge, the VoxBlink dataset is one of the largest publicly available speaker verification datasets. Leveraging the VoxCeleb and VoxBlink-clean datasets together, we employ diverse speaker verification models with multiple architectural backbones to conduct comprehensive evaluations on the VoxCeleb test sets. Experimental results indicate a substantial enhancement in performance,ranging from 12% to 30% relatively, across various backbone architectures upon incorporating the VoxBlink-clean into the training process. The details of the dataset can be found on http://voxblink.github.io

eess.AS↗

arXiv2023

R2-Talker: Realistic Real-Time Talking Head Synthesis with Hash Grid Landmarks Encoding and Progressive Multilayer Conditioning

Dynamic NeRFs have recently garnered growing attention for 3D talking portrait synthesis. Despite advances in rendering speed and visual quality, challenges persist in enhancing efficiency and effectiveness. We present R2-Talker, an efficient and effective framework enabling realistic real-time talking head synthesis. Specifically, using multi-resolution hash grids, we introduce a novel approach for encoding facial landmarks as conditional features. This approach losslessly encodes landmark structures as conditional features, decoupling input diversity, and conditional spaces by mapping arbitrary landmarks to a unified feature space. We further propose a scheme of progressive multilayer conditioning in the NeRF rendering pipeline for effective conditional feature fusion. Our new approach has the following advantages as demonstrated by extensive experiments compared with the state-of-the-art works: 1) The lossless input encoding enables acquiring more precise features, yielding superior visual quality. The decoupling of inputs and conditional spaces improves generalizability. 2) The fusing of conditional features and MLP outputs at each MLP layer enhances conditional impact, resulting in more accurate lip synthesis and better visual quality. 3) It compactly structures the fusion of conditional features, significantly enhancing computational efficiency.

cs.CV↗

arXiv2023

PMMTalk: Speech-Driven 3D Facial Animation from Complementary Pseudo Multi-modal Features

Speech-driven 3D facial animation has improved a lot recently while most related works only utilize acoustic modality and neglect the influence of visual and textual cues, leading to unsatisfactory results in terms of precision and coherence. We argue that visual and textual cues are not trivial information. Therefore, we present a novel framework, namely PMMTalk, using complementary Pseudo Multi-Modal features for improving the accuracy of facial animation. The framework entails three modules: PMMTalk encoder, cross-modal alignment module, and PMMTalk decoder. Specifically, the PMMTalk encoder employs the off-the-shelf talking head generation architecture and speech recognition technology to extract visual and textual information from speech, respectively. Subsequently, the cross-modal alignment module aligns the audio-image-text features at temporal and semantic levels. Then PMMTalk decoder is employed to predict lip-syncing facial blendshape coefficients. Contrary to prior methods, PMMTalk only requires an additional random reference face image but yields more accurate results. Additionally, it is artist-friendly as it seamlessly integrates into standard animation production workflows by introducing facial blendshape coefficients. Finally, given the scarcity of 3D talking face datasets, we introduce a large-scale 3D Chinese Audio-Visual Facial Animation (3D-CAVFA) dataset. Extensive experiments and user studies show that our approach outperforms the state of the art. We recommend watching the supplementary video.

cs.CV↗

arXiv2023

Incompressible limit of the Ericksen-Leslie hyperbolic liquid crystal model in compressible flow

Ericksen and Leslie proposed a hydrodynamic model for liquid crystals in the format of conservation laws in the 1960s. Their original model includes inertial and compressibility effects, which makes the model a coupled parabolic-hyperbolic system. In this paper we build up the connection between the compressible and incompressible parabolic-hyperbolic liquid crystal model in the framework of classical solutions. We first derive the scaled Ericksen-Leslie system with dimensionless numbers, including Mach, Reynolds, and Ericksen numbers. In particular, we introduce the so-called inertial constant $χ$ which characterizes the inertial effect of the liquid crystal molecular. Next, we establish the energy estimates uniform in the Mach number $ε$ for both the compressible system and its time-derivative system with small data. Then, we pass to the limit $ε\rightarrow 0$ in the compressible system, so that we establish the global classical solution of the incompressible system by the compactness arguments. Moreover, we also obtain the convergence rates associated with $L^2$-norm in the case of well-prepared initial data. This is the first result on the incompressible limit of the compressible parabolic-hyperbolic liquid crystal model and confirms the relations of different parabolic-hyperbolic liquid crystal model rigorously.

math.AP↗

arXiv2023

Dissonance in harmony: The UV/optical periodic outbursts of ASASSN-14ko exhibit repeated bumps and rebrightenings

ASASSN-14ko was identified as an abnormal periodic nuclear transient with a potential decreasing period. Its outbursts in the optical and UV bands have displayed a consistent and smooth "fast-rise and slow-decay" pattern since its discovery, which has recently experienced an unexpected alteration in the last two epochs, as revealed by our proposed high-cadence Swift observations. The new light curve profiles show a bump during the rising stages and a rebrightening during the declining stages, making them much broader and symmetrical than the previous ones. In the last two epochs, there is no significant difference in the X-ray spectral slope compared to the previous one, and its overall luminosity is lower than those of the previous epochs. The energy released in the early bump and rebrightening phases ($\sim10^{50}$ erg) could be due to collision of the stripped stream from partial tidal disruption events (pTDEs) with an expanded accretion disk. We also discussed other potential explanations, such as disk instability and star-disk collisions. Further high-cadence multi-wavelength observations of subsequent cycles are encouraged to comprehend the unique periodic source with its new intriguing features.

astro-ph.HE↗

arXiv2023

Haha-Pod: An Attempt for Laughter-based Non-Verbal Speaker Verification

It is widely acknowledged that discriminative representation for speaker verification can be extracted from verbal speech. However, how much speaker information that non-verbal vocalization carries is still a puzzle. This paper explores speaker verification based on the most ubiquitous form of non-verbal voice, laughter. First, we use a semi-automatic pipeline to collect a new Haha-Pod dataset from open-source podcast media. The dataset contains over 240 speakers' laughter clips with corresponding high-quality verbal speech. Second, we propose a Two-Stage Teacher-Student (2S-TS) framework to minimize the within-speaker embedding distance between verbal and non-verbal (laughter) signals. Considering Haha-Pod as a test set, two trials (S2L-Eval) are designed to verify the speaker's identity through laugh sounds. Experimental results demonstrate that our method can significantly improve the performance of the S2L-Eval test set with only a minor degradation on the VoxCeleb1 test set. The resources for the Haha-Pod dataset can be found at https://github.com/nevermoreLin/HahaPod.

eess.AS↗