arXiv subjects

Hui Zhan

Publications and source records attributed to Hui Zhan.

12 recordsLinked to original sources

arXiv2025

Ming-UniAudio: Speech LLM for Joint Understanding, Generation and Editing with Unified Representation

Existing speech models suffer from competing requirements on token representations by understanding and generation tasks. This discrepancy in representation prevents speech language models from performing instruction-based free-form editing. To solve this challenge, we introduce a novel framework that unifies speech understanding, generation, and editing. The core of our unified model is a unified continuous speech tokenizer MingTok-Audio, the first continuous tokenizer to effectively integrate semantic and acoustic features, which makes it suitable for both understanding and generation tasks. Based on this unified continuous audio tokenizer, we developed the speech language model Ming-UniAudio, which achieved a balance between generation and understanding capabilities. Ming-UniAudio sets new state-of-the-art (SOTA) records on 8 out of 12 metrics on the ContextASR benchmark. Notably, for Chinese voice cloning, it achieves a highly competitive Seed-TTS-WER of 0.95. Leveraging this foundational model, we further trained a dedicated speech editing model Ming-UniAudio-Edit, the first speech language model that enables universal, free-form speech editing guided solely by natural language instructions, handling both semantic and acoustic modifications without timestamp condition. To rigorously assess the editing capability and establish a foundation for future research, we introduce Ming-Freeform-Audio-Edit, the first comprehensive benchmark tailored for instruction-based free-form speech editing, featuring diverse scenarios and evaluation dimensions spanning semantic correctness, acoustic quality, and instruction alignment. We open-sourced the continuous audio tokenizer, the unified foundational model, and the free-form instruction-based editing model to facilitate the development of unified audio understanding, generation, and manipulation.

cs.CL↗

arXiv2023

CALM: Contrastive Cross-modal Speaking Style Modeling for Expressive Text-to-Speech Synthesis

To further improve the speaking styles of synthesized speeches, current text-to-speech (TTS) synthesis systems commonly employ reference speeches to stylize their outputs instead of just the input texts. These reference speeches are obtained by manual selection which is resource-consuming, or selected by semantic features. However, semantic features contain not only style-related information, but also style irrelevant information. The information irrelevant to speaking style in the text could interfere the reference audio selection and result in improper speaking styles. To improve the reference selection, we propose Contrastive Acoustic-Linguistic Module (CALM) to extract the Style-related Text Feature (STF) from the text. CALM optimizes the correlation between the speaking style embedding and the extracted STF with contrastive learning. Thus, a certain number of the most appropriate reference speeches for the input text are selected by retrieving the speeches with the top STF similarities. Then the style embeddings are weighted summarized according to their STF similarities and used to stylize the synthesized speech of TTS. Experiment results demonstrate the effectiveness of our proposed approach, with both objective evaluations and subjective evaluations on the speaking styles of the synthesized speeches outperform a baseline approach with semantic-feature-based reference selection.

cs.SD↗

arXiv2022

Boundedness and exponential stabilization for time-space fractional parabolic-elliptic Keller-Segel model in higher dimensions

For the time-space fractional degenerate Keller-Segel equation \begin{equation*} \begin{cases} \partial _{t}^{\beta }u=-(-\Delta )^{\frac{\alpha}{2}}(\rho (v)u),& t>0\\ (-\Delta )^{\frac{\alpha}{2}} v+v=u,& t>0 \end{cases} \end{equation*} $x\in\Omega, \Omega \subset \mathbb{R}^{n}, \beta\in (0,1),\alpha\in (1,2)$, we consider for $n\geq 3$ the problem of finding a time-independent upper bound of the classical solution such that as $\theta>0,C>0$ \begin{equation*} \left \| u(\cdot ,t)-\overline{u_{0}} \right \|_{L^{\infty }(\Omega )}+\left \| v(\cdot ,t)-\overline{u_{0}} \right \|_{W^{1,\infty }(\Omega )}\leq Ce^{(-\theta)^{1/\beta}t}, \end{equation*} where $\overline{u_{0}}=\frac{1}{\left | \Omega \right |}\int _{\Omega }u_{0}dx$. We find such solution in the special cases of time-independent upper bound of the concentration with Alikakos-Moser iteration and fractional differential inequality. In those cases the problem is reduced to a time-space fractional parabolic-elliptic equation which is treated with Lyapunov functional methods. A key element in our construction is a proof of the exponential stabilization toward the constant steady states by using fractional Duhamel type integral equation.

math.AP↗

arXiv2022

Positive effects of multiplicative noise on the explosion of nonlinear fractional stochastic differential equations

For the nonlinear stochastic partial differential equation which is driven by multiplicative noise of the form \[D_t^\beta u = \left[ { - {{\left( { - \Delta } \right)}^s}u + \zeta \left( u \right)} \right]dt + A\sum\limits_{m \in Z_0^d} {\sum\limits_{j = 1}^{d - 1} {{\theta _m}{\sigma _{m,j}}\left( x \right)} } \circ dW_t^{m,j},\;\; s \ge 1,\;\;\frac{1}{2} < \beta < 1,\] where $D_{t}^\beta $ denotes the Caputo derivative, $A>0$ is a constant depending on the noise intensity, $\circ$ represent the Stratonovich-type stochastic differential, we consider the blow-up time of its solutions. We find that the introduction of noise can effectively delay the blow-up time of the solution to the deterministic differential equation when $\zeta$ in the above equation satisfies some assumptions. A key element in our construction is using the Galerkin approximation and a priori estimates methods to prove the existence and uniqueness of the solutions to the above stochastic equations, which can be regarded as the fractional order extension of the conclusions in \cite{flandoli2021delayed}. We also verify the validation of hypotheses in the time fractional Keller-Segel and time fractional Fisher-KPP equations in 3D case.

math.PR↗

arXiv2022

The Cauchy problem of non-local space-time reaction-diffusion equation involving fractional $p$-Laplacian

For the non-local space-time reaction-diffusion equation involving fractional $p$-Laplacian \begin{equation*} \begin{cases} \frac{\partial^{\alpha }u}{\partial t^{\alpha }}+(-\Delta)_{p}^{s} u=\mu u^{2}(1-kJ*u)-\gamma u,&(x,t)\in\mathbb{R}^{N}\times(0,T)\\ u(x,0)=u_{0}(x),& x\in\mathbb{R}^{N} \end{cases} \end{equation*} $\mu>0 ,k>0,\gamma\geq 1,\alpha\in(0,1),s\in(0,1),1<p$, we consider for $N\leq2$ the problem of finding a global boundedness of the weak solution by virtue of Gagliardo-Nirenberg inequality and fractional Duhamel's formula. Moreover, we prove such weak solution converge to $0$ exponentially or locally uniformly as $t \rightarrow \infty$ for small $\mu$ values with the comparison principle and local Lyapunov type functional. In those cases the problem is reduced to fractional $p$-Laplacian equation in the non-local reaction-diffusion range which is treated with the symmetry and other properties of the kernel of $(-\Delta)_{p}^{s}$. Finally, a key element in our construction is a proof of global bounded weak solution with the fractional nonlinear diffusion terms $(-\Delta)_{p}^{s}u^{m}(2-\frac{2}{N}<m\leq 3,1<p<\frac{4}{3})$ by using Moser iteration and fractional differential inequality.

math.AP↗

arXiv2022

Global boundedness and asymptotic behavior of time-space fractional nonlocal reaction-diffusion equation

The global boundedness and asymptotic behavior are investigate for the solution of time-space fractional non-local reaction-diffusion equation (TSFNRDE) $$ \frac{\partial^{\alpha }u}{\partial t^{\alpha }}=-(-\Delta)^{s} u+\mu u^{2}(1-kJ*u)-\gamma u, \qquad(x,t)\in\mathbb{R}^{N}\times(0,+\infty),$$ where $s\in(0,1),\alpha\in(0,1), N \leq 2$. The operator $\partial_{t}^{\alpha }$ is the Caputo fractional derivative, which $-(-\Delta)^{s}$ is the fractional Laplacian operator. For appropriate assumptions on $J$, it is proved that for homogeneous Dirichlet boundary condition, this problem admits a global bounded weak solution for $N=1$, while for $N=2$, global bounded weak solution exists for large $k$ values by Gagliardo-Nirenberg inequality and fractional differential inequality. With further assumptions on the initial datum, for small $\mu$ values, the solution is shown to converge to $0$ exponentially or locally uniformly as $t \rightarrow \infty$. Furthermore, under the condition of $J \equiv 1$, it is proved that the nonlinear TSFNRDE has a unique weak solution which is global bounded in fractional Sobolev space with the nonlinear fractional diffusion terms $-(-\Delta)^{s} u^{m}\, (2-\frac{2}{N}<m<1)$.

math.AP↗

arXiv2022

Global existence, uniqueness and $L^{\infty}$-bound of weak solutions of fractional time-space Keller-Segel system

This paper studies the properties of weak solutions to a class of space-time fractional parabolic-elliptic Keller-Segel equations with logistic source terms in $\mathbb{R}^{n}$, $n\geq 2$. The global existence and $L^{\infty}$-bound of weak solutions are established. We mainly divide the damping coefficient into two cases: (i) $b>1-\frac{\alpha}{n}$, for any initial value and birth rate; (ii) $0 q$, the $L^{h}$-norms of the weak solution blow up at the same time.

math.AP↗

arXiv2022

Global boundedness and Allee effect for a nonlocal time fractional p-Laplacian reaction-diffusion equation

The global boundedness and asymptotic behavior are investigated for the solutions of a nonlocal time fractional p-Laplacian reaction-diffusion equation (NTFPLRDE) $$ \frac{\partial^{\alpha }u}{\partial t^{\alpha }}=\Delta_{p} u+\mu u^{2}(1-kJ*u) -\gamma u, \qquad(x,t)\in\mathbb{R}^{N}\times(0,+\infty)$$ with $0<\alpha <1,\beta, \mu ,k>0,N\leq 2$ and $\Delta_{p}u =div(\left| \bigtriangledown u \right|^{p-2}\bigtriangledown u)$. Under appropriate assumptions on $J$ and the conditions of $1<p<2$, it is proved that for any nonnegative and bounded initial conditions, the problem has a global bounded classical solution if $k^{*}=0$ for $N=1$ or $k^{*}=(\mu C^{2}_{GN}+1)\eta^{-1}$ for $N=2$, where $C_{GN}$ is the constant in Gagliardo-Nirenberg inequality. With further assumptions on the initial datum, for small $\mu$ values, the solution is shown to converge to $0$ exponentially or locally uniformly as $t \rightarrow \infty$, which is referred as the Allee effect in sense of Caputo derivative. Moreover, under the condition of $J \equiv 1$, it is proved that the nonlinear NTFPLRDE has a global bounded solution in any dimensional space with the nonlinear p-Laplacian diffusion terms $\Delta_{p} u^{m}\, (2-\frac{2}{N}< m\leq 3)$.

math.AP↗

arXiv2021

Global boundedness and Allee effect for a nonlocal time fractional reaction-diffusion equation

The global boundedness and asymptotic behavior are investigated for the solutions of a nonlocal time fractional reaction-diffusion equation (NTFRDE) $$ \frac{\partial^{\alpha }u}{\partial t^{\alpha }}=\Delta u+\mu u^{2}(1-kJ*u)-\gamma u, \qquad(x,t)\in\mathbb{R}^{N}\times(0,+\infty)$$ with $0<\alpha <1,\beta, \mu ,k>0,N\leq 2$ and $u(x,0)=u_{0}(x)$. Under appropriate assumptions on $J$ and the property of time fractional derivative, it is proved that for any nonnegative and bounded initial conditions, the problem has a global bounded classical solution if $k^{*}=0$ for $N=1$ or $k^{*}=(\mu C^{2}_{GN}+1)\eta^{-1}$ for $N=2$, where $C_{GN}$ is the constant in Gagliardo-Nirenberg inequality. With further assumptions on the initial datum, for small $\mu$ values, the solution is shown to converge to $0$ exponentially or locally uniformly as $t \rightarrow \infty$, which is referred as the Allee effect in sense of Caputo derivative. Moreover, under the condition of $J \equiv 1$, it is proved that the nonlinear NTFRDE has a global bounded solution in any dimensional space with the nonlinear diffusion terms $\Delta u^{m}\, (2-\frac{2}{N}< m\leq 3)$.

math.AP↗

arXiv2021

Deep Music Retrieval for Fine-Grained Videos by Exploiting Cross-Modal-Encoded Voice-Overs

Recently, the witness of the rapidly growing popularity of short videos on different Internet platforms has intensified the need for a background music (BGM) retrieval system. However, existing video-music retrieval methods only based on the visual modality cannot show promising performance regarding videos with fine-grained virtual contents. In this paper, we also investigate the widely added voice-overs in short videos and propose a novel framework to retrieve BGM for fine-grained short videos. In our framework, we use the self-attention (SA) and the cross-modal attention (CMA) modules to explore the intra- and the inter-relationships of different modalities respectively. For balancing the modalities, we dynamically assign different weights to the modal features via a fusion gate. For paring the query and the BGM embeddings, we introduce a triplet pseudo-label loss to constrain the semantics of the modal embeddings. As there are no existing virtual-content video-BGM retrieval datasets, we build and release two virtual-content video datasets HoK400 and CFM400. Experimental results show that our method achieves superior performance and outperforms other state-of-the-art methods with large margins.

cs.IR↗

arXiv2019

Unsupervised Multi-stream Highlight detection for the Game "Honor of Kings"

With the increasing popularity of E-sport live, Highlight Flashback has been a critical functionality of live platforms, which aggregates the overall exciting fighting scenes in a few seconds. In this paper, we introduce a novel training strategy without any additional annotation to automatically generate highlights for game video live. Considering that the existing manual edited clips contain more highlights than long game live videos, we perform pair-wise ranking constraints across clips from edited and long live videos. A multi-stream framework is also proposed to fuse spatial, temporal as well as audio features extracted from videos. To evaluate our method, we test on long game live videos with an average length of about 15 minutes. Extensive experimental results on videos demonstrate its satisfying performance on highlights generation and effectiveness by the fusion of three streams.

cs.CV↗

arXiv2018

Prediction of Alzheimer's disease-associated genes by integration of GWAS summary data and expression data

Alzheimer's disease is the most common cause of dementia. It is the fifth-leading cause of death among elderly people. With high genetic heritability (79%), finding disease causal genes is a crucial step in find treatment for AD. Following the International Genomics of Alzheimer's Project (IGAP), many disease-associated genes have been identified; however, we don't have enough knowledge about how those disease-associated genes affect gene expression and disease-related pathways. We integrated GWAS summary data from IGAP and five different expression level data by using TWAS method and identified 15 disease causal genes under strict multiple testing (alpha<0.05), 4 genes are newly identified; identified additional 29 potential disease causal genes under false discovery rate(alpha < 0.05), 21 of them are newly identified. Many genes we identified are also associated with some autoimmune disorder.

stat.AP↗