arXiv subjects

Zhuo Chen

Publications and source records attributed to Zhuo Chen.

At least 289 records · Page 16Linked to original sources

arXiv2023

Universal Multi-modal Entity Alignment via Iteratively Fusing Modality Similarity Paths

The objective of Entity Alignment (EA) is to identify equivalent entity pairs from multiple Knowledge Graphs (KGs) and create a more comprehensive and unified KG. The majority of EA methods have primarily focused on the structural modality of KGs, lacking exploration of multi-modal information. A few multi-modal EA methods have made good attempts in this field. Still, they have two shortcomings: (1) inconsistent and inefficient modality modeling that designs complex and distinct models for each modality; (2) ineffective modality fusion due to the heterogeneous nature of modalities in EA. To tackle these challenges, we propose PathFusion, consisting of two main components: (1) MSP, a unified modeling approach that simplifies the alignment process by constructing paths connecting entities and modality nodes to represent multiple modalities; (2) IRF, an iterative fusion method that effectively combines information from different modalities using the path as an information carrier. Experimental results on real-world datasets demonstrate the superiority of PathFusion over state-of-the-art methods, with 22.4%-28.9% absolute improvement on Hits@1, and 0.194-0.245 absolute improvement on MRR.

cs.CL↗

arXiv2023

The second multi-channel multi-party meeting transcription challenge (M2MeT) 2.0): A benchmark for speaker-attributed ASR

With the success of the first Multi-channel Multi-party Meeting Transcription challenge (M2MeT), the second M2MeT challenge (M2MeT 2.0) held in ASRU2023 particularly aims to tackle the complex task of \emph{speaker-attributed ASR (SA-ASR)}, which directly addresses the practical and challenging problem of ``who spoke what at when" at typical meeting scenario. We particularly established two sub-tracks. The fixed training condition sub-track, where the training data is constrained to predetermined datasets, but participants can use any open-source pre-trained model. The open training condition sub-track, which allows for the use of all available data and models without limitation. In addition, we release a new 10-hour test set for challenge ranking. This paper provides an overview of the dataset, track settings, results, and analysis of submitted systems, as a benchmark to show the current state of speaker-attributed ASR.

cs.SD↗

arXiv2023

On decoder-only architecture for speech-to-text and large language model integration

Large language models (LLMs) have achieved remarkable success in the field of natural language processing, enabling better human-computer interaction using natural language. However, the seamless integration of speech signals into LLMs has not been explored well. The "decoder-only" architecture has also not been well studied for speech processing tasks. In this research, we introduce Speech-LLaMA, a novel approach that effectively incorporates acoustic information into text-based large language models. Our method leverages Connectionist Temporal Classification and a simple audio encoder to map the compressed acoustic features to the continuous semantic space of the LLM. In addition, we further probe the decoder-only architecture for speech-to-text tasks by training a smaller scale randomly initialized speech-LLaMA model from speech-text paired data alone. We conduct experiments on multilingual speech-to-text translation tasks and demonstrate a significant improvement over strong baselines, highlighting the potential advantages of decoder-only models for speech-to-text conversion.

eess.AS↗

arXiv2023

t-SOT FNT: Streaming Multi-talker ASR with Text-only Domain Adaptation Capability

Token-level serialized output training (t-SOT) was recently proposed to address the challenge of streaming multi-talker automatic speech recognition (ASR). T-SOT effectively handles overlapped speech by representing multi-talker transcriptions as a single token stream with $\langle \text{cc}\rangle$ symbols interspersed. However, the use of a naive neural transducer architecture significantly constrained its applicability for text-only adaptation. To overcome this limitation, we propose a novel t-SOT model structure that incorporates the idea of factorized neural transducers (FNT). The proposed method separates a language model (LM) from the transducer's predictor and handles the unnatural token order resulting from the use of $\langle \text{cc}\rangle$ symbols in t-SOT. We achieve this by maintaining multiple hidden states and introducing special handling of the $\langle \text{cc}\rangle$ tokens within the LM. The proposed t-SOT FNT model achieves comparable performance to the original t-SOT model while retaining the ability to reduce word error rate (WER) on both single and multi-talker datasets through text-only adaptation.

eess.AS↗

arXiv2023

Knowledge-informed Molecular Learning: A Survey on Paradigm Transfer

Machine learning, notably deep learning, has significantly propelled molecular investigations within the biochemical sphere. Traditionally, modeling for such research has centered around a handful of paradigms. For instance, the prediction paradigm is frequently deployed for tasks such as molecular property prediction. To enhance the generation and decipherability of purely data-driven models, scholars have integrated biochemical domain knowledge into these molecular study models. This integration has sparked a surge in paradigm transfer, which is solving one molecular learning task by reformulating it as another one. With the emergence of Large Language Models, these paradigms have demonstrated an escalating trend towards harmonized unification. In this work, we delineate a literature survey focused on knowledge-informed molecular learning from the perspective of paradigm transfer. We classify the paradigms, scrutinize their methodologies, and dissect the contribution of domain knowledge. Moreover, we encapsulate prevailing trends and identify intriguing avenues for future exploration in molecular learning.

cs.LG↗

arXiv2023

Drinfeld Modular Curves Subordinate to Conjugacy Classes of Nilpotent Upper-Triangular Matrices

We introduce normalized Drinfeld modular curves that parameterize rank $m$ Drinfeld modules compatible with a $T$-torsion structure arising from a given conjugacy class of nilpotent upper-triangular $n\times n$ matrices with rank $\geqslant n-m$ over a finite field $\mathbb{F}_q$. This creates a deep link connecting the classification of nilpotent upper-triangular matrices and the decomposition of Drinfeld modular curves. The conjugacy classes of nilpotent upper-triangular matrices one-to-one corresponds to certain $T$-torsion flags, and form a tree structure. As a result, the associated Drinfeld modular curves are organized in the same tree. This generalizes the tower structure introduced by Bassa, Beelen, Garcia, Stichtenoth, and others. Additionally,we prove the geometric irreducibility of $(3,2)$-type normalized Drinfeld modular curves, and characterize their associated function fields.

math.NT↗

arXiv2023

MACO: A Modality Adversarial and Contrastive Framework for Modality-missing Multi-modal Knowledge Graph Completion

Recent years have seen significant advancements in multi-modal knowledge graph completion (MMKGC). MMKGC enhances knowledge graph completion (KGC) by integrating multi-modal entity information, thereby facilitating the discovery of unobserved triples in the large-scale knowledge graphs (KGs). Nevertheless, existing methods emphasize the design of elegant KGC models to facilitate modality interaction, neglecting the real-life problem of missing modalities in KGs. The missing modality information impedes modal interaction, consequently undermining the model's performance. In this paper, we propose a modality adversarial and contrastive framework (MACO) to solve the modality-missing problem in MMKGC. MACO trains a generator and discriminator adversarially to generate missing modality features that can be incorporated into the MMKGC model. Meanwhile, we design a cross-modal contrastive loss to improve the performance of the generator. Experiments on public benchmarks with further explorations demonstrate that MACO could achieve state-of-the-art results and serve as a versatile framework to bolster various MMKGC models. Our code and benchmark data are available at https://github.com/zjukg/MACO.

cs.CL↗

arXiv2023

Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment

As a crucial extension of entity alignment (EA), multi-modal entity alignment (MMEA) aims to identify identical entities across disparate knowledge graphs (KGs) by exploiting associated visual information. However, existing MMEA approaches primarily concentrate on the fusion paradigm of multi-modal entity features, while neglecting the challenges presented by the pervasive phenomenon of missing and intrinsic ambiguity of visual images. In this paper, we present a further analysis of visual modality incompleteness, benchmarking latest MMEA models on our proposed dataset MMEA-UMVM, where the types of alignment KGs covering bilingual and monolingual, with standard (non-iterative) and iterative training paradigms to evaluate the model performance. Our research indicates that, in the face of modality incompleteness, models succumb to overfitting the modality noise, and exhibit performance oscillations or declines at high rates of missing modality. This proves that the inclusion of additional multi-modal data can sometimes adversely affect EA. To address these challenges, we introduce UMAEA , a robust multi-modal entity alignment approach designed to tackle uncertainly missing and ambiguous visual modalities. It consistently achieves SOTA performance across all 97 benchmark splits, significantly surpassing existing baselines with limited parameters and time consumption, while effectively alleviating the identified limitations of other models. Our code and benchmark data are available at https://github.com/zjukg/UMAEA.

cs.AI↗

arXiv2023

ELFNet: Evidential Local-global Fusion for Stereo Matching

Although existing stereo matching models have achieved continuous improvement, they often face issues related to trustworthiness due to the absence of uncertainty estimation. Additionally, effectively leveraging multi-scale and multi-view knowledge of stereo pairs remains unexplored. In this paper, we introduce the \textbf{E}vidential \textbf{L}ocal-global \textbf{F}usion (ELF) framework for stereo matching, which endows both uncertainty estimation and confidence-aware fusion with trustworthy heads. Instead of predicting the disparity map alone, our model estimates an evidential-based disparity considering both aleatoric and epistemic uncertainties. With the normal inverse-Gamma distribution as a bridge, the proposed framework realizes intra evidential fusion of multi-level predictions and inter evidential fusion between cost-volume-based and transformer-based stereo matching. Extensive experimental results show that the proposed framework exploits multi-view information effectively and achieves state-of-the-art overall performance both on accuracy and cross-domain generalization. The codes are available at https://github.com/jimmy19991222/ELFNet.

cs.CV↗

arXiv2023

MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid

Multi-modal entity alignment (MMEA) aims to discover identical entities across different knowledge graphs (KGs) whose entities are associated with relevant images. However, current MMEA algorithms rely on KG-level modality fusion strategies for multi-modal entity representation, which ignores the variations of modality preferences of different entities, thus compromising robustness against noise in modalities such as blurry images and relations. This paper introduces MEAformer, a multi-modal entity alignment transformer approach for meta modality hybrid, which dynamically predicts the mutual correlation coefficients among modalities for more fine-grained entity-level modality fusion and alignment. Experimental results demonstrate that our model not only achieves SOTA performance in multiple training scenarios, including supervised, unsupervised, iterative, and low-resource settings, but also has a limited number of parameters, efficient runtime, and interpretability. Our code is available at https://github.com/zjukg/MEAformer.

cs.AI↗

arXiv2023

Target-oriented Sentiment Classification with Sequential Cross-modal Semantic Graph

Multi-modal aspect-based sentiment classification (MABSC) is task of classifying the sentiment of a target entity mentioned in a sentence and an image. However, previous methods failed to account for the fine-grained semantic association between the image and the text, which resulted in limited identification of fine-grained image aspects and opinions. To address these limitations, in this paper we propose a new approach called SeqCSG, which enhances the encoder-decoder sentiment classification framework using sequential cross-modal semantic graphs. SeqCSG utilizes image captions and scene graphs to extract both global and local fine-grained image information and considers them as elements of the cross-modal semantic graph along with tokens from tweets. The sequential cross-modal semantic graph is represented as a sequence with a multi-modal adjacency matrix indicating relationships between elements. Experimental results show that the approach outperforms existing methods and achieves state-of-the-art performance on two standard datasets. Further analysis has demonstrated that the model can implicitly learn the correlation between fine-grained information of the image and the text with the given target. Our code is available at https://github.com/zjukg/SeqCSG.

cs.CV↗

arXiv2023

The weak Lie 2-algebra of multiplicative forms on a quasi-Poisson groupoid

Berwick-Evens and Lerman recently showed that the category of vector fields on a geometric stack has the structure of a Lie $2$-algebra. Motivated by this work, we present a construction of graded weak Lie $2$-algebras associated with quasi-Poisson groupoids based on the space of multiplicative forms on the groupoid and differential forms on the base manifold. We also establish a morphism between the Lie $2$-algebra of multiplicative multivector fields and the weak Lie $2$-algebra of multiplicative forms, allowing us to compare and relate different aspects of Lie $2$-algebra theory within the context of quasi-Poisson geometry. As an infinitesimal analogy, we explicitly determine the associated weak Lie $2$-algebra structure of IM $1$-forms along with differential $1$-forms on the base manifold for any quasi-Lie bialgebroid.

math.DG↗

arXiv2023

AD-AutoGPT: An Autonomous GPT for Alzheimer's Disease Infodemiology

In this pioneering study, inspired by AutoGPT, the state-of-the-art open-source application based on the GPT-4 large language model, we develop a novel tool called AD-AutoGPT which can conduct data collection, processing, and analysis about complex health narratives of Alzheimer's Disease in an autonomous manner via users' textual prompts. We collated comprehensive data from a variety of news sources, including the Alzheimer's Association, BBC, Mayo Clinic, and the National Institute on Aging since June 2022, leading to the autonomous execution of robust trend analyses, intertopic distance maps visualization, and identification of salient terms pertinent to Alzheimer's Disease. This approach has yielded not only a quantifiable metric of relevant discourse but also valuable insights into public focus on Alzheimer's Disease. This application of AD-AutoGPT in public health signifies the transformative potential of AI in facilitating a data-rich understanding of complex health narratives like Alzheimer's Disease in an autonomous manner, setting the groundwork for future AI-driven investigations in global health landscapes.

cs.CL↗

arXiv2023

Handling Trade-Offs in Speech Separation with Sparsely-Gated Mixture of Experts

Employing a monaural speech separation (SS) model as a front-end for automatic speech recognition (ASR) involves balancing two kinds of trade-offs. First, while a larger model improves the SS performance, it also requires a higher computational cost. Second, an SS model that is more optimized for handling overlapped speech is likely to introduce more processing artifacts in non-overlapped-speech regions. In this paper, we address these trade-offs with a sparsely-gated mixture-of-experts (MoE) architecture. Comprehensive evaluation results obtained using both simulated and real meeting recordings show that our proposed sparsely-gated MoE SS model achieves superior separation capabilities with less speech distortion, while involving only a marginal run-time cost increase.

eess.AS↗

arXiv2023

On the reduced space of multiplicative multivectors

A strict Lie $2$-algebra $Γ(\wedge^\bullet A) \stackrel{T}{\rightarrow} \mathfrak{X}_{\mathrm{mult}}^\bullet(\mathcal{G})$ is associated with any Lie groupoid $\mathcal{G}$. Here, $Γ(\wedge^\bullet A)$ is the Schouten algebra of the tangent Lie algebroid $A$ of $\mathcal{G}$ and $\mathfrak{X}_{\mathrm{mult}}^\bullet(\mathcal{G})$ is the space of multiplicative multivectors on $\mathcal{G}$. The quotient ${R}_{\mathrm{mult}}^\bullet:=\mathfrak{X}_{\mathrm{mult}}^\bullet(\mathcal{G})/\mathrm{Img} T$, a Morita invariant of $\mathcal{G}$, is called the reduced space of multiplicative multivectors. We prove a canonical decomposition formula of elements in $\mathfrak{X}_{\mathrm{mult}}^\bullet(\mathcal{G})$ and establish a key relation between ${R}_{\mathrm{mult}}^k$ and the cohomology $\mathrm{H} ^1(\mathfrak{J} \mathcal{G},\wedge^k A)$ where $\mathfrak{J} \mathcal{G}$ is the jet groupoid of $\mathcal{G}$ and $1\leqslant k\leqslant \mathrm{rank} A$. We also study ${R}_{\mathrm{diff}}^\bullet $, the reduced space of Lie algebroid differentials on $A$. By taking infinitesimals, $\barδ: $ ${R}_{\mathrm{mult}}^\bullet $ $\to $ ${R}_{\mathrm{diff}}^\bullet $, the two reduced spaces are related. We find that the kernel of $\barδ$ is isomorphic to the kernel of the Van Est map $\mathrm{H}^1(\mathcal{G},\wedge^k \kerρ)\to \mathrm{H}^1(A,\wedge^k \kerρ)$, where $ρ$ is the anchor of $A$.

math.AG↗

arXiv2023

Adapting Multi-Lingual ASR Models for Handling Multiple Talkers

State-of-the-art large-scale universal speech models (USMs) show a decent automatic speech recognition (ASR) performance across multiple domains and languages. However, it remains a challenge for these models to recognize overlapped speech, which is often seen in meeting conversations. We propose an approach to adapt USMs for multi-talker ASR. We first develop an enhanced version of serialized output training to jointly perform multi-talker ASR and utterance timestamp prediction. That is, we predict the ASR hypotheses for all speakers, count the speakers, and estimate the utterance timestamps at the same time. We further introduce a lightweight adapter module to maintain the multilingual property of the USMs even when we perform the adaptation with only a single language. Experimental results obtained using the AMI and AliMeeting corpora show that our proposed approach effectively transfers the USMs to a strong multilingual multi-talker ASR model with timestamp prediction capability.

eess.AS↗

arXiv2023

VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation

Recent research shows a big convergence in model architecture, training objectives, and inference methods across various tasks for different modalities. In this paper, we propose VioLA, a single auto-regressive Transformer decoder-only network that unifies various cross-modal tasks involving speech and text, such as speech-to-text, text-to-text, text-to-speech, and speech-to-speech tasks, as a conditional codec language model task via multi-task learning framework. To accomplish this, we first convert all the speech utterances to discrete tokens (similar to the textual data) using an offline neural codec encoder. In such a way, all these tasks are converted to token-based sequence conversion problems, which can be naturally handled with one conditional language model. We further integrate task IDs (TID) and language IDs (LID) into the proposed model to enhance the modeling capability of handling different languages and tasks. Experimental results demonstrate that the proposed VioLA model can support both single-modal and cross-modal tasks well, and the decoder-only model achieves a comparable and even better performance than the strong baselines.

cs.CL↗

arXiv2023

Multiplicative forms on Poisson groupoids

Given a Lie groupoid $\mathcal{G}$ over $M$, $A$ the tangent Lie algebroid of $\mathcal{G}$, and $ρ: A\rightarrow TM$ the anchor map, we provide a formula that decomposes an arbitrary multiplicative $k$-form $Θ$ on $\mathcal{G}$ into two parts. The first part is $e$, a $1$-cocycle of $\mathfrak{J}\mathcal{G}$ valued in $\wedge^k T^*M$, and the second part is $θ\in Γ(A^*\otimes (\wedge^{k-1} T^*M))$ which is $ρ$-compatible, meaning that $ι_{ρ(u)}θ(u)=0$ for all $u\in A$. We call this pair of data $(e,θ)$ the $(0,k)$-characteristic pair of $Θ$. Next, we prove that if $\mathcal{G}$ is a Poisson Lie groupoid, then the space $Ω^{\bullet}_{\mathrm{mult}}(\mathcal{G})$ of multiplicative forms on $\mathcal{G}$ has a differential graded Lie algebra (DGLA) structure. Furthermore, when combined with $Ω^\bullet(M)$, which is the space of forms on the base manifold $M$, $Ω^{\bullet}_{\mathrm{mult}}(\mathcal{G})$ forms a canonical DGLA crossed module. This supplements a previously known fact that multiplicative multivector fields on $\mathcal{G}$ form a DGLA crossed module with the Schouten algebra $Γ(\wedge^\bullet A)$ stemming from the tangent Lie algebroid $A$.

math.DG↗