arXiv subjects

Zhaolin Li

Publications and source records attributed to Zhaolin Li.

16 recordsLinked to original sources

arXiv2026

From Speech to Interaction: Analyzing Multimodal Systems in Cocktail-Party Scenarios

Humans have the remarkable ability to engage in spontaneous informal conversations and selectively attend to individual speakers while filtering out competing speech from nearby conversations. This "cocktail party" scenario still presents severe challenges to speech recognition systems. The CHiME-9 MCoRec task provides a testbed where systems must recognize groups of speakers and transcribe each of their conversations from audio-visual input. In this work, we analyze a diverse set of systems, representing different design directions for addressing the cocktail-party scenario, where the best system achieves up to 57% relative error reduction. We identify three main strategies: (1) explicit or implicit audio-visual target speech separation, (2) improved audio-visual speech recognition for each target speaker, and (3) the use of large language models to group speakers into conversations and enhance conversational consistency. Our analysis shows that these directions address complementary failure modes of the cocktail-party problem, and that high speech overlap alone does not explain performance differences, challenging the common assumption that overlap is the primary source of difficulty in cocktail-party recognition.

cs.CL↗

arXiv2026

Physics-Guided Deep Unfolding for Blind Cross-Sensor Spectral Super-Resolution via Learning the Spectral Transformation Function

Hyperspectral imaging provides rich spectral information for quantitative remote sensing, yet hyperspectral sensors remain costly and thus unavailable in many UAV deployments. Spectral super-resolution (SSR) seeks to reconstruct hyperspectral images (HSIs) from multispectral images (MSIs). Most existing SSR methods assume a fixed and known spectral response function (SRF) and are therefore limited to single-sensor settings. In practical cross-sensor scenarios, the spectral degradation from HSI to MSI is unknown and varies with sensor characteristics and scene content, which renders HSI reconstruction ill-posed. This paper proposes a physics-guided deep unfolding network, termed PGU-Net, to address blind cross-sensor SSR by jointly estimating the HSI and a learnable spectral transformation function (STF). PGU-Net unrolls an alternating optimization procedure into an end-to-end trainable architecture with stages, where each stage sequentially updates the HSI and the STF. Both modules combine learnable proximal networks with differentiable closed-form solvers, enabling physical interpretability while retaining strong representation capacity. Experiments on benchmark datasets (CAVE and NTIRE 2022) with multiple SRFs demonstrate accurate recovery of the STF (degradation operator) and improved reconstruction performance over state-of-the-art SSR methods. Furthermore, evaluations on a real UAV cross-sensor dataset (Headwall Nano HSI and DJI P4 Multispectral MSI) verify the effectiveness and robustness of PGU-Net under truly blind conditions, and suggest that the estimated STF may exhibit land-cover-related differences.

cs.CV↗

arXiv2026

In-context Language Learning for Endangered Languages in Speech Recognition

With approximately 7,000 languages spoken worldwide, current large language models (LLMs) support only a small subset. Prior research indicates LLMs can learn new languages for certain tasks without supervised data. We extend this investigation to speech recognition, investigating whether LLMs can learn unseen, low-resource languages through in-context learning (ICL). With experiments on four diverse endangered languages that LLMs have not been trained on, we find that providing more relevant text samples enhances performance in both language modelling and Automatic Speech Recognition (ASR) tasks. Furthermore, we show that the probability-based approach outperforms the traditional instruction-based approach in language learning. Lastly, we show ICL enables LLMs to achieve ASR performance that is comparable to or even surpasses dedicated language models trained specifically for these languages, while preserving the original capabilities of the LLMs. Our code is publicly available.

cs.CL↗

arXiv2026

Fundamental Lemma for Rank One Spherical Varieties of Classical Types

According to the relative Langlands functoriality conjecture, an admissible morphism between the $L$-groups of spherical varieties should induce a functorial transfer of the corresponding local and global automorphic spectra. Via the relative trace formula approach, two basic problems are the local transfer and the fundamental lemma on the geometric side of the relative trace formulae. In this paper, we consider the rank one spherical variety case, where the admissible morphism between the $L$-groups is the identity morphism, in which case, Y. Sakellaridis has already established the local transfer. We formulate the statement of the fundamental lemma for the general rank one spherical variety case and prove the fundamental lemma for the rank one spherical varieties of classical types.

math.NT↗

arXiv2026

Multimodal In-context Learning for ASR of Low-resource Languages

Automatic speech recognition (ASR) still covers only a small fraction of the world's languages, mainly due to supervised data scarcity. In-context learning (ICL) with large language models (LLMs) addresses this problem, but prior work largely focuses on high-resource languages covered during training and text-only settings. This paper investigates whether speech LLMs can learn unseen languages with multimodal ICL (MICL), and how this learning can be used to improve ASR. We conduct experiments with two speech LLMs, Phi-4 and Qwen3-Omni, on three diverse endangered languages. Firstly, we find that MICL is effective for unseen languages, leveraging both speech and text modalities. We further show that cross-lingual transfer learning improves MICL efficiency on target languages without training on them. Moreover, we analyze attention patterns to interpret MICL mechanisms, and we observe layer-dependent preferences between audio and text context, with an overall bias towards text. Finally, we show that prompt-based ASR with speech LLMs performs poorly on unseen languages, motivating a simple ASR system that combines a stronger acoustic model with a speech LLM via MICL-based selection of acoustic hypotheses. Results show that MICL consistently improves ASR performance, and that cross-lingual transfer learning matches or outperforms corpus-trained language models without using target-language data. Our code is publicly available.

cs.CL↗

arXiv2025

KIT's Low-resource Speech Translation Systems for IWSLT2025: System Enhancement with Synthetic Data and Model Regularization

This paper presents KIT's submissions to the IWSLT 2025 low-resource track. We develop both cascaded systems, consisting of Automatic Speech Recognition (ASR) and Machine Translation (MT) models, and end-to-end (E2E) Speech Translation (ST) systems for three language pairs: Bemba, North Levantine Arabic, and Tunisian Arabic into English. Building upon pre-trained models, we fine-tune our systems with different strategies to utilize resources efficiently. This study further explores system enhancement with synthetic data and model regularization. Specifically, we investigate MT-augmented ST by generating translations from ASR data using MT models. For North Levantine, which lacks parallel ST training data, a system trained solely on synthetic data slightly surpasses the cascaded system trained on real data. We also explore augmentation using text-to-speech models by generating synthetic speech from MT data, demonstrating the benefits of synthetic data in improving both ASR and ST performance for Bemba. Additionally, we apply intra-distillation to enhance model performance. Our experiments show that this approach consistently improves results across ASR, MT, and ST tasks, as well as across different pre-trained models. Finally, we apply Minimum Bayes Risk decoding to combine the cascaded and end-to-end systems, achieving an improvement of approximately 1.5 BLEU points.

cs.CL↗

arXiv2025

End-to-End Evaluation for Low-Latency Simultaneous Speech Translation

The challenge of low-latency speech translation has recently draw significant interest in the research community as shown by several publications and shared tasks. Therefore, it is essential to evaluate these different approaches in realistic scenarios. However, currently only specific aspects of the systems are evaluated and often it is not possible to compare different approaches. In this work, we propose the first framework to perform and evaluate the various aspects of low-latency speech translation under realistic conditions. The evaluation is carried out in an end-to-end fashion. This includes the segmentation of the audio as well as the run-time of the different components. Secondly, we compare different approaches to low-latency speech translation using this framework. We evaluate models with the option to revise the output as well as methods with fixed output. Furthermore, we directly compare state-of-the-art cascaded as well as end-to-end systems. Finally, the framework allows to automatically evaluate the translation quality as well as latency and also provides a web interface to show the low-latency model outputs to the user.

cs.CL↗

arXiv2025

Improved Semi-Parametric Bounds for Tail Probability and Expected Loss: Theory and Applications

Many management decisions involve accumulated random realizations for which only the first and second moments of their distribution are available. The sharp Chebyshev-type bound for the tail probability and Scarf bound for the expected loss are widely used in this setting. We revisit the tail behavior of such quantities with a focus on independence. Conventional primal-dual approaches from optimization are ineffective in this setting. Instead, we use probabilistic inequalities to derive new bounds and offer new insights. For non-identical distributions attaining the tail probability bounds, we show that the extreme values are equidistant regardless of the distributional differences. For the bound on the expected loss, we show that the impact of each random variable on the expected sum can be isolated using an extension of the Korkine identity. We illustrate how these new results open up abundant practical applications, including improved pricing of product bundles, more precise option pricing, more efficient insurance design, and better inventory management. For example, we establish a new solution to the optimal bundling problem, yielding a 17% uplift in per-bundle profits, and a new solution to the inventory problem, yielding a 5.6% cost reduction for a model with 20 retailers.

econ.EM↗

arXiv2025

Beyond Endoscopy: Poisson Summation Formula and Kuznetsov Trace Formula on $\mathrm{GL}_2$

In the first part of this paper, we present a direct proof of a Poisson summation formula on the Whittaker space of $\mathrm{GL}_2$, which underlies the local Hankel transform computed by H. Jacquet. In the second part, we derive a Kuznetsov-type trace formula for incorporating non-standard test functions. This formulation reveals that the Poisson summation formula naturally yields the functional equation for the standard $L$-functions of $\mathrm{GL}_2$.

math.NT↗

arXiv2024

On the Braverman-Kazhdan-Ngo Triples

In the Braverman-Kazhdan proposal and certain refinement of Ngo for automorphic $L$-functions, the reductive group $G$ and the representations $ρ$ of the Langlands dual group $G^\vee$ are taken with certain assumptions. We introduce the notion of the Braverman-Kazhdan-Ngo triples $(G,G^\vee,ρ)$ and show that for general automorphic $L$-functions, it is enough to consider the Braverman-Kazhdan-Ngo triples. We also verify that for a given Braverman-Kazhdan-Ngo triple, the reductive monoid constructed from the Vinberg method and that constructed from the Putcha-Renner method are isomorphic.

math.NT↗

arXiv2024

SciEx: Benchmarking Large Language Models on Scientific Exams with Human Expert Grading and Automatic Grading

With the rapid development of Large Language Models (LLMs), it is crucial to have benchmarks which can evaluate the ability of LLMs on different domains. One common use of LLMs is performing tasks on scientific topics, such as writing algorithms, querying databases or giving mathematical proofs. Inspired by the way university students are evaluated on such tasks, in this paper, we propose SciEx - a benchmark consisting of university computer science exam questions, to evaluate LLMs ability on solving scientific tasks. SciEx is (1) multilingual, containing both English and German exams, and (2) multi-modal, containing questions that involve images, and (3) contains various types of freeform questions with different difficulty levels, due to the nature of university exams. We evaluate the performance of various state-of-the-art LLMs on our new benchmark. Since SciEx questions are freeform, it is not straightforward to evaluate LLM performance. Therefore, we provide human expert grading of the LLM outputs on SciEx. We show that the free-form exams in SciEx remain challenging for the current LLMs, where the best LLM only achieves 59.4\% exam grade on average. We also provide detailed comparisons between LLM performance and student performance on SciEx. To enable future evaluation of new LLMs, we propose using LLM-as-a-judge to grade the LLM answers on SciEx. Our experiments show that, although they do not perform perfectly on solving the exams, LLMs are decent as graders, achieving 0.948 Pearson correlation with expert grading.

cs.CL↗

arXiv2024

Augmenting Automatic Speech Recognition Models with Disfluency Detection

Speech disfluency commonly occurs in conversational and spontaneous speech. However, standard Automatic Speech Recognition (ASR) models struggle to accurately recognize these disfluencies because they are typically trained on fluent transcripts. Current research mainly focuses on detecting disfluencies within transcripts, overlooking their exact location and duration in the speech. Additionally, previous work often requires model fine-tuning and addresses limited types of disfluencies. In this work, we present an inference-only approach to augment any ASR model with the ability to detect open-set disfluencies. We first demonstrate that ASR models have difficulty transcribing speech disfluencies. Next, this work proposes a modified Connectionist Temporal Classification(CTC)-based forced alignment algorithm from \cite{kurzinger2020ctc} to predict word-level timestamps while effectively capturing disfluent speech. Additionally, we develop a model to classify alignment gaps between timestamps as either containing disfluent speech or silence. This model achieves an accuracy of 81.62% and an F1-score of 80.07%. We test the augmentation pipeline of alignment gap detection and classification on a disfluent dataset. Our results show that we captured 74.13% of the words that were initially missed by the transcription, demonstrating the potential of this pipeline for downstream tasks.

cs.CL↗

arXiv2024

Blending LLMs into Cascaded Speech Translation: KIT's Offline Speech Translation System for IWSLT 2024

Large Language Models (LLMs) are currently under exploration for various tasks, including Automatic Speech Recognition (ASR), Machine Translation (MT), and even End-to-End Speech Translation (ST). In this paper, we present KIT's offline submission in the constrained + LLM track by incorporating recently proposed techniques that can be added to any cascaded speech translation. Specifically, we integrate Mistral-7B\footnote{mistralai/Mistral-7B-Instruct-v0.1} into our system to enhance it in two ways. Firstly, we refine the ASR outputs by utilizing the N-best lists generated by our system and fine-tuning the LLM to predict the transcript accurately. Secondly, we refine the MT outputs at the document level by fine-tuning the LLM, leveraging both ASR and MT predictions to improve translation quality. We find that integrating the LLM into the ASR and MT systems results in an absolute improvement of $0.3\%$ in Word Error Rate and $0.65\%$ in COMET for tst2019 test set. In challenging test sets with overlapping speakers and background noise, we find that integrating LLM is not beneficial due to poor ASR performance. Here, we use ASR with chunked long-form decoding to improve context usage that may be unavailable when transcribing with Voice Activity Detection segmentation alone.

cs.CL↗

arXiv2024

The Voronoi Summation Formula for $\mathrm{GL}_n$ and the Godement-Jacquet Kernels

Let $\mathbb{A}$ be the ring of adeles of a number field $k$ and $π$ be an irreducible cuspidal automorphic representation of $\mathrm{GL}_n(\mathbb{A})$. In the previous work of the first author with Zhilin Luo, they introduced $π$-Schwartz space $\mathcal{S}_π(\mathbb{A}^\times)$ and $π$-Fourier transform $\mathcal{F}_{π,ψ}$ with a non-trivial additive character $ψ$ of $k\backslash\mathbb{A}$, proved the associated Poisson summation formula over $\mathbb{A}^\times$, based on the Godement-Jacquet theory for the standard $L$-functions $L(s,π)$, and provided interesting applications. In this paper, in addition to the further development of the local theory, we found two global applications. First, we find a Poisson summation formula proof of the Voronoi summation formula for $\mathrm{GL}_n$ over a number field, which was first proved by A. Ichino and N. Templier. Then we introduce the notion of the Godement-Jacquet kernels $H_{π,s}$ and their dual kernels $K_{π,s}$ for any irreducible cuspidal automorphic representation $π$ of $\mathrm{GL}_n(\mathbb{A})$ and show that $H_{π,s}$ and $K_{π,1-s}$ are related by the nonlinear $π_\infty$-Fourier transform if and only if $s\in\mathbb{C}$ is a zero of $L_f(s,π_f)=0$, the finite part of the standard automorphic $L$-function $L(s,π)$, which are the $(\mathrm{GL}_n,π)$-versions of a Clozel's Theorem, where the Tate kernel with $n=1$ and $π$ the trivial character are considered.

math.NT↗

arXiv2023

The Uniqueness of the Ginzburg-Rallis Model: the Non-Archimedean Case

We prove the uniqueness of the Ginzburg-Rallis models over $p$-adic local fields of characteristic zero, which completes the local uniqueness problem for the Ginzburg-Rallis models starting from the work of C.-F. Nien in \cite{MR2709083} that proves the non-split case, and the work of D. Jiang, B. Sun and C. Zhu in \cite{MR2763736} that proves the general case over Archimedean local fields. Our proof extends the strategy of \cite{MR2763736} to the $p$-adic case with the help of the refined structure of the wavefront sets of $\mathfrak {z}$-finite distributions as developed by A. Aizenbud, D. Gourevitch and E. Sayag in \cite{MR3406530}.

math.RT↗

arXiv2022

Efficient Speech Translation with Pre-trained Models

When building state-of-the-art speech translation models, the need for large computational resources is a significant obstacle due to the large training data size and complex models. The availability of pre-trained models is a promising opportunity to build strong speech translation systems efficiently. In a first step, we investigate efficient strategies to build cascaded and end-to-end speech translation systems based on pre-trained models. Using this strategy, we can train and apply the models on a single GPU. While the end-to-end models show superior translation performance to cascaded ones, the application of this technology has a limitation on the need for additional end-to-end training data. In a second step, we proposed an additional similarity loss to encourage the model to generate similar hidden representations for speech and transcript. Using this technique, we can increase the data efficiency and improve the translation quality by 6 BLEU points in scenarios with limited end-to-end training data.

cs.CL↗