Searcharxiv⌕ Search

arXiv subjects

Weixin Zhu

Publications and source records attributed to Weixin Zhu.

8 recordsLinked to original sources

Structured Speaker Conditioning for Diarization-Guided Target-Speaker ASR

We propose structured speaker conditioning (SSC) for diarization-guided target-speaker recognition. SSC combines local activity structure with session-level acoustic identity to construct identity-consistent target conditions that control the acoustic encoder. Target-centric relational encoding (TCRE) retains per-speaker activity histories and factorized second- and third-order relations, capturing overlap and turn changes with aggregation cost linear in the number of tracks. Reference-based speaker conditioning (RSC) supplies session-derived identities that disambiguate local activity patterns within TCRE. Conditional low-rank modulation (CLRM) converts the joint condition into channel-wise affine modulation and a low-rank cross-channel residual with a condition-dependent core. On Qwen3-ASR-0.6B with oracle diarization, SSC uses 1.46M conditioning parameters and achieves a 6.23% mean relative reduction in concatenated minimum-permutation error rate (cpER) over four-state diagonal conditioning across four meeting test sets. SSC also transfers across audio foundation models. With predicted diarization and target-error group sequence policy optimization, the FireRedAudio text system achieves mean relative error reductions of 17.9% and 17.3% over DiCoW and MOSS-TD on the same four sets, and also outperforms both on three unseen domains. Scenario analyses identify gains under challenging overlap conditions, while fixed-checkpoint interventions support the contributions of session identity and the condition-dependent core. A timestamp variant jointly produces sentence boundaries and text. With predicted diarization, macro-average cpER across the four sets is 15.60% for text and 15.96% with timestamps.

eess.AS↗

DM-ASR: Diarization-aware Multi-speaker ASR with Unified Offline and Online Decoding with Large Language Models

Multi-speaker automatic speech recognition (ASR) aims to recognize not only what was said, but also who said it and when. Recent Speech-LLMs have shown promising results in structured multi-speaker transcription, but jointly modeling speaker attribution, temporal information, and lexical content remains challenging and often requires substantial model capacity and training data. To address this challenge, we propose DM-ASR, a Diarization-aware Multi-speaker ASR framework that reformulates multi-speaker recognition as diarization-conditioned multi-turn dialogue generation. By leveraging diarization-derived speaker-time cues as explicit structural priors, DM-ASR decomposes transcription into a sequence of conditioned queries while preserving mixed acoustic information and conversational context. We further introduce optional word-level timestamps, fine-tune the model with Group Sequence Policy Optimization (GSPO), and analyze the complementary roles of diarization cues and language-model predictions. The proposed formulation also enables unified offline and causal sentence-level online decoding through flexible acoustic context and dialogue history management. Experiments on Mandarin and English benchmarks demonstrate competitive structured transcription performance with relatively compact models and training data, while effectively supporting both offline and online recognition.

eess.AS↗

Sequence-to-Sequence Refinement for EEND-VC Diarization with a Shared Encoder

End-to-end neural diarization with vector clustering (EEND-VC) separates local segmentation from global speaker assignment, leaving speaker confusion (SC) despite strong segmentation. We propose shared-encoder sequence-to-sequence (S2S) refinement, which reuses DiariZen's self-supervised learning (SSL) representations for speaker-conditioned decoding and prior-constrained assignment. We study frozen-feature refinement with WavLM and structurally pruned w2v-BERT 2.0, and extend the latter to joint training of local segmentation and refinement heads. The heads learn separate layer mixtures while their losses update one acoustic encoder. Across eight domains, frozen-feature refinement reduces macro diarization error rate (DER) from 12.95% to 12.40% with WavLM and from 12.35% to 12.08% with w2v-BERT 2.0, with relative SC reductions of 17.4% and 14.6%. It adds approximately 15% to system real-time factor. The joint system further lowers macro DER to 11.90%, improving seven of eight domains over non-joint refinement. Error decomposition separates SC-driven refinement gains from detection-driven joint-system gains.

eess.AS↗

S2SND-v2: Unified Online and Offline Speaker Diarization with Speaker-Adapted w2v-BERT 2.0

This paper presents S2SND-v2, a unified online and offline speaker diarization framework built on S2SND that improves acoustic feature extraction and the use of the resulting representations. For acoustic feature extraction, we introduce a speaker-adapted w2v-BERT 2.0 front end and compare full and pruned construction routes. To better use these features, overlap-aware activity-query masking improves speaker embedding extraction under overlap, while late-speaker training aligns unknown-speaker training with online inference conditions. Combining these improvements, the pruned system achieves competitive average diarization error rates of 14.52% online and 12.03% offline across eight test sets.

cs.SD↗

Intelligent Perception Assisted Light Modulation for real-time and program-free nanofabrication and particle manipulation

In this paper, we present an Intelligent Perception-Assisted Light Modulation (IPALM) technique, for femtosecond laser nanofabrication and particle manipulation. IPALM technique integrates real-time hand-motion recognition with dynamic spatial light modulation to achieve programming-free laser beam control. In contrast to the conventional programmed laser fabrication techniques which need setup a project prior to fabrication, IPALM offers a direct "mind-to-matter" pathway for laser nanostructuring and biological cells handling with high flexibility and multiple degree of freedom. In nanofabrication, IPALM provides high resolution with feature dimensions down to 280 nm. By mind-driven hand gesture control, the laser beam is regulated to enable direct writing of micro/nanostructures with a minimum feature size down to 280 nm via IPALM. In precise particle manipulation, multiple cells can be simultaneously moved to achieve cell coalescence. With these examples, IPALM showcases its potential for applications in photonics, biomedicine, and microfluidics, for high-dimension and flexible laser applications.

physics.optics↗

TEA-PSE 3.0: Tencent-Ethereal-Audio-Lab Personalized Speech Enhancement System For ICASSP 2023 DNS Challenge

This paper introduces the Unbeatable Team's submission to the ICASSP 2023 Deep Noise Suppression (DNS) Challenge. We expand our previous work, TEA-PSE, to its upgraded version -- TEA-PSE 3.0. Specifically, TEA-PSE 3.0 incorporates a residual LSTM after squeezed temporal convolution network (S-TCN) to enhance sequence modeling capabilities. Additionally, the local-global representation (LGR) structure is introduced to boost speaker information extraction, and multi-STFT resolution loss is used to effectively capture the time-frequency characteristics of the speech signals. Moreover, retraining methods are employed based on the freeze training strategy to fine-tune the system. According to the official results, TEA-PSE 3.0 ranks 1st in both ICASSP 2023 DNS-Challenge track 1 and track 2.

eess.AS↗

spatial-dccrn: dccrn equipped with frame-level angle feature and hybrid filtering for multi-channel speech enhancement

Recently, multi-channel speech enhancement has drawn much interest due to the use of spatial information to distinguish target speech from interfering signal. To make full use of spatial information and neural network based masking estimation, we propose a multi-channel denoising neural network -- Spatial DCCRN. Firstly, we extend S-DCCRN to multi-channel scenario, aiming at performing cascaded sub-channel and full-channel processing strategy, which can model different channels separately. Moreover, instead of only adopting multi-channel spectrum or concatenating first-channel's magnitude and IPD as the model's inputs, we apply an angle feature extraction module (AFE) to extract frame-level angle feature embeddings, which can help the model to apparently perceive spatial information. Finally, since the phenomenon of residual noise will be more serious when the noise and speech exist in the same time frequency (TF) bin, we particularly design a masking and mapping filtering method to substitute the traditional filter-and-sum operation, with the purpose of cascading coarsely denoising, dereverberation and residual noise suppression. The proposed model, Spatial-DCCRN, has surpassed EaBNet, FasNet as well as several competitive models on the L3DAS22 Challenge dataset. Not only the 3D scenario, Spatial-DCCRN outperforms state-of-the-art (SOTA) model MIMO-UNet by a large margin in multiple evaluation metrics on the multi-channel ConferencingSpeech2021 Challenge dataset. Ablation studies also demonstrate the effectiveness of different contributions.

eess.AS↗

Multispectral Image Intrinsic Decomposition via Low Rank Constraint

Multispectral images contain many clues of surface characteristics of the objects, thus can be widely used in many computer vision tasks, e.g., recolorization and segmentation. However, due to the complex illumination and the geometry structure of natural scenes, the spectra curves of a same surface can look very different. In this paper, a Low Rank Multispectral Image Intrinsic Decomposition model (LRIID) is presented to decompose the shading and reflectance from a single multispectral image. We extend the Retinex model, which is proposed for RGB image intrinsic decomposition, for multispectral domain. Based on this, a low rank constraint is proposed to reduce the ill-posedness of the problem and make the algorithm solvable. A dataset of 12 images is given with the ground truth of shadings and reflectance, so that the objective evaluations can be conducted. The experiments demonstrate the effectiveness of proposed method.

cs.CV↗