Large Language Model Guided Decoding for Self-Supervised Speech Recognition

Abstract

Self-supervised automatic speech recognition (SSL-ASR) is an ASR approach that uses speech encoders pretrained on large amounts of unlabeled audio (e.g., wav2vec2.0 or HuBERT) and then fine-tunes them with limited labeled data to perform transcription. Decoding is usually performed with a CTC decoder, whose hypotheses are scored and refined using an external language model (LM), typically an n-gram or neural LM, which guides beam search to produce the final transcription. Using Large Language Models (LLMs) as external LMs remains a challenge, as their word probabilities are overly confident. The proposed method integrates an LLM with an SSL acoustic model by using the LLM's decoding mechanism to generate a set of candidate next tokens. For each candidate, the SSL model provides an acoustic score by aligning it to the input acoustics of the SSL model. A combined acoustic and LLM score is then calculated based on decomposing the MAP estimator of words given the acoustic signal. The tokens with the highest combined scores are maintained in a beam, which is then used to proceed to the next decoding step. We illustrate the effectiveness of our method through a comprehensive comparison with the current state-of-the-art LLM-based decoding, post-processing, and error-correcting methods across multiple datasets. Our approach proves particularly effective when processing challenging inputs such as complex speech sentences, acronyms, and domain-specific vocabulary.