arXiv subjects

Vikas Joshi

Publications and source records attributed to Vikas Joshi.

At least 19 recordsLinked to original sources

arXiv2025

Length Aware Speech Translation for Video Dubbing

In video dubbing, aligning translated audio with the source audio is a significant challenge. Our focus is on achieving this efficiently, tailored for real-time, on-device video dubbing scenarios. We developed a phoneme-based end-to-end length-sensitive speech translation (LSST) model, which generates translations of varying lengths short, normal, and long using predefined tags. Additionally, we introduced length-aware beam search (LABS), an efficient approach to generate translations of different lengths in a single decoding pass. This approach maintained comparable BLEU scores compared to a baseline without length awareness while significantly enhancing synchronization quality between source and target audio, achieving a mean opinion score (MOS) gain of 0.34 for Spanish and 0.65 for Korean, respectively.

cs.CL↗

arXiv2025

Building English ASR model with regional language support

In this paper, we present a novel approach to developing an English Automatic Speech Recognition (ASR) system that can effectively handle Hindi queries, without compromising its performance on English. We propose a novel acoustic model (AM), referred to as SplitHead with Attention (SHA) model, features shared hidden layers across languages and language-specific projection layers combined via a self-attention mechanism. This mechanism estimates the weight for each language based on input data and weighs the corresponding language-specific projection layers accordingly. Additionally, we propose a language modeling approach that interpolates n-gram models from both English and transliterated Hindi text corpora. Our results demonstrate the effectiveness of our approach, with a 69.3% and 5.7% relative reduction in word error rate on Hindi and English test sets respectively when compared to a monolingual English model.

eess.AS↗

arXiv2025

Addressing speaker gender bias in large scale speech translation systems

This study addresses the issue of speaker gender bias in Speech Translation (ST) systems, which can lead to offensive and inaccurate translations. The masculine bias often found in large-scale ST systems is typically perpetuated through training data derived from Machine Translation (MT) systems. Our approach involves two key steps. First, we employ Large Language Models (LLMs) to rectify translations based on the speaker's gender in a cost-effective manner. Second, we fine-tune the ST model with the corrected data, enabling the model to generate gender-specific translations directly from audio cues, without the need for explicit gender input. Additionally, we propose a three-mode fine-tuned model for scenarios where the speaker's gender is either predefined or should not be inferred from speech cues. We demonstrate a 70% improvement in translations for female speakers compared to our baseline and other large-scale ST systems, such as Seamless M4T and Canary, on the MuST-SHE test set.

cs.CL↗

arXiv2024

Detectability of Supernova Remnants with the Southern Wide-field Gamma-ray Observatory

Supernova remnants (SNRs) are likely sources of hadronic particle acceleration within our galaxy, contributing to the galactic cosmic ray flux. Next-generation instruments, such as the Southern Wide-field Gamma-ray Observatory (SWGO), will be of crucial importance in identifying new candidate SNRs. SWGO will observe two-thirds of the gamma-ray sky, covering the energy range between a few hundreds of GeV and a PeV. In this work, we apply a model of SNR evolution to a catalogue of SNRs in order to predict their gamma-ray spectra, explore the SNR emission phase space, and quantify detection prospects for SWGO. Finally, we validate our model for sources observed with current-generation instruments, fitting it using a Monte-Carlo Markov Chain technique to the observed gamma-ray emission from four SNRs. We anticipate that at least 6, and potentially as many as 11 SNRs will be detected by SWGO within 1 year.

astro-ph.HE↗

arXiv2024

Streaming Bilingual End-to-End ASR model using Attention over Multiple Softmax

Even with several advancements in multilingual modeling, it is challenging to recognize multiple languages using a single neural model, without knowing the input language and most multilingual models assume the availability of the input language. In this work, we propose a novel bilingual end-to-end (E2E) modeling approach, where a single neural model can recognize both languages and also support switching between the languages, without any language input from the user. The proposed model has shared encoder and prediction networks, with language-specific joint networks that are combined via a self-attention mechanism. As the language-specific posteriors are combined, it produces a single posterior probability over all the output symbols, enabling a single beam search decoding and also allowing dynamic switching between the languages. The proposed approach outperforms the conventional bilingual baseline with 13.3%, 8.23% and 1.3% word error rate relative reduction on Hindi, English and code-mixed test sets, respectively.

eess.AS↗

arXiv2023

Application of Graph Networks to background rejection in Imaging Air Cherenkov Telescopes

Imaging Air Cherenkov Telescopes (IACTs) are essential to ground-based observations of gamma rays in the GeV to TeV regime. One particular challenge of ground-based gamma-ray astronomy is an effective rejection of the hadronic background. We propose a new deep-learning-based algorithm for classifying images measured using single or multiple Imaging Air Cherenkov Telescopes. We interpret the detected images as a collection of triggered sensors that can be represented by graphs and analyzed by graph convolutional networks. For images cleaned of the light from the night sky, this allows for an efficient algorithm design that bypasses the challenge of sparse images in deep learning approaches based on computer vision techniques such as convolutional neural networks. We investigate different graph network architectures and find a promising performance with improvements to previous machine-learning and deep-learning-based methods.

astro-ph.IM↗

arXiv2023

Can AI Put Gamma-Ray Astrophysicists Out of a Job?

In what will likely be a litany of generative-model-themed arXiv submissions celebrating April the 1st, we evaluate the capacity of state-of-the-art transformer models to create a paper detailing the detection of a Pulsar Wind Nebula with a non-existent Imaging Atmospheric Cherenkov Telescope (IACT) Array. We do this to evaluate the ability of such models to interpret astronomical observations and sources based on language information alone, and to assess potential means by which fraudulently generated scientific papers could be identified during peer review (given that reliable generative model watermarking has yet to be deployed for these tools). We conclude that our jobs as astronomers are safe for the time being. From this point on, prompts given to ChatGPT and Stable Diffusion are shown in orange, text generated by ChatGPT is shown in black, whereas analysis by the (human) authors is in blue.

physics.pop-ph↗

arXiv2023

Revisiting HESS J1809$-$193 -- a very-high-energy gamma-ray source in a fascinating environment

HESS J1809$-$193 is one of the unidentified very-high-energy gamma-ray sources in the H.E.S.S. Galactic Plane Survey (HGPS). It is located in a rich environment, with an energetic pulsar and associated X-ray pulsar wind nebula, several supernova remnants, and molecular clouds in the vicinity. Furthermore, HESS J1809$-$193 was recently detected at energies above 56 TeV with HAWC, which makes it a PeVatron candidate, that is, a source capable of accelerating cosmic rays up to PeV energies. We present a new analysis of the TeV gamma-ray emission of HESS J1809$-$193 with H.E.S.S., based on improved analysis techniques. We find that the emission is best described by two components with distinct morphologies and energy spectra. We complement this study with an analysis of Fermi-LAT data in the same region. Finally, taking into account further multi-wavelength data, we interpret our results both in a hadronic and leptonic framework.

astro-ph.HE↗

arXiv2022

WavFT: Acoustic model finetuning with labelled and unlabelled data

Unsupervised and self-supervised learning methods have leveraged unlabelled data to improve the pretrained models. However, these methods need significantly large amount of unlabelled data and the computational cost of training models with such large amount of data can be prohibitively high. We address this issue by using unlabelled data during finetuning, instead of pretraining. We propose acoustic model finetuning (FT) using labelled and unlabelled data. The model is jointly trained to learn representations to classify senones, as well as learn contextual acoustic representations. Our training objective is a combination of cross entropy loss, suitable for classification task, and contrastive loss, suitable to learn acoustic representations. The proposed approach outperforms conventional finetuning with 11.2% and 9.19% word error rate relative (WERR) reduction on Gujarati and Bengali languages respectively.

cs.CL↗

arXiv2021

Standardized Formats for Gamma-Ray Analysis Applied to HAWC Observatory Data

A wide range of data formats and proprietary software have traditionally been used in gamma-ray astronomy, usually developed for a single specific mission or experiment. However, in recent years there has been an increasing effort towards making astronomical data open and easily accessible. Within the gamma-ray community this has translated to the creation of a common data format across different gamma-ray observatories: the "gamma-astro-data-format" (GADF). Based on a similar premise, open-source analysis packages, such as Gammapy, are being developed and aim to provide a single, robust tool which suits the needs of many experiments at once. In this contribution we show that data from the High-Altitude Water Cherenkov (HAWC) observatory can be made compatible with the GADF and present the first GADF-based production of event lists and instrument response functions for a ground-based wide-field instrument. We use these data products to reproduce with excellent agreement the published HAWC Crab spectrum using Gammapy. Having a common data format and analysis tools facilitates joint analysis between different experiments and effective data sharing. This will be especially important for next-generation instruments, such as the proposed Southern Wide-field Gamma-ray Observatory (SWGO) and the planned Cherenkov Telescope Array (CTA).

astro-ph.IM↗

arXiv2020

Transfer Learning Approaches for Streaming End-to-End Speech Recognition System

Transfer learning (TL) is widely used in conventional hybrid automatic speech recognition (ASR) system, to transfer the knowledge from source to target language. TL can be applied to end-to-end (E2E) ASR system such as recurrent neural network transducer (RNN-T) models, by initializing the encoder and/or prediction network of the target language with the pre-trained models from source language. In the hybrid ASR system, transfer learning is typically done by initializing the target language acoustic model (AM) with source language AM. Several transfer learning strategies exist in the case of the RNN-T framework, depending upon the choice of the initialization model for encoder and prediction networks. This paper presents a comparative study of four different TL methods for RNN-T framework. We show 17% relative word error rate reduction with different TL methods over randomly initialized RNN-T model. We also study the impact of TL with varying amount of training data ranging from 50 hours to 1000 hours and show the efficacy of TL for languages with small amount of training data.

eess.AS↗

arXiv2020

Learning not to Discriminate: Task Agnostic Learning for Improving Monolingual and Code-switched Speech Recognition

Recognizing code-switched speech is challenging for Automatic Speech Recognition (ASR) for a variety of reasons, including the lack of code-switched training data. Recently, we showed that monolingual ASR systems fine-tuned on code-switched data deteriorate in performance on monolingual speech recognition, which is not desirable as ASR systems deployed in multilingual scenarios should recognize both monolingual and code-switched speech with high accuracy. Our experiments indicated that this loss in performance could be mitigated by using certain strategies for fine-tuning and regularization, leading to improvements in both monolingual and code-switched ASR. In this work, we present further improvements over our previous work by using domain adversarial learning to train task agnostic models. We evaluate the classification accuracy of an adversarial discriminator and show that it can learn shared layer parameters that are task agnostic. We train end-to-end ASR systems starting with a pooled model that uses monolingual and code-switched data along with the adversarial discriminator. Our proposed technique leads to reductions in Word Error Rates (WER) in monolingual and code-switched test sets across three language pairs.

eess.AS↗

arXiv2020

Learning to Recognize Code-switched Speech Without Forgetting Monolingual Speech Recognition

Recently, there has been significant progress made in Automatic Speech Recognition (ASR) of code-switched speech, leading to gains in accuracy on code-switched datasets in many language pairs. Code-switched speech co-occurs with monolingual speech in one or both languages being mixed. In this work, we show that fine-tuning ASR models on code-switched speech harms performance on monolingual speech. We point out the need to optimize models for code-switching while also ensuring that monolingual performance is not sacrificed. Monolingual models may be trained on thousands of hours of speech which may not be available for re-training a new model. We propose using the Learning Without Forgetting (LWF) framework for code-switched ASR when we only have access to a monolingual model and do not have the data it was trained on. We show that it is possible to train models using this framework that perform well on both code-switched and monolingual test sets. In cases where we have access to monolingual training data as well, we propose regularization strategies for fine-tuning models for code-switching without sacrificing monolingual accuracy. We report improvements in Word Error Rate (WER) in monolingual and code-switched test sets compared to baselines that use pooled data and simple fine-tuning.

eess.AS↗

arXiv2019

Air shower reconstruction using HAWC and the Outrigger array

The High Altitude Water Cherenkov (HAWC) gamma-ray observatory detects cosmic- and gamma-ray initiated air showers in the TeV energy range using 300 water Cherenkov detectors (WCDs). To improve its sensitivity at the highest energies, HAWC has been upgraded with a sparse array of 345 small WCDs (outrigger array) around the HAWC main array. The outrigger array increases the instrumented area of HAWC by a factor of 4 and has started taking data since August 2018. A new gamma-ray reconstruction method has been developed to improve the reconstruction of the air showers which combines the data of mixed type particle detector arrays. In this contribution, we will show the first results of the combined air shower reconstruction of HAWC and its outrigger array using Monte Carlo simulations and the first combined experimental data set.

astro-ph.IM↗

arXiv2019

A Template-based $γ$-ray Reconstruction Method for Air Shower Arrays

We introduce a new Monte Carlo template-based reconstruction method for air shower arrays, with a focus on shower core and energy reconstruction of $γ$-ray induced air showers. The algorithm fits an observed lateral amplitude distribution of an extensive air shower against an expected probability distribution using a likelihood approach. A full Monte Carlo air shower simulation in combination with the detector simulation is used to generate the expected probability distributions. The goodness of fit can be used to discriminate between $γ$-ray and hadron induced air showers. As an example, we apply this method to the High Altitude Water Cherenkov $γ$-ray Observatory and its recently installed high-energy upgrade. The performance of this method and the applicability to air shower arrays with mixed detector types makes it a promising reconstruction approach for current and future instruments.

astro-ph.IM↗

arXiv2017

HAWC High Energy Upgrade with a Sparse Outrigger Array

The High Altitude Water Cherenkov (HAWC) gamma-ray observatory consists of 300 water Cherenkov detectors and has been fully operational since March 2015 in central Mexico. It detects cosmic- and gamma-ray showers in the TeV energy range. For multi-TeV energies, the shower reconstruction and hence the performance of the detector is affected by the partial containment of the showers within the array. To improve the sensitivity at the highest energies, HAWC is being upgraded with an outrigger array. It consists of 350 comparably much smaller water Cherenkov detectors, sparsely distributed around the HAWC main array. It will increase the instrumented area by a factor of 4-5. In this contribution, we will present the current status of the upgrade as well as simulation results on anticipated improvements in the performance of the observatory.

astro-ph.IM↗

arXiv2017

HAWC High Energy Upgrade with a Sparse Array

The High Altitude Water Cherenkov (HAWC) gamma-ray observatory has been fully operational since March 2015. To improve its sensitivity at the highest energies, it is being upgraded with an additional sparse array called outrigger array. We will discuss in this contribution, the different outrigger array components, and the simulation results to optimize it.

astro-ph.IM↗

arXiv2016

HAWC Upgrade for Multi-TeV γ-ray Detection

The High Altitude Water Cherenkov (HAWC) high-energy γ-ray observatory was completed in march 2015 in central Mexico. The detector, consisting of 300 water tanks, is currently being upgraded to improve its performance at Multi-TeV energies, with a sparse array of small water Cherenkov tanks. It will extend the instrumental area by a factor of 4, and enhance the sensitivity at the highest energies. In this contribution, the current status of the observatory is presented, as well as the coming upgrade. The electronics and the readout system for the new sparse array of small water tanks are also desrcibed, and results from simulations performed to optimize the performance of the array are discussed.

astro-ph.IM↗