arXiv subjects

Leili Tavabi

Publications and source records attributed to Leili Tavabi.

5 recordsLinked to original sources

arXiv2026

Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence

We introduce Nemotron 3 Nano Omni, the latest model in the Nemotron multimodal series and the first to natively support audio inputs alongside text, images, and video. Nemotron 3 Nano Omni delivers consistent accuracy improvements over its predecessor, Nemotron Nano V2 VL, across all modalities, enabled by advances in architecture, training data and recipes. In particular, Nemotron 3 delivers leading results in real-world document understanding, long audio-video comprehension, and agentic computer use. Built on the highly efficient Nemotron 3 Nano 30B-A3B backbone, Nemotron 3 Nano Omni further incorporates innovative multimodal token-reduction techniques to deliver substantially lower inference latency and higher throughput than other models of similar size. We are releasing model checkpoints in BF16, FP8, and FP4 formats, along with portions of the training data and codebase to facilitate further research and development.

cs.LG↗

arXiv2025

Align2Speak: Improving TTS for Low Resource Languages via ASR-Guided Online Preference Optimization

Developing high-quality text-to-speech (TTS) systems for low-resource languages is challenging due to the scarcity of paired text and speech data. In contrast, automatic speech recognition (ASR) models for such languages are often more accessible, owing to large-scale multilingual pre-training efforts. We propose a framework based on Group Relative Policy Optimization (GRPO) to adapt an autoregressive, multilingual TTS model to new languages. Our method first establishes a language-agnostic foundation for TTS synthesis by training a multilingual baseline with International Phonetic Alphabet (IPA) tokens. Next, we fine-tune this model on limited paired data of the new languages to capture the target language's prosodic features. Finally, we apply GRPO to optimize the model using only unpaired text and speaker prompts, guided by a multi-objective reward from pretrained ASR, speaker verification, and audio quality estimation models. Experiments demonstrate that this pipeline produces intelligible and speaker-consistent speech in low-resource languages, substantially outperforming fine-tuning alone. Furthermore, our GRPO-based framework also improves TTS performance in high-resource languages, surpassing offline alignment methods such as Direct Preference Optimization (DPO) yielding superior intelligibility, speaker similarity, and audio quality.

cs.AI↗

arXiv2021

Speaker Turn Modeling for Dialogue Act Classification

Dialogue Act (DA) classification is the task of classifying utterances with respect to the function they serve in a dialogue. Existing approaches to DA classification model utterances without incorporating the turn changes among speakers throughout the dialogue, therefore treating it no different than non-interactive written text. In this paper, we propose to integrate the turn changes in conversations among speakers when modeling DAs. Specifically, we learn conversation-invariant speaker turn embeddings to represent the speaker turns in a conversation; the learned speaker turn embeddings are then merged with the utterance embeddings for the downstream task of DA classification. With this simple yet effective mechanism, our model is able to capture the semantics from the dialogue content while accounting for different speaker turns in a conversation. Validation on three benchmark public datasets demonstrates superior performance of our model.

cs.CL↗

arXiv2019

AVEC 2019 Workshop and Challenge: State-of-Mind, Detecting Depression with AI, and Cross-Cultural Affect Recognition

The Audio/Visual Emotion Challenge and Workshop (AVEC 2019) "State-of-Mind, Detecting Depression with AI, and Cross-cultural Affect Recognition" is the ninth competition event aimed at the comparison of multimedia processing and machine learning methods for automatic audiovisual health and emotion analysis, with all participants competing strictly under the same conditions. The goal of the Challenge is to provide a common benchmark test set for multimodal information processing and to bring together the health and emotion recognition communities, as well as the audiovisual processing communities, to compare the relative merits of various approaches to health and emotion recognition from real-life data. This paper presents the major novelties introduced this year, the challenge guidelines, the data used, and the performance of the baseline systems on the three proposed tasks: state-of-mind recognition, depression assessment with AI, and cross-cultural affect sensing, respectively.

cs.HC↗

arXiv2016

Ranking Research Institutions Based On Related Academic Conferences

The detection of influential nodes in a social network is an active research area with many valuable applications including marketing and advertisement. As a new application in academia, KDD Cup 2016 shed light on the lack of an existing objective ranking for institutions within their respective research areas and proposed a solution for it. In this problem, the academic fields are defined as social networks whose nodes are the active institutions within the field, with the most influential nodes representing the highest contributors. The solution is able to provide a ranking of active institutions within their specific domains. The problem statement provided an annual scoring mechanism for institutions based on their publications and encouraged the use of any publicly available dataset such as the Microsoft Academic Graph (MAG). The contest was focused on research publications in selected conferences and asked for a prediction of the ranking for active institutions within those conferences in 2016. It should be noted that the results of the paper submissions and therefore the ground truths for KDD Cup were unknown at the time of the contest. Each team's final ranking list was evaluated by a metric called NDCG@20 after the results were released. This metric was used to indicate the distance between each team's proposed ranking and the actual one once it was known. After computing the scores of institutions for each year starting from 2011, we aggregated the rankings by summing the normalized scores across the years and using the final score set to provide the final ranking. Since the 2016 ground truths were unknown, we utilized the scores from 2011-2014 and used the 2015 publications as a test bed for evaluating our aggregation method. Based on the testing, summing the normalized scores got us closest to the actual 2015 rankings and using same heuristic for predicting the 2016 results.

cs.SI↗