arXiv subjects

Yanru Chen

Publications and source records attributed to Yanru Chen.

29 records · Page 2Linked to original sources

arXiv2025

MoE-TTS: Enhancing Out-of-Domain Text Understanding for Description-based TTS via Mixture-of-Experts

Description-based text-to-speech (TTS) models exhibit strong performance on in-domain text descriptions, i.e., those encountered during training. However, in real-world applications, the diverse range of user-generated descriptions inevitably introduces numerous out-of-domain inputs that challenge the text understanding capabilities of these systems. To address this issue, we propose MoE-TTS, a description-based TTS model designed to enhance the understanding of out-of-domain text descriptions. MoE-TTS employs a modality-based mixture-of-experts (MoE) approach to augment a pre-trained textual large language model (LLM) with a set of specialized weights adapted to the speech modality while maintaining the original LLM frozen during training. This approach allows MoE-TTS to effectively leverage the pre-trained knowledge and text understanding abilities of textual LLMs. Our experimental results indicate that: first, even the most advanced closed-source commercial products can be challenged by carefully designed out-of-domain description test sets; second, MoE-TTS achieves superior performance in generating speech that more accurately reflects the descriptions. We encourage readers to listen to the demos at https://welkinyang.github.io/MoE-TTS/.

eess.AS↗

arXiv2025

Kimi-VL Technical Report

We present Kimi-VL, an efficient open-source Mixture-of-Experts (MoE) vision-language model (VLM) that offers advanced multimodal reasoning, long-context understanding, and strong agent capabilities - all while activating only 2.8B parameters in its language decoder (Kimi-VL-A3B). Kimi-VL demonstrates strong performance across challenging domains: as a general-purpose VLM, Kimi-VL excels in multi-turn agent tasks (e.g., OSWorld), matching flagship models. Furthermore, it exhibits remarkable capabilities across diverse challenging vision language tasks, including college-level image and video comprehension, OCR, mathematical reasoning, and multi-image understanding. In comparative evaluations, it effectively competes with cutting-edge efficient VLMs such as GPT-4o-mini, Qwen2.5-VL-7B, and Gemma-3-12B-IT, while surpassing GPT-4o in several key domains. Kimi-VL also advances in processing long contexts and perceiving clearly. With a 128K extended context window, Kimi-VL can process diverse long inputs, achieving impressive scores of 64.5 on LongVideoBench and 35.1 on MMLongBench-Doc. Its native-resolution vision encoder, MoonViT, further allows it to see and understand ultra-high-resolution visual inputs, achieving 83.2 on InfoVQA and 34.5 on ScreenSpot-Pro, while maintaining lower computational cost for common tasks. Building upon Kimi-VL, we introduce an advanced long-thinking variant: Kimi-VL-Thinking-2506. Developed through long chain-of-thought (CoT) supervised fine-tuning (SFT) and reinforcement learning (RL), the latest model exhibits strong long-horizon reasoning capabilities (64.0 on MMMU, 46.3 on MMMU-Pro, 56.9 on MathVision, 80.1 on MathVista, 65.2 on VideoMMMU) while obtaining robust general abilities. Code and models are publicly accessible at https://github.com/MoonshotAI/Kimi-VL.

cs.CV↗

arXiv2025

Kimi k1.5: Scaling Reinforcement Learning with LLMs

Language model pretraining with next token prediction has proved effective for scaling compute but is limited to the amount of available training data. Scaling reinforcement learning (RL) unlocks a new axis for the continued improvement of artificial intelligence, with the promise that large language models (LLMs) can scale their training data by learning to explore with rewards. However, prior published work has not produced competitive results. In light of this, we report on the training practice of Kimi k1.5, our latest multi-modal LLM trained with RL, including its RL training techniques, multi-modal data recipes, and infrastructure optimization. Long context scaling and improved policy optimization methods are key ingredients of our approach, which establishes a simplistic, effective RL framework without relying on more complex techniques such as Monte Carlo tree search, value functions, and process reward models. Notably, our system achieves state-of-the-art reasoning performance across multiple benchmarks and modalities -- e.g., 77.5 on AIME, 96.2 on MATH 500, 94-th percentile on Codeforces, 74.9 on MathVista -- matching OpenAI's o1. Moreover, we present effective long2short methods that use long-CoT techniques to improve short-CoT models, yielding state-of-the-art short-CoT reasoning results -- e.g., 60.8 on AIME, 94.6 on MATH500, 47.3 on LiveCodeBench -- outperforming existing short-CoT models such as GPT-4o and Claude Sonnet 3.5 by a large margin (up to +550%).

cs.AI↗

arXiv2025

Kimi-Audio Technical Report

We present Kimi-Audio, an open-source audio foundation model that excels in audio understanding, generation, and conversation. We detail the practices in building Kimi-Audio, including model architecture, data curation, training recipe, inference deployment, and evaluation. Specifically, we leverage a 12.5Hz audio tokenizer, design a novel LLM-based architecture with continuous features as input and discrete tokens as output, and develop a chunk-wise streaming detokenizer based on flow matching. We curate a pre-training dataset that consists of more than 13 million hours of audio data covering a wide range of modalities including speech, sound, and music, and build a pipeline to construct high-quality and diverse post-training data. Initialized from a pre-trained LLM, Kimi-Audio is continual pre-trained on both audio and text data with several carefully designed tasks, and then fine-tuned to support a diverse of audio-related tasks. Extensive evaluation shows that Kimi-Audio achieves state-of-the-art performance on a range of audio benchmarks including speech recognition, audio understanding, audio question answering, and speech conversation. We release the codes, model checkpoints, as well as the evaluation toolkits in https://github.com/MoonshotAI/Kimi-Audio.

eess.AS↗

arXiv2025

Muon is Scalable for LLM Training

Recently, the Muon optimizer based on matrix orthogonalization has demonstrated strong results in training small-scale language models, but the scalability to larger models has not been proven. We identify two crucial techniques for scaling up Muon: (1) adding weight decay and (2) carefully adjusting the per-parameter update scale. These techniques allow Muon to work out-of-the-box on large-scale training without the need of hyper-parameter tuning. Scaling law experiments indicate that Muon achieves $\sim\!2\times$ computational efficiency compared to AdamW with compute optimal training. Based on these improvements, we introduce Moonlight, a 3B/16B-parameter Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon. Our model improves the current Pareto frontier, achieving better performance with much fewer training FLOPs compared to prior models. We open-source our distributed Muon implementation that is memory optimal and communication efficient. We also release the pretrained, instruction-tuned, and intermediate checkpoints to support future research.

cs.LG↗

arXiv2025

MoBA: Mixture of Block Attention for Long-Context LLMs

Scaling the effective context length is essential for advancing large language models (LLMs) toward artificial general intelligence (AGI). However, the quadratic increase in computational complexity inherent in traditional attention mechanisms presents a prohibitive overhead. Existing approaches either impose strongly biased structures, such as sink or window attention which are task-specific, or radically modify the attention mechanism into linear approximations, whose performance in complex reasoning tasks remains inadequately explored. In this work, we propose a solution that adheres to the ``less structure'' principle, allowing the model to determine where to attend autonomously, rather than introducing predefined biases. We introduce Mixture of Block Attention (MoBA), an innovative approach that applies the principles of Mixture of Experts (MoE) to the attention mechanism. This novel architecture demonstrates superior performance on long-context tasks while offering a key advantage: the ability to seamlessly transition between full and sparse attention, enhancing efficiency without the risk of compromising performance. MoBA has already been deployed to support Kimi's long-context requests and demonstrates significant advancements in efficient attention computation for LLMs. Our code is available at https://github.com/MoonshotAI/MoBA.

cs.LG↗

arXiv2024

Level of Regions for Deformed Braid Arrangements

This paper primarily investigates a specific type of deformation of the braid arrangement $\mathcal{B}_n$ in $\mathbb{R}^n$, denoted by $\mathcal{B}_n^A$ and defined in (1.2). Let $r_l(\mathcal{B}_n^A)$ be the number of regions of level $l$ in $\mathcal{B}_n^A$ with the corresponding exponential generating function $R_l(A;x)$. Using the weighted digraph model introduced by Hetyei [11], we establish a bijection between regions of level $l$ in $\mathcal{B}_n^A$ and valid $m$-acyclic weighted digraphs on the vertex set $[n]$ with exactly $l$ strong components. Based on this bijection, we obtain a property analogous to a polynomial sequence of binomial type, that is, $R_l(A;x)$ satisfies the relation \[ R_l(A;x)=\big(R_1(A;x)\big)^l=R_k(A;x)R_{l-k}(A;x). \] Furthermore, the values $r_l(\mathcal{B}_n^A)$ yield a combinatorial interpretation for the coefficients in the expansion of the characteristic polynomial $χ_{\mathcal{B}_n^A}(t)$ in the basis elements $\binom{t}{l}$, that is, \[χ_{\mathcal{B}_n^A}(t)=\sum_{l=0}^n(-1)^{n-l}r_l(\mathcal{B}_n^A)\binom{t}{l}.\] If $n$, $a$ and $b$ are non-negative integers with $n\ge 2$ and $b-a\ge n-1$, for the deformation $\mathcal{B}_n^{[-a,b]}$ defined in (1.3), its characteristic polynomial has a single real root $0$ of multiplicity one when $n$ is odd, and has one more real root $\frac{n(a+b+1)}{2}$ of multiplicity one when $n$ is even.

math.CO↗

arXiv2023

Quasiparticle and Transport Properties of Disordered Bilayer Graphene

In recent experimental and theoretical studies of graphene, disorder scattering processes have been suggested to play an important role in its electronic and transport properties. In the preceding paper, it has been shown that the nonperturbative momentum-space Lanczos method is able to accurately describe all the multiple impurity scattering events and account for the quasiparticle and transport properties of disordered monolayer graphene. In the present study, we expand the range of applicability of this recursive method by numerically investigating the quasiparticle and transport properties of Bernal-stacked bilayer graphene in the presence of scalar Anderson disorder. The results are further compared with the findings of the same system using a self-consistent Born approximation, as well as the central findings in the preceding paper for monolayer graphene. It is found that in both systems, proper inclusions of all the scattering events are needed in order to reliably capture the role of disorder via multiple impurity scattering. In particular, the quasiparticle residue is shown to decrease sharply near the charge neutrality point, suggesting that the system is either a marginal Fermi liquid or a non-Fermi liquid. Furthermore, we reveal the dependences of the transport properties of disordered bilayer graphene on the carrier density and temperature, and explore the role of interlayer scattering at varying strengths. Our findings help to provide some new angles into the quasiparticle and transport properties of disordered bilayer graphene.

cond-mat.dis-nn↗

arXiv2023

Disorder Effects on the Quasiparticle and Transport Properties of Two-Dimensional Dirac Fermionic Systems

Despite extensive existing studies, a complete understanding of the role of disorder in affecting the physical properties of two-dimensional Dirac fermionic systems remains a standing challenge, largely due to obstacles encountered in treating multiple scattering events for such inherently strong scattering systems. Using graphene as an example and a nonperturbative numerical technique, here we reveal that the low energy quasiparticle properties are considerably modified by multiple scattering processes even in the presence of weak scalar potentials. We extract unified power-law energy dependences of the self-energy with fractional exponents from the weak scattering limit to the strong scattering limit from our numerical analysis, leading to sharp reductions of the quasiparticle residues near the Dirac point, eventually vanishing at the Dirac point. The central findings stay valid when the Anderson-type impurities are replaced by correlated Gaussian- or Yukawa-type disorder with varying correlation lengths. The improved understanding gained here also enables us to provide better interpretations of the experimental observations surrounding the temperature and carrier density dependences of the conductivity in ultra-high mobility graphene samples. The approach demonstrated here is expected to find broad applicability in understanding the role of various other types of impurities in two-dimensional Dirac systems.

cond-mat.mes-hall↗

arXiv2023

A Comparison of Self-Supervised Pretraining Approaches for Predicting Disease Risk from Chest Radiograph Images

Deep learning is the state-of-the-art for medical imaging tasks, but requires large, labeled datasets. For risk prediction, large datasets are rare since they require both imaging and follow-up (e.g., diagnosis codes). However, the release of publicly available imaging data with diagnostic labels presents an opportunity for self and semi-supervised approaches to improve label efficiency for risk prediction. Though several studies have compared self-supervised approaches in natural image classification, object detection, and medical image interpretation, there is limited data on which approaches learn robust representations for risk prediction. We present a comparison of semi- and self-supervised learning to predict mortality risk using chest x-ray images. We find that a semi-supervised autoencoder outperforms contrastive and transfer learning in internal and external validation.

eess.IV↗

arXiv2022

Prompt-Based Metric Learning for Few-Shot NER

Few-shot named entity recognition (NER) targets generalizing to unseen labels and/or domains with few labeled examples. Existing metric learning methods compute token-level similarities between query and support sets, but are not able to fully incorporate label semantics into modeling. To address this issue, we propose a simple method to largely improve metric learning for NER: 1) multiple prompt schemas are designed to enhance label semantics; 2) we propose a novel architecture to effectively combine multiple prompt-based representations. Empirically, our method achieves new state-of-the-art (SOTA) results under 16 of the 18 considered settings, substantially outperforming the previous SOTA by an average of 8.84% and a maximum of 34.51% in relative gains of micro F1. Our code is available at https://github.com/AChen-qaq/ProML.

cs.CL↗