arXiv subjects

Xinran Xu

Publications and source records attributed to Xinran Xu.

27 records · Page 2Linked to original sources

arXiv2025

Kimi k1.5: Scaling Reinforcement Learning with LLMs

Language model pretraining with next token prediction has proved effective for scaling compute but is limited to the amount of available training data. Scaling reinforcement learning (RL) unlocks a new axis for the continued improvement of artificial intelligence, with the promise that large language models (LLMs) can scale their training data by learning to explore with rewards. However, prior published work has not produced competitive results. In light of this, we report on the training practice of Kimi k1.5, our latest multi-modal LLM trained with RL, including its RL training techniques, multi-modal data recipes, and infrastructure optimization. Long context scaling and improved policy optimization methods are key ingredients of our approach, which establishes a simplistic, effective RL framework without relying on more complex techniques such as Monte Carlo tree search, value functions, and process reward models. Notably, our system achieves state-of-the-art reasoning performance across multiple benchmarks and modalities -- e.g., 77.5 on AIME, 96.2 on MATH 500, 94-th percentile on Codeforces, 74.9 on MathVista -- matching OpenAI's o1. Moreover, we present effective long2short methods that use long-CoT techniques to improve short-CoT models, yielding state-of-the-art short-CoT reasoning results -- e.g., 60.8 on AIME, 94.6 on MATH500, 47.3 on LiveCodeBench -- outperforming existing short-CoT models such as GPT-4o and Claude Sonnet 3.5 by a large margin (up to +550%).

cs.AI↗

arXiv2025

Kimi-Audio Technical Report

We present Kimi-Audio, an open-source audio foundation model that excels in audio understanding, generation, and conversation. We detail the practices in building Kimi-Audio, including model architecture, data curation, training recipe, inference deployment, and evaluation. Specifically, we leverage a 12.5Hz audio tokenizer, design a novel LLM-based architecture with continuous features as input and discrete tokens as output, and develop a chunk-wise streaming detokenizer based on flow matching. We curate a pre-training dataset that consists of more than 13 million hours of audio data covering a wide range of modalities including speech, sound, and music, and build a pipeline to construct high-quality and diverse post-training data. Initialized from a pre-trained LLM, Kimi-Audio is continual pre-trained on both audio and text data with several carefully designed tasks, and then fine-tuned to support a diverse of audio-related tasks. Extensive evaluation shows that Kimi-Audio achieves state-of-the-art performance on a range of audio benchmarks including speech recognition, audio understanding, audio question answering, and speech conversation. We release the codes, model checkpoints, as well as the evaluation toolkits in https://github.com/MoonshotAI/Kimi-Audio.

eess.AS↗

arXiv2025

Muon is Scalable for LLM Training

Recently, the Muon optimizer based on matrix orthogonalization has demonstrated strong results in training small-scale language models, but the scalability to larger models has not been proven. We identify two crucial techniques for scaling up Muon: (1) adding weight decay and (2) carefully adjusting the per-parameter update scale. These techniques allow Muon to work out-of-the-box on large-scale training without the need of hyper-parameter tuning. Scaling law experiments indicate that Muon achieves $\sim\!2\times$ computational efficiency compared to AdamW with compute optimal training. Based on these improvements, we introduce Moonlight, a 3B/16B-parameter Mixture-of-Expert (MoE) model trained with 5.7T tokens using Muon. Our model improves the current Pareto frontier, achieving better performance with much fewer training FLOPs compared to prior models. We open-source our distributed Muon implementation that is memory optimal and communication efficient. We also release the pretrained, instruction-tuned, and intermediate checkpoints to support future research.

cs.LG↗

arXiv2025

MoBA: Mixture of Block Attention for Long-Context LLMs

Scaling the effective context length is essential for advancing large language models (LLMs) toward artificial general intelligence (AGI). However, the quadratic increase in computational complexity inherent in traditional attention mechanisms presents a prohibitive overhead. Existing approaches either impose strongly biased structures, such as sink or window attention which are task-specific, or radically modify the attention mechanism into linear approximations, whose performance in complex reasoning tasks remains inadequately explored. In this work, we propose a solution that adheres to the ``less structure'' principle, allowing the model to determine where to attend autonomously, rather than introducing predefined biases. We introduce Mixture of Block Attention (MoBA), an innovative approach that applies the principles of Mixture of Experts (MoE) to the attention mechanism. This novel architecture demonstrates superior performance on long-context tasks while offering a key advantage: the ability to seamlessly transition between full and sparse attention, enhancing efficiency without the risk of compromising performance. MoBA has already been deployed to support Kimi's long-context requests and demonstrates significant advancements in efficient attention computation for LLMs. Our code is available at https://github.com/MoonshotAI/MoBA.

cs.LG↗

arXiv2024

Fully Automated CTC Detection, Segmentation and Classification for Multi-Channel IF Imaging

Liquid biopsies (eg., blood draws) offer a less invasive and non-localized alternative to tissue biopsies for monitoring the progression of metastatic breast cancer (mBCa). Immunofluoresence (IF) microscopy is a tool to image and analyze millions of blood cells in a patient sample. By detecting and genetically sequencing circulating tumor cells (CTCs) in the blood, personalized treatment plans are achievable for various cancer subtypes. However, CTCs are rare (about 1 in 2M), making manual CTC detection very difficult. In addition, clinicians rely on quantitative cellular biomarkers to manually classify CTCs. This requires prior tasks of cell detection, segmentation and feature extraction. To assist clinicians, we have developed a fully automated machine learning-based production-level pipeline to efficiently detect, segment and classify CTCs in multi-channel IF images. We achieve over 99% sensitivity and 97% specificity on 9,533 cells from 15 mBCa patients. Our pipeline has been successfully deployed on real mBCa patients, reducing a patient average of 14M detected cells to only 335 CTC candidates for manual review.

cs.CV↗

arXiv2024

Constraining vector dark matter and dark photon with degenerate mass in a hidden local SU(2) model

We discuss degenerate vector dark matter and dark photon that are induced from hidden $SU(2)_H$ gauge sector where it is spontaneously broken by vacuum expectation value of $SU(2)_H$ doublet. Kinetic mixing between $SU(2)_H$ and $U(1)_Y$ gauge fields can be generated by introducing dimension six operator realizing dark photon interactions. In estimating relic density we focus on the process in which dark matter annihilates into dark photons, and search for the region of dark matter mass and gauge coupling realizing observed relic density. We then discuss constraints from dark photon physics, thermalization of dark sector and direct detection of dark matter. It is then found that constraints from direct detection experiments give us the strongest upper limits on the dark photon interactions.

hep-ph↗

arXiv2020

EPIHC: Improving Enhancer-Promoter Interaction Prediction by using Hybrid features and Communicative learning

Enhancer-promoter interactions (EPIs) regulate the expression of specific genes in cells, and EPIs are important for understanding gene regulation, cell differentiation and disease mechanisms. EPI identification through the wet experiments is costly and time-consuming, and computational methods are in demand. In this paper, we propose a deep neural network-based method EPIHC based on sequence-derived features and genomic features for the EPI prediction. EPIHC extracts features from enhancer and promoter sequences respectively using convolutional neural networks (CNN), and then design a communicative learning module to captures the communicative information between enhancer and promoter sequences. EPIHC also take the genomic features of enhancers and promoters into account. At last, EPIHC combines sequence-derived features and genomic features to predict EPIs. The computational experiments show that EPIHC outperforms the existing state-of-the-art EPI prediction methods on the benchmark datasets and chromosome-split datasets, and the study reveal that the communicative learning module can bring explicit information about EPIs, which is ignore by CNN. Moreover, we consider two strategies to improve performances of EPIHC in the cross-cell line prediction, and experimental results show that EPIHC constructed on training cell lines exhibit improved performances for the other cell lines.

q-bio.GN↗

arXiv2020

Elucidating the Origin of Heterogeneous Anomalous Diffusion in the Cytoplasm of Mammalian Cells

Diffusion of tracer particles in the cytoplasm of mammalian cells is often anomalous with a marked heterogeneity even within individual particle trajectories. Despite considerable efforts, the mechanisms behind these observations have remained largely elusive. To tackle this problem, we performed extensive single-particle tracking experiments on quantum dots in the cytoplasm of living mammalian cells at varying conditions. Analyses of the trajectories reveal a strong, microtubule-dependent subdiffusion with antipersistent increments and a substantial heterogeneity. Furthermore, particles stochastically switch between different mobility states, most likely due to transient associations with the cytoskeleton-shaken endoplasmic reticulum network. Comparison to simulations highlight that all experimental observations can be fully described by an intermittent fractional Brownian motion, alternating between two states of different mobility.

physics.bio-ph↗

arXiv2018

Power spectral density of a single Brownian trajectory: What one can and cannot learn from it

The power spectral density (PSD) of any time-dependent stochastic processes $X_t$ is a meaningful feature of its spectral content. In its text-book definition, the PSD is the Fourier transform of the covariance function of $X_t$ over an infinitely large observation time $T$, that is, it is defined as an ensemble-averaged property taken in the limit $T \to \infty$. A legitimate question is what information on the PSD can be reliably obtained from single-trajectory experiments, if one goes beyond the standard definition and analyzes the PSD of a \textit{single} trajectory recorded for a \textit{finite} observation time $T$. In quest for this answer, for a $d$-dimensional Brownian motion we calculate the probability density function of a single-trajectory PSD for arbitrary frequency $f$, finite observation time $T$ and arbitrary number $k$ of projections of the trajectory on different axes. We show analytically that the scaling exponent for the frequency-dependence of the PSD specific to an ensemble of Brownian motion trajectories can be already obtained from a single trajectory, while the numerical amplitude in the relation between the ensemble-averaged and single-trajectory PSDs is a fluctuating property which varies from realization to realization. The distribution of this amplitude is calculated exactly and is discussed in detail. Our results are confirmed by numerical simulations and single particle tracking experiments, with remarkably good agreement. In addition we consider a truncated Wiener representation of Brownian motion, and the case of a discrete-time lattice random walk. We highlight some differences in the behavior of a single-trajectory PSD for Brownian motion and for the two latter situations. The framework developed herein will allow for meaningful physical analysis of experimental stochastic trajectories.

cond-mat.stat-mech↗