arXiv subjects

Hailin Zhang

Publications and source records attributed to Hailin Zhang.

At least 37 records · Page 2Linked to original sources

arXiv2025

MiMo-Audio: Audio Language Models are Few-Shot Learners

Existing audio language models typically rely on task-specific fine-tuning to accomplish particular audio tasks. In contrast, humans are able to generalize to new audio tasks with only a few examples or simple instructions. GPT-3 has shown that scaling next-token prediction pretraining enables strong generalization capabilities in text, and we believe this paradigm is equally applicable to the audio domain. By scaling MiMo-Audio's pretraining data to over one hundred million of hours, we observe the emergence of few-shot learning capabilities across a diverse set of audio tasks. We develop a systematic evaluation of these capabilities and find that MiMo-Audio-7B-Base achieves SOTA performance on both speech intelligence and audio understanding benchmarks among open-source models. Beyond standard metrics, MiMo-Audio-7B-Base generalizes to tasks absent from its training data, such as voice conversion, style transfer, and speech editing. MiMo-Audio-7B-Base also demonstrates powerful speech continuation capabilities, capable of generating highly realistic talk shows, recitations, livestreaming and debates. At the post-training stage, we curate a diverse instruction-tuning corpus and introduce thinking mechanisms into both audio understanding and generation. MiMo-Audio-7B-Instruct achieves open-source SOTA on audio understanding benchmarks (MMSU, MMAU, MMAR, MMAU-Pro), spoken dialogue benchmarks (Big Bench Audio, MultiChallenge Audio) and instruct-TTS evaluations, approaching or surpassing closed-source models. Model checkpoints and full evaluation suite are available at https://github.com/XiaomiMiMo/MiMo-Audio.

cs.CL↗

arXiv2025

Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers

Reinforcement learning (RL) has emerged as a crucial approach for enhancing the capabilities of large language models. However, in Mixture-of-Experts (MoE) models, the routing mechanism often introduces instability, even leading to catastrophic RL training collapse. We analyze the training-inference consistency of MoE models and identify a notable discrepancy in routing behaviors between the two phases. Moreover, even under identical conditions, the routing framework can yield divergent expert selections across repeated forward passes. To address this foundational inconsistency, we propose Rollout Routing Replay (R3), a method that records routing distributions from the inference engine and replays them during training. R3 significantly reduces training-inference policy KL divergence and mitigates extreme discrepancies without compromising training speed. Extensive experiments on various settings confirm that R3 succeeds in stabilizing RL training, preventing collapse and outperforming methods such as GSPO and TIS. We believe this work can offer a new solution for stabilizing RL in MoE models.

cs.CL↗

arXiv2025

Airway Mucus Rheology: Physical Insights for Navigating through Health to Pathology and Clinical Applications

Airway mucus is a complex gel with an anisotropic three-dimensional network structure. As a crucial component of the respiratory defense barrier, it plays a vital role in maintaining airway hydration and supporting the function of airway epithelial cells. Through linear and nonlinear rheological mechanisms such as ciliary motion and coughing, airway mucus expels foreign pathogens and toxic nano- and microparticles while selectively allowing the passage of specific nutrients and proteins. These protective and clearance functions depend on the proper rheological properties of mucus under normal physiological conditions. However, in respiratory disease such as CF, COPD, asthma, and COVID-19, excessive mucus secretion is often accompanied by abnormal rheological behaviors. This leads to impaired mucus flow, airway obstruction, and potentially life-threatening conditions. Therefore, this review examines the rheological behaviors of airway mucus in relation to health and disease, focusing on both macrorheology and microrheology. The review highlights those changes in the chemical composition and microstructure of airway mucus, especially under pathological conditions, that can significantly alter its rheological behavior. Rheological parameters can also serve as biological indicators to study the role of mucus in clearance functions and aid in developing pulmonary drug delivery systems. By integrating findings from both macro- and microrheological studies, this review aims to enhance our understanding of the complex behavior of airway mucus, supporting better diagnosis, treatment, and management of chronic respiratory diseases.

physics.med-ph↗

arXiv2025

Multi-Frequency Resonating Based Magnetic Induction Underground Emergency Communications with Diverse Mediums

Magnetic induction (MI) communication is an effective underground emergency communication technique after disasters such as landslides, mine collapses, and earthquakes, due to its advantages in mediums such as soil, concrete, and metals. However, the propagation mediums in practical MI based underground emergency communications are usually diverse and composed randomly due to the impact of disasters, which poses a challenge for MI communication in practical applications. In this paper, we formulate a statistical fading channel model, which reflects the random composition of diverse mediums and is shown to follow a lognormal distribution. To mitigate the impact of diverse medium fading, Multi-frequency Resonating Compensation (MuReC) based coils are used to achieve multiband transmission. Then, we analyze the performance of MuReC based multi-band MI communication with diverse medium fading and derive the expressions of signal-to-noise ratio (SNR) probability density functions, ergodic capacities, average bit error rates (BERs), and outage probabilities for both multiplexing and diversity cases. Numerical results show that MuReC based multiband transmission schemes can effectively reduce the impact of diverse medium fading and enhance the performance.

eess.SP↗

arXiv2025

Robust Beamforming for Magnetic Induction Based Underground Emergency Communications

Magnetic induction (MI) communication is an effective underground emergency communication technique after disasters such as landslides, mine collapses, and earthquakes, due to its advantages in mediums such as soil, concrete, and metals. Based on channel state information (CSI), magnetic beamforming can significantly improve the performance of MI communication. However, in post-disaster underground communication, channel estimation may suffer from errors due to factors such as complex environmental interferences. Taking channel estimation error into account, we formulate a beamforming optimization problem for multi-user MI underground emergency communications, which aims to minimize the power consumption under the constraints of sum rate and signal to interference plus noise ratio (SINR) of each user. Based on the worst-case optimization criterion and the S-procedure, the non-convex optimization problem is transformed into convex and solved. Numerical results show that the proposed robust beamforming scheme can effectively enhance communication reliability and effective throughput in the presence of channel estimation errors.

eess.SP↗

arXiv2025

MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttraining

We present MiMo-7B, a large language model born for reasoning tasks, with optimization across both pre-training and post-training stages. During pre-training, we enhance the data preprocessing pipeline and employ a three-stage data mixing strategy to strengthen the base model's reasoning potential. MiMo-7B-Base is pre-trained on 25 trillion tokens, with additional Multi-Token Prediction objective for enhanced performance and accelerated inference speed. During post-training, we curate a dataset of 130K verifiable mathematics and programming problems for reinforcement learning, integrating a test-difficulty-driven code-reward scheme to alleviate sparse-reward issues and employing strategic data resampling to stabilize training. Extensive evaluations show that MiMo-7B-Base possesses exceptional reasoning potential, outperforming even much larger 32B models. The final RL-tuned model, MiMo-7B-RL, achieves superior performance on mathematics, code and general reasoning tasks, surpassing the performance of OpenAI o1-mini. The model checkpoints are available at https://github.com/xiaomimimo/MiMo.

cs.CL↗

arXiv2025

MiMo-VL Technical Report

We open-source MiMo-VL-7B-SFT and MiMo-VL-7B-RL, two powerful vision-language models delivering state-of-the-art performance in both general visual understanding and multimodal reasoning. MiMo-VL-7B-RL outperforms Qwen2.5-VL-7B on 35 out of 40 evaluated tasks, and scores 59.4 on OlympiadBench, surpassing models with up to 78B parameters. For GUI grounding applications, it sets a new standard with 56.1 on OSWorld-G, even outperforming specialized models such as UI-TARS. Our training combines four-stage pre-training (2.4 trillion tokens) with Mixed On-policy Reinforcement Learning (MORL) integrating diverse reward signals. We identify the importance of incorporating high-quality reasoning data with long Chain-of-Thought into pre-training stages, and the benefits of mixed RL despite challenges in simultaneous multi-domain optimization. We also contribute a comprehensive evaluation suite covering 50+ tasks to promote reproducibility and advance the field. The model checkpoints and full evaluation suite are available at https://github.com/XiaomiMiMo/MiMo-VL.

cs.CL↗

arXiv2025

SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling

Many advanced Large Language Model (LLM) applications require long-context processing, but the self-attention module becomes a bottleneck during the prefilling stage of inference due to its quadratic time complexity with respect to sequence length. Existing sparse attention methods accelerate attention computation by skipping less significant regions of the attention map. However, these approaches typically perform coarse-grained inspection of the attention map, rendering considerable loss in model accuracy. In this paper, we propose SALE, a fine-grained sparse attention method that accelerates the long-context prefilling stage of LLM with negligible loss in model accuracy. SALE achieves fast and accurate fine-grained attention weight estimation through 4-bit quantized query-key products, followed by block-sparse attention to accelerate prefilling computations. For importance evaluation for query-key pairs, we adopt our Relative Attention Score metric, which offers significantly higher efficiency within our framework. We implement a custom CUDA kernel optimized for our approach for hardware efficiency, reducing the additional overhead to approximately 11% of the full attention latency. Notably, SALE requires no parameter training and can be seamlessly integrated into existing systems with trivial code modifications. Experiments on long-context benchmarks demonstrate that our method outperforms existing approaches in accuracy-efficiency trade-offs, achieving at least 3.36x speedups on Llama-3.1-8B for sequences longer than 64K while maintaining model quality.

cs.LG↗

arXiv2025

A High-Efficiency Reconfigurable Bidirectional Array Antenna Based on Transmit-Reflect Switchable Metasurface

This paper proposes a reconfigurable bidirectional array antenna with high-efficiency radiations and flexible beam-switching capability by employing a novel transmit-reflect switchable metasurface (TRSM). To realize the electromagnetic (EM) wave transmitted or reflected manipulation, a dedicated transmit-reflect switch layer (TRSL) with periodically soldered PIN diodes is introduced between two transmitted metasurfaces. By switching ON/OFF the embedded diodes, the TRSL performs as a mesh-type ground layer or polarization-grid layer, exhibiting a reflect or transmit property to the incident wave respectively. Further, utilizing the above TRSM configuration in conjunction with a microstrip feed antenna, bidirectional radiations are obtained at the same frequency and polarization. To further reduce the number of PIN diodes and control complexity, an enhanced TRSM using a single diode to control two unit cells is also investigated, resulting in half PIN diodes reduction. Since the bidirectional beam-switching is achieved by only controlling PIN diodes integrated in the ground plane instead of directly acting on the radiation element, which reduces insertion loss and avoids phase quantization errors, the proposed antenna can maintain a high aperture efficiency. To verify this concept, a prototype was designed, fabricated, and measured, demonstrating a successful realization of backward and forward patterns with peak gains of 22.3 and 22.1 dBi, and aperture efficiencies of 47.2% and 43.8%. The 3-dB gain bandwidths of reflected and transmitted modes are 13.7% and 12.3%. This antenna has the advantages of high gain, high aperture efficiency, simple configuration, cost-effectiveness, and flexible and digital beam control.

eess.SY↗

arXiv2025

Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelization

As the scale of models and training data continues to grow, there is an expanding reliance on more GPUs to train large-scale models, which inevitably increases the likelihood of encountering dynamic stragglers that some devices lag behind in performance occasionally. However, hybrid parallel training, one of the de facto paradigms to train large models, is typically sensitive to the stragglers. This paper presents Malleus, a straggler-resilient hybrid parallel training framework for large-scale models. Malleus quantifies the stragglers at the nuanced, per-GPU granularity during training, and develops a novel planning algorithm to deduce the optimal parallelization of GPU devices, pipeline stages, model layers, and training data, maximizing training efficiency when stragglers exist. In addition, once a shift in the straggler situation is detected, Malleus adaptively adjusts the parallelization via a re-planning process, and seamlessly and efficiently migrates the model states on the fly, without sacrificing the stability of the training tasks. Empirical results on large language models with up to 110B parameters show that Malleus consistently outperforms existing parallel training frameworks under various straggler situations, delivering on average 2.63-5.28 times of efficiency improvement.

cs.DC↗

arXiv2025

PQCache: Product Quantization-based KVCache for Long Context LLM Inference

As the field of Large Language Models (LLMs) continues to evolve, the context length in inference is steadily growing. Key-Value Cache (KVCache), the intermediate representations of tokens within LLM inference, has now become the primary memory bottleneck due to limited GPU memory. Current methods selectively determine suitable keys and values for self-attention computation in LLMs to address the issue. However, they either fall short in maintaining model quality or result in high serving latency. Drawing inspiration from advanced embedding retrieval techniques prevalent in the data management community, we consider the storage and retrieval of KVCache as a typical embedding retrieval problem. We propose PQCache, which employs Product Quantization (PQ) to manage KVCache, maintaining model quality while ensuring low serving latency. During the prefilling phase, we apply PQ to tokens' keys for each LLM layer and head. During the autoregressive decoding phase, we use PQ codes and centroids to approximately identify important preceding tokens, then fetch the corresponding key-value pairs for self-attention computation. Through meticulous design of overlapping and caching, we minimize any additional computation and communication overhead during both phases. Extensive experiments demonstrate that PQCache achieves both effectiveness and efficiency, with 4.60% score improvement over existing methods on InfiniteBench and low system latency in both prefilling and decoding.

cs.CL↗

arXiv2025

MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training

Nowadays, Large Language Models (LLMs) have been trained using extended context lengths to foster more creative applications. However, long context training poses great challenges considering the constraint of GPU memory. It not only leads to substantial activation memory consumption during training, but also incurs considerable memory fragmentation. To facilitate long context training, existing frameworks have adopted strategies such as recomputation and various forms of parallelisms. Nevertheless, these techniques rely on redundant computation or extensive communication, resulting in low Model FLOPS Utilization (MFU). In this paper, we propose MEMO, a novel LLM training framework designed for fine-grained activation memory management. Given the quadratic scaling of computation and linear scaling of memory with sequence lengths when using FlashAttention, we offload memory-consuming activations to CPU memory after each layer's forward pass and fetch them during the backward pass. To maximize the swapping of activations without hindering computation, and to avoid exhausting limited CPU memory, we implement a token-wise activation recomputation and swapping mechanism. Furthermore, we tackle the memory fragmentation issue by employing a bi-level Mixed Integer Programming (MIP) approach, optimizing memory reuse across transformer layers. Empirical results demonstrate that MEMO achieves an average of 1.97x and 1.80x MFU compared to Megatron-LM and DeepSpeed, respectively. This improvement is attributed to MEMO's ability to minimize memory fragmentation, reduce recomputation and intensive communication, and circumvent the delays associated with the memory reorganization process due to fragmentation. By leveraging fine-grained activation memory management, MEMO facilitates efficient training of 7B LLM with 1 million sequence length on just 8 A800 GPUs, achieving an MFU of 52.30%.

cs.LG↗

arXiv2024

Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling

In current deep learning tasks, Adam style optimizers such as Adam, Adagrad, RMSProp, Adafactor, and Lion have been widely used as alternatives to SGD style optimizers. These optimizers typically update model parameters using the sign of gradients, resulting in more stable convergence curves. The learning rate and the batch size are the most critical hyperparameters for optimizers, which require careful tuning to enable effective convergence. Previous research has shown that the optimal learning rate increases linearly or follows similar rules with batch size for SGD style optimizers. However, this conclusion is not applicable to Adam style optimizers. In this paper, we elucidate the connection between optimal learning rates and batch sizes for Adam style optimizers through both theoretical analysis and extensive experiments. First, we raise the scaling law between batch sizes and optimal learning rates in the sign of gradient case, in which we prove that the optimal learning rate first rises and then falls as the batch size increases. Moreover, the peak value of the surge will gradually move toward the larger batch size as training progresses. Second, we conducted experiments on various CV and NLP tasks and verified the correctness of the scaling law.

cs.LG↗

arXiv2024

Optimal UCA Design for OAM Based Wireless Backhaul Transmission

Orbital angular momentum (OAM), which is considered as a novel way to achieve high capacity, has been attracted much attention recently. OAM signals emitted by uniform circular array (UCA) are widely regarded to go through the Bessel-form channels. However, the channel gains corresponding to the Bessel-form channels are with low signal-to-noise-ratio (SNR) on OAM-modes and it is difficult to achieve high capacity using all OAM modes. To achieve maximum capacity offered by OAM multiplexing for wireless backhaul communications, in this paper we propose the optimal UCA design, which selects the optimal OAM-modes and radius of receive UCA. We formulate the capacity maximization problem and divide it into two subproblems for obtaining the corresponding optimal UCA design for OAM multiplexing based wireless backhaul communications. In particular, the optimal radius of the receive UCA is firstly derived. Then, we propose an mode selection scheme to choose appropriate OAM-modes for data transmission to maximize the capacity. Extensive simulations obtained validate that the capacity of OAM multiplexing can be significantly increased with our developed scheme.

cs.IT↗

arXiv2024

Orbital-Angular-Momentum Versus MIMO: Orthogonality, Degree of Freedom,and Capacity

The plane wave based wireless communications have becoming more and more matured, along with the well utilization of the traditional resources such as time and frequency. To further increase the capacity for rapidly increasing capacity demand of wireless communications, it is potential to use the twist wave, which has the orbital angular momentum (OAM). In this paper, we discuss the OAM based wireless communications in the aspect of orthogonality, degree of freedom (DoF), and capacity, where both the transmitter and the receiver use uniform circular array (UCA) antennas. In particular, we compare OAM based wireless communications with multiple-input-multiple-output (MIMO) based wireless communications in terms of DoF and capacity. Numerical results are presented to validate and evaluate that the DoF of OAM based wireless communications is greater than or equal to that of correlated MIMO based wireless communications when the transmitter and the receiver antennas are aligned well. The OAM based wireless communications can achieve larger capacity than the correlated MIMO in high signal-to-noise ratio (SNR) region under line-of-sight scenario.

eess.SP↗

arXiv2024

Orbital-Angular-Momentum Embedded Massive MIMO: Achieving Multiplicative Spectrum-Efficiency for mmWave Communications

By enabling very high bandwidth for radio communications, the millimeter-wave (mmWave), which can easily be integrated with massive-multiple-input-multiple-output (massive-MIMO) due to small antenna size, has been attracting growing attention as a candidate for the fifth-generation (5G) and 5G-beyond wireless communications networks. On the other hand, the communication over the orthogonal states/modes of orbital angular momentum (OAM) is a subset of the solutions offered by massive-MIMO communications. Traditional massive-MIMO based mmWave communications did not concern the potential spectrum-efficiency-gain (SE-gain) offered by orthogonal states of OAM. However, the highly expecting maximum SE-gain for OAM and massive-MIMO communications is the product of SE-gains offered by OAM and multiplexing-MIMO. In this paper, we propose the OAM-embedded-MIMO (OEM) communication framework to obtain the multiplicative SE-gain for joint OAM and massive-MIMO based mmWave wireless communications. We design the parabolic antenna for each uniform circular array antenna to converge OAM signals. Then, we develop the mode-decomposition and multiplexing-detection scheme to obtain the transmit signal on each OAM-mode of each transmit antenna. Also, we develop the OEM-water-filling power allocation policy to achieve the maximum multiplicative SE-gain for OEM communications. The extensive simulations obtained validate and evaluate our developed parabolic antenna based converging method, mode-decomposition and multiplexing-detection scheme, and OEM-water-filling policy, showing that our proposed OEM mmWave communications can significantly increase the spectrum-efficiency as compared with traditional massive-MIMO based mmWave communications.

eess.SP↗

arXiv2024

Achieving Practical OAM Based Wireless Communications With Misaligned Transceiver

Orbital angular momentum (OAM) has attracted much attention for radio vortex wireless communications due to the orthogonality among different OAM-modes. To maintain the orthogonality among different OAM modes at the receiver, the strict alignment between transmit and receive antennas is highly demanded. However, it is not practical to guarantee the transceiver alignment in wireless communications. The phase turbulence, resulting from the misaligned transceivers, leads to serious inter-mode interference among different OAM modes and therefore fail for signals detection of multiple OAM modes at the receiver. To achieve practical OAM based wireless communications, in this paper we investigate the radio vortex wireless communications with misaligned transmit and receive antennas. We propose a joint Beamforming and Pre-detection (BePre) scheme, which uses two unitary matrices to convert the channel matrix into the equivalent circulant matrix for keeping the orthogonality among OAM-modes at the receiver. Then, the OAM signals can be detected with the mode-decomposition scheme at the misaligned receiver. Extensive simulations obtained validate and evaluate that our developed joint BePre scheme can efficiently detect the signals of multiple OAM-modes for the misaligned transceiver and can significantly increase the spectrum efficiency.

eess.SP↗

arXiv2024

Concentric UCAs Based Low-Order OAM for High Capacity in Radio Vortex Wireless Communications

Due to the potential capacity-boosting for wireless communications, the Radio vOrtex Wireless COMMunication (RowComm) over orthogonal states/modes of Orbital Angular Momentum (OAM) has been paid much attention in recent years. Uniform circular array (UCA), as an efficient and convenient antenna structure, can transmit/receive multiple OAM beams with different OAM-modes simultaneously when the transmitter and receiver are aligned. However, for high-order OAM-modes, the OAM beams are divergent accompanied by severe attenuations. Thus, it is difficult to directly use high-order OAM-modes to achieve high capacity for RowComms. To obtain high capacity potentially offered by OAM-modes, in this paper we transform the singular UCA into the concentric UCAs, where high capacity can be achieved using multiple parallel low-order OAM-modes instead of all high-order OAM-modes, to increase the capacity of transmitter-receiver aligned RowComms. In particular, we study two cases: 1). Concentric UCAs based RowComms without co-mode interference; 2). Concentric UCAs based RowComms with co-mode interference. We propose a co-mode-interference-free and a co-mode-interference-contained mode-decomposition schemes to recover transmit signals for these two cases, respectively. Also, we develop the optimal power allocation schemes to maximize the capacity for these two cases. Numerical simulations are presented to validate and evaluate that our developed concentric UCAs based low-order RowComms can significantly increase the capacity as compared with that of singular UCA based RowComms.

eess.SP↗