arXiv subjects

Chen Zhang

Publications and source records attributed to Chen Zhang.

At least 235 records · Page 13Linked to original sources

arXiv2025

AudioGen-Omni: A Unified Multimodal Diffusion Transformer for Video-Synchronized Audio, Speech, and Song Generation

We present AudioGen-Omni - a unified approach based on multimodal diffusion transformers (MMDit), capable of generating high-fidelity audio, speech, and song coherently synchronized with the input video. AudioGen-Omni introduces a novel joint training paradigm that seamlessly integrates large-scale video-text-audio corpora, enabling a model capable of generating semantically rich, acoustically diverse audio conditioned on multimodal inputs and adaptable to a wide range of audio generation tasks. AudioGen-Omni employs a unified lyrics-transcription encoder that encodes graphemes and phonemes from both song and spoken inputs into dense frame-level representations. Dense frame-level representations are fused using an AdaLN-based joint attention mechanism enhanced with phase-aligned anisotropic positional infusion (PAAPI), wherein RoPE is selectively applied to temporally structured modalities to ensure precise and robust cross-modal alignment. By unfreezing all modalities and masking missing inputs, AudioGen-Omni mitigates the semantic constraints of text-frozen paradigms, enabling effective cross-modal conditioning. This joint training approach enhances audio quality, semantic alignment, and lip-sync accuracy, while also achieving state-of-the-art results on Text-to-Audio/Speech/Song tasks. With an inference time of 1.91 seconds for 8 seconds of audio, it offers substantial improvements in both efficiency and generality.

cs.SD↗

arXiv2025

A Design for an Early Quantum Network

With the rapid advancement of quantum information technology, quantum networks have become essential for supporting diverse applications, which often have stringent demands for key metrics such as fidelity and request completion time. In this work, we propose a design for early-stage quantum networks that is compatible with the three existing quantum repeater technologies. The design aims to maximize the ability of the network to accommodate the diverse needs of quantum applications, even under conditions of limited quantum resources and suboptimal network performance. We have also described the required identifiers in the quantum network and the specific process for implementing quantum requests. To assess the feasibility of our design, we conduct simulations based on discrete-event modeling of quantum networks. The simulations consider various types of noise and imperfect parameters that might exist in early-stage networks. We analyze the impact of these parameters on the fidelity of the generated entangled states and the request completion time. Furthermore, we investigated additional decisions that the central controller can make beyond path selection, such as the choice of cutoff time and the allocation of network resources to requests.

quant-ph↗

arXiv2025

HumanDiT: Pose-Guided Diffusion Transformer for Long-form Human Motion Video Generation

Human motion video generation has advanced significantly, while existing methods still struggle with accurately rendering detailed body parts like hands and faces, especially in long sequences and intricate motions. Current approaches also rely on fixed resolution and struggle to maintain visual consistency. To address these limitations, we propose HumanDiT, a pose-guided Diffusion Transformer (DiT)-based framework trained on a large and wild dataset containing 14,000 hours of high-quality video to produce high-fidelity videos with fine-grained body rendering. Specifically, (i) HumanDiT, built on DiT, supports numerous video resolutions and variable sequence lengths, facilitating learning for long-sequence video generation; (ii) we introduce a prefix-latent reference strategy to maintain personalized characteristics across extended sequences. Furthermore, during inference, HumanDiT leverages Keypoint-DiT to generate subsequent pose sequences, facilitating video continuation from static images or existing videos. It also utilizes a Pose Adapter to enable pose transfer with given sequences. Extensive experiments demonstrate its superior performance in generating long-form, pose-accurate videos across diverse scenarios.

cs.CV↗

arXiv2025

ForgeHLS: A Large-Scale, Open-Source Dataset for High-Level Synthesis

High-Level Synthesis (HLS) plays a crucial role in modern hardware design by transforming high-level code into optimized hardware implementations. However, progress in applying machine learning (ML) to HLS optimization has been hindered by a shortage of sufficiently large and diverse datasets. To bridge this gap, we introduce ForgeHLS, a large-scale, open-source dataset explicitly designed for ML-driven HLS research. ForgeHLS comprises over 400k diverse designs generated from 846 kernels covering a broad range of application domains, consuming over 200k CPU hours during dataset construction. Each kernel includes systematically automated pragma insertions (loop unrolling, pipelining, array partitioning), combined with extensive design space exploration using Bayesian optimization. Compared to existing datasets, ForgeHLS significantly enhances scale, diversity, and design coverage. We further define and evaluate representative downstream tasks in Quality of Result (QoR) prediction and automated pragma exploration, clearly demonstrating ForgeHLS utility for developing and improving ML-based HLS optimization methodologies. The dataset and code are public at https://github.com/zedong-peng/ForgeHLS.

cs.AR↗

arXiv2025

SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codec

Speech codecs serve as a crucial bridge in unifying speech and text language models. Existing codec methods face several challenges in semantic encoding, such as residual paralinguistic information (e.g., timbre, emotion), insufficient semantic completeness, limited reconstruction capability, and lack of support for streaming. To address these challenges, we propose SecoustiCodec, a cross-modal aligned low-bitrate streaming speech codec that disentangles semantic and paralinguistic information in a single-codebook space. To ensure semantic completeness and reconstruction fidelity, paralinguistic encoding is introduced to bridge the information gap between semantic and acoustic encoding. A semantic-only efficient quantization method based on VAE (Variational Autoencoder) and FSQ (Finite Scalar Quantization) is proposed. This approach alleviates the long-tail distribution problem of tokens while maintaining high codebook utilization. A semantic disentanglement method based on contrastive learning is proposed, which aligns text and speech in a joint multimodal frame-level space, effectively removing paralinguistic information from semantic encoding. An acoustic-constrained multi-stage optimization strategy is proposed to ensure robust and stable convergence. Figure~\ref{fig:pesq_kbps_below_2kbps} shows SecoustiCodec achieves SOTA (state-of-the-art) reconstruction quality (PESQ) of 1.77/2.58 at 0.27/1 kbps. The code and model weights for SecoustiCodec will be open-sourced upon the completion of the peer-review process. We've open-sourced SecoustiCodec's demo, code, and model weights.

eess.AS↗

arXiv2025

Indication for a compact object next to a LIGO-Virgo binary black hole merger

The astrophysical origin of binary black hole (BBH) mergers remains uncertain, although many events have been observed by the LIGO-Virgo-KAGRA network. Such mergers are potentially originated in the vicinity of massive black holes (MBHs). GW190814, due to its secondary mass and mass ratio being beyond the expectations of isolated stellar evolution theories, is a promising event that has occurred in an active galactic nucleus (AGN) disk. In this model, a compact object resides in the vicinity of a merging BBH. Here we report multiple pieces of evidence suggesting that GW190814 is a BBH merging near a compact object. The orbital motion of BBHs around a third body produces a line-of-sight acceleration (LSA) and induces a varying Doppler shift. Using a waveform template that considers LSA, we perform Bayesian inference on a few BBH events with a high signal-to-noise ratio in the gravitational-wave (GW) transient catalog. Compared to the model for isolated BBH mergers, we obtain significantly higher network signal-to-noise ratios for GW190814 with the inclusion of LSA, constraining the LSA to $a = 0.0015^{+0.0008}_{-0.0008} ~c~\mathrm{s}^{-1}$ at a $90 \%$ confidence level. Additionally, the Bayes factor for the LSA case over the isolated case is $58/1$, indicating that the GW data strongly prefer the LSA model. We conclude that this is the first indication showing merging BBHs are located near a third compact object.

astro-ph.HE↗

arXiv2025

Towards the Law of Capacity Gap in Distilling Language Models

Language model (LM) distillation aims at distilling the knowledge in a large teacher LM to a small student one. As a critical issue facing LM distillation, a superior student often arises from a teacher of a relatively small scale instead of a larger one, especially in the presence of substantial capacity gap between the teacher and student. This issue, often referred to as the \textit{curse of capacity gap}, suggests that there is likely an optimal teacher yielding the best-performing student along the scaling course of the teacher. Consequently, distillation trials on teachers of a wide range of scales are called for to determine the optimal teacher, which becomes computationally intensive in the context of large LMs (LLMs). This paper addresses this critical bottleneck by providing the \textit{law of capacity gap} inducted from a preliminary study on distilling a broad range of small-scale (<3B) LMs, where the optimal teacher consistently scales linearly with the student scale across different model and data scales. By extending the law to LLM distillation on a larger scale (7B), we succeed in obtaining versatile LLMs that outperform a wide array of competitors.

cs.CL↗

arXiv2025

Hypernetworks for Model-Heterogeneous Personalized Federated Learning

Recent advances in personalized federated learning have focused on addressing client model heterogeneity. However, most existing methods still require external data, rely on model decoupling, or adopt partial learning strategies, which can limit their practicality and scalability. In this paper, we revisit hypernetwork-based methods and leverage their strong generalization capabilities to design a simple yet effective framework for heterogeneous personalized federated learning. Specifically, we propose MH-pFedHN, which leverages a server-side hypernetwork that takes client-specific embedding vectors as input and outputs personalized parameters tailored to each client's heterogeneous model. To promote knowledge sharing and reduce computation, we introduce a multi-head structure within the hypernetwork, allowing clients with similar model sizes to share heads. Furthermore, we further propose MH-pFedHNGD, which integrates an optional lightweight global model to improve generalization. Our framework does not rely on external datasets and does not require disclosure of client model architectures, thereby offering enhanced privacy and flexibility. Extensive experiments on multiple benchmarks and model settings demonstrate that our approach achieves competitive accuracy, strong generalization, and serves as a robust baseline for future research in model-heterogeneous personalized federated learning.

cs.LG↗

arXiv2025

TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput

Large Language Model (LLM) serving systems batch concurrent user requests to achieve efficient serving. However, in real-world deployments, such inter-request parallelism from batching is often limited by external factors such as low request rates or memory constraints. Recent works focus on intra-request parallelism from speculative decoding as a solution to this problem. Unfortunately, benefits from intra-request parallelism are often fragile, as speculative decoding causes overhead, and speculated tokens may miss. We observe that speculative decoding may degrade LLM serving performance if added naively without tuning to the incoming requests and the speculation method. To alleviate the need for expert tuning and make speculative decoding more robust, we present TurboSpec, a speculation control system that automatically profiles the execution environment and utilizes a feedback-based algorithm to dynamically adjust the amount of intra-request parallelism in LLM serving. TurboSpec predicts "goodput" - the amount of successfully generated tokens - to evaluate and adjust intra-request parallelism amount to that with the highest goodput in runtime. We implement TurboSpec on a real-world LLM serving system vLLM and demonstrate its effectiveness across diverse workloads and hardware configurations, providing consistent performance improvements across all test scenarios.

cs.AI↗

arXiv2025

Asymmetric real topology of conduction and valence bands

Previously, it was believed that conduction and valence bands exhibit a symmetry: They possess opposite topological invariants (e.g., the Chern numbers of conduction and valence bands for the Chern insulator are $\pm C$). However, we present a counterexample: The second Stiefel-Whitney numbers for conduction and valence bands over the Klein bottle may be asymmetric, with one being nontrivial while the other trivial. Here, the Stiefel-Whitney classes are the characteristic classes for real Bloch functions under $PT$ symmetry with $(PT)^2=1$, and the Klein bottle is the momentum-space unit under the projective anticommutation relation of the mirror reflection reversing $x$ and the translation along the $y$ direction. The asymmetry originates from the algebraic difference of real cohomology classes over the Klein bottle and torus. This discovery is rooted in the foundation of topological band theory, and has the potential to fundamentally refresh our current understanding of topological phases.

cond-mat.mes-hall↗

arXiv2025

AI-Based Impedance Encoding-Decoding Method for Online Impedance Network Construction of Wind Farms

The impedance network (IN) model is gaining popularity in the oscillation analysis of wind farms. However, the construction of such an IN model requires impedance curves of each wind turbine under their respective operating conditions, making its online application difficult due to the transmission of numerous high-density impedance curves. To address this issue, this paper proposes an AI-based impedance encoding-decoding method to facilitate the online construction of IN model. First, an impedance encoder is trained to compress impedance curves by setting the number of neurons much smaller than that of frequency points. Then, the compressed data of each turbine are uploaded to the wind farm and an impedance decoder is trained to reconstruct original impedance curves. At last, based on the nodal admittance matrix (NAM) method, the IN model of the wind farm can be obtained. The proposed method is validated via model training and real-time simulations, demonstrating that the encoded impedance vectors enable fast transmission and accurate reconstruction of the original impedance curves.

eess.SP↗

arXiv2025

Unraveling Magneto-Phononic Coupling and Photoinduced Magnetic Control in Antiferromagnetic Kondo Semimetal CeBi

We report an ultrafast optical spectroscopy study on the coherent phonon dynamics in the topological semimetal CeBi and its nonmagnetic isostructural compound LaBi, revealing profound insights into their electronic and magnetic interactions. Both materials exhibit prominent $A_{1g}$ longitudinal optical phonons with characteristic anharmonic temperature dependencies. However, in CeBi, the $A_{1g}$ phonon frequency and amplitude show clear anomalies near its antiferromagnetic (AFM) ordering temperatures ($T_{N1}$ $\simeq$ 25 K and $T_{N2}$ $\simeq$ 12 K), which unequivocally demonstrate strong magneto-phononic coupling. Crucially, in the AFM state at 4 K, CeBi exhibits a pump fluence threshold of $F_C$ $\approx$ 44 $μ$J/cm$^2$, above which the rate of phonon softening accelerates and the amplitude increases sharply. This unique threshold, absent in paramagnetic CeBi and LaBi, points to a photoinduced, non-thermal quenching of the AFM order. Our findings establish coherent phonons as highly sensitive probes of intertwined orders in heavy fermion systems, highlighting the transformative potential of ultrafast pulses in dynamically controlling magnetic states in correlated electron materials and paving the way for the manipulation of emergent quantum phases.

cond-mat.str-el↗

arXiv2025

Beyond Connectivity: Higher-Order Network Framework for Capturing Memory-Driven Mobility Dynamics

Understanding and predicting mobility dynamics in transportation networks is critical for infrastructure planning, resilience analysis, and traffic management. Traditional graph-based models typically assume memoryless movement, limiting their ability to capture sequential dependencies inherent in real-world mobility patterns. In this study, we introduce a novel higher-order network framework for modeling memory-dependent dynamics in transportation systems. By extending classical graph representations through higher-order Markov chains and de Bruijn graph structures, our framework encodes the spatial and temporal ordering of traversed paths, enabling the analysis of structurally and functionally critical components with improved fidelity. We generalize key network analytics, including betweenness centrality, PageRank, and next-step prediction, to this higher-order setting and validate our approach on the Sioux Falls transportation network using agent-based trajectory data generated with MATSim. Experimental results demonstrate that higher-order models outperform first-order baselines across multiple tasks, with the third-order model achieving an optimal balance between predictive accuracy and model complexity. These findings highlight the importance of incorporating memory effects into network-based transportation analysis and offer a scalable, data-driven methodology for capturing complex mobility behaviors in infrastructure systems.

cs.SI↗

arXiv2025

Seamless Interaction: Dyadic Audiovisual Motion Modeling and Large-Scale Dataset

Human communication involves a complex interplay of verbal and nonverbal signals, essential for conveying meaning and achieving interpersonal goals. To develop socially intelligent AI technologies, it is crucial to develop models that can both comprehend and generate dyadic behavioral dynamics. To this end, we introduce the Seamless Interaction Dataset, a large-scale collection of over 4,000 hours of face-to-face interaction footage from over 4,000 participants in diverse contexts. This dataset enables the development of AI technologies that understand dyadic embodied dynamics, unlocking breakthroughs in virtual agents, telepresence experiences, and multimodal content analysis tools. We also develop a suite of models that utilize the dataset to generate dyadic motion gestures and facial expressions aligned with human speech. These models can take as input both the speech and visual behavior of their interlocutors. We present a variant with speech from an LLM model and integrations with 2D and 3D rendering methods, bringing us closer to interactive virtual agents. Additionally, we describe controllable variants of our motion models that can adapt emotional responses and expressivity levels, as well as generating more semantically-relevant gestures. Finally, we discuss methods for assessing the quality of these dyadic motion models, which are demonstrating the potential for more intuitive and responsive human-AI interactions.

cs.CV↗

arXiv2025

Mathematical Computation on High-dimensional Data via Array Programming and Parallel Acceleration

While deep learning excels in natural image and language processing, its application to high-dimensional data faces computational challenges due to the dimensionality curse. Current large-scale data tools focus on business-oriented descriptive statistics, lacking mathematical statistics support for advanced analysis. We propose a parallel computation architecture based on space completeness, decomposing high-dimensional data into dimension-independent structures for distributed processing. This framework enables seamless integration of data mining and parallel-optimized machine learning methods, supporting scientific computations across diverse data types like medical and natural images within a unified system.

cs.LG↗

arXiv2025

In-flight calibration of the Lobster Eye Imager for Astronomy

The Lobster Eye Imager for Astronomy (LEIA), as a pathfinder of the Wide-field X-ray Telescope (WXT) onboard the Einstein Probe (EP) satellite, is the first lobster-eye focusing X-ray telescope with a considerably large field-of-view (FoV) ever flown. During the two and half years of operations, a series of calibration observations were performed, to fully characterize its performance and calibrate the instrumental properties. In this paper, we present the results of the in-flight calibration campaign of LEIA, focusing on the properties of the PSF, source positional accuracy, effective area, energy response and the instrumental background. The calibration sources used are the Crab nebula, Sco X-1 and Cassiopeia A supernova remnant. Specifically, it is found that the spatial resolution remains almost unchanged compared to the pre-launch values, ranging from 3.6'-9.3' with a median of 5.9'. The post-calibration source positional accuracy is found to be ~2' (at the 90% C.L.). The Crab spectra can be well reproduced by the absorbed power-law model with the best-fit parameters in large agreement with the literature values, indicating that the in-orbit effective area is overall consistent with the model predictions and ground measurements. The effective area exhibits a systematic of $\lesssim10\%$ (at the 68% C.L.), and a mild deterioration of ~15% at the lower energy end after one year of operation. The Cas A spectral analysis shows that the energy scale and spectral resolution of the detectors are generally consistent with ground values. The instrumental background is found to be largely consistent among the four detectors, with strong modulations by the geomagnetic activity and the spectrum qualitatively consistent with our previous simulations. These instrumental performances well meet the design requirements. This work paves the way for the in-orbit calibration of the EP-WXT.

astro-ph.IM↗

arXiv2025

Commonsense Generation and Evaluation for Dialogue Systems using Large Language Models

This paper provides preliminary results on exploring the task of performing turn-level data augmentation for dialogue system based on different types of commonsense relationships, and the automatic evaluation of the generated synthetic turns. The proposed methodology takes advantage of the extended knowledge and zero-shot capabilities of pretrained Large Language Models (LLMs) to follow instructions, understand contextual information, and their commonsense reasoning capabilities. The approach draws inspiration from methodologies like Chain-of-Thought (CoT), applied more explicitly to the task of prompt-based generation for dialogue-based data augmentation conditioned on commonsense attributes, and the automatic evaluation of the generated dialogues. To assess the effectiveness of the proposed approach, first we extracted 200 randomly selected partial dialogues, from 5 different well-known dialogue datasets, and generate alternative responses conditioned on different event commonsense attributes. This novel dataset allows us to measure the proficiency of LLMs in generating contextually relevant commonsense knowledge, particularly up to 12 different specific ATOMIC [10] database relations. Secondly, we propose an evaluation framework to automatically detect the quality of the generated dataset inspired by the ACCENT [26] metric, which offers a nuanced approach to assess event commonsense. However, our method does not follow ACCENT's complex eventrelation tuple extraction process. Instead, we propose an instruction-based prompt for each commonsense attribute and use state-of-the-art LLMs to automatically detect the original attributes used when creating each augmented turn in the previous step. Preliminary results suggest that our approach effectively harnesses LLMs capabilities for commonsense reasoning and evaluation in dialogue systems.

cs.CL↗

arXiv2025

Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation

We propose Kling-Foley, a large-scale multimodal Video-to-Audio generation model that synthesizes high-quality audio synchronized with video content. In Kling-Foley, we introduce multimodal diffusion transformers to model the interactions between video, audio, and text modalities, and combine it with a visual semantic representation module and an audio-visual synchronization module to enhance alignment capabilities. Specifically, these modules align video conditions with latent audio elements at the frame level, thereby improving semantic alignment and audio-visual synchronization. Together with text conditions, this integrated approach enables precise generation of video-matching sound effects. In addition, we propose a universal latent audio codec that can achieve high-quality modeling in various scenarios such as sound effects, speech, singing, and music. We employ a stereo rendering method that imbues synthesized audio with a spatial presence. At the same time, in order to make up for the incomplete types and annotations of the open-source benchmark, we also open-source an industrial-level benchmark Kling-Audio-Eval. Our experiments show that Kling-Foley trained with the flow matching objective achieves new audio-visual SOTA performance among public models in terms of distribution matching, semantic alignment, temporal alignment and audio quality.

eess.AS↗