arXiv subjects

Ziqin Zhou

Publications and source records attributed to Ziqin Zhou.

At least 19 recordsLinked to original sources

arXiv2026

MTAVG-Bench 2.0: Diagnosing Failure Modes of Cinematic Expressiveness in Multi-Talker Audio-Video Generation

In recent years, Multi-Talker Audio-Video Generation (MTAVG) models have shown promising performance on fundamental metrics such as lip-sync and audio-visual alignment. However, these metrics remain insufficient for assessing cinematic expressiveness in scene-level generation. In multi-character scenes, generation models must go beyond audio-visual realism to convey coherent character performance and other higher-level cinematic qualities. To fill this gap, we introduce MTAVG-Bench 2.0, a benchmark for diagnosing failure modes of cinematic expressiveness in multi-talker audio-video generation. Unlike prior settings that mainly focus on the quality of basic multi-turn dialogue, MTAVG-Bench 2.0 targets short-drama and scene-level generation, and establishes a high-level failure taxonomy spanning acting, narrative, atmosphere, and audio-visual language. Based on this taxonomy, we construct more than 10,000 question-answering evaluation instances, together with subsets for short-drama-level assessment and temporal localization of failure modes, to systematically evaluate the ability of omni large language models to diagnose high-level audio-visual failures. Experimental results show that commercial omni models such as Gemini substantially outperform other evaluators, yet even the strongest models continue to struggle with complex failures in our benchmark. These results demonstrate that MTAVG-Bench 2.0 provides a systematic benchmark for failure diagnosis in cinematic multi-talker audio-video generation.

cs.AI↗

arXiv2026

A Numerical PDEs Approach to Evolution Equations in Shape Analysis Based on Regularized Morphoelasticity

This work studies a variational formulation and numerical solution of a regularized morphoelasticity problem of shape evolution. The foundation of our analysis is based on the governing equations of linear elasticity, extended to account for volumetric growth. In the morphoelastic framework, the total deformation is decomposed into an elastic component and a growth component, represented by a growth tensor $G$. While the forward one-step problem -- computing displacement given a growth tensor -- is well-established, a more challenging and relevant question in biological modeling is the inverse problem in a continuous sense. While this problem is fundamentally ill-posed without additional constraints, we will explore parametrized growth models inscribed within an optimal control problem inspired by the Large Deformation Diffeomorphic Metric Mapping (LDDMM) framework. By treating the growth process as a path within a shape space, we can define a physically meaningful metric and seek the most plausible, energy-efficient trajectory between configurations. In the construction, a high-order regularization term is introduced. This elevates the governing equations to a high-order elliptic system, ensuring the existence of a smooth solution. This dissertation focuses on the issue of solving this equation efficiently, as this is a key requirement for the feasibility of the overall approach. This will be achieved with the help of finite element solvers, notably from the FEniCSx library in Python. Also, we implement a Mixed Finite Element Method, which decomposes the problem into a system of coupled second-order equations as a treatment of these high-order systems that have significant computational challenges.

math.NA↗

arXiv2026

MTAVG-Bench: A Diagnostic Benchmark for Multi-Talker Dialogue-Centric Audio-Video Generation

Recent advances in text-to-audio-video (T2AV) generation have enabled models to synthesize audio-visual videos with multi-participant dialogues. However, existing evaluation benchmarks remain largely designed for human-recorded videos or single-speaker settings. As a result, structural failures in generated multi-talker dialogue videos, such as identity drift, unnatural turn transitions, and audio-visual misalignment, cannot be effectively diagnosed. To address this issue, we introduce MTAVG-Bench, a failure-driven diagnostic benchmark for multi-talker dialogue-centric audio-video generation. MTAVG-Bench is built via a semi-automatic pipeline, where 1.8k videos are generated using mainstream T2AV models with carefully designed prompts, yielding 2.4k manually annotated QA pairs for fine-grained failure diagnosis. The benchmark evaluates multi-speaker dialogue generation at four levels: audio-visual signal fidelity, temporal attribute consistency, social interaction, and cinematic expression. Built on a hierarchical failure taxonomy and a targeted QA protocol, MTAVG-Bench is primarily designed to evaluate whether proprietary and open-source omni-models can reliably identify failure modes in multi-speaker T2AV outputs. We benchmark 12 proprietary and open-source omni-models on MTAVG-Bench, with Gemini 3 Pro achieving the strongest overall performance, while leading open-source models remain competitive in signal fidelity and consistency. Overall, MTAVG-Bench enables fine-grained failure analysis for rigorous model comparison and targeted video generation refinement.

cs.MM↗

arXiv2026

Training-free Motion Factorization for Compositional Video Generation

Compositional video generation aims to synthesize multiple instances with diverse appearance and motion. However, current approaches mainly focus on binding semantics, neglecting to understand diverse motion categories specified in prompts. In this paper, we propose a motion factorization framework that decomposes complex motion into three primary categories: motionlessness, rigid motion, and non-rigid motion. Specifically, our framework follows a planning before generation paradigm. (1) During planning, we reason about motion laws on the motion graph to obtain frame-wise changes in the shape and position of each instance. This alleviates semantic ambiguities in the user prompt by organizing it into a structured representation of instances and their interactions. (2) During generation, we modulate the synthesis of distinct motion categories in a disentangled manner. Conditioned on the motion cues, guidance branches stabilize appearance in motionless regions, preserve rigid-body geometry, and regularize local non-rigid deformations. Crucially, our two modules are model-agnostic, which can be seamlessly incorporated into various diffusion model architectures. Extensive experiments demonstrate that our framework achieves impressive performance in motion synthesis on real-world benchmarks. Code is available at https://github.com/ZixuanWang0525/MF-CVG.

cs.CV↗

arXiv2026

Dipolar order mapping based on spin-lock magnetic resonance imaging

Purpose: Inhomogeneous magnetization transfer (ihMT) effect reflects dipolar order with a dipolar relaxation time ($T_{1D}$), specific to motion-restricted macromolecules. We aim to quantify $T_{1D}$ using spin-lock MRI technique. Methods: In the proposed method, we introduce a $T_{1D}$-specific ratio, denoted as $RATIO_{dosl}$. This ratio is derived from the distinct relaxation rate $R_{dosl}$, calculated as the difference between dual-frequency relaxation $R_{1ρ}^{dual}$ and single-frequency $R_{1ρ}^{single}$ relaxation measurements. A novel rotary-echo spin-lock sequence was developed to enable dual-frequency spin-lock acquisition. We established a framework to estimate $T_{1D}$, as well as the macromolecular pool fraction (MPF) map. The proposed approach was validated via numerical simulations, phantom studies, and demonstrated in vivo in human white matter. Results: Simulations revealed the high sensitivity of $RATIO_{dosl}$ to $T_{1D}$, and substantiated the accuracy and robustness of the proposed methods. Phantom experiments demonstrated robust ihMT contrast and confirmed the capability of $T_{1D}$ quantification via $RATIO_{dosl}$. In vivo studies supported the clinical viability of this approcah, achieving simultaneous $T_{1D}$ and MPF mapping using only three spin-lock prepared images. Across ten healthy volunteers, the mean white matter $T_{1D}$ ranged from approximately 3.70 to 4.80 ms. Conclustion: We propose a novel method for $T_{1D}$ quantification based on spin-lock MRI. By requiring only three contrast-prepared images, this technique provides a promising pathway for robust, rapid, and simultaneous $T_{1D}$ and MPF quantification with fewer confounds

physics.med-ph↗

arXiv2025

Digital Twin-Assisted High-Precision Massive MIMO Localization in Urban Canyons

High-precision wireless localization in urban canyons is challenged by noisy measurements and severe non-line-of-sight (NLOS) propagation. This paper proposes a robust three-stage algorithm synergizing a digital twin (DT) model with the random sample consensus (RANSAC) algorithm to overcome these limitations. The method leverages the DT for geometric path association and employs RANSAC to identify reliable line-of-sight (LOS) and single-bounce NLOS paths while rejecting multi-bounce outliers. A final optimization on the resulting inlier set estimates the user's position and clock bias. Simulations validate that by effectively turning NLOS paths into valuable geometric information via the DT, the approach enables accurate localization, reduces reliance on direct LOS, and significantly lowers system deployment costs, making it suitable for practical deployment.

eess.SP↗

arXiv2025

Cutting the Skip: Training Residual-Free Transformers

Transformers have achieved remarkable success across a wide range of applications, a feat often attributed to their scalability. Yet training them without skip (residual) connections remains notoriously difficult. While skips stabilize optimization, they also disrupt the hierarchical structure of representations, raising the long-standing question of whether transformers can be trained efficiently without them. In this work, we address this problem by analyzing the Jacobian of a skipless transformer block, showing why skips improve conditioning and revealing that their stabilization benefits can be recovered through a principled initialization strategy. Building on this insight, we introduce the first method that enables stable and efficient training of skipless transformers without altering the standard architecture. We validate our approach on Vision Transformers (ViTs) in both supervised and self-supervised settings, demonstrating that skipless ViTs trained with our initialization overcome the usual optimization barriers, learn richer hierarchical representations, and outperform strong baselines, that incorporate skip connections, on dense prediction benchmarks. These results show that skip connections are not a fundamental requirement for training ViTs and open new avenues for hierarchical representation learning in vision models.

cs.LG↗

arXiv2025

EZIGen: Enhancing zero-shot personalized image generation with precise subject encoding and decoupled guidance

Zero-shot personalized image generation models aim to produce images that align with both a given text prompt and subject image, requiring the model to incorporate both sources of guidance. Existing methods often struggle to capture fine-grained subject details and frequently prioritize one form of guidance over the other, resulting in suboptimal subject encoding and imbalanced generation. In this study, we uncover key insights into overcoming such drawbacks, notably that 1) the choice of the subject image encoder critically influences subject identity preservation and training efficiency, and 2) the text and subject guidance should take effect at different denoising stages. Building on these insights, we introduce a new approach, EZIGen, that employs two main components: leveraging a fixed pre-trained Diffusion UNet itself as subject encoder, following a process that balances the two guidances by separating their dominance stage and revisiting certain time steps to bootstrap subject transfer quality. Through these two components, EZIGen, initially built upon SD2.1-base, achieved state-of-the-art performances on multiple personalized generation benchmarks with a unified model, while using 100 times less training data. Moreover, by further migrating our design to SDXL, EZIGen is proven to be a versatile model-agnostic solution for personalized generation. Demo Page: zichengduan.github.io/pages/EZIGen/index.html

cs.CV↗

arXiv2025

HiTVideo: Hierarchical Tokenizers for Enhancing Text-to-Video Generation with Autoregressive Large Language Models

Text-to-video generation poses significant challenges due to the inherent complexity of video data, which spans both temporal and spatial dimensions. It introduces additional redundancy, abrupt variations, and a domain gap between language and vision tokens while generation. Addressing these challenges requires an effective video tokenizer that can efficiently encode video data while preserving essential semantic and spatiotemporal information, serving as a critical bridge between text and vision. Inspired by the observation in VQ-VAE-2 and workflows of traditional animation, we propose HiTVideo for text-to-video generation with hierarchical tokenizers. It utilizes a 3D causal VAE with a multi-layer discrete token framework, encoding video content into hierarchically structured codebooks. Higher layers capture semantic information with higher compression, while lower layers focus on fine-grained spatiotemporal details, striking a balance between compression efficiency and reconstruction quality. Our approach efficiently encodes longer video sequences (e.g., 8 seconds, 64 frames), reducing bits per pixel (bpp) by approximately 70\% compared to baseline tokenizers, while maintaining competitive reconstruction quality. We explore the trade-offs between compression and reconstruction, while emphasizing the advantages of high-compressed semantic tokens in text-to-video tasks. HiTVideo aims to address the potential limitations of existing video tokenizers in text-to-video generation tasks, striving for higher compression ratios and simplify LLMs modeling under language guidance, offering a scalable and promising framework for advancing text to video generation. Demo page: https://ziqinzhou66.github.io/project/HiTVideo.

cs.CV↗

arXiv2024

Orientation independent quantification of macromolecular proton fraction in tissues with suppression of residual dipolar coupling

Quantitative magnetization transfer (MT) imaging enables non-invasive characterization of the macromolecular environment of tissues. However, recent work has highlighted that the quantification of MT parameters exhibits orientation dependence in ordered tissue structures, potentially confounding its clinical applications. Notably, in tissues with ordered structures, such as articular cartilage and myelin, the residual dipolar coupling (RDC) effect can arise owing to incomplete averaging of dipolar-dipolar interactions of water protons. In this study, we demonstrated the confounding effect of RDC on quantitative MT imaging in ordered tissues can be suppressed by using an emerging technique known as macromolecular proton fraction mapping based on spin-lock (MPF-SL). The off-resonance spin-lock pulse in MPF-SL could be designed to generate a strong effective spin-lock field to suppress RDC without violating the specific absorption rate and hardware limitations in clinical scans. Furthermore, removing the water signal in MPF-SL enabled the application of a strong effective spin-lock field without any confounding signal from direct water saturation. Our findings were experimentally validated using human knee specimens and healthy human cartilage. The results demonstrated that MPF-SL exhibits lower sensitivity to tissue orientation compared with R2, R1rho, and saturation-pulse-based MT imaging. Thus, MPF-SL could serve as a valuable orientation-independent technique for quantifying MPF.

physics.med-ph↗

arXiv2024

MuseBarControl: Enhancing Fine-Grained Control in Symbolic Music Generation through Pre-Training and Counterfactual Loss

Automatically generating symbolic music-music scores tailored to specific human needs-can be highly beneficial for musicians and enthusiasts. Recent studies have shown promising results using extensive datasets and advanced transformer architectures. However, these state-of-the-art models generally offer only basic control over aspects like tempo and style for the entire composition, lacking the ability to manage finer details, such as control at the level of individual bars. While fine-tuning a pre-trained symbolic music generation model might seem like a straightforward method for achieving this finer control, our research indicates challenges in this approach. The model often fails to respond adequately to new, fine-grained bar-level control signals. To address this, we propose two innovative solutions. First, we introduce a pre-training task designed to link control signals directly with corresponding musical tokens, which helps in achieving a more effective initialization for subsequent fine-tuning. Second, we implement a novel counterfactual loss that promotes better alignment between the generated music and the control prompts. Together, these techniques significantly enhance our ability to control music generation at the bar level, showing a 13.06\% improvement over conventional methods. Our subjective evaluations also confirm that this enhanced control does not compromise the musical quality of the original pre-trained generative model.

cs.SD↗

arXiv2024

Source-Free Unsupervised Domain Adaptation with Hypothesis Consolidation of Prediction Rationale

Source-Free Unsupervised Domain Adaptation (SFUDA) is a challenging task where a model needs to be adapted to a new domain without access to target domain labels or source domain data. The primary difficulty in this task is that the model's predictions may be inaccurate, and using these inaccurate predictions for model adaptation can lead to misleading results. To address this issue, this paper proposes a novel approach that considers multiple prediction hypotheses for each sample and investigates the rationale behind each hypothesis. By consolidating these hypothesis rationales, we identify the most likely correct hypotheses, which we then use as a pseudo-labeled set to support a semi-supervised learning procedure for model adaptation. To achieve the optimal performance, we propose a three-step adaptation process: model pre-adaptation, hypothesis consolidation, and semi-supervised learning. Extensive experimental results demonstrate that our approach achieves state-of-the-art performance in the SFUDA task and can be easily integrated into existing approaches to improve their performance. The codes are available at \url{https://github.com/GANPerf/HCPR}.

cs.CV↗

arXiv2023

Integrating Sensing, Communication, and Power Transfer: Multiuser Beamforming Design

In the sixth-generation (6G) networks, massive low-power devices are expected to sense environment and deliver tremendous data. To enhance the radio resource efficiency, the integrated sensing and communication (ISAC) technique exploits the sensing and communication functionalities of signals, while the simultaneous wireless information and power transfer (SWIPT) techniques utilizes the same signals as the carriers for both information and power delivery. The further combination of ISAC and SWIPT leads to the advanced technology namely integrated sensing, communication, and power transfer (ISCPT). In this paper, a multi-user multiple-input multiple-output (MIMO) ISCPT system is considered, where a base station equipped with multiple antennas transmits messages to multiple information receivers (IRs), transfers power to multiple energy receivers (ERs), and senses a target simultaneously. The sensing target can be regarded as a point or an extended surface. When the locations of IRs and ERs are separated, the MIMO beamforming designs are optimized to improve the sensing performance while meeting the communication and power transfer requirements. The resultant non-convex optimization problems are solved based on a series of techniques including Schur complement transformation and rank reduction. Moreover, when the IRs and ERs are co-located, the power splitting factors are jointly optimized together with the beamformers to balance the performance of communication and power transfer. To better understand the performance of ISCPT, the target positioning problem is further investigated. Simulations are conducted to verify the effectiveness of our proposed designs, which also reveal a performance tradeoff among sensing, communication, and power transfer.

cs.IT↗

arXiv2023

Beamforming Design for RIS-Aided THz Wideband Communication Systems

Benefiting from tens of GHz of bandwidth, terahertz (THz) communications has become a promising technology for future 6G networks. However, the conventional hybrid beamforming architecture based on frequency-independent phase-shifters is not able to cope with the beam split effect (BSE) in THz massive multiple-input multiple-output (MIMO) systems. Despite some work introducing the frequency-dependent phase shifts via the time delay network to mitigate the beam splitting in THz wideband communications, the corresponding issue in reconfigurable intelligent surface (RIS)-aided communications has not been well investigated. In this paper, the BSE in THz massive MIMO is quantified by analyzing the array gain loss. A new beamforming architecture has been proposed to mitigate this effect under RIS-aided communications scenarios. Simulations are performed to evaluate the effectiveness of the proposed system architecture in combating the array gain loss.

cs.IT↗

arXiv2023

ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation

Recently, CLIP has been applied to pixel-level zero-shot learning tasks via a two-stage scheme. The general idea is to first generate class-agnostic region proposals and then feed the cropped proposal regions to CLIP to utilize its image-level zero-shot classification capability. While effective, such a scheme requires two image encoders, one for proposal generation and one for CLIP, leading to a complicated pipeline and high computational cost. In this work, we pursue a simpler-and-efficient one-stage solution that directly extends CLIP's zero-shot prediction capability from image to pixel level. Our investigation starts with a straightforward extension as our baseline that generates semantic masks by comparing the similarity between text and patch embeddings extracted from CLIP. However, such a paradigm could heavily overfit the seen classes and fail to generalize to unseen classes. To handle this issue, we propose three simple-but-effective designs and figure out that they can significantly retain the inherent zero-shot capacity of CLIP and improve pixel-level generalization ability. Incorporating those modifications leads to an efficient zero-shot semantic segmentation system called ZegCLIP. Through extensive experiments on three public benchmarks, ZegCLIP demonstrates superior performance, outperforming the state-of-the-art methods by a large margin under both "inductive" and "transductive" zero-shot settings. In addition, compared with the two-stage method, our one-stage ZegCLIP achieves a speedup of about 5 times faster during inference. We release the code at https://github.com/ZiqinZhou66/ZegCLIP.git.

cs.CV↗

arXiv2022

Joint Sensing and Communication-Rate Control for Energy Efficient Mobile Crowd Sensing

Driven by the rapid growth of Internet of Things applications, tremendous data need to be collected by sensors and uploaded to the servers for further process. As a promising solution, mobile crowd sensing enables controllable sensing and transmission processes for multiple types of data in a single device. In this paper, a typical user is considered that is required to sense and transmit data to a server, while it is assumed to remain busy and incapable of sensing data during an interval. An optimization problem is formulated to minimize the energy consumption of data sensing and transmission by controlling the sensing and transmission rates over time, subject to the constraints on the sensing data sizes, transmission data sizes, data casualty, and sensing busy time. This problem is highly challenging, due to the coupling between the rates as well as the existence of the busy time. To deal with this problem, we first show that it can be equivalently decomposed into two subproblems, corresponding to a search for the amount of data size that needs to be sensed before the busy time (referred to as the height), as well as the sensing and transmission rate control given the height. Next, we show that the latter problem can be efficiently solved by using the classical string-pulling method, while an efficient algorithm is proposed to progressively find the optimal height without the exhaustive search. Moreover, the solution approach is extended to a more complex scenario where there is a finite-size buffer at the server for receiving data. Last, simulations are conducted to evaluate the performance of the proposed design.

cs.IT↗

arXiv2022

Integrated Sensing, Communication, and Computation Over-the-Air: MIMO Beamforming Design

To support the unprecedented growth of the Internet of Things (IoT) applications, tremendous data need to be collected by the IoT devices and delivered to the server for further computation. By utilizing the same signals for both radar sensing and data communication, the integrated sensing and communication (ISAC) technique has broken the barriers between data collection and delivery in the physical layer. By exploiting the analog-wave addition in a multi-access channel, over-the-air computation (AirComp) enables function computation via transmissions in the physical layer. The promising performance of ISAC and AirComp motivates the current work on developing a framework called integrated sensing, communication, and computation over-the-air (ISCCO). The performance metrics of radar sensing and AirComp are evaluated by the mean squared errors of the estimated target response matrix and the received computation results, respectively. The design challenge of MIMO ISCCO lies in the joint optimization of beamformers for sensing, communication, and computation at both the IoT devices and the server, which results in a non-convex problem. To solve this problem, an algorithmic solution based on the technique of semidefinite relaxation is proposed. The use case of target location estimation based on ISCCO is demonstrated in simulation to show the performance superiority.

cs.IT↗

arXiv2021

Data Partition and Rate Control for Learning and Energy Efficient Edge Intelligence

The rapid development of artificial intelligence together with the powerful computation capabilities of the advanced edge servers make it possible to deploy learning tasks at the wireless network edge, which is dubbed as edge intelligence (EI). The communication bottleneck between the data resource and the server results in deteriorated learning performance as well as tremendous energy consumption. To tackle this challenge, we explore a new paradigm called learning-and-energy-efficient (LEE) EI, which simultaneously maximizes the learning accuracies and energy efficiencies of multiple tasks via data partition and rate control. Mathematically, this results in a multi-objective optimization problem. Moreover, the continuous varying rates over the whole transmission duration introduce infinite variables. To solve this complex problem, we consider the case with infinite server buffer capacity and one-shot data arrival at sensor. First, the number of variables are reduced to a finite level by exploiting the optimality of constant-rate transmission in each epoch. Second, the optimal solution is found by applying stratified sequencing or objectives merging. By assuming higher priority of learning efficiency in stratified sequencing, the closed form of optimal data partition is derived by the Lagrange method, while the optimal rate control is proved to have the structure of directional water filling (DWF), based on which a string-pulling (SP) algorithm is proposed to obtain the numerical values. The DWF structure of rate control is also proved to be optimal in objectives merging via weighted summation. By exploiting the optimal rate changing properties, the SP algorithm is further extended to account for the cases with limited server buffer capacity or bursty data arrival at sensor. The performance of the proposed design is examined by extensive experiments based on public datasets.

cs.IT↗