HEART: Exploiting Head Heterogeneity in Sparse Attention for Video Diffusion

Abstract

Sparse attention accelerates video diffusion by allowing each attention head to focus on only a small subset of interactions. Existing methods already construct head-specific sparse patterns conditioned on the input. However, we find that these heads also differ in two less obvious but practically important ways. First, for some heads, sparse attention masks remain stable across many denoising steps, whereas others change rapidly. Second, heads differ substantially in their sensitivity to sparsification: applying the same threshold can induce markedly different errors in the final denoising velocity. Ignoring these differences leads to redundant mask prediction and suboptimal threshold calibration across heads under a global sparsity budget. We present HEART, short for Heterogeneity-Exploiting Adaptive Refresh and Thresholding, a training-free framework that exploits both forms of head heterogeneity. First, Temporal Mask Reuse (TMR) uses a lightweight per-head query-key drift signal to determine whether a cached sparse mask remains reliable across denoising steps, refreshing it only when the drift exceeds a prescribed threshold. Second, Error-guided Budgeted Calibration (EBC) evaluates candidate thresholds offline using a frequency-weighted denoising-velocity error, and assigns each head an appropriate threshold under a global sparsity budget. HEART requires no retraining, weight modification, or sparse-kernel changes, and can be integrated directly into existing sparse-attention pipelines. Across Wan2.1-1.3B, Wan2.1-14B, and HunyuanVideo-13B, HEART consistently pushes the quality--efficiency frontier of advanced sparse attention methods such as XAttention and SVG2 outward.