Patch Attention Wisely: Understanding Patch Length in Attention-Based Forecasting

Junyu Chen; Likai Chen; Weining Wang

Home/Archive/arXiv

arXiv · 2611.08160

Patch Attention Wisely: Understanding Patch Length in Attention-Based Forecasting

Junyu Chen·Likai Chen·Weining Wang

View original source ↗Read source document ↗Export metadata ↓

Abstract

Attention-based time series forecasters commonly group consecutive observations into patches. Patch length is known to have a substantial impact on forecasting performance, yet its selection remains largely empirical. We develop a theoretical framework that explains why patch length matters for forecasting accuracy and provides a principled basis for its selection. For linear state-space models, we derive an exact factorization of the forecasting coefficient matrix that separates forecast dynamics from filtering dynamics. We show that dominance of a single forecast component at long horizons does not generally imply a rank-one representation. This explains why longer horizons alone need not reduce representational complexity. We combine the factorization with estimation bounds to develop a patch-selection criterion that balances representation and estimation errors under temporal dependence. Our analysis establishes conditions under which balanced patching is justified and clarifies how predictive structure, head count, training sample size, and long-run dependence enter the selection criterion.

Explore related subjects

stat.ML cs.LG econ.EM

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Junyu Chen, Likai Chen, Weining Wang. 2026-11-05. Patch Attention Wisely: Understanding Patch Length in Attention-Based Forecasting. https://arxiv.org/abs/2611.08160

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Identifying Drift, Diffusion, and Causal Structure from Temporal Snapshots

Stochastic differential equations (SDEs) are a fundamental tool for modelling dynamic processes, including gene regulatory networks (GRNs), contaminant transport, financial markets, and image generation. However, learning the underlying SDE from data is a challenging task, especially if individual trajectories are not observable. Motivated by burgeoning research in single-cell datasets, we present the first comprehensive approach for jointly identifying the drift and diffusion of an SDE from its temporal marginals. Assuming linear drift and additive diffusion, we show that non-identifiability can only arise if the initial distribution possesses generalized rotational symmetries. We further prove that even if this condition holds, the drift and diffusion can almost always be recovered from the marginals. Additionally, we show that the causal graph of any SDE with additive diffusion can be recovered from the identified SDE parameters. To complement this theory, we adapt entropy-regularized optimal transport to handle anisotropic diffusion and introduce APPEX (Alternating Projection Parameter Estimation from $X_0$), an iterative algorithm designed to estimate the drift, diffusion, and causal graph of an additive noise SDE, solely from temporal marginals, and demonstrate its effectiveness on simulated data from linear additive noise SDEs.

stat.ML↗

arXiv2026

Sparse Network Inference under Imperfect Detection and its Application to Ecological Networks

Recovering latent structure from count data has received considerable attention in network inference, particularly when one seeks both cross-group interactions and within-group similarity patterns in bipartite networks, which are widely used in ecology research. Such networks are often sparse and prone to imperfect detection. Existing models mainly focus on interaction recovery, while the induced similarity graphs are much less studied. Moreover, existing models for under-counted data place sparsity, if at all, on the detection model rather than on these graphs. To address these issues, we propose a nonnegative low-rank factorization framework that jointly estimates detection probabilities and promotes sparse within- and cross-group graphs through $\ell_{1/2}$ regularization. For the resulting nonconvex, nonsmooth problem, we develop an ADMM-based algorithm with adaptive penalization, and establish asymptotic feasibility and KKT stationarity of cluster points under stated assumptions. We also give sufficient conditions for exact recovery of within- and cross-group graphs. On synthetic data with up to 20% missing entries, the method recovers the graphs and detection probabilities with competitive accuracy. Across three ecological datasets, it matches the baseline held-out predictions while estimating detection probabilities and within- and cross-group graphs.

stat.ML↗

arXiv2026

Hypergraph Generation via Structured Stochastic Diffusion

Hypergraphs model higher-order interactions, but realistic hypergraph generation remains difficult because incidence, hyperedge-size heterogeneity, and overlap structure are not faithfully captured by pairwise reductions. We propose \HEDGE, a generative model defined directly on relaxed incidence matrices via a structured stochastic diffusion. The forward process combines a hypergraph-specific two-sided heat operator with an Ornstein--Uhlenbeck component, preserving structure-aware noising near the data while yielding an explicit Gaussian terminal law. Conditional on an observed hypergraph, this forward process is linear-Gaussian, so conditional means, covariances, scores, and reverse-drift targets are available in closed form. We therefore learn a permutation-equivariant state-only reverse-drift field in incidence space by regressing onto exact conditional targets, and generate samples by simulating a learned reverse-time SDE from the Gaussian base law. We establish exactness in the ideal state-only setting together with finite-horizon stability guarantees, and empirically show improved hypergraph generation quality relative to strong baselines.

stat.ML↗