arXiv subjects

Jiasong Han

Publications and source records attributed to Jiasong Han.

5 recordsLinked to original sources

arXiv2026

M3Former: Fine-Grained Cross-Layer Mixing for Transformers

Residual connections give each Transformer block access to all earlier layers only through one fixed sum, broadcast to all of the block's inputs. This broadcast bottleneck imposes three constraints: the query, key, value and residual inputs of a block, which we call its roles, read the same vector; all hidden dimensions within a role share one set of weights over earlier layers; and every earlier layer has the same fixed weight for every token. We propose the Multi-Role Multi-Group Mixing Residual (M3), which removes all three: every role, and every group of hidden dimensions within a role, reads its own token-dependent mixture of all earlier layers. M3 starts as an exact residual connection and adds as little as 1.4% parameters and 1.8% FLOPs. From 100M to 1.4B parameters, M3Former outperforms the Transformer and coarse-grained cross-layer connections at every size, matching a Transformer 1.19-1.34x the size of the one it modifies, trained with 1.4-1.8x the compute. Relaxing only some of the constraints recovers only part of this gain. M3Former also improves sparse mixture-of-experts models of up to 4.6B parameters and trains more stably in deeper networks. Code: https://github.com/Kingchet10/m3former

cs.LG↗

arXiv2026

A Positional Realizability Theory of Length Generalization in Single-Head Attention

Why does length generalization (LG) depend so sharply on the positional encoding (PE)? Relative schemes extrapolate fixed offsets, absolute schemes fix positions, and neither handles growing offsets. We give a realizability theory for a deliberately controlled abstraction, a single positional-only attention head, that derives this well-documented empirical pattern and says precisely where it holds and breaks. The device is a simulation transfer operator $T_L$ that re-evaluates a selected length-$L_0$-realizing positional kernel on the longer grid, with a scalar $α_L \in [0,1]$ read off before any training from $(Φ, t_n)$ alone. Here $α_L$ serves as the computable witness of that taxonomy's derivation, not as a stronger empirical predictor. Our results are a relative-head deep-row characterization for kernels with a strict unique global maximizer (Theorem 2), an absolute/relative duality that derives the pattern (Proposition 4), a learned-absolute underdetermination result, and a static expressivity bound; a sufficiency theorem (Theorem 1) bridges $α_L$ to LG, and the general deterministic case is a precise open conjecture (correcting a tempting but invalid "outside-colspan" argument). Experiments confirm the structure: in the controlled single-head regime $α_L$ exactly separates the taxonomy's generalizing and non-generalizing (PE, task) cells (2625 trained-model measurements; stable at vocab 64/d=128). Together, these results turn the received PE taxonomy into a positional-kernel transfer theorem, and identify the content and depth mechanisms that escape it. The theory concerns the positional channel alone: tasks whose targets depend on token content, including the standard length-generalization suite such as addition and sorting, fall outside Theorem 2.

cs.LG↗

arXiv2026

Channel Estimation with Hierarchical Sparse Bayesian Learning for ODDM Systems

Orthogonal delay-Doppler division multiplexing (ODDM) is a promising modulation technique for reliable communications in high-mobility scenarios. However, the existing channel estimation frameworks for ODDM systems cannot achieve both high accuracy and low complexity simultaneously, due to the inherent coupling of delay and Doppler parameters. To address this problem, a two-dimensional (2D) hierarchical sparse Bayesian learning (HSBL) based channel estimation framework is proposed in this paper. Specifically, we address the inherent coupling between delay and Doppler dimensions in ODDM by developing a partially-decoupled 2D sparse signal recovery (SSR) formulation on a virtual sampling grid defined in the delay-Doppler (DD) domain. With the help of the partially-decoupled formulation, the proposed 2D HSBL framework first performs low-complexity coarse on-grid 2D sparse Bayesian learning (SBL) estimation to identify potential channel paths. Then, high-resolution fine grids are constructed around these regions, where an off-grid 2D SBL estimation is applied to achieve accurate channel estimation. Simulation results demonstrate that the proposed framework achieves performance superior to conventional off-grid 2D SBL with significantly reduced computational complexity.

econ.EM↗

arXiv2026

FilterLoss: A Transfer Learning Approach for Communication Scene Recognition

Communication scene recognition has been widely applied in practice, but using deep learning to address this problem faces challenges such as insufficient data and imbalanced data distribution. To address this, we designed a weighted loss function structure, named FilterLoss, which assigns different loss function weights to different sample points. This allows the deep learning model to focus primarily on high-value samples while appropriately accounting for noisy, boundary-level data points. Additionally, we developed a matching weight filtering algorithm that evaluates the quality of sample points in the input dataset and assigns different weight values to samples based on their quality. By applying this method, when using transfer learning on a highly imbalanced new dataset, the accuracy of the transferred model was restored to 92.34% of the original model's performance. Our experiments also revealed that using this loss function structure allowed the model to maintain good stability despite insufficient and imbalanced data.

econ.EM↗

arXiv2025

Dynamic SINR-Guided Iterative Interference Cancellation for ODDM Systems in Doubly Dispersive Channels

Orthogonal delay-Doppler division multiplexing (ODDM) modulation has recently gained significant attention as a promising candidate to promote the communication reliability in high-mobility environments. Low complexity signal detection is one of the most significant challenges for ODDM over general physical channels, due to the large channel spreading caused by the fractional delay and Doppler shifts. In this paper, we investigate the low-complexity data detection for ODDM system by utilizing iterative interference cancellation. Based on the theoretical analysis of signal to interference plus noise ratio (SINR) during the iteration, a dynamic SINR-guided approach is proposed to provide a better initialization result. Specifically, we analyze the SINR of each time domain sample before initial estimate with consideration of off-grid delay and Doppler shifts. The iteration is then started from the multi-carrier symbol index which has the best SINR. The corresponding interference is then eliminated for other time domain samples while the SINR for symbol awaiting detection is also updated. Based on the updated SINR, the next multi-carrier symbol index is selected for the same processing until all data symbols have been initialized. Finally, we update the SINR synchronously until the end of the initialization. Simulation experiments indicate that our proposed algorithms demonstrate satisfying convergence and error performance while avoiding the huge complexity introduced by full linear minimum mean squared error (LMMSE) initialization.

econ.TH↗