arXiv subjects

Bonan Xu

Publications and source records attributed to Bonan Xu.

9 recordsLinked to original sources

arXiv2026

Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing

Sparse mixture-of-experts (MoE) language models route each token to multiple experts, suggesting a geometric account of their benefit: co-selected experts should contribute distinct representation directions. Existing evidence often conflates route coherence, candidate quality, and candidate-by-context interaction. We distinguish these quantities using an Expert Subspace Separation Index (ESSI), matched-route residuals, and a prefix-controlled $2\times2$ factorial; frozen-route interventions and a controlled Top-$k$ study assess functional value. Three paired contrasts organize the findings. First, across six MoE architectures, expert subspaces overlap substantially, yet actual routes explain token representations better than matched alternatives. Second, across the 39 factorial cells in OLMoE, Mixtral, and DeepSeek, the selected candidate explains more of the residual representation than the strongest unselected rival in every cell, yet the actual prefix narrows this advantage throughout: all interactions are negative, and every 95% confidence interval lies below zero. Third, this geometric narrowing does not imply functional redundancy: adding later experts improves next-token prediction in 24 of 39 frozen-route comparisons, while the other 15 estimates are inconclusive; a controlled training study also favors Top-2 over Top-1 in all three seeds. We call this joint pattern coherent overlap: routing selects token-relevant experts from a shared geometric neighborhood, while useful multi-expert computation persists without disjoint linear coverage. Separating these quantities clarifies why geometric similarity alone cannot determine redundancy or pruning value.

cs.LG↗

arXiv2026

A Compression-Directional Entropic Stress Method for Shock-Regularized Compressible Flow

We introduce the Compression-Directional Entropic Stress (CoDeS) method inspired by information geometric regularization. CoDeS replaces scalar multidimensional entropic pressure with a tensor stress aligned with the principal directions of compression. The stress has the form $\boldsymbol{\Pi}_{\Sigma}=\sigma\boldsymbol{M}$, where $\sigma$ is obtained from a modified-Helmholtz equation and $\boldsymbol{M}$ is constructed from the compressive eigenspace of the symmetric velocity-gradient tensor. The source is gated by volumetric and principal-strain compression, so the regularization vanishes in smooth expansion, rigid-body rotation, and ideal contacts, while recovering the compressive one-dimensional IGR mechanism at planar shocks. The same tensor stress is used in the conservative momentum flux and the stress-work energy flux. CoDeS is tested on one-, two-, and three-dimensional problems including smooth expansion, double rarefaction, the Sod shock tube, multidimensional Riemann flow, a viscous shock tube, a two-fluid triple point, a Mach-3 slot jet, and a supersonic Taylor--Green vortex. The results show that CoDeS remains inactive in expansive and contact regions, supplies localized stress at shocks, and concentrates regularization along compressive wave structures while remaining weak in shear- and vorticity-dominated regions. At matched resolutions, the three-dimensional Taylor--Green results are comparable to or more energetic than seventh-order WENO/TENO references. These results indicate that CoDeS provides a compression-selective shock regularization compatible with high-order finite-volume resolution of contacts, interfaces, shear layers, and vortical structures. All the code, case settings, and code for plotting figures of this paper are available at https://github.com/xubonan/code\_for\_CoDeS.

physics.flu-dyn↗

arXiv2025

From Per-Image Low-Rank to Encoding Mismatch: Rethinking Feature Distillation in Vision Transformers

Feature-map knowledge distillation (KD) transfers internal representations well between comparably sized Vision Transformers (ViTs), but it often fails in compression. We revisit this failure and uncover a paradox. Sample-wise SVD shows that each image is highly compressible, which seems to suggest that a narrow student with a linear projector should match the teacher "in principle". However, a dataset-level view contradicts this intuition: PCA shows that the teacher is a union of low-rank subspaces with significant subspace rotation across inputs. We further introduce token-level Spectral Energy Patterns (SEP) and find an architecture-invariant encoding law: tokens spread energy broadly across channel modes even when they live in low-rank subspace, creating a bandwidth mismatch. We refer to this combined phenomenon as an encoding mismatch. We propose two minimal remedies, Lift or WideLast: (i) Lift retains a lightweight lifting projector at inference to provide wider channel, or (ii) WideLast widens only the student's last block, enabling an input-dependent expansion. On ImageNet-1K, these fixes revive feature KD for ViT compression, improving DeiT-Tiny distilled from CaiT-S24 from 74.86% to 77.53%/78.23% top-1 accuracy, and they also strengthen students trained without distillation. Our analyses clarify when and why feature-map KD fails and then how to fix it. Code and raw data are provided in https://github.com/thy960112/From-Per-Image-Low-Rank-to-Encoding-Mismatch.

cs.CV↗

arXiv2025

Distillation Dynamics: Towards Understanding Feature-Based Distillation in Vision Transformers

While feature-based knowledge distillation has proven highly effective for compressing CNNs, these techniques unexpectedly fail when applied to Vision Transformers (ViTs), often performing worse than simple logit-based distillation. We provide the first comprehensive analysis of this phenomenon through a novel analytical framework termed as "distillation dynamics", combining frequency spectrum analysis, information entropy metrics, and activation magnitude tracking. Our investigation reveals that ViTs exhibit a distinctive U-shaped information processing pattern: initial compression followed by expansion. We identify the root cause of negative transfer in feature distillation: a fundamental representational paradigm mismatch between teacher and student models. Through frequency-domain analysis, we show that teacher models employ distributed, high-dimensional encoding strategies in later layers that smaller student models cannot replicate due to limited channel capacity. This mismatch causes late-layer feature alignment to actively harm student performance. Our findings reveal that successful knowledge transfer in ViTs requires moving beyond naive feature mimicry to methods that respect these fundamental representational constraints, providing essential theoretical guidance for designing effective ViTs compression strategies. All source code and experimental logs are provided at https://github.com/thy960112/Distillation-Dynamics.

cs.CV↗

arXiv2025

A Central Differential Flux with High-Order Dissipation for Robust Simulations of Transcritical Flows

The simulation of transcritical flows remains challenging due to strong thermodynamic nonlinearities that induce spurious pressure oscillations in conventional schemes.While primitive-variable formulations offer improved robustness under such conditions, they are always limited by energy conservation errors and the absence of systematic high-order treatments for numerical fluxes. In this paper, we introduce the Central Differential flux with High-Order Dissipation (CDHD), a novel numerical flux solver designed for primitive-variable discretization. This method combines a central flux for advection with a minimal, upwind-biased dissipation term to stabilize the simulation while maintaining formal accuracy. The dissipation term effectively suppresses oscillations and improves stability in transcritical flows. Compared to traditional primitive-variable approaches, CDHD reduces the energy conservation error in two order of magnitude. When incorporated into a hybrid framework with a conservative shock-capturing scheme, the method robustly handles both smooth transcritical phenomena and shock waves. Numerical tests validate the accuracy, stability, and energy-preserving capabilities of CDHD, demonstrating its potential as a reliable tool for complex real-gas flow simulations.

physics.flu-dyn↗

arXiv2024

SpectralKD: A Unified Framework for Interpreting and Distilling Vision Transformers via Spectral Analysis

Knowledge Distillation (KD) has achieved widespread success in compressing large Vision Transformers (ViTs), but a unified theoretical framework for both ViTs and KD is still lacking. In this paper, we propose SpectralKD, a novel unified analytical framework that offers deeper insights into ViTs and optimizes KD via spectral analysis. Our model-wise analysis reveals that CaiT concentrates information in their first and last few layers, informing optimal layer selection for KD. Surprisingly, our layer-wise analysis discovers that Swin Transformer and CaiT exhibit similar spectral encoding patterns despite their architectural differences, leading to feature map alignment guideline. Building on these insights, we propose a simple yet effective spectral alignment method for KD. Benefiting from the deeper understanding by above analysis results, even such a simple strategy achieves state-of-the-art performance on ImageNet-1K without introducing any trainable parameters, improving DeiT-Tiny by $+5.2\%$ and Swin-Tiny by $+1.4\%$ in top-1 accuracy. Furthermore, our post-training analysis reveals that distilled students can reproduce spectral patterns similar to their teachers, opening a new area we term ``distillation dynamics". Code and experimental logs are available in https://github.com/thy960112/SpectralKD.

cs.CV↗

arXiv2023

Self-supervised learning based on Transformer for flow reconstruction and prediction

Machine learning has great potential for efficient reconstruction and prediction of flow fields. However, existing datasets may have highly diversified labels for different flow scenarios, which are not applicable for training a model. To this end, we make a first attempt to apply the self-supervised learning (SSL) technique to fluid dynamics, which disregards data labels for pre-training the model. The SSL technique embraces a large amount of data ($8000$ snapshots) at Reynolds numbers of $Re=200$, $300$, $400$, $500$ without discriminating between them, which improves the generalization of the model. The Transformer model is pre-trained via a specially designed pretext task, where it reconstructs the complete flow fields after randomly masking $20\%$ data points in each snapshot. For the downstream task of flow reconstruction, the pre-trained model is fine-tuned separately with $256$ snapshots for each Reynolds number. The fine-tuned models accurately reconstruct the complete flow fields based on less than $5\%$ random data points within a limited window even for $Re=250$ and $600$, whose data were not seen in the pre-trained phase. For the other downstream task of flow prediction, the pre-training model is fine-tuned separately with $128$ consecutive snapshot pairs for each corresponding Reynolds number. The fine-tuned models then correctly predict the evolution of the flow fields over many periods of cycles. We compare all results generated by models trained via SSL and models trained via supervised learning, where the former has unequivocally superior performance. We expect that the methodology presented here will have wider applications in fluid mechanics

physics.flu-dyn↗

arXiv2022

An adaptive primitive-conservative scheme for high speed transcritical flow with an arbitrary equation of state

When fully conservative methods are used to simulate transcritical flow, spurious pressure oscillations and numerical instability are generated. The strength and speed of propagation of shock waves cannot be represented correctly using a semi-conservative or primitive method. In this research, an adaptive primitive-conservative scheme is designed to overcome the aforesaid two difficulties. The underlying cause for pressure oscillation is analyzed within the framework of Finite Volume Method (FVM). We found that the nonlinearity of the thermodynamic properties of transcritical fluids renders standard conservative numerical methods ineffective. In smooth regions, schemes based on primitive variable are used to eliminate spurious pressure oscillations. For the purpose of correctly capturing shock waves, the modified Roe Riemann solver for real fluid is utilized in regions where shock waves induce discontinuity. The adaptive numerical approach relies only on the speed of sound, eliminating the requirement to calculate the derivatives of thermodynamic quantities. A large number of numerical test cases conducted in one- and two-dimensional spaces have shown the robustness and accuracy of the proposed adaptive scheme for the simulations of high speed transcritical flows.

physics.flu-dyn↗

arXiv2021

Transcritical transition of the fluid around the interface

In this letter, we provide fundamental insights into the dynamic transcritical transition process using molecular dynamics simulations. A transcritical region, which covers three different fluid states, was discovered as a substitute for the traditional interface. The physical properties, such as temperature and density, exhibited a highly non-linear distribution in the transcritical region. Meanwhile, the surface tension was found to exist throughout the transcritical region, and the magnitude was directly proportional to $ - ρ\cdot {\nabla ^2}ρ$

physics.flu-dyn↗