arXiv subjects

Mingquan Feng

Publications and source records attributed to Mingquan Feng.

8 recordsLinked to original sources

arXiv2026

RINS: Residual-Image Neural Subspace Solvers for Large Sparse Linear Systems

Neural subspace solvers offer a promising way to accelerate the large sparse linear systems arising from PDE discretizations, but their value depends on generating effective correction directions at low computational overheads. We study this problem from a residual-image viewpoint: a correction subspace is useful when its image under the system operator captures the current residual. Guided by this view, we propose Gate-RINS, which generates multiple correction directions from shared polynomial residual probes and modulates them with a lightweight residual- and coordinate-dependent pointwise gate. We also introduce a hybrid schedule that uses GRANS-style graph controllers for the first correction block and Gate-RINS thereafter, combining an expressive graph based initial step with a lower-cost corrector as the solve progress. Theoretical analysis establishes a unified residual-image and dynamic approximate-inverse perspective for Gate-RINS and G-RANS, showing that a single neural-generated direction can emulate the residual quality of fixed-step GMRES. Numerical experiments on six PDE benchmarks demonstrate that Gate-RINS reaches a relative residual of $10^{-5}$ with 1.6-2.3$\times$ and 2.2-3.0$\times$ less wall-clock time than G-RANS and GMRES respectively. For Helmholtz systems with meshsize up to $10^5$, Gate-RINS also achieves the lowest residual and outperforms a GPU-accelerated parallel GMRES, while GRANS fails to reach relative residual of $10^{-4}$ in the same time budget.

cs.CE↗

arXiv2026

KO: Kinetics-inspired Neural Optimizer with PDE Simulation Approaches

The design of effective optimization algorithms for neural networks remains a fundamental challenge, and most existing methods rely on heuristic extensions of gradient-based updates. We introduce KO (Kinetics-inspired Optimizer), a plug-and-play optimization module grounded in kinetic theory and partial differential equations. KO models parameter dynamics as a particle system, augmenting standard gradient updates with stochastic interactions induced by a discretization of the Boltzmann transport equation. This mechanism naturally promotes parameter diversity and mitigates weight condensation, the tendency of parameters to collapse into low-dimensional subspaces, a phenomenon closely associated with degraded generalization. We provide both a rigorous theoretical analysis and a physical interpretation, showing that KO provably increases parameter diversity while preserving convergence guarantees. Extensive experiments on image classification benchmarks (CIFAR-10/100, ImageNet) and large-scale language model pretraining demonstrate that KO consistently improves accuracy over competitive baselines with negligible additional computational cost.

cs.LG↗

arXiv2026

KITINet: KInetic Theory Inspired Inter-Channel Information Exchange for Residual Networks

Residual connections are foundational to modern neural networks, yet the standard additive update provides no explicit mechanism for structured inter-channel interaction. We introduce KITINet (KInetics Theory Inspired Network), a training-time module with no additional trainable parameters that augments residual updates with stochastic pairwise feature interactions inspired by collision sampling in kinetic theory. KITINet reshapes channel groups as particles and mixes their features according to relative distance and velocity, while leaving the underlying residual layer trainable in the usual way. The module is disabled at inference, so the deployed model is identical to the original backbone and incurs no additional inference cost. Across language pre-training and fine-tuning, image classification, and PDE operator learning, KITINet demonstrates broad and robust performance gains across diverse tasks and architectures. We further observe enhanced parameter condensation across several settings and develop a simplified stochastic analysis that provides a mechanistic account of how collision-inspired interactions can promote condensation dynamics.

cs.LG↗

arXiv2025

ssToken: Self-modulated and Semantic-aware Token Selection for LLM Fine-tuning

Data quality plays a critical role in enhancing supervised fine-tuning (SFT) for large language models (LLMs), and token-level data selection has emerged as a promising direction for its fine-grained nature. Despite their strong empirical performance, existing token-level selection methods share two key limitations: (1) requiring training or accessing an additional reference model, and (2) relying solely on loss information for token selection, which cannot well preserve semantically important tokens that are not favored by loss-based metrics. To address these challenges, we propose ssToken, a Self-modulated and Semantic-aware Token Selection approach. ssToken leverages readily accessible history models to compute the per-token loss difference with the current model, which serves as a self-modulated signal that enables the model to adaptively select tokens along its optimization trajectory, rather than relying on excess loss from an offline-trained reference model as in prior works. We further introduce a semantic-aware, attention-based token importance estimation metric, orthogonal to loss-based selection and providing complementary semantic information for more effective filtering. Extensive experiments across different model families and scales demonstrate that both self-modulated selection and semantic-aware selection alone outperform full-data fine-tuning, while their integration--ssToken--achieves synergistic gains and further surpasses prior token-level selection methods, delivering performance improvements while maintaining training efficiency.

cs.AI↗

arXiv2025

Rethinking LLM Evaluation: Can We Evaluate LLMs with 200x Less Data?

As the demand for comprehensive evaluations of diverse model capabilities steadily increases, benchmark suites have correspondingly grown significantly in scale. Despite notable advances in redundancy reduction and subset-level performance prediction, a systematic framework that effectively integrates these methods to ensure both prediction accuracy and ranking consistency is still largely elusive. In this paper, we first perform a sample-level analysis of benchmark redundancy and identify several highly similar samples that can be eliminated. Besides, we frame benchmark compression as an optimization problem with the aim of score reconstruction. Building on these, we then propose EssenceBench, a coarse-to-fine framework utilizing an iterative Genetic Algorithm (GA), which takes the advantages of fitness-based subset search and attribution-based sample search. Compared to previous methods, our approach yields superior compression results with lower reconstruction error and markedly higher efficiency. In particular, on the HellaSwag benchmark (10K samples), our method preserves the ranking of all models shifting within 5% using 25x fewer samples, and achieves 95% ranking preservation shifting within 5% using only 200x fewer samples.

cs.CL↗

arXiv2024

Optimal Control Operator Perspective and a Neural Adaptive Spectral Method

Optimal control problems (OCPs) involve finding a control function for a dynamical system such that a cost functional is optimized. It is central to physical systems in both academia and industry. In this paper, we propose a novel instance-solution control operator perspective, which solves OCPs in a one-shot manner without direct dependence on the explicit expression of dynamics or iterative optimization processes. The control operator is implemented by a new neural operator architecture named Neural Adaptive Spectral Method (NASM), a generalization of classical spectral methods. We theoretically validate the perspective and architecture by presenting the approximation error bounds of NASM for the control operator. Experiments on synthetic environments and a real-world dataset verify the effectiveness and efficiency of our approach, including substantial speedup in running time, and high-quality in- and out-of-distribution generalization.

eess.SY↗

arXiv2022

Learning Neural Hamiltonian Dynamics: A Methodological Overview

The past few years have witnessed an increased interest in learning Hamiltonian dynamics in deep learning frameworks. As an inductive bias based on physical laws, Hamiltonian dynamics endow neural networks with accurate long-term prediction, interpretability, and data-efficient learning. However, Hamiltonian dynamics also bring energy conservation or dissipation assumptions on the input data and additional computational overhead. In this paper, we systematically survey recently proposed Hamiltonian neural network models, with a special emphasis on methodologies. In general, we discuss the major contributions of these models, and compare them in four overlapping directions: 1) generalized Hamiltonian system; 2) symplectic integration, 3) generalized input form, and 4) extended problem settings. We also provide an outlook of the fundamental challenges and emerging opportunities in this area.

cs.LG↗

arXiv2020

Perception Improvement for Free: Exploring Imperceptible Black-box Adversarial Attacks on Image Classification

Deep neural networks are vulnerable to adversarial attacks. White-box adversarial attacks can fool neural networks with small adversarial perturbations, especially for large size images. However, keeping successful adversarial perturbations imperceptible is especially challenging for transfer-based black-box adversarial attacks. Often such adversarial examples can be easily spotted due to their unpleasantly poor visual qualities, which compromises the threat of adversarial attacks in practice. In this study, to improve the image quality of black-box adversarial examples perceptually, we propose structure-aware adversarial attacks by generating adversarial images based on psychological perceptual models. Specifically, we allow higher perturbations on perceptually insignificant regions, while assigning lower or no perturbation on visually sensitive regions. In addition to the proposed spatial-constrained adversarial perturbations, we also propose a novel structure-aware frequency adversarial attack method in the discrete cosine transform (DCT) domain. Since the proposed attacks are independent of the gradient estimation, they can be directly incorporated with existing gradient-based attacks. Experimental results show that, with the comparable attack success rate (ASR), the proposed methods can produce adversarial examples with considerably improved visual quality for free. With the comparable perceptual quality, the proposed approaches achieve higher attack success rates: particularly for the frequency structure-aware attacks, the average ASR improves more than 10% over the baseline attacks.

cs.CV↗