arXiv subjects

Hongsheng Liu

Publications and source records attributed to Hongsheng Liu.

At least 19 recordsLinked to original sources

arXiv2026

How Much Dense Attention is Necessary? Oracle-Guided Sparse Prefill for Full/GQA Layers in Hybrid Long-Context Models

Long-context prefill remains expensive because full/GQA layers still score the historical sequence, even in hybrid models with local, sparse, linear, or recurrent components. We study how much dense attention is needed to preserve task-level behavior under explicit support granularity and top-k budgets. We introduce an attention-mass top-k oracle for existing GQA checkpoints: for each layer and query position, it computes dense attention, selects head-averaged token support, and recomputes attention only on that support. The oracle is a diagnostic reference, not a deployable accelerator, and separates sparse-budget feasibility from indexer error and runtime realization effects. On Qwen-family retrieval-heavy evaluations, the longest per-query oracle rows stay within 1 point of dense, and a Qwen3.5-9B RULER-style sweep from 4K to 100K stays within 0.48 points. Guided by the oracle, we derive a head-collapsed auxiliary indexer trained by KL distillation from dense attention-mass distributions while keeping the backbone frozen. With separately distilled Qwen3.5-0.8B and Qwen3.5-9B indexers, the reported 16K/32K validation macro gaps are +2.04 and +1.13 points, treated as quality preservation rather than improvement; fused selection-block-shared support can introduce a larger realization gap. Preliminary single-card TTFT measurements show distilled-indexer sparse serving speedups of 1.71x for Qwen3.5-0.8B on NPU and 1.93x for Qwen3.5-9B on GPU against its dense FlashAttention-2 baseline. Additional random-init stress rows reach 3.44x, indicating sparse-runtime headroom but not validated output quality. This first release separates oracle feasibility, distilled-indexer quality, and runtime headroom, leaving a fully matched quality-latency frontier to future work.

cs.LG↗

arXiv2026

LVSA: Training-Free Sparse Attention for Long Video Diffusion

Dense self-attention is the compute and quality bottleneck of long-video diffusion inference: cost grows quadratically with the sequence length, and beyond the training horizon the model converges to near-static output, that is, "frozen" repetitive video. State of the art approaches are either too costly, e.g., they require retraining, or fail to satisfy both performance and quality objectives in a scalable manner. To this end, we introduce Long Video Sparse Attention (LVSA), a training-free model-agnostic block-sparse attention for video diffusion transformers that combines a structured window pattern with rotating global anchors, thus removing the fixed-grid bias which causes long-range temporal artifacts. LVSA, combined with a FlashInfer kernel, reduces compute up to 3.17x on Wan 2.1 1.3B at a 6x horizon, 2.98x on Wan 2.1 14B at a 6x horizon, and 3.33x on HunyuanVideo 1.5 at a 1.5x horizon, compared to dense attention. Beyond reducing compute, LVSA enables HunyuanVideo 1.5 generation at a 2x horizon, which is otherwise out-of-memory on a single GPU. Moreover, LVSA provides speedups up to 2.41x compared to RIFLEx and 3.27x compared to UltraViCo on Wan 2.1 1.3B. To demonstrate applicability across diverse platforms, we apply LVSA on NPUs and achieve speedups up to 2.71x on Wan 2.2 A14B and 3.24x on Wan 2.1 1.3B compared to dense attention. To evaluate quality in a fair way, we introduce VQeval, a tool properly scoring loopy video failures, which instead are rewarded in state of the art evaluators like VBench-Long. LVSA is quality-neutral for generation at training horizon length and quality-positive at extended lengths.

cs.CV↗

arXiv2026

vLLM-Omni: Fully Disaggregated Serving for Any-to-Any Multimodal Models

Any-to-any multimodal models that jointly handle text, images, video, and audio represent a significant advance in multimodal AI. However, their complex architectures (typically combining multiple autoregressive LLMs, diffusion transformers, and other specialized components) pose substantial challenges for efficient model serving. Existing serving systems are mainly tailored to a single paradigm, such as autoregressive LLMs for text generation or diffusion transformers for visual generation. They lack support for any-to-any pipelines that involve multiple interconnected model components. As a result, developers must manually handle cross-stage interactions, leading to huge performance degradation. We present vLLM-Omni, a fully disaggregated serving system for any-to-any models. vLLM-Omni features a novel stage abstraction that enables users to decompose complex any-to-any architectures into interconnected stages represented as a graph, and a disaggregated stage execution backend that optimizes resource utilization and throughput across stages. Each stage is independently served by an LLM or diffusion engine with per-stage request batching, flexible GPU allocation, and unified inter-stage connectors for data routing. Experimental results demonstrate that vLLM-Omni reduces job completion time (JCT) by up to 91.4% compared to baseline methods. The code is public available at https://github.com/vllm-project/vllm-omni.

cs.DC↗

arXiv2025

From cluster to nanocrystal: the continuous evolution and critical size of copper clusters revealed by machine learning

The evolution of cluster structure with size and the critical size for the transition from cluster to nanocrystal have long been fundamental problems in nanoscience. Due to limitations of experimental technology and computational methods, the exploration of the continuous evolution of clusters towards nanocrystal is still a big challenge. Here, we proposed a machine learning force field (MLFF) that can generalize well to various copper systems ranging from small clusters to large clusters and bulk. The continuous evolution of copper clusters CuN towards nanocrystal was revealed by investigating clusters in a wide size range (7 <= N <= 17885) based on MLFF simulated annealing. For small CuN (N < 40), electron counting rule plays a major role in stability. For large CuN (N > 80), geometric magic number rule plays a dominant role and the evolution of clusters is based on the formation of more and more icosahedral shells. For medium size CuN (40 <= N <= 80), both rules contribute. The critical size from cluster to nanocrystal was calculated to be around 8000 atoms (about 6 nm in diameter). Our work terminates the long-term challenge in nanoscience, and lay the methodological foundation for subsequent research on other cluster systems.

cond-mat.mtrl-sci↗

arXiv2025

PDEformer-2: A Versatile Foundation Model for Two-Dimensional Partial Differential Equations

Partial differential equations (PDEs) play a central role in describing many physical phenomena. Various scientific and engineering applications demand a versatile and differentiable PDE solver that can quickly generate solutions with adequate accuracy, and limitations of the traditional solvers and specialized neural operators motivate the development of foundation models for solving PDEs. This paper introduces PDEformer-2, a versatile foundation model for two-dimensional PDEs. Based on our previous one-dimensional PDEformer-1 model, PDEformer-2 receives the PDE form as network input via computational graph representation, which has the flexibility to encode most common PDEs. The mesh-free predicted solutions can be directly queried at arbitrary spatio-temporal coordinates. A large (40TB) diverse dataset is employed to pretrain the current model, making it capable of simultaneously addressing PDEs with different symbolic forms, domain shapes, boundary conditions, number of variables, and time-dependency. Accurate zero-shot prediction is allowed for PDEs that resemble the pretraining ones. When adapted to new unseen PDEs, PDEformer-2 demonstrates faster learning than many specialized models, and has smaller errors given limited (less than 100) samples. Additionally, PDEformer-2 can be employed in the inverse problems thanks to its fast and differentiable nature and produces reasonable results in our experiments to recover coefficient scalars and fields of a PDE.

math.NA↗

arXiv2025

Learnable-Differentiable Finite Volume Solver for Accelerated Simulation of Flows

Simulation of fluid flows is crucial for modeling physical phenomena like meteorology, aerodynamics, and biomedicine. Classical numerical solvers often require fine spatiotemporal grids to satisfy stability, consistency, and convergence conditions, leading to substantial computational costs. Although machine learning has demonstrated better efficiency, they typically suffer from issues of interpretability, generalizability, and data dependency. Hence, we propose a learnable and differentiable finite volume solver, called LDSolver, designed for efficient and accurate simulation of fluid flows on spatiotemporal coarse grids. LDSolver comprises two key components: (1) a differentiable finite volume solver, and (2) an learnable module providing equivalent approximation for fluxes (derivatives and interpolations), and temporal error correction on coarse grids. Even with limited training data (e.g., only a few trajectories), our model could accelerate the simulation while maintaining a high accuracy with superior generalizability. Experiments on different flow systems (e.g., Burgers, decaying, forced and shear flows) show that LDSolver achieves state-of-the-art performance, surpassing baseline models with notable margins.

cs.LG↗

arXiv2025

SlotPi: Physics-informed Object-centric Reasoning Models

Understanding and reasoning about dynamics governed by physical laws through visual observation, akin to human capabilities in the real world, poses significant challenges. Currently, object-centric dynamic simulation methods, which emulate human behavior, have achieved notable progress but overlook two critical aspects: 1) the integration of physical knowledge into models. Humans gain physical insights by observing the world and apply this knowledge to accurately reason about various dynamic scenarios; 2) the validation of model adaptability across diverse scenarios. Real-world dynamics, especially those involving fluids and objects, demand models that not only capture object interactions but also simulate fluid flow characteristics. To address these gaps, we introduce SlotPi, a slot-based physics-informed object-centric reasoning model. SlotPi integrates a physical module based on Hamiltonian principles with a spatio-temporal prediction module for dynamic forecasting. Our experiments highlight the model's strengths in tasks such as prediction and Visual Question Answering (VQA) on benchmark and fluid datasets. Furthermore, we have created a real-world dataset encompassing object interactions, fluid dynamics, and fluid-object interactions, on which we validated our model's capabilities. The model's robust performance across all datasets underscores its strong adaptability, laying a foundation for developing more advanced world models.

cs.CV↗

arXiv2025

Copper Vapor Catalyzing Role in the Growth of Graphene

Cu is most used substrate to grow monolayer graphene under a temperature near melting point. In this study, we elaborated a remarkable amount of Cu clusters were continuously evaporated during the graphene growth, resulting into the vapor pressure comparable with the CH4. Importantly, the decomposition barrier of CH4 on Cu clusters is similar or even lower than on Cu surface. CuCH4 serves as the primary active cluster in complex intermediates, exhibiting a growth-promoting effect. Particularly after the first graphene layer coverage, it may emerge as a dominant catalytic factor for multilayer growth by supplying critical growth species. Through controlled seed incorporation, this mechanism is expected to enable large-area controllable growth of bilayer and multilayer graphene structures.

cond-mat.mtrl-sci↗

arXiv2025

MultiPDENet: PDE-embedded Learning with Multi-time-stepping for Accelerated Flow Simulation

Solving partial differential equations (PDEs) by numerical methods meet computational cost challenge for getting the accurate solution since fine grids and small time steps are required. Machine learning can accelerate this process, but struggle with weak generalizability, interpretability, and data dependency, as well as suffer in long-term prediction. To this end, we propose a PDE-embedded network with multiscale time stepping (MultiPDENet), which fuses the scheme of numerical methods and machine learning, for accelerated simulation of flows. In particular, we design a convolutional filter based on the structure of finite difference stencils with a small number of parameters to optimize, which estimates the equivalent form of spatial derivative on a coarse grid to minimize the equation's residual. A Physics Block with a 4th-order Runge-Kutta integrator at the fine time scale is established that embeds the structure of PDEs to guide the prediction. To alleviate the curse of temporal error accumulation in long-term prediction, we introduce a multiscale time integration approach, where a neural network is used to correct the prediction error at a coarse time scale. Experiments across various PDE systems, including the Navier-Stokes equations, demonstrate that MultiPDENet can accurately predict long-term spatiotemporal dynamics, even given small and incomplete training data, e.g., spatiotemporally down-sampled datasets. MultiPDENet achieves the state-of-the-art performance compared with other neural baseline models, also with clear speedup compared to classical numerical methods.

math.NA↗

arXiv2024

Conservation-informed Graph Learning for Spatiotemporal Dynamics Prediction

Data-centric methods have shown great potential in understanding and predicting spatiotemporal dynamics, enabling better design and control of the object system. However, deep learning models often lack interpretability, fail to obey intrinsic physics, and struggle to cope with the various domains. While geometry-based methods, e.g., graph neural networks (GNNs), have been proposed to further tackle these challenges, they still need to find the implicit physical laws from large datasets and rely excessively on rich labeled data. In this paper, we herein introduce the conservation-informed GNN (CiGNN), an end-to-end explainable learning framework, to learn spatiotemporal dynamics based on limited training data. The network is designed to conform to the general conservation law via symmetry, where conservative and non-conservative information passes over a multiscale space enhanced by a latent temporal marching strategy. The efficacy of our model has been verified in various spatiotemporal systems based on synthetic and real-world datasets, showing superiority over baseline models. Results demonstrate that CiGNN exhibits remarkable accuracy and generalizability, and is readily applicable to learning for prediction of various spatiotemporal dynamics in a spatial domain with complex geometry.

cs.LG↗

arXiv2024

P$^2$C$^2$Net: PDE-Preserved Coarse Correction Network for efficient prediction of spatiotemporal dynamics

When solving partial differential equations (PDEs), classical numerical methods often require fine mesh grids and small time stepping to meet stability, consistency, and convergence conditions, leading to high computational cost. Recently, machine learning has been increasingly utilized to solve PDE problems, but they often encounter challenges related to interpretability, generalizability, and strong dependency on rich labeled data. Hence, we introduce a new PDE-Preserved Coarse Correction Network (P$^2$C$^2$Net) to efficiently solve spatiotemporal PDE problems on coarse mesh grids in small data regimes. The model consists of two synergistic modules: (1) a trainable PDE block that learns to update the coarse solution (i.e., the system state), based on a high-order numerical scheme with boundary condition encoding, and (2) a neural network block that consistently corrects the solution on the fly. In particular, we propose a learnable symmetric Conv filter, with weights shared over the entire model, to accurately estimate the spatial derivatives of PDE based on the neural-corrected system state. The resulting physics-encoded model is capable of handling limited training data (e.g., 3--5 trajectories) and accelerates the prediction of PDE solutions on coarse spatiotemporal grids while maintaining a high accuracy. P$^2$C$^2$Net achieves consistent state-of-the-art performance with over 50\% gain (e.g., in terms of relative prediction error) across four datasets covering complex reaction-diffusion processes and turbulent flows.

math.NA↗

arXiv2024

PhyMPGN: Physics-encoded Message Passing Graph Network for spatiotemporal PDE systems

Solving partial differential equations (PDEs) serves as a cornerstone for modeling complex dynamical systems. Recent progresses have demonstrated grand benefits of data-driven neural-based models for predicting spatiotemporal dynamics (e.g., tremendous speedup gain compared with classical numerical methods). However, most existing neural models rely on rich training data, have limited extrapolation and generalization abilities, and suffer to produce precise or reliable physical prediction under intricate conditions (e.g., irregular mesh or geometry, complex boundary conditions, diverse PDE parameters, etc.). To this end, we propose a new graph learning approach, namely, Physics-encoded Message Passing Graph Network (PhyMPGN), to model spatiotemporal PDE systems on irregular meshes given small training datasets. Specifically, we incorporate a GNN into a numerical integrator to approximate the temporal marching of spatiotemporal dynamics for a given PDE system. Considering that many physical phenomena are governed by diffusion processes, we further design a learnable Laplace block, which encodes the discrete Laplace-Beltrami operator, to aid and guide the GNN learning in a physically feasible solution space. A boundary condition padding strategy is also designed to improve the model convergence and accuracy. Extensive experiments demonstrate that PhyMPGN is capable of accurately predicting various types of spatiotemporal dynamics on coarse unstructured meshes, consistently achieves the state-of-the-art results, and outperforms other baselines with considerable gains.

cs.LG↗

arXiv2024

PDEformer-1: A Foundation Model for One-Dimensional Partial Differential Equations

This paper introduces PDEformer-1, a versatile neural solver capable of simultaneously addressing various partial differential equations (PDEs). With the PDE represented as a computational graph, we facilitate the seamless integration of symbolic and numeric information inherent in a PDE. A graph Transformer and an implicit neural representation (INR) are employed subsequently to generate mesh-free predicted solutions. We generated a dataset with up to three million samples involving diverse one-dimensional PDEs to pretrain our model. Compared with baseline models trained specifically on benchmark datasets, our pretrained model achieves comparable accuracy via zero-shot inference, and the advantage expands after finetuning. For PDEs new or unseen in the pretraining stage, our model can adapt quickly by finetuning on a relatively small set of examples from the target equation. Additionally, PDEformer-1 demonstrates promising results in the inverse problem of PDE scalar coefficient recovery and coefficient field recovery.

math.NA↗

arXiv2024

Intermediates of Forming Transition Metal Dichalcogenide Heterostructures Revealed by Machine Learning Simulations

Two-dimensional (2D) transition metal dichalcogenide (TMD) van der Waals heterostructures (vdWHs) hold promise for high-performance electronics, but their large-scale synthesis remains limited by size constraints and alloying contaminations. Recently, a two-step vapor deposition method was reported for growing wafer-size TMD vdWHs with minimal impurities. In this study, we develop a machine learning potential (MLP) that accurately captures the atomic-scale dynamic growth process of bilayer MoS$_2$/WS$_2$ vdWHs under feasible growth conditions. Our simulations uncover a crucial metastable SMMS (M = Mo or W) intermediate structure that facilitates metal atom swap and alloying. Eliminating the alloying contamination requires preventing the embedding of bare metal atoms. The results also show that the SMMS structure exhibits favourable electronic properties and emerges as a low Schottky barrier contact electrode for MoS$_2$ field-effect transistors (FETs).

cond-mat.mtrl-sci↗

arXiv2024

PDEformer: Towards a Foundation Model for One-Dimensional Partial Differential Equations

This paper introduces PDEformer, a neural solver for partial differential equations (PDEs) capable of simultaneously addressing various types of PDEs. We propose to represent the PDE in the form of a computational graph, facilitating the seamless integration of both symbolic and numerical information inherent in a PDE. A graph Transformer and an implicit neural representation (INR) are employed to generate mesh-free predicted solutions. Following pretraining on data exhibiting a certain level of diversity, our model achieves zero-shot accuracies on benchmark datasets that is comparable to those of specifically trained expert models. Additionally, PDEformer demonstrates promising results in the inverse problem of PDE coefficient recovery.

math.NA↗

arXiv2024

Meta-Auto-Decoder: A Meta-Learning Based Reduced Order Model for Solving Parametric Partial Differential Equations

Many important problems in science and engineering require solving the so-called parametric partial differential equations (PDEs), i.e., PDEs with different physical parameters, boundary conditions, shapes of computational domains, etc. Typical reduced order modeling techniques accelarate solution of the parametric PDEs by projecting them onto a linear trial manifold constructed in the offline stage. These methods often need a predefined mesh as well as a series of precomputed solution snapshots, andmay struggle to balance between efficiency and accuracy due to the limitation of the linear ansatz. Utilizing the nonlinear representation of neural networks, we propose Meta-Auto-Decoder (MAD) to construct a nonlinear trial manifold, whose best possible performance is measured theoretically by the decoder width. Based on the meta-learning concept, the trial manifold can be learned in a mesh-free and unsupervised way during the pre-training stage. Fast adaptation to new (possibly heterogeneous) PDE parameters is enabled by searching on this trial manifold, and optionally fine-tuning the trial manifold at the same time. Extensive numerical experiments show that the MAD method exhibits faster convergence speed without losing accuracy than other deep learning-based methods.

math.NA↗

arXiv2023

Sumanene monolayer of pure carbon: a two-dimensional Kagome-analogy lattice with desirable band gap, ultrahigh carrier mobility and strong exciton binding energy

Design and synthesis of novel two-dimensional (2D) materials that possess robust structural stability and unusual physical properties may open up enormous opportunities for device and engineering applications. Herein we propose a 2D sumanene lattice that be regarded as a derivative of the conventional Kagome lattice. Our tight-binding analysis demonstrates sumanene lattice contains two sets of Dirac cones and two sets of flat bands near the Fermi surface, distinctively different from the Kagome lattice. Using first-principles calculations, we theoretically suggest two possible routines for realization of stable 2D sumanene monolayers (named as a phase and b phase), and a-sumanene monolayer can be experimentally synthesized with chemical vapor deposition using C21H12 as a precursor. Small binding energies on Au(111) surface signify the possibility of their peel-off after grown on the noble metal substrate. Importantly, our GW plus Bethe-Salpeter equation calculations demonstrate both monolayers have moderate band gaps (1.94 eV for a) and ultrahigh carrier mobilities (3.4*104 cm2/Vs for a). In particular, a-sumanene monolayer possesses a strong exciton binding energy of 0.73 eV, suggesting potential applications in optics.

cond-mat.mtrl-sci↗

arXiv2023

Robust Sandwiched B/TM/B Structures by Metal Intercalating into Bilayer Borophene Leading to Excellent Hydrogen Evolution Reaction

Bilayer borophene, very recently synthesized on Ag and Cu, possesses extremely flat large surface and excellent conductivity. Besides, the van der Waals gap of bilayer borophene can be intercalated by metal atoms, thereby tailoring the properties of bilayer borophene. Herein, we propose that sandwiched B/TM/B (TM=Co, Ni, Cu, Pd) could be a new 2D formation by transiton metal atoms intercalated into bilayer borophene network, it is quiet robust with both energetic, structural and thermal stability, and exhibits heat resistance of at least 1300 K. Besides, it is novel platform for electrocatalytic hydrogen evolution reaction (HER). The interecalation metal atom serves as single-atomic catalyst, which acting the nonmetal boron layers. Beyond that, the transtion metal is protected by outside boron layers from being corroded by acidic/alkaline solution. B/Cux/B, B/Pdx/B and B/Alx/B with different metal coverage exhibit defect-independent extremely low HER free energy in the range of -0.162 ~ 0.179 eV, -0.134 ~ 0.183 eV and -0.082 ~ 0.086 eV which are comparable to noble metal Pt. Combining excellent conduction, high structural and thermal stability, low resistance to intercalated behaviour, effortless water splitting process, excellent defect-independent catalytic performance, cheapness and abundance of raw materials, free of corrodation, 2D sandwiched B/TM/B (TM=Co, Ni, Cu, Pd) is believed to promising for electrocatalytic HER applications.

cond-mat.mtrl-sci↗