arXiv subjects

Bin Gao

Publications and source records attributed to Bin Gao.

At least 55 records · Page 3Linked to original sources

arXiv2025

AdaSkip: Adaptive Sublayer Skipping for Accelerating Long-Context LLM Inference

Long-context large language models (LLMs) inference is increasingly critical, motivating a number of studies devoted to alleviating the substantial storage and computational costs in such scenarios. Layer-wise skipping methods are promising optimizations but rarely explored in long-context inference. We observe that existing layer-wise skipping strategies have several limitations when applied in long-context inference, including the inability to adapt to model and context variability, disregard for sublayer significance, and inapplicability for the prefilling phase. This paper proposes \sysname, an adaptive sublayer skipping method specifically designed for long-context inference. \sysname adaptively identifies less important layers by leveraging on-the-fly similarity information, enables sublayer-wise skipping, and accelerates both the prefilling and decoding phases. The effectiveness of \sysname is demonstrated through extensive experiments on various long-context benchmarks and models, showcasing its superior inference performance over existing baselines.

cs.CL↗

arXiv2024

Desingularization of bounded-rank tensor sets

Low-rank tensors appear to be prosperous in many applications. However, the sets of bounded-rank tensors are non-smooth and non-convex algebraic varieties, rendering the low-rank optimization problems to be challenging. To this end, we delve into the geometry of bounded-rank tensor sets, including Tucker and tensor train formats. We propose a desingularization approach for bounded-rank tensor sets by introducing slack variables, resulting in a low-dimensional smooth manifold embedded in a higher-dimensional space while preserving the structure of low-rank tensor formats. Subsequently, optimization on tensor varieties can be reformulated to optimization on smooth manifolds, where the methods and convergence are well explored. We reveal the relationship between the landscape of optimization on varieties and that of optimization on manifolds. Numerical experiments on tensor completion illustrate that the proposed methods are in favor of others under different rank parameters.

math.OC↗

arXiv2024

Infeasible Deterministic, Stochastic, and Variance-Reduction Algorithms for Optimization under Orthogonality Constraints

Orthogonality constraints naturally appear in many machine learning problems, from principal component analysis to robust neural network training. They are usually solved using Riemannian optimization algorithms, which minimize the objective function while enforcing the constraint. However, enforcing the orthogonality constraint can be the most time-consuming operation in such algorithms. Recently, Ablin & Peyré (2022) proposed the landing algorithm, a method with cheap iterations that does not enforce the orthogonality constraints but is attracted towards the manifold in a smooth manner. This article provides new practical and theoretical developments for the landing algorithm. First, the method is extended to the Stiefel manifold, the set of rectangular orthogonal matrices. We also consider stochastic and variance reduction algorithms when the cost function is an average of many functions. We demonstrate that all these methods have the same rate of convergence as their Riemannian counterparts that exactly enforce the constraint, and converge to the manifold. Finally, our experiments demonstrate the promise of our approach to an array of machine-learning problems that involve orthogonality constraints.

stat.ML↗

arXiv2024

Absence of Acoustic Phonon Anomaly in a Kagome Metal with Short-ranged Structural Modulation

Kagome lattice $A$V$_3$Sb$_5$ ($A$ = K, Rb, and Cs) superconductors without magnetism from vanadium $d$-electrons are intriguing because they have a novel charge density wave (CDW) order around 90 K and display superconductivity at $\sim$3 K that competes with the CDW order. Recently, CsCr$_3$Sb$_5$, isostructural to $A$V$_3$Sb$_5$, was found to have concurrent structural and magnetic phase transition at $T^{\ast}\approx$ 55 K that can be suppressed by pressure to induce superconductivity [Liu \textit{et al.}, \href{https://doi.org/10.1038/s41586-024-07761-x}{Nature \textbf{632}, 1032 (2024)}]. Here, we use elastic and inelastic X-ray scattering to study the microscopic origin of the structural transition in CsCr$_3$Sb$_5$. Although our elastic measurements confirm the 4$\times$1$\times$1 superlattice order below $T^{\ast}$, its underlying correlation is rather short-ranged. Moreover, our inelastic measurements at the superlattice wavevectors around (3, 0, 0) find no evidence of a significant acoustic phonon anomaly below $T^{\ast}$, similar to the case of $A$V$_3$Sb$_5$. The absence of acoustic phonon anomaly indicates a weak electron-phonon coupling in CsCr$_3$Sb$_5$, suggesting that the structural transition is likely associated with an unconventional CDW order.

cond-mat.str-el↗

arXiv2024

Spin Excitation Continuum in the Exactly Solvable Triangular-Lattice Spin Liquid CeMgAl11O19

In magnetically ordered insulators, elementary quasiparticles manifest as spin waves - collective motions of localized magnetic moments propagating through the lattice - observed via inelastic neutron scattering. In effective spin-1/2 systems where geometric frustrations suppress static magnetic order, spin excitation continua can emerge, either from degenerate classical spin ground states or from entangled quantum spins characterized by emergent gauge fields and deconfined fractionalized excitations. Comparing the spin Hamiltonian with theoretical models can unveil the microscopic origins of these zero-field spin excitation continua. Here, we use neutron scattering to study spin excitations of the two-dimensional (2D) triangular-lattice effective spin-1/2 antiferromagnet CeMgAl11O19. Analyzing the spin waves in the field-polarized ferromagnetic state, we find that the spin Hamiltonian is close to an exactly solvable 2D triangular-lattice XXZ model, where degenerate 120$^\circ$ ordered ground states - umbrella states - develop in the zero temperature limit. We then find that the observed zero-field spin excitation continuum matches the calculated ensemble of spin waves from the umbrella state manifold, and thus conclude that CeMgAl11O19 is the first example of an exactly solvable spin liquid on a triangular lattice where the spin excitation continuum arises from the ground state degeneracy.

cond-mat.str-el↗

arXiv2024

On the analysis of optimization with fixed-rank matrices: a quotient geometric view

We study a type of Riemannian gradient descent (RGD) algorithm, designed through Riemannian preconditioning, for optimization on $\mathcal{M}_k^{m\times n}$ -- the set of $m\times n$ real matrices with a fixed rank $k$. Our analysis is based on a quotient geometric view of $\mathcal{M}_k^{m\times n}$: by identifying this set with the quotient manifold of a two-term product space $\mathbb{R}_*^{m\times k}\times \mathbb{R}_*^{n\times k}$ of matrices with full column rank via matrix factorization, we find an explicit form for the update rule of the RGD algorithm, which leads to a novel approach to analysing their convergence behavior in rank-constrained optimization. We then deduce some interesting properties that reflect how RGD distinguishes from other matrix factorization algorithms such as those based on the Euclidean geometry. In particular, we show that the RGD algorithm is not only faster than Euclidean gradient descent but also does not rely on balancing techniques to ensure its efficiency while the latter does. We further show that this RGD algorithm is guaranteed to solve matrix sensing and matrix completion problems with linear convergence rate under the restricted positive definiteness property. Numerical experiments on matrix sensing and completion are provided to demonstrate these properties.

math.OC↗

arXiv2024

Low-rank optimization on Tucker tensor varieties

In the realm of tensor optimization, the low-rank Tucker decomposition is crucial for reducing the number of parameters and for saving storage. We explore the geometry of Tucker tensor varieties -- the set of tensors with bounded Tucker rank -- which is notably more intricate than the well-explored matrix varieties. We give an explicit parametrization of the tangent cone of Tucker tensor varieties and leverage its geometry to develop provable gradient-related line-search methods for optimization on Tucker tensor varieties. To the best of our knowledge, this is the first work concerning geometry and optimization on Tucker tensor varieties. In practice, low-rank tensor optimization suffers from the difficulty of choosing a reliable rank parameter. To this end, we incorporate the established geometry and propose a Tucker rank-adaptive method that aims to identify an appropriate rank with guaranteed convergence. Numerical experiments on tensor completion reveal that the proposed methods are in favor of recovering performance over other state-of-the-art methods. The rank-adaptive method performs the best across various rank parameter selections and is indeed able to find an appropriate rank.

math.OC↗

arXiv2024

Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention

Interacting with humans through multi-turn conversations is a fundamental feature of large language models (LLMs). However, existing LLM serving engines executing multi-turn conversations are inefficient due to the need to repeatedly compute the key-value (KV) caches of historical tokens, incurring high serving costs. To address the problem, this paper proposes CachedAttention, a new attention mechanism that enables reuse of KV caches across multi-turn conversations, significantly reducing the repetitive computation overheads. CachedAttention maintains a hierarchical KV caching system that leverages cost-effective memory/storage mediums to save KV caches for all requests. To reduce KV cache access overheads from slow mediums, CachedAttention employs layer-wise pre-loading and asynchronous saving schemes to overlap the KV cache access with the GPU computation. To ensure that the KV caches to be accessed are placed in the fastest hierarchy, CachedAttention employs scheduler-aware fetching and eviction schemes to consciously place the KV caches in different layers based on the hints from the inference job scheduler. To avoid the invalidation of the saved KV caches incurred by context window overflow, CachedAttention enables the saved KV caches to remain valid via decoupling the positional encoding and effectively truncating the KV caches. Extensive experimental results demonstrate that CachedAttention significantly decreases the time to the first token (TTFT) by up to 87%, improves the prompt prefilling throughput by up to 7.8$\times$ for multi-turn conversations, and reduces the end-to-end inference cost by up to 70%.

cs.CL↗

arXiv2024

Symplectic Stiefel manifold: tractable metrics, second-order geometry and Newton's methods

Optimization under the symplecticity constraint is an approach for solving various problems in quantum physics and scientific computing. Building on the results that this optimization problem can be transformed into an unconstrained problem on the symplectic Stiefel manifold, we construct geometric ingredients for Riemannian optimization with a new family of Riemannian metrics called tractable metrics and develop Riemannian Newton schemes. The newly obtained ingredients do not only generalize several existing results but also provide us with freedom to choose a suitable metric for each problem. To the best of our knowledge, this is the first try to develop the explicit second-order geometry and Newton's methods on the symplectic Stiefel manifold. For the Riemannian Newton method, we first consider novel operator-valued formulas for computing the Riemannian Hessian of a~cost function, which further allows the manifold to be endowed with a weighted Euclidean metric that can provide a preconditioning effect. We then solve the resulting Newton equation, as the central step of Newton's methods, directly via transforming it into a~saddle point problem followed by vectorization, or iteratively via applying any matrix-free iterative method either to the operator Newton equation or its saddle point formulation. Finally, we propose a hybrid Riemannian Newton optimization algorithm that enjoys both global convergence and quadratic/superlinear local convergence at the final stage. Various numerical experiments are presented to validate the proposed methods.

math.OC↗

arXiv2024

Spin-charge-lattice coupling across the charge density wave transition in a Kagome lattice antiferromagnet

Understanding spin and lattice excitations in a metallic magnetic ordered system form the basis to unveil the magnetic and lattice exchange couplings and their interactions with itinerant electrons. Kagome lattice antiferromagnet FeGe is interesting because it displays rare charge density wave (CDW) deep inside the antiferromagnetic ordered phase that interacts with the magnetic order. We use neutron scattering to study the evolution of spin and lattice excitations across the CDW transition $T_{\rm CDW}$ in FeGe. While spin excitations below $\sim$100 meV can be well described by spin waves of a spin-1 Heisenberg Hamiltonian, spin excitations at higher energies are centered around the Brillouin zone boundary and extend up to $\sim180$ meV consistent with quasiparticle excitations across spin-polarized electron-hole Fermi surfaces. Furthermore, $c$-axis spin wave dispersion and Fe-Ge optical phonon modes show a clear hardening below $T_{\rm CDW}$ due to spin-charge-lattice coupling but with no evidence for a phonon Kohn anomaly. By comparing our experimental results with density functional theory calculations in absolute units, we conclude that FeGe is a Hund's metal in the intermediate correlated regime where magnetism has contributions from both itinerant and localized electrons arising from spin polarized electronic bands near the Fermi level.

cond-mat.str-el↗

arXiv2024

Emergent photons and fractionalized excitations in a quantum spin liquid

A quantum spin liquid (QSL) arises from a highly entangled superposition of many degenerate classical ground states in a frustrated magnet, and is characterized by emergent gauge fields and deconfined fractionalized excitations (spinons). Because such a novel phase of matter is relevant to high-transition-temperature superconductivity and quantum computation, the microscopic understanding of QSL states is a long-sought goal in condensed matter physics. The 3D pyrochlore lattice of corner-sharing tetrahedra can host a QSL with U(1) gauge fields called quantum spin ice (QSI), which is a quantum (with effective $S=1/2$) analog of the classical (with large effective moment) spin ice. A key difference between QSI and classical spin ice is the predicted presence of the linearly dispersing collective excitations near zero energy, dubbed the "photons", arising from emergent quantum electrodynamics, in addition to the spinons at higher energies. Recently, 3D pyrochlore systems Ce2M2O7 (M = Sn, Zr, Hf) have been suggested as effective $S=1/2$ QSI candidates, but there has been no evidence of quasielastic magnetic scattering signals from photons, a key signature for a QSI. Here, we use polarized neutron scattering experiments on single crystals of Ce2Zr2O7 to conclusively demonstrate the presence of magnetic excitations near zero energy at 50 mK in addition to signatures of spinons at higher energies. By comparing the energy (E), wave vector (Q), and polarization dependence of the magnetic excitations with theoretical calculations, we conclude that Ce2Zr2O7 is the first example of a dipolar-octupolar $π$ flux QSI with dominant dipolar Ising interactions, therefore identifying a microscopic Hamiltonian responsible for a QSL.

cond-mat.str-el↗

arXiv2024

Symmetry breaking and ascending in the magnetic kagome metal FeGe

Spontaneous symmetry breaking-the phenomenon where an infinitesimal perturbation can cause the system to break the underlying symmetry-is a cornerstone concept in the understanding of interacting solid-state systems. In a typical series of temperature-driven phase transitions, higher temperature phases are more symmetric due to the stabilizing effect of entropy that becomes dominant as the temperature is increased. However, the opposite is rare but possible when there are multiple degrees of freedom in the system. Here, we present such an example of a symmetry-ascending phenomenon in a magnetic kagome metal FeGe by utilizing neutron Larmor diffraction and Raman spectroscopy. In the paramagnetic state at 460K, we confirm that the crystal structure is indeed hexagonal kagome lattice. On cooling to TN, the crystal structure changes from hexagonal to monoclinic with in-plane lattice distortions on the order of 10^(-4) and the associated splitting of the double degenerate phonon mode of the pristine kagome lattice. Upon further cooling to TCDW, the kagome lattice shows a small negative thermal expansion, and the crystal structure becomes more symmetric gradually upon further cooling. Increasing the crystalline symmetry upon cooling is unusual, it originates from an extremely weak structural instability that coexists and competes with the CDW and magnetic orders. These observations are against the expectations for a simple model with a single order parameter, hence can only be explained by a Landau free energy expansion that takes into account multiple lattice, charge, and spin degrees of freedom. Thus, the determination of the crystalline lattice symmetry as well as the unusual spin-lattice coupling is a first step towards understanding the rich electronic and magnetic properties of the system and sheds new light on intertwined orders where the lattice degree of freedom is no longer dominant.

cond-mat.str-el↗

arXiv2024

numaPTE: Managing Page-Tables and TLBs on NUMA Systems

Memory management operations that modify page-tables, typically performed during memory allocation/deallocation, are infamous for their poor performance in highly threaded applications, largely due to process-wide TLB shootdowns that the OS must issue due to the lack of hardware support for TLB coherence. We study these operations in NUMA settings, where we observe up to 40x overhead for basic operations such as munmap or mprotect. The overhead further increases if page-table replication is used, where complete coherent copies of the page-tables are maintained across all NUMA nodes. While eager system-wide replication is extremely effective at localizing page-table reads during address translation, we find that it creates additional penalties upon any page-table changes due to the need to maintain all replicas coherent. In this paper, we propose a novel page-table management mechanism, called numaPTE, to enable transparent, on-demand, and partial page-table replication across NUMA nodes in order to perform address translation locally, while avoiding the overheads and scalability issues of system-wide full page-table replication. We then show that numaPTE's precise knowledge of page-table sharers can be leveraged to significantly reduce the number of TLB shootdowns issued upon any memory-management operation. As a result, numaPTE not only avoids replication-related slowdowns, but also provides significant speedup over the baseline on memory allocation/deallocation and access control operations. We implement numaPTEin Linux on x86_64, evaluate it on 4- and 8-socket systems, and show that numaPTE achieves the full benefits of eager page-table replication on a wide range of applications, while also achieving a 12% and 36% runtime improvement on Webserver and Memcached respectively due to a significant reduction in TLB shootdowns.

cs.OS↗

arXiv2024

Noncollinear magnetic order, in-plane anisotropy, and magnetoelectric coupling in a pyroelectric honeycomb antiferromagnet Ni$_{2}$Mo$_{3}$O$_{8}$

Ni$_{2}$Mo$_{3}$O$_{8}$ is a pyroelectric honeycomb antiferromagnet exhibiting peculiar changes of its electric polarization at magnetic transitions. Ni$_{2}$Mo$_{3}$O$_{8}$ stands out from the isostructural magnetic compounds, showing an anomalously low magnetic transition temperature and unique magnetic anisotropy. We determine the magnetic structure of Ni$_{2}$Mo$_{3}$O$_{8}$ utilizing high-resolution powder and single-crystal neutron diffraction. A noncollinear stripy antiferromagnetic order is found in the honeycomb planes. The magnetic space group is \textit{P$_C$na}2$_1$. The in-plane magnetic connection is of the stripy type both for the $ab$-plane and the $c$-axis spin components. This is a simpler connection than the one proposed previously. The ferromagnetic interlayer order of the $c$-axis spin components in our model is also distinct. The magnetic anisotropy of Ni$_{2}$Mo$_{3}$O$_{8}$ is characterized by orientation-dependent magnetic susceptibility measurements on a single crystal, consistent with neutron diffraction analysis. The local magnetoelectric tensor analysis using our magnetic models provides new insights into its magnetoelectric coupling and polarization. Thus, our results deliver essential information for understanding both the unusual magnetoelectric properties of Ni$_{2}$Mo$_{3}$O$_{8}$ and the prospects for observation of exotic nonreciprocal, Hall, and magnonic effects characteristic to this compound family.

cond-mat.str-el↗

arXiv2024

Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunities

The rise of large foundation models, trained on extensive datasets, is revolutionizing the field of AI. Models such as SAM, DALL-E2, and GPT-4 showcase their adaptability by extracting intricate patterns and performing effectively across diverse tasks, thereby serving as potent building blocks for a wide range of AI applications. Autonomous driving, a vibrant front in AI applications, remains challenged by the lack of dedicated vision foundation models (VFMs). The scarcity of comprehensive training data, the need for multi-sensor integration, and the diverse task-specific architectures pose significant obstacles to the development of VFMs in this field. This paper delves into the critical challenge of forging VFMs tailored specifically for autonomous driving, while also outlining future directions. Through a systematic analysis of over 250 papers, we dissect essential techniques for VFM development, including data preparation, pre-training strategies, and downstream task adaptation. Moreover, we explore key advancements such as NeRF, diffusion models, 3D Gaussian Splatting, and world models, presenting a comprehensive roadmap for future research. To empower researchers, we have built and maintained https://github.com/zhanghm1995/Forge_VFM4AD, an open-access repository constantly updated with the latest advancements in forging VFMs for autonomous driving.

cs.CV↗

arXiv2023

Segment Anything in Defect Detection

Defect detection plays a crucial role in infrared non-destructive testing systems, offering non-contact, safe, and efficient inspection capabilities. However, challenges such as low resolution, high noise, and uneven heating in infrared thermal images hinder comprehensive and accurate defect detection. In this study, we propose DefectSAM, a novel approach for segmenting defects on highly noisy thermal images based on the widely adopted model, Segment Anything (SAM)\cite{kirillov2023segany}. Harnessing the power of a meticulously curated dataset generated through labor-intensive lab experiments and valuable prompts from experienced experts, DefectSAM surpasses existing state-of-the-art segmentation algorithms and achieves significant improvements in defect detection rates. Notably, DefectSAM excels in detecting weaker and smaller defects on complex and irregular surfaces, reducing the occurrence of missed detections and providing more accurate defect size estimations. Experimental studies conducted on various materials have validated the effectiveness of our solutions in defect detection, which hold significant potential to expedite the evolution of defect detection tools, enabling enhanced inspection capabilities and accuracy in defect identification.

cs.CV↗

arXiv2023

Riemannian preconditioned algorithms for tensor completion via tensor ring decomposition

We propose Riemannian preconditioned algorithms for the tensor completion problem via tensor ring decomposition. A new Riemannian metric is developed on the product space of the mode-2 unfolding matrices of the core tensors in tensor ring decomposition. The construction of this metric aims to approximate the Hessian of the cost function by its diagonal blocks, paving the way for various Riemannian optimization methods. Specifically, we propose the Riemannian gradient descent and Riemannian conjugate gradient algorithms. We prove that both algorithms globally converge to a stationary point. In the implementation, we exploit the tensor structure and adopt an economical procedure to avoid large matrix formulation and computation in gradients, which significantly reduces the computational cost. Numerical experiments on various synthetic and real-world datasets -- movie ratings, hyperspectral images, and high-dimensional functions -- suggest that the proposed algorithms are more efficient and have better reconstruction ability than other candidates.

math.OC↗

arXiv2023

Alternating minimization algorithms for graph regularized tensor completion

We consider a Canonical Polyadic (CP) decomposition approach to low-rank tensor completion (LRTC) by incorporating external pairwise similarity relations through graph Laplacian regularization on the CP factor matrices. The usage of graph regularization entails benefits in the learning accuracy of LRTC, but at the same time, induces coupling graph Laplacian terms that hinder the optimization of the tensor completion model. In order to solve graph-regularized LRTC, we propose efficient alternating minimization algorithms by leveraging the block structure of the underlying CP decomposition-based model. For the subproblems of alternating minimization, a linear conjugate gradient subroutine is specifically adapted to graph-regularized LRTC. Alternatively, we circumvent the complicating coupling effects of graph Laplacian terms by using an alternating directions method of multipliers. Based on the Kurdyka-Łojasiewicz property, we show that the sequence generated by the proposed algorithms globally converges to a critical point of the objective function. Moreover, the complexity and convergence rate are also derived. In addition, numerical experiments including synthetic data and real data show that the graph regularized tensor completion model has improved recovery results compared to those without graph regularization, and that the proposed algorithms achieve gains in time efficiency over existing algorithms.

math.NA↗