arXiv subjects

Liqian Peng

Publications and source records attributed to Liqian Peng.

13 recordsLinked to original sources

arXiv2026

Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens

Large language models (LLMs) have demonstrated impressive reasoning capabilities by scaling test-time compute via long Chain-of-Thought (CoT). However, recent findings suggest that raw token counts are unreliable proxies for reasoning quality: increased generation length does not consistently correlate with accuracy and may instead signal "overthinking," leading to performance degradation. In this work, we quantify inference-time effort by identifying deep-thinking tokens -- tokens where internal predictions undergo significant revisions in deeper model layers prior to convergence. Across four challenging mathematical and scientific benchmarks (AIME 24/25, HMMT 25, and GPQA-diamond) and a diverse set of reasoning-focused models (GPT-OSS, DeepSeek-R1, and Qwen3), we show that deep-thinking ratio (the proportion of deep-thinking tokens in a generated sequence) exhibits a robust and consistently positive correlation with accuracy, substantially outperforming both length-based and confidence-based baselines. Leveraging this insight, we introduce Think@n, a test-time scaling strategy that prioritizes samples with high deep-thinking ratios. We demonstrate that Think@n matches or exceeds standard self-consistency performance while significantly reducing inference costs by enabling the early rejection of unpromising generations based on short prefixes.

cs.CL↗

arXiv2025

Compressing Many-Shots in In-Context Learning

Large Language Models (LLMs) have been shown to be able to learn different tasks without explicit finetuning when given many input-output examples / demonstrations through In-Context Learning (ICL). Increasing the number of examples, called ``shots'', improves downstream task performance but incurs higher memory and computational costs. In this work, we study an approach to improve the memory and computational efficiency of ICL inference by compressing the many-shot prompts. Given many shots comprising t tokens, our goal is to generate a m soft-token summary, where m < t. We first show that existing prompt compression methods are ineffective for many-shot compression, and simply using fewer shots as a baseline is surprisingly strong. To achieve effective compression, we find that: (a) a stronger compressor model with more trainable parameters is necessary, and (b) compressing many-shot representations at each transformer layer enables more fine-grained compression by providing each layer with its own compressed representation. Based on these insights, we propose MemCom, a layer-wise compression method. We systematically evaluate various compressor models and training approaches across different model sizes (2B and 7B), architectures (Gemma and Mistral), many-shot sequence lengths (3k-6k tokens), and compression ratios (3x to 8x). MemCom outperforms strong baselines across all compression ratios on multiple classification tasks with large label sets. Notably, while baseline performance degrades sharply at higher compression ratios, often by over 20-30%, MemCom maintains high accuracy with minimal degradation, typically dropping by less than 10%.

cs.LG↗

arXiv2025

You Only Fine-tune Once: Many-Shot In-Context Fine-Tuning for Large Language Models

Large language models (LLMs) possess a remarkable ability to perform in-context learning (ICL), which enables them to handle multiple downstream tasks simultaneously without requiring task-specific fine-tuning. Recent studies have shown that even moderately sized LLMs, such as Mistral 7B, Gemma 7B and Llama-3 8B, can achieve ICL through few-shot in-context fine-tuning of all tasks at once. However, this approach still lags behind dedicated fine-tuning, where a separate model is trained for each individual task. In this paper, we propose a novel approach, Many-Shot In-Context Fine-tuning (ManyICL), which significantly narrows this performance gap by extending the principles of ICL to a many-shot setting. To unlock the full potential of ManyICL and address the inherent inefficiency of processing long sequences with numerous in-context examples, we propose a novel training objective. Instead of solely predicting the final answer, our approach treats every answer within the context as a supervised training target. This effectively shifts the role of many-shot examples from prompts to targets for autoregressive learning. Through extensive experiments on diverse downstream tasks, including classification, summarization, question answering, natural language inference, and math, we demonstrate that ManyICL substantially outperforms zero/few-shot fine-tuning and approaches the performance of dedicated fine-tuning. Furthermore, ManyICL significantly mitigates catastrophic forgetting issues observed in zero/few-shot fine-tuning. The code will be made publicly available upon publication.

cs.CL↗

arXiv2025

Privacy-preserved LLM Cascade via CoT-enhanced Policy Learning

Large Language Models (LLMs) have gained significant attention in on-device applications due to their remarkable performance across real-world tasks. However, on-device LLMs often suffer from suboptimal performance due to hardware limitations. A promising solution to this challenge is cascading a weaker local (on-device) LLM with a more powerful server LLM. While existing research on LLM cascade primarily optimizes the performance-cost trade-off, real-world applications impose additional requirements, such as privacy preservation, which remain largely unaddressed. In this work, we move beyond existing confidence- and logit-based LLM cascade methods and propose $\mathbf{P^{3}Defer}$, a novel Chain-of-Thought (CoT)-enhanced \textbf{p}olicy learning framework for \textbf{p}rivacy-\textbf{p}reserved \textbf{defer}ral decision-making. Our approach effectively improves cascade efficiency while mitigating privacy risks. Extensive experiments on three benchmark datasets demonstrate the effectiveness and superiority of $\mathbf{P^{3}Defer}$ over existing methods.

cs.CL↗

arXiv2021

Non-intrusive Nonlinear Model Reduction via Machine Learning Approximations to Low-dimensional Operators

Although projection-based reduced-order models (ROMs) for parameterized nonlinear dynamical systems have demonstrated exciting results across a range of applications, their broad adoption has been limited by their intrusivity: implementing such a reduced-order model typically requires significant modifications to the underlying simulation code. To address this, we propose a method that enables traditionally intrusive reduced-order models to be accurately approximated in a non-intrusive manner. Specifically, the approach approximates the low-dimensional operators associated with projection-based reduced-order models (ROMs) using modern machine-learning regression techniques. The only requirement of the simulation code is the ability to export the velocity given the state and parameters as this functionality is used to train the approximated low-dimensional operators. In addition to enabling nonintrusivity, we demonstrate that the approach also leads to very low computational complexity, achieving up to $1000\times$ reduction in run time. We demonstrate the effectiveness of the proposed technique on two types of PDEs.

cs.LG↗

arXiv2018

Recovering missing CFD data for high-order discretizations using deep neural networks and dynamics learning

Data I/O poses a significant bottleneck in large-scale CFD simulations; thus, practitioners would like to significantly reduce the number of times the solution is saved to disk, yet retain the ability to recover any field quantity (at any time instance) a posteriori. The objective of this work is therefore to accurately recover missing CFD data a posteriori at any time instance, given that the solution has been written to disk at only a relatively small number of time instances. We consider in particular high-order discretizations (e.g., discontinuous Galerkin), as such techniques are becoming increasingly popular for the simulation of highly separated flows. To satisfy this objective, this work proposes a methodology consisting of two stages: 1) dimensionality reduction and 2) dynamics learning. For dimensionality reduction, we propose a novel hierarchical approach. First, the method reduces the number of degrees of freedom within each element of the high-order discretization by applying autoencoders from deep learning. Second, the methodology applies principal component analysis to compress the global vector of encodings. This leads to a low-dimensional state, which associates with a nonlinear embedding of the original CFD data. For dynamics learning, we propose to apply regression techniques (e.g., kernel methods) to learn the discrete-time velocity characterizing the time evolution of this low-dimensional state. A numerical example on a large-scale CFD example characterized by nearly 13 million degrees of freedom illustrates the suitability of the proposed method in an industrial setting.

physics.comp-ph↗

arXiv2017

Structure-preserving model reduction for marginally stable LTI systems

This work proposes a structure-preserving model reduction method for marginally stable linear time-invariant (LTI) systems. In contrast to Lyapunov-stability-based approaches---which ensure the poles of the reduced system remain in the open left-half plane---the proposed method preserves marginal stability by reducing the subsystem with poles on the imaginary axis in a manner that ensures those poles remain purely imaginary. In particular, the proposed method decomposes a marginally stable LTI system into (1) an asymptotically stable subsystem with eigenvalues in the open left-half plane and (2) a pure marginally stable subsystem with a purely imaginary spectrum. We propose a method based on inner-product projection and the Lyapunov inequality to reduce the first subsystem while preserving asymptotic stability. In addition, we demonstrate that the pure marginally stable subsystem is a generalized Hamiltonian system; we then propose a method based on symplectic projection to reduce this subsystem while preserving pure marginal stability. In addition, we propose both inner-product and symplectic balancing methods that balance the operators associated with two quadratic energy functionals while preserving asymptotic and pure marginal stability, respectively. We formulate a geometric perspective that enables a unified comparison of the proposed inner-product and symplectic projection methods. Numerical examples illustrate the ability of the method to reduce the dimensionality of marginally stable LTI systems while retaining accuracy and preserving marginal stability; further, the resulting reduced-order model yields a finite infinite-time energy, which arises from the pure marginally stable subsystem.

math.DS↗

arXiv2016

Structure-Preserving Model Reduction of Forced Hamiltonian Systems

This paper reports a development in the proper symplectic decomposition (PSD) for model reduction of forced Hamiltonian systems. As an analogy to the proper orthogonal decomposition (POD), PSD is designed to build a symplectic subspace to fit empirical data. Our aim is two-fold. First, to achieve computational savings for large-scale Hamiltonian systems with external forces. Second, to simultaneously preserve the symplectic structure and the forced structure of the original system. We first reformulate d'Alembert's principle in the Hamiltonian form. Corresponding to the integral and local forms of d'Alembert's principle, we propose two different structure-preserving model reduction approaches to reconstruct low-dimensional systems, based on the variational principle and on the structure-preserving projection, respectively. These two approaches are proven to yield the same reduced system. Moreover, by incorporating the vector field into the data ensemble, we provided several algorithms for energy preservation. In a special case when the external force is described by the Rayleigh dissipative function, the proposed method automatically preserves the dissipativity, boundedness, and stability of the original system. The stability, accuracy, and efficiency of the proposed method are illustrated through numerical simulations of a dissipative wave equation.

math.NA↗

arXiv2014

Symplectic Model Reduction of Hamiltonian Systems

In this paper, a symplectic model reduction technique, proper symplectic decomposition (PSD) with symplectic Galerkin projection, is proposed to save the computational cost for the simplification of large-scale Hamiltonian systems while preserving the symplectic structure. As an analogy to the classical proper orthogonal decomposition (POD)-Galerkin approach, PSD is designed to build a symplectic subspace to fit empirical data, while the symplectic Galerkin projection constructs a reduced Hamiltonian system on the symplectic subspace. For practical use, we introduce three algorithms for PSD, which are based upon: the cotangent lift, complex singular value decomposition, and nonlinear programming. The proposed technique has been proven to preserve system energy and stability. Moreover, PSD can be combined with the discrete empirical interpolation method to reduce the computational cost for nonlinear Hamiltonian systems. Owing to these properties, the proposed technique is better suited than the classical POD-Galerkin approach for model reduction of Hamiltonian systems, especially when long-time integration is required. The stability, accuracy, and efficiency of the proposed technique are illustrated through numerical simulations of linear and nonlinear wave equations.

math.NA↗

arXiv2014

An Online Manifold Learning Approach for Model Reduction of Dynamical Systems

This article discusses a newly developed online manifold learning method, subspace iteration using reduced models (SIRM), for the dimensionality reduction of dynamical systems. This method may be viewed as subspace iteration combined with a model reduction procedure. Specifically, starting with a test solution, the method solves a reduced model to obtain a more precise solution, and it repeats this process until sufficient accuracy is achieved. The reduced model is obtained by projecting the full model onto a subspace that is spanned by the dominant modes of an extended data ensemble. The learning procedure is computed in the online stage, as opposed to being computed offline, which is used in many projection-based model reduction techniques that require prior calculations or experiments. After providing an error bound of the classical POD-Galerkin method in terms of the projection error and the initial condition error, we prove that the sequence of approximate solutions converge to the actual solution of the original system as long as the vector field of the full model is locally Lipschitz on an open set that contains the solution trajectory. Good accuracy of the proposed method has been demonstrated in two numerical examples, from a linear advection-diffusion equation to a nonlinear Burgers equation. In order to save computational cost, the SIRM method is extended to a local model reduction approach by partitioning the entire time domain into several subintervals and obtaining a series of local reduced models of much lower dimensionality. The accuracy and efficiency of the local SIRM are shown through the numerical simulation of the Navier--Stokes equation in a lid-driven cavity flow problem.

math.DS↗

arXiv2013

Nonlinear Model Reduction via an Adaptive Weighting of Snapshots

In this paper, we propose a new approach to model reduction of parameterized partial differential equations (PDEs) based on the concept of adaptive reduced bases. The presented approach is particularly suited for large-scale nonlinear systems characterized by parameter variations. Instead of using a global basis to construct a global reduced model, the proposed method approximates the original system by multiple lower-dimensional subspaces. Each localized reduced basis is generated by the SVD of a weighted snapshot ensemble; here, each weighting coefficient is a function of the input parameter. Compared with a global model reduction method, such as the classical POD, the adaptive model reduction method could yield a more accurate solution with a fixed subspace dimension. Moreover, we combine the adaptive reduced model with the chord iteration to solve elliptic PDEs in a computationally efficient fashion. The potential of the method for achieving large speedups, while maintaining good accuracy, is demonstrated for both elliptic and parabolic PDEs in a few numerical examples.

math.NA↗

arXiv2009

Consensus of self-driven agents with avoidance of collisions

In recent years, many efforts have been addressed on collision avoidance of collectively moving agents. In this paper, we propose a modified version of the Vicsek model with adaptive speed, which can guarantee the absence of collisions. However, this strategy leads to an aggregated state with slowly moving agents. We therefore further introduce a certain repulsion, which results in both faster consensus and longer safe distance among agents, and thus provides a powerful mechanism for collective motions in biological and technological multi-agent systems.

physics.data-an↗

arXiv2008

Accelerating consensus of self-driven swarm via adaptive speed

In resent years, Vicsek model has attracted more and more attention and been well developed. However, the in-depth analysis on the convergence time are scarce thus far. In this paper, we study some certain factors that mainly govern the convergence time of Vicsek model. By extensively numerical simulations, we find the convergence time scales in a power law with $r^2\ln N$ in the noise-free case, where $r$ and $N$ are horizon radius and the number of particles. Furthermore, to accelerate the convergence, we propose a new model in which the speed of each particle is variable. The convergence time can be remarkably shortened compared with the standard Vicsek model.

cond-mat.dis-nn↗