arXiv subjects

Yan Li

Publications and source records attributed to Yan Li.

At least 235 records · Page 13Linked to original sources

arXiv2025

A Training-Free Length Extrapolation Approach for LLMs: Greedy Attention Logit Interpolation (GALI)

Transformer-based Large Language Models (LLMs) struggle with inputs exceeding their training context window due to positional out-of-distribution (O.O.D.) issues that disrupt attention. Existing solutions, including fine-tuning and training-free methods, face challenges like inefficiency, redundant interpolation, logit outliers, or loss of local positional information. We propose Greedy Attention Logit Interpolation (GALI), a training-free method that improves length extrapolation by greedily reusing pretrained positional intervals and interpolating attention logit to eliminate outliers. GALI achieves stable and superior performance across a wide range of long-context tasks without requiring input-length-specific tuning. Our analysis further reveals that LLMs interpret positional intervals unevenly and that restricting interpolation to narrower ranges improves performance, even on short-context tasks. GALI represents a step toward more robust and generalizable long-text processing in LLMs. Our implementation of GALI, along with the experiments from our paper, is open-sourced at https://github.com/adlnlp/Gali.

cs.CL↗

arXiv2025

SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought

Chain of Thought (CoT) prompting improves the reasoning performance of large language models (LLMs) by encouraging step by step thinking. However, CoT-based methods depend on intermediate reasoning steps, which limits scalability and generalization. Recent work explores recursive reasoning, where LLMs reuse internal layers across iterations to refine latent representations without explicit CoT supervision. While promising, these approaches often require costly pretraining and lack a principled framework for how reasoning should evolve across iterations. We address this gap by introducing Flow Chain of Thought (Flow CoT), a reasoning paradigm that models recursive inference as a progressive trajectory of latent cognitive states. Flow CoT frames each iteration as a distinct cognitive stage deepening reasoning across iterations without relying on manual supervision. To realize this, we propose SCOUT (Stepwise Cognitive Optimization Using Teachers), a lightweight fine tuning framework that enables Flow CoT style reasoning without the need for pretraining. SCOUT uses progressive distillation to align each iteration with a teacher of appropriate capacity, and a cross attention based retrospective module that integrates outputs from previous iterations while preserving the models original computation flow. Experiments across eight reasoning benchmarks show that SCOUT consistently improves both accuracy and explanation quality, achieving up to 1.8% gains under fine tuning. Qualitative analyses further reveal that SCOUT enables progressively deeper reasoning across iterations refining both belief formation and explanation granularity. These results not only validate the effectiveness of SCOUT, but also demonstrate the practical viability of Flow CoT as a scalable framework for enhancing reasoning in LLMs.

cs.AI↗

arXiv2025

Risk-averse formulations of Stochastic Optimal Control and Markov Decision Processes

The aim of this paper is to investigate risk-averse and distributionally robust modeling of Stochastic Optimal Control (SOC) and Markov Decision Process (MDP). We discuss construction of conditional nested risk functionals, a particular attention is given to the Value-at-Risk measure. Necessary and sufficient conditions for existence of non-randomized optimal policies in the framework of robust SOC and MDP are derived. We also investigate sample complexity of optimization problems involving the Value-at-Risk measure.

math.OC↗

arXiv2025

Regularized Fingerprinting with Linearly Optimal Weight Matrix in Detection and Attribution of Climate Change

Climate change detection and attribution play a central role in establishing the causal influence of human activities on global warming. The dominant framework, optimal fingerprinting, is a linear errors-in-variables model in which each covariate is subject to measurement error with covariance proportional to that of the regression error. The reliability of such analyses depends critically on accurate inference of the regression coefficients. The optimal weight matrix for estimating these coefficients is the precision matrix of the regression error, which is typically unknown and must be estimated from climate model simulations. However, existing regularized optimal fingerprinting approaches often yield underestimated uncertainties and overly narrow confidence intervals that fail to attain nominal coverage, thereby compromising the reliability of analysis. In this paper, we first propose consistent variance estimators for the regression coefficients within the class of linear shrinkage weight matrices, addressing undercoverage in conventional methods. Building on this, we derive a linearly optimal weight matrix that directly minimizes the asymptotic variances of the estimated scaling factors. Numerical studies confirm improved empirical coverage and shorter interval lengths. When applied to annual mean temperature data, the proposed method produces narrower, more reliable intervals and provides new insights into detection and attribution across different regions.

stat.ME↗

arXiv2025

Tianyi: A Traditional Chinese Medicine all-rounder language model and its Real-World Clinical Practice

Natural medicines, particularly Traditional Chinese Medicine (TCM), are gaining global recognition for their therapeutic potential in addressing human symptoms and diseases. TCM, with its systematic theories and extensive practical experience, provides abundant resources for healthcare. However, the effective application of TCM requires precise syndrome diagnosis, determination of treatment principles, and prescription formulation, which demand decades of clinical expertise. Despite advancements in TCM-based decision systems, machine learning, and deep learning research, limitations in data and single-objective constraints hinder their practical application. In recent years, large language models (LLMs) have demonstrated potential in complex tasks, but lack specialization in TCM and face significant challenges, such as too big model scale to deploy and issues with hallucination. To address these challenges, we introduce Tianyi with 7.6-billion-parameter LLM, a model scale proper and specifically designed for TCM, pre-trained and fine-tuned on diverse TCM corpora, including classical texts, expert treatises, clinical records, and knowledge graphs. Tianyi is designed to assimilate interconnected and systematic TCM knowledge through a progressive learning manner. Additionally, we establish TCMEval, a comprehensive evaluation benchmark, to assess LLMs in TCM examinations, clinical tasks, domain-specific question-answering, and real-world trials. The extensive evaluations demonstrate the significant potential of Tianyi as an AI assistant in TCM clinical practice and research, bridging the gap between TCM knowledge and practical application.

cs.CL↗

arXiv2025

Paragraph-to-Image Generation with Information-Enriched Diffusion Model

Text-to-image (T2I) models have recently experienced rapid development, achieving astonishing performance in terms of fidelity and textual alignment capabilities. However, given a long paragraph (up to 512 words), these generation models still struggle to achieve strong alignment and are unable to generate images depicting complex scenes. In this paper, we introduce an information-enriched diffusion model for paragraph-to-image generation task, termed ParaDiffusion, which delves into the transference of the extensive semantic comprehension capabilities of large language models to the task of image generation. At its core is using a large language model (e.g., Llama V2) to encode long-form text, followed by fine-tuning with LORA to alignthe text-image feature spaces in the generation task. To facilitate the training of long-text semantic alignment, we also curated a high-quality paragraph-image pair dataset, namely ParaImage. This dataset contains a small amount of high-quality, meticulously annotated data, and a large-scale synthetic dataset with long text descriptions being generated using a vision-language model. Experiments demonstrate that ParaDiffusion outperforms state-of-the-art models (SD XL, DeepFloyd IF) on ViLG-300 and ParaPrompts, achieving up to 15% and 45% human voting rate improvements for visual appeal and text faithfulness, respectively. The code and dataset will be released to foster community research on long-text alignment.

cs.CV↗

arXiv2025

Deep Learning Model Inversion Attacks and Defenses: A Comprehensive Survey

The rapid adoption of deep learning in sensitive domains has brought tremendous benefits. However, this widespread adoption has also given rise to serious vulnerabilities, particularly model inversion (MI) attacks, posing a significant threat to the privacy and integrity of personal data. The increasing prevalence of these attacks in applications such as biometrics, healthcare, and finance has created an urgent need to understand their mechanisms, impacts, and defense methods. This survey aims to fill the gap in the literature by providing a structured and in-depth review of MI attacks and defense strategies. Our contributions include a systematic taxonomy of MI attacks, extensive research on attack techniques and defense mechanisms, and a discussion about the challenges and future research directions in this evolving field. By exploring the technical and ethical implications of MI attacks, this survey aims to offer insights into the impact of AI-powered systems on privacy, security, and trust. In conjunction with this survey, we have developed a comprehensive repository to support research on MI attacks and defenses. The repository includes state-of-the-art research papers, datasets, evaluation metrics, and other resources to meet the needs of both novice and experienced researchers interested in MI attacks and defenses, as well as the broader field of AI security and privacy. The repository will be continuously maintained to ensure its relevance and utility. It is accessible at https://github.com/overgter/Deep-Learning-Model-Inversion-Attacks-and-Defenses.

cs.CR↗

arXiv2025

ESDiff: Encoding Strategy-inspired Diffusion Model with Few-shot Learning for Color Image Inpainting

Image inpainting is a technique used to restore missing or damaged regions of an image. Traditional methods primarily utilize information from adjacent pixels for reconstructing missing areas, while they struggle to preserve complex details and structures. Simultaneously, models based on deep learning necessitate substantial amounts of training data. To address this challenge, an encoding strategy-inspired diffusion model with few-shot learning for color image inpainting is proposed in this paper. The main idea of this novel encoding strategy is the deployment of a "virtual mask" to construct high-dimensional objects through mutual perturbations between channels. This approach enables the diffusion model to capture diverse image representations and detailed features from limited training samples. Moreover, the encoding strategy leverages redundancy between channels, integrates with low-rank methods during iterative inpainting, and incorporates the diffusion model to achieve accurate information output. Experimental results indicate that our method exceeds current techniques in quantitative metrics, and the reconstructed images quality has been improved in aspects of texture and structural integrity, leading to more precise and coherent results.

cs.CV↗

arXiv2025

Full dynamics of two-membrane cavity optomechanics

In a two-membrane cavity optomechanical setup, two semi-transparent membranes placed within an optical Fabry-Pérot cavity yield a nontrivial dependence of the frequency of a mode of the optical cavity on the membranes' positions, which is due to interference. However, the system dynamics is typically described by a radiation pressure force treatment in which the frequency shift is expanded stopping at first order in the membrane displacements. In this paper, we study the full dynamics of the system obtained by considering the exact nonlinear dependence of the optomechanical interaction between two membranes' vibrational modes and the driven cavity mode. We then compare this dynamics with the standard treatment based on the Hamiltonian linear interaction, and we find the conditions under which the two dynamics may significantly depart from each other. In particular, we see that a parameter regime exists in which the customary first-order treatment provides distinct and incorrect predictions for the synchronization of two self-sustained mechanical limit-cycles, and for Gaussian entanglement of the two membranes in the case of two-tone driving.

quant-ph↗

arXiv2025

A Bring-Your-Own-Model Approach for ML-Driven Storage Placement in Warehouse-Scale Computers

Storage systems account for a major portion of the total cost of ownership (TCO) of warehouse-scale computers, and thus have a major impact on the overall system's efficiency. Machine learning (ML)-based methods for solving key problems in storage system efficiency, such as data placement, have shown significant promise. However, there are few known practical deployments of such methods. Studying this problem in the context of real-world hyperscale data centers at Google, we identify a number of challenges that we believe cause this lack of practical adoption. Specifically, prior work assumes a monolithic model that resides entirely within the storage layer, an unrealistic assumption in real-world deployments with frequently changing workloads. To address this problem, we introduce a cross-layer approach where workloads instead ''bring their own model''. This strategy moves ML out of the storage system and instead allows each workload to train its own lightweight model at the application layer, capturing the workload's specific characteristics. These small, interpretable models generate predictions that guide a co-designed scheduling heuristic at the storage layer, enabling adaptation to diverse online environments. We build a proof-of-concept of this approach in a production distributed computation framework at Google. Evaluations in a test deployment and large-scale simulation studies using production traces show improvements of as much as 3.47$\times$ in TCO savings compared to state-of-the-art baselines.

cs.DC↗

arXiv2025

The Policy Paradox: Government Debt Servicing and Local Bank Risk Growth

The issue of local government debt is widely recognized as one of the "gray rhinos" affecting the stable development of China's economy. Government debt can transmit risks to local banks, which are among the primary holders of local debt, thereby triggering systemic financial risks. Consequently, exploring debt resolution pathways and evaluating the systematic effects of debt servicing policies has become critically important. This study employs panel data from 348 local commercial banks across 29 provincial-level administrative regions in China from 2010 to 2023, and constructs a difference-in-differences (DID) model to investigate the impact of the State Council's special supervision of debt servicing on local bank risks. The findings indicate that the government's debt servicing policy essentially represents a shift of government debt from explicit to implicit forms, significantly increasing the risks faced by local banks and producing outcomes contrary to the policy's original intent. This effect is particularly pronounced for rural commercial banks and banks with high customer concentration and fewer branches. Mechanism analysis reveals two key insights. First, local banks are heavily influenced by local government control; the government's debt servicing requires banks to support the government by purchasing government bonds and other financial instruments, which leads to a deterioration in asset quality and an expansion of risk exposure. Second, government debt crowds out private credit from local banks, weakening the region's repayment capacity and ultimately increasing bank risk. Our research uncovers the counterintuitive effects of government debt servicing and offers corresponding policy recommendations.

econ.GN↗

arXiv2025

ProtoGS: Efficient and High-Quality Rendering with 3D Gaussian Prototypes

3D Gaussian Splatting (3DGS) has made significant strides in novel view synthesis but is limited by the substantial number of Gaussian primitives required, posing challenges for deployment on lightweight devices. Recent methods address this issue by compressing the storage size of densified Gaussians, yet fail to preserve rendering quality and efficiency. To overcome these limitations, we propose ProtoGS to learn Gaussian prototypes to represent Gaussian primitives, significantly reducing the total Gaussian amount without sacrificing visual quality. Our method directly uses Gaussian prototypes to enable efficient rendering and leverage the resulting reconstruction loss to guide prototype learning. To further optimize memory efficiency during training, we incorporate structure-from-motion (SfM) points as anchor points to group Gaussian primitives. Gaussian prototypes are derived within each group by clustering of K-means, and both the anchor points and the prototypes are optimized jointly. Our experiments on real-world and synthetic datasets prove that we outperform existing methods, achieving a substantial reduction in the number of Gaussians, and enabling high rendering speed while maintaining or even enhancing rendering fidelity.

cs.CV↗

arXiv2025

Learning Coarse-Grained Dynamics on Graph

We consider a Graph Neural Network (GNN) non-Markovian modeling framework to identify coarse-grained dynamical systems on graphs. Our main idea is to systematically determine the GNN architecture by inspecting how the leading term of the Mori-Zwanzig memory term depends on the coarse-grained interaction coefficients that encode the graph topology. Based on this analysis, we found that the appropriate GNN architecture that will account for $K$-hop dynamical interactions has to employ a Message Passing (MP) mechanism with at least $2K$ steps. We also deduce that the memory length required for an accurate closure model decreases as a function of the interaction strength under the assumption that the interaction strength exhibits a power law that decays as a function of the hop distance. Supporting numerical demonstrations on two examples, a heterogeneous Kuramoto oscillator model and a power system, suggest that the proposed GNN architecture can predict the coarse-grained dynamics under fixed and time-varying graph topologies.

math.NA↗

arXiv2025

Manipulating magnetization by orbital current from a light metal Ti

The orbital Hall effect, which does not rely on the spin-orbit coupling, has recently emerged as a promising mechanism for electrically manipulating magnetization in thin-film ferromagnets. Despite its potential, direct experimental observation of magnetization switching driven by orbital currents has been challenging, primarily because there is no direct exchange coupling between orbital angular momentum and local spin based magnetic moments. In this study, we present a compensated design to directly probe the contribution of orbital currents in the most promising light metal titanium (Ti), where symmetric layer structures allow zeroing out of the net spin current. By varying the thickness of the Ti layer in Ti(t)/Pt/Co/Pt/Co/Pt multilayers, we demonstrate the ability to control the magnetization switching polarity. We deduce the orbital charge conversion efficiency of the Ti layer to be approximately 0.17. These findings not only confirm the presence of the orbital Hall effect in Ti but also suggest that orbital currents may be promising candidates for developing energy-efficient magnetic devices with enhanced performance and scalability.

cond-mat.mtrl-sci↗

arXiv2025

Cream of the Crop: Harvesting Rich, Scalable and Transferable Multi-Modal Data for Instruction Fine-Tuning

The hypothesis that pretrained large language models (LLMs) necessitate only minimal supervision during the fine-tuning (SFT) stage (Zhou et al., 2024) has been substantiated by recent advancements in data curation and selection research. However, their stability and generalizability are compromised due to the vulnerability to experimental setups and validation protocols, falling short of surpassing random sampling (Diddee & Ippolito, 2024; Xia et al., 2024b). Built upon LLMs, multi-modal LLMs (MLLMs), combined with the sheer token volume and heightened heterogeneity of data sources, amplify both the significance and complexity of data selection. To harvest multi-modal instructional data in a robust and efficient manner, we re-define the granularity of the quality metric by decomposing it into 14 vision-language-related capabilities, and introduce multi-modal rich scorers to evaluate the capabilities of each data candidate. To promote diversity, in light of the inherent objective of the alignment stage, we take interaction style as diversity indicator and use a multi-modal rich styler to identify data instruction patterns. In doing so, our multi-modal rich scorers and styler (mmSSR) guarantee that high-scoring information is conveyed to users in diversified forms. Free from embedding-based clustering or greedy sampling, mmSSR efficiently scales to millions of data with varying budget constraints, supports customization for general or specific capability acquisition, and facilitates training-free generalization to new domains for curation. Across 10+ experimental settings, validated by 14 multi-modal benchmarks, we demonstrate consistent improvements over random sampling, baseline strategies and state-of-the-art selection methods, achieving 99.1% of full performance with only 30% of the 2.6M data.

cs.CV↗

arXiv2025

Single Atom Catalysts with Halogen Ligands: Elevating the HER Performance of Pd-anchored MoS2 monolayer

Single-atom catalysts (SACs) have attracted ever-growing interest due to their high atom-utilization efficiency and potential for cost-effective of hydrogen production. However, enhancing the hydrogen evolution reaction (HER) performance remains a key challenge in developing SACs for HER technology. Herein, we employed first-principles calculations in conjunction with the climbing-image nudged elastic band (CI-NEB) method to explore the effect of surface ligands (F, Cl, Br, I) on the HER performance and mechanism of single-atom (Pd or Cu)-anchored MoS2 monolayer. The results indicate that the relative Gibbs free energy for the adsorbed hydrogen atom in the I-Pd@MoS2 system is an exceptionally low value of -0.13 eV, which is not only comparable to that of Pt-based catalysts but also significantly more favorable than the calculated 0.84 eV for Pd@MoS2. However, the introduction of ligands to Cu@MoS2 deteriorates HER performance due to strong coupling between the absorbed H and ligands. It reveals that the ligand I restructures the local chemical microenvironment surrounding the SAC Pd, leading to impurity bands near the Fermi level that couple favorably with the s states of H atoms, yielding numerous highly active sites to enhance catalytic performance. Furthermore, the CI-NEB method elucidates that the enhanced HER mechanism for the I-Pd@MoS2 catalyst should belong to the coexistence of the Volmer-Tafel and Volmer-Heyrovsky reactions. This investigation provides a valuable framework for the experimental design and development of innovative single-atom catalysts.

cond-mat.mtrl-sci↗

arXiv2025

Physics-based machine learning framework for predicting NOx emissions from compression ignition engines using on-board diagnostics data

This work presents a physics-based machine learning framework to predict and analyze oxides of nitrogen (NOx) emissions from compression-ignition engine-powered vehicles using on-board diagnostics (OBD) data as input. Accurate NOx prediction from OBD datasets is difficult because NOx formation inside an engine combustion chamber is governed by complex processes occurring on timescales much shorter than the data collection rate. Thus, emissions generally cannot be predicted accurately using simple empirically derived physics models. Black box models like genetic algorithms or neural networks can be more accurate, but have poor interpretability. The transparent model presented in this paper has both high accuracy and can explain potential sources of high emissions. The proposed framework consists of two major steps: a physics-based NOx prediction model combined with a novel Divergent Window Co-occurrence (DWC) Pattern detection algorithm to analyze operating conditions that are not adequately addressed by the physics-based model. The proposed framework is validated for generalizability with a second vehicle OBD dataset, a sensitivity analysis is performed, and model predictions are compared with that from a deep neural network. The results show that NOx emissions predictions using the proposed model has around 55% better root mean square error, and around 60% higher mean absolute error compared to the baseline NOx prediction model from previously published work. The DWC Pattern Detection Algorithm identified low engine power conditions to have high statistical significance, indicating an operating regime where the model can be improved. This work shows that the physics-based machine learning framework is a viable method for predicting NOx emissions from engines that do not incorporate NOx sensing.

cs.LG↗

arXiv2025

Scalable Low-overhead Superconducting Non-local Coupler with Exponentially Enhanced Connectivity

Quantum error correction codes with non-local connections such as quantum low-density parity-check (qLDPC) incur lower overhead and outperform surface codes on large-scale devices. These codes are not applicable on current superconducting devices with nearest-neighbor connections. To rectify the deficiency in connectivity of superconducting circuit system, we experimentally demonstrate a convenient on-chip coupler of centimeters long and propose an extra coupler layer to map the qubit array to a binary-tree connecting graph. This mapping layout reduces the average qubit entangling distance from O(N) to O(logN), demonstrating an exponentially enhanced connectivity with eliminated crosstalk. The entangling gate with the coupler is performed between two fluxonium qubits, reaching a fidelity of 99.37 % while the system static ZZ rate remains as low as 144 Hz without active cancellation or circuit parameter targeting. With the scalable binary tree structure and high-fidelity non-local entanglement, novel quantum algorithms can be implemented on the superconducting qubit system, positioning it as a strong competitor to other physics systems regarding circuit connectivity.

quant-ph↗