arXiv subjects

Yankun Hong

Publications and source records attributed to Yankun Hong.

2 recordsLinked to original sources

arXiv2026

KeepLoRA++: Continual Learning with Layer-Scaled Residual Gradient Adaptation

Continual learning for pre-trained vision-language models requires balancing three competing objectives: retaining pre-trained knowledge, preserving knowledge from a sequence of learned tasks, and maintaining the plasticity to acquire new knowledge. This paper presents KeepLoRA++, balancing these objectives through a unified dual-dimensional knowledge retention mechanism. We analyze knowledge distribution of Transformer architecture from both inter-layer and intra-layer perspectives. The inter-layer perspective examines how retention is distributed across layers, while the intra-layer perspective focuses on the parameter space within each layer. Our analysis reveals a structural property: general transferable knowledge is mainly encoded in the shallow layers and the principal subspace of the parameters, while task-specific adaptations are localized in the deep layers and the residual subspace. Motivated by this insight, KeepLoRA++ introduces a layer-scaled residual gradient adaptation method. New tasks are learned by restricting LoRA parameter updates to the residual subspace, combined with a shallow-to-deep layer scaling, to prevent interference with previously acquired capabilities. Specifically, the gradient of a new task is projected onto a subspace orthogonal to both the principal subspace of the pre-trained model and the dominant directions of previous task features, while simultaneously assigning smaller update magnitudes to shallow layers and larger ones to deeper layers. Our theoretical analysis and empirical evaluations confirm that KeepLoRA++ successfully balances these three competing objectives, consistently outperforming representative baselines across image classification, visual question answering, and video understanding tasks.

cs.CV↗

arXiv2024

A Score-based Generative Solver for PDE-constrained Inverse Problems with Complex Priors

In the field of inverse estimation for systems modeled by partial differential equations (PDEs), challenges arise when estimating high- (or even infinite-) dimensional parameters. Typically, the ill-posed nature of such problems necessitates leveraging prior information to achieve well-posedness. In most existing inverse solvers, the prior distribution is assumed to be of either Gaussian or Laplace form which, in many practical scenarios, is an oversimplification. In case the prior is complex and the likelihood model is computationally expensive (e.g., due to expensive forward models), drawing the sample from such posteriors can be computationally intractable, especially when the unknown parameter is high-dimensional. In this work, to sample efficiently, we propose a score-based diffusion model, which combines a score-based generative sampling tool with a noising and denoising process driven by stochastic differential equations. This tool is used for iterative sample generation in accordance with the posterior distribution, while simultaneously learning and leveraging the underlying information and constraints inherent in the given complex prior. A time-varying time schedule is proposed to adapt the method for posterior sampling. To expedite the simulation of non-parameterized PDEs and enhance the generalization capacity, we introduce a physics-informed convolutional neural network (CNN) surrogate for the forward model. Finally, numerical experiments, including a hyper-elastic problem and a multi-scale mechanics problem, demonstrate the efficacy of the proposed approach. In particular, the score-based diffusion model, coupled with the physics-informed CNN surrogate, effectively learns geometrical features from provided prior samples, yielding better inverse estimation results compared to the state-of-the-art techniques.

cs.CE↗