arXiv subjects

Jinglong Yang

Publications and source records attributed to Jinglong Yang.

5 recordsLinked to original sources

arXiv2026

Curvature-Guided Mixing for MLLM Adaptation

Fine-tuning Multimodal Large Language Models (MLLMs) on specialized tasks often leads to catastrophic forgetting of their general capabilities. Existing model merging methods to combat this are often heuristic or use sub-optimal objectives. We propose CurvatureGuided Mixing (CGM), a theoretically grounded framework that merges pre-trained and fine-tuned models. CGM formulates a joint optimization objective and uses a second-order (Hessian) approximation of the loss landscapes to analytically derive an optimal, closed-form "soft mixing" ratio. This ratio intelligently blends parameters based on their relative task-specific curvatures. We also introduce CGM$\dagger$, a robust "hard mixing" variant that performs sparse parameter selection guided by a novel, curvature-aware score. Experiments on LLaVA-1.5 and Qwen2.5VL across multiple downstream tasks show that CGM and CGM$\dagger$ consistently improve the trade-off between task specialization and general knowledge retention over existing methods. Code is available at github.com/zzsyjl/CGM-ECCV-2026.

cs.CV↗

arXiv2026

Dynamic Moir\'e Potentials and Robust Wigner Crystallization in Large-Scale Twisted Transition Metal Dichalcogenides

Understanding the dynamical evolution of large-scale moir\'e systems is crucial for connecting theoretical predictions with experimental observations. Here we develop a machine-learning-based workflow, integrating DeePMD and DeepH frameworks with first-principles calculations, to efficiently investigate time-dependent structural and electronic responses in twisted bilayer transition metal dichalcogenides (TMDs) with experimentally relevant moir\'e supercells containing over 3000 atoms. Using $\mathrm{WS_2}$ as a representative system, we show that low-temperature lattice vibrations and relaxation deepen the moir\'e potential wells, narrow the lowest conduction band, and facilitate the formation of strongly localized electronic states. Based on DFT-derived moir\'e potentials that incorporate these dynamical effects, density-matrix-renormalization-group (DMRG) simulations reveal robust Wigner crystallization and a kagom\'e-patterned three-electron state, consistent with recent experimental observations. Our workflow provides a practical route for exploring large moir\'e supercells beyond static configurations and offers new insight into the interplay between lattice dynamics, electronic localization, and emergent correlated states in twisted two-dimensional materials.

cond-mat.mtrl-sci↗

arXiv2024

Continual Learning for Segment Anything Model Adaptation

Although the current different types of SAM adaptation methods have achieved promising performance for various downstream tasks, such as prompt-based ones and adapter-based ones, most of them belong to the one-step adaptation paradigm. In real-world scenarios, we are generally confronted with the dynamic scenario where the data comes in a streaming manner. Driven by the practical need, in this paper, we first propose a novel Continual SAM adaptation (CoSAM) benchmark with 8 different task domains and carefully analyze the limitations of the existing SAM one-step adaptation methods in the continual segmentation scenario. Then we propose a novel simple-yet-effective Mixture of Domain Adapters (MoDA) algorithm which utilizes the Global Feature Tokens (GFT) and Global Assistant Tokens (GAT) modules to help the SAM encoder extract well-separated features for different task domains, and then provide the accurate task-specific information for continual learning. Extensive experiments demonstrate that our proposed MoDA obviously surpasses the existing classic continual learning methods, as well as prompt-based and adapter-based approaches for continual segmentation. Moreover, after sequential learning on the CoSAM benchmark with diverse data distributions, our MoDA maintains highly competitive results in the natural image domain, approaching the zero-shot performance of the original SAM, demonstrating its superior capability in knowledge preservation. Notably, the proposed MoDA can be seamlessly integrated into various one-step adaptation methods of SAM, which can consistently bring obvious performance gains. Code is available at \url{https://github.com/yangjl1215/CoSAM}

cs.CV↗

arXiv2024

On shape optimization for fourth order Steklov eigenvalue problems

We study three types of fourth-order Steklov eigenvalue problems. For the first two of them, we derive the asymptotic expansion of their spectra on Euclidean annular domains $\mathbb{B}^n_1\setminus \overline{\mathbb{B}^n_\epsilon}$ as $\epsilon \to 0$, leading to conclusions on shape optimization. For these two problems, we also compute their spectra on cylinders over closed Riemannian manifolds. Last, for the third problem, we obtain a sharp upper bound for its first non-zero eigenvalue on star-shaped and mean convex Euclidean domains.

math.AP↗

arXiv2024

Using Left and Right Brains Together: Towards Vision and Language Planning

Large Language Models (LLMs) and Large Multi-modality Models (LMMs) have demonstrated remarkable decision masking capabilities on a variety of tasks. However, they inherently operate planning within the language space, lacking the vision and spatial imagination ability. In contrast, humans utilize both left and right hemispheres of the brain for language and visual planning during the thinking process. Therefore, we introduce a novel vision-language planning framework in this work to perform concurrent visual and language planning for tasks with inputs of any form. Our framework incorporates visual planning to capture intricate environmental details, while language planning enhances the logical coherence of the overall system. We evaluate the effectiveness of our framework across vision-language tasks, vision-only tasks, and language-only tasks. The results demonstrate the superior performance of our approach, indicating that the integration of visual and language planning yields better contextually aware task execution.

cs.CV↗