arXiv subjects

Jiajie Diao

Publications and source records attributed to Jiajie Diao.

3 recordsLinked to original sources

arXiv2026

Hierarchical Pre-Training of Vision Encoders with Large Language Model

The field of computer vision has experienced significant advancements through scalable vision encoders and multimodal pre-training frameworks. However, existing approaches often treat vision encoders and large language models (LLMs) as independent modules, limiting the integration of hierarchical visual features. In this work, we propose HIVE (Hierarchical Pre-Training of Vision Encoders), a novel framework that enhances vision-language alignment by introducing hierarchical cross-attention between the vision encoder and LLM. Unlike conventional methods that flatten image embeddings, HIVE enables structured feature fusion across multiple layers, improving gradient flow and representation learning. To optimize this interaction, we introduce a three-stage training strategy that progressively aligns the vision encoder with the LLM, ensuring stable optimization and effective multimodal fusion. Empirical evaluations demonstrate that HIVE achieves superior performance not only in image classification but also on various vision-language tasks, outperforming self-attention-based methods in benchmarks such as MME, GQA, OK-VQA, and ScienceQA. Our results highlight the benefits of hierarchical feature integration, paving the way for more efficient and expressive vision-language models.

cs.CV↗

arXiv2026

Learning to Select Visual In-Context Demonstrations

Multimodal Large Language Models (MLLMs) adapt to visual tasks via in-context learning (ICL), which relies heavily on demonstration quality. The dominant demonstration selection strategy is unsupervised k-Nearest Neighbor (kNN) search. While simple, this similarity-first approach is sub-optimal for complex factual regression tasks; it selects redundant examples that fail to capture the task's full output range. We reframe selection as a sequential decision-making problem and introduce Learning to Select Demonstrations (LSD), training a Reinforcement Learning agent to construct optimal demonstration sets. Using a Dueling DQN with a query-centric Transformer Decoder, our agent learns a policy that maximizes MLLM downstream performance. Evaluating across five visual regression benchmarks, we uncover a crucial dichotomy: while kNN remains optimal for subjective preference tasks, LSD significantly outperforms baselines on objective, factual regression tasks. By balancing visual relevance with diversity, LSD better defines regression boundaries, illuminating when learned selection is strictly necessary for visual ICL.

cs.LG↗

arXiv2018

Probing single protein dynamics on liposome surfaces

It is crucial to measure position and conformational changes of a membrane-interacting protein relative to the membrane surface. This is however challenging because the thickness of a membrane is usually only about 4 nm. We developed a fluorescence method which makes use of the principle of FRET between a fluorophore and a cloud of quenchers encapsulated in a liposome, hence the name LipoFRET. LipoFRET can readily locate a fluorophore in different depths inside and at different heights above the membrane. We applied LipoFRET to study α-synuclein, a key player in the pathology of Parkinson's disease. Our approach yielded quantita-tive information about the dynamics of different regions of α-syn in lipid membranes, which has never been explored before.

physics.bio-ph↗