arXiv subjects

Jifeng Song

Publications and source records attributed to Jifeng Song.

3 recordsLinked to original sources

arXiv2026

FigEx2: Visual-Conditioned Panel Detection and Captioning for Scientific Compound Figures

Scientific compound figures combine multiple labeled panels into a single image, and downstream pretraining and retrieval require panel-aligned visual-text pairs. However, in a PubMed Central (PMC)-scale crawl of 346,567 compound figures, 16.3% have no caption and are discarded by existing caption-decomposition pipelines. We propose FigEx2, a visual-conditioned framework that takes only a compound figure as input and jointly produces labeled panel boxes and panel-wise captions. FigEx2 introduces an Entity-Attention Kullback-Leibler (KL) regularizer that aligns the detector's cross-attention with scientific entities annotated for each panel, providing a stable conditioning signal that also improves localization, and applies Group Relative Policy Optimization (GRPO) with a panel-level Entity-F1 reward to optimize scientific faithfulness. We curate BioSci-Fig-Cap for in-domain supervision and contribute physics and chemistry test suites for cross-disciplinary evaluation. FigEx2 achieves 0.751 mAP@0.5:0.95 on BioSci-Fig-Cap, and outperforms Qwen3-VL-8B by 6.80 Entity-F1 on MedICaT for captioning. It also transfers zero-shot to out-of-distribution domains. The source code is available at https://github.com/Huang-AI4Medicine-Lab/FigEx2.

cs.CV↗

arXiv2026

HERO: Hierarchical Evidential Reasoning Optimization for Radiology Report Generation via Reason-then-Summarize

Multimodal Large Language Models (MLLMs) have substantially advanced Radiology Report Generation (RRG), yet aligning them through reinforcement learning (RL) remains challenging due to heterogeneous medical supervision. Vanilla Group Relative Policy Optimization (GRPO) assigns uniform credit across the entire generation, leading to segment interference, token dilution, and evidence--diagnosis decoupling, which exacerbates clinical hallucinations. We propose HERO (Hierarchical Evidential Reasoning Optimization), a factorized policy optimization framework that aligns heterogeneous supervision with three optimization granularities. HERO separately optimizes reasoning, diagnosis, and evidence grounding through complementary segment-, token-, and completion-level optimization with a heterogeneous reward formulation covering diagnostic accuracy, reasoning quality, and think--answer consistency. Experiments on MIMIC-CXR and IU-Xray show that HERO outperforms strong supervised and reinforcement learning baselines, achieving state-of-the-art clinical efficacy while producing more evidence-grounded and think--answer-consistent reports, thereby substantially mitigating clinical hallucinations.

cs.LG↗

arXiv2024

Achieving Sparse Activation in Small Language Models

Sparse activation, which selectively activates only an input-dependent set of neurons in inference, is a useful technique to reduce the computing cost of Large Language Models (LLMs) without retraining or adaptation efforts. However, whether it can be applied to the recently emerging Small Language Models (SLMs) remains questionable, because SLMs are generally less over-parameterized than LLMs. In this paper, we aim to achieve sparse activation in SLMs. We first show that the existing sparse activation schemes in LLMs that build on neurons' output magnitudes cannot be applied to SLMs, and activating neurons based on their attribution scores is a better alternative. Further, we demonstrated and quantified the large errors of existing attribution metrics when being used for sparse activation, due to the interdependency among attribution scores of neurons across different layers. Based on these observations, we proposed a new attribution metric that can provably correct such errors and achieve precise sparse activation. Experiments over multiple popular SLMs and datasets show that our approach can achieve 80% sparsification ratio with <5% model accuracy loss, comparable to the sparse activation achieved in LLMs. The source code is available at: https://github.com/pittisl/Sparse-Activation.

cs.CL↗