arXiv subjects

Xukun Zhang

Publications and source records attributed to Xukun Zhang.

At least 19 recordsLinked to original sources

arXiv2026

Vis2Reg: Visibility-Aware Landmark-Free Geometric 3D--2D Registration for Liver Laparoscopy

Accurate 3D--2D liver registration, which aligns preoperative 3D models to partial, view-dependent intraoperative surface observations, is critical for AR-guided laparoscopic surgery but remains challenging due to severe occlusion, limited visibility, and the lack of 3D ground-truth supervision. Existing landmark-free approaches perform partial-to-complete geometric alignment, yet robust self-supervision under extreme partial visibility remains difficult. We propose Vis2Reg, a visibility-aware registration framework that explicitly constrains deformation using mask-consistent visible regions. We introduce a visibility-aware self-supervision that derives a visible-domain 3D supervision signal from intraoperative masks, enabled by differentiable point rasterization and mask-guided back-projection. This formulation improves robustness under severe occlusion while maintaining fully self-supervised learning. Vis2Reg combines a robust geometric rigid initialization module with an implicit neural deformation field for stable alignment. Vis2Reg achieves a Dice score of 92.6\% and a Chamfer Distance of 1.43 mm on real intraoperative datasets, with 111 ms per-frame inference time, demonstrating both accuracy and practical efficiency.

cs.CV↗

arXiv2025

MSCPT: Few-shot Whole Slide Image Classification with Multi-scale and Context-focused Prompt Tuning

Multiple instance learning (MIL) has become a standard paradigm for the weakly supervised classification of whole slide images (WSIs). However, this paradigm relies on using a large number of labeled WSIs for training. The lack of training data and the presence of rare diseases pose significant challenges for these methods. Prompt tuning combined with pre-trained Vision-Language models (VLMs) is an effective solution to the Few-shot Weakly Supervised WSI Classification (FSWC) task. Nevertheless, applying prompt tuning methods designed for natural images to WSIs presents three significant challenges: 1) These methods fail to fully leverage the prior knowledge from the VLM's text modality; 2) They overlook the essential multi-scale and contextual information in WSIs, leading to suboptimal results; and 3) They lack exploration of instance aggregation methods. To address these problems, we propose a Multi-Scale and Context-focused Prompt Tuning (MSCPT) method for FSWC task. Specifically, MSCPT employs the frozen large language model to generate pathological visual language prior knowledge at multiple scales, guiding hierarchical prompt tuning. Additionally, we design a graph prompt tuning module to learn essential contextual information within WSI, and finally, a non-parametric cross-guided instance aggregation module has been introduced to derive the WSI-level features. Extensive experiments, visualizations, and interpretability analyses were conducted on five datasets and three downstream tasks using three VLMs, demonstrating the strong performance of our MSCPT. All codes have been made publicly accessible at https://github.com/Hanminghao/MSCPT.

cs.CV↗

arXiv2025

Adaptive Reinforcement for Open-ended Medical Reasoning via Semantic-Guided Reward Collapse Mitigation

Reinforcement learning (RL) with rule-based reward functions has recently shown great promise in enhancing the reasoning depth and generalization ability of vision-language models (VLMs), while maintaining computational efficiency. In spite of these advances, its adoption in medical imaging remains limited. Current reinforcement fine-tuning (RFT) efforts in this field mainly focus on closed-ended visual question answering (VQA), restricting their applicability to realistic clinical reasoning. However, open-ended medical VQA better mirrors clinical diagnostic workflows but remains underexplored. Although several studies have attempted to bridge the two formats through semantically guided RL, model-driven semantic rewards often suffer from reward collapse, where responses with distinct semantics yield nearly identical scores. To overcome this limitation, we introduce Adaptive Reinforcement for Medical Reasoning (ARMed), a novel RL framework tailored for open-ended medical VQA. ARMed first injects domain expertise through supervised fine-tuning (SFT) on chain-of-thought annotations, followed by reinforcement optimization using textual correctness and adaptive semantic rewards to refine reasoning consistency and factual accuracy. Extensive experiments on six challenging medical VQA benchmarks demonstrate that ARMed substantially improves both accuracy and generalization. These findings underscore the importance of reward discriminability in medical RL and highlight the potential of adaptive semantic rewards for building robust, clinically reliable multimodal reasoning systems.

cs.CV↗

arXiv2025

VGAT: A Cancer Survival Analysis Framework Transitioning from Generative Visual Question Answering to Genomic Reconstruction

Multimodal learning combining pathology images and genomic sequences enhances cancer survival analysis but faces clinical implementation barriers due to limited access to genomic sequencing in under-resourced regions. To enable survival prediction using only whole-slide images (WSI), we propose the Visual-Genomic Answering-Guided Transformer (VGAT), a framework integrating Visual Question Answering (VQA) techniques for genomic modality reconstruction. By adapting VQA's text feature extraction approach, we derive stable genomic representations that circumvent dimensionality challenges in raw genomic data. Simultaneously, a cluster-based visual prompt module selectively enhances discriminative WSI patches, addressing noise from unfiltered image regions. Evaluated across five TCGA datasets, VGAT outperforms existing WSI-only methods, demonstrating the viability of genomic-informed inference without sequencing. This approach bridges multimodal research and clinical feasibility in resource-constrained settings. The code link is https://github.com/CZZZZZZZZZZZZZZZZZ/VGAT.

cs.CV↗

arXiv2025

VLM-based Prompts as the Optimal Assistant for Unpaired Histopathology Virtual Staining

In histopathology, tissue sections are typically stained using common H&E staining or special stains (MAS, PAS, PASM, etc.) to clearly visualize specific tissue structures. The rapid advancement of deep learning offers an effective solution for generating virtually stained images, significantly reducing the time and labor costs associated with traditional histochemical staining. However, a new challenge arises in separating the fundamental visual characteristics of tissue sections from the visual differences induced by staining agents. Additionally, virtual staining often overlooks essential pathological knowledge and the physical properties of staining, resulting in only style-level transfer. To address these issues, we introduce, for the first time in virtual staining tasks, a pathological vision-language large model (VLM) as an auxiliary tool. We integrate contrastive learnable prompts, foundational concept anchors for tissue sections, and staining-specific concept anchors to leverage the extensive knowledge of the pathological VLM. This approach is designed to describe, frame, and enhance the direction of virtual staining. Furthermore, we have developed a data augmentation method based on the constraints of the VLM. This method utilizes the VLM's powerful image interpretation capabilities to further integrate image style and structural information, proving beneficial in high-precision pathological diagnostics. Extensive evaluations on publicly available multi-domain unpaired staining datasets demonstrate that our method can generate highly realistic images and enhance the accuracy of downstream tasks, such as glomerular detection and segmentation. Our code is available at: https://github.com/CZZZZZZZZZZZZZZZZZ/VPGAN-HARBOR

eess.IV↗

arXiv2024

Towards Unified Molecule-Enhanced Pathology Image Representation Learning via Integrating Spatial Transcriptomics

Recent advancements in multimodal pre-training models have significantly advanced computational pathology. However, current approaches predominantly rely on visual-language models, which may impose limitations from a molecular perspective and lead to performance bottlenecks. Here, we introduce a Unified Molecule-enhanced Pathology Image REpresentationn Learning framework (UMPIRE). UMPIRE aims to leverage complementary information from gene expression profiles to guide the multimodal pre-training, enhancing the molecular awareness of pathology image representation learning. We demonstrate that this molecular perspective provides a robust, task-agnostic training signal for learning pathology image embeddings. Due to the scarcity of paired data, approximately 4 million entries of spatial transcriptomics gene expression were collected to train the gene encoder. By leveraging powerful pre-trained encoders, UMPIRE aligns the encoders across over 697K pathology image-gene expression pairs. The performance of UMPIRE is demonstrated across various molecular-related downstream tasks, including gene expression prediction, spot classification, and mutation state prediction in whole slide images. Our findings highlight the effectiveness of multimodal data integration and open new avenues for exploring computational pathology enhanced by molecular perspectives. The code and pre-trained weights are available at https://github.com/Hanminghao/UMPIRE.

cs.CV↗

arXiv2024

Primordial black holes and scalar induced density perturbations: the effects of probability density functions

We investigate the second order energy density perturbation $δ^{(2)}$ induced by small-scale Gaussian and local-type non-Gaussian primordial curvature perturbations. The relative abundance of primordial black hole is calculated in terms of the probability density function of total energy density perturbation $δ_r=δ^{(1)}+\frac{1}{2}δ^{(2)}$. The effects of second order density perturbation greatly reduce the upper bounds of small-scale power spectra of primordial curvature perturbations by one to two orders of magnitude. For log-normal primordial power spectrum, its amplitude $A_ζ$ is constrained to be about $A_ζ\sim 3\times10^{-3}$. And for local-type non-Gaussianity with $f_{\mathrm{NL}}=10$, the upper bound of $A_ζ$ is about $2.5\times10^{-4}$.

astro-ph.CO↗

arXiv2024

MaskBEV: Towards A Unified Framework for BEV Detection and Map Segmentation

Accurate and robust multimodal multi-task perception is crucial for modern autonomous driving systems. However, current multimodal perception research follows independent paradigms designed for specific perception tasks, leading to a lack of complementary learning among tasks and decreased performance in multi-task learning (MTL) due to joint training. In this paper, we propose MaskBEV, a masked attention-based MTL paradigm that unifies 3D object detection and bird's eye view (BEV) map segmentation. MaskBEV introduces a task-agnostic Transformer decoder to process these diverse tasks, enabling MTL to be completed in a unified decoder without requiring additional design of specific task heads. To fully exploit the complementary information between BEV map segmentation and 3D object detection tasks in BEV space, we propose spatial modulation and scene-level context aggregation strategies. These strategies consider the inherent dependencies between BEV segmentation and 3D detection, naturally boosting MTL performance. Extensive experiments on nuScenes dataset show that compared with previous state-of-the-art MTL methods, MaskBEV achieves 1.3 NDS improvement in 3D object detection and 2.7 mIoU improvement in BEV map segmentation, while also demonstrating slightly leading inference speed.

cs.CV↗

arXiv2024

HybridOcc: NeRF Enhanced Transformer-based Multi-Camera 3D Occupancy Prediction

Vision-based 3D semantic scene completion (SSC) describes autonomous driving scenes through 3D volume representations. However, the occlusion of invisible voxels by scene surfaces poses challenges to current SSC methods in hallucinating refined 3D geometry. This paper proposes HybridOcc, a hybrid 3D volume query proposal method generated by Transformer framework and NeRF representation and refined in a coarse-to-fine SSC prediction framework. HybridOcc aggregates contextual features through the Transformer paradigm based on hybrid query proposals while combining it with NeRF representation to obtain depth supervision. The Transformer branch contains multiple scales and uses spatial cross-attention for 2D to 3D transformation. The newly designed NeRF branch implicitly infers scene occupancy through volume rendering, including visible and invisible voxels, and explicitly captures scene depth rather than generating RGB color. Furthermore, we present an innovative occupancy-aware ray sampling method to orient the SSC task instead of focusing on the scene surface, further improving the overall performance. Extensive experiments on nuScenes and SemanticKITTI datasets demonstrate the effectiveness of our HybridOcc on the SSC task.

cs.CV↗

arXiv2024

Multi-Scale Heterogeneity-Aware Hypergraph Representation for Histopathology Whole Slide Images

Survival prediction is a complex ordinal regression task that aims to predict the survival coefficient ranking among a cohort of patients, typically achieved by analyzing patients' whole slide images. Existing deep learning approaches mainly adopt multiple instance learning or graph neural networks under weak supervision. Most of them are unable to uncover the diverse interactions between different types of biological entities(\textit{e.g.}, cell cluster and tissue block) across multiple scales, while such interactions are crucial for patient survival prediction. In light of this, we propose a novel multi-scale heterogeneity-aware hypergraph representation framework. Specifically, our framework first constructs a multi-scale heterogeneity-aware hypergraph and assigns each node with its biological entity type. It then mines diverse interactions between nodes on the graph structure to obtain a global representation. Experimental results demonstrate that our method outperforms state-of-the-art approaches on three benchmark datasets. Code is publicly available at \href{https://github.com/Hanminghao/H2GT}{https://github.com/Hanminghao/H2GT}.

cs.CV↗

arXiv2024

An objective comparison of methods for augmented reality in laparoscopic liver resection by preoperative-to-intraoperative image fusion

Augmented reality for laparoscopic liver resection is a visualisation mode that allows a surgeon to localise tumours and vessels embedded within the liver by projecting them on top of a laparoscopic image. Preoperative 3D models extracted from CT or MRI data are registered to the intraoperative laparoscopic images during this process. In terms of 3D-2D fusion, most of the algorithms make use of anatomical landmarks to guide registration. These landmarks include the liver's inferior ridge, the falciform ligament, and the occluding contours. They are usually marked by hand in both the laparoscopic image and the 3D model, which is time-consuming and may contain errors if done by a non-experienced user. Therefore, there is a need to automate this process so that augmented reality can be used effectively in the operating room. We present the Preoperative-to-Intraoperative Laparoscopic Fusion Challenge (P2ILF), held during the Medical Imaging and Computer Assisted Interventions (MICCAI 2022) conference, which investigates the possibilities of detecting these landmarks automatically and using them in registration. The challenge was divided into two tasks: 1) A 2D and 3D landmark detection task and 2) a 3D-2D registration task. The teams were provided with training data consisting of 167 laparoscopic images and 9 preoperative 3D models from 9 patients, with the corresponding 2D and 3D landmark annotations. A total of 6 teams from 4 countries participated, whose proposed methods were evaluated on 16 images and two preoperative 3D models from two patients. All the teams proposed deep learning-based methods for the 2D and 3D landmark segmentation tasks and differentiable rendering-based methods for the registration task. Based on the experimental outcomes, we propose three key hypotheses that determine current limitations and future directions for research in this domain.

cs.CV↗

arXiv2024

Primordial gravitational waves and curvature perturbations induced energy density perturbation

We study the second order scalar and density perturbations generated by the Gaussian curvature perturbations and primordial gravitational waves in the radiation-dominated era. After presenting all the possible second-order source terms, we obtain the explicit expressions of the kernel functions and the power spectra of the second order scalar perturbations. It shows that the primordial gravitational waves might affect the second order energy density perturbation significantly. The effects of the primordial gravitational waves are studied in terms of different kinds of primordial power spectra.

astro-ph.CO↗

arXiv2023

Tissue Segmentation of Thick-Slice Fetal Brain MR Scans with Guidance from High-Quality Isotropic Volumes

Accurate tissue segmentation of thick-slice fetal brain magnetic resonance (MR) scans is crucial for both reconstruction of isotropic brain MR volumes and the quantification of fetal brain development. However, this task is challenging due to the use of thick-slice scans in clinically-acquired fetal brain data. To address this issue, we propose to leverage high-quality isotropic fetal brain MR volumes (and also their corresponding annotations) as guidance for segmentation of thick-slice scans. Due to existence of significant domain gap between high-quality isotropic volume (i.e., source data) and thick-slice scans (i.e., target data), we employ a domain adaptation technique to achieve the associated knowledge transfer (from high-quality volumes to thick-slice scans). Specifically, we first register the available high-quality isotropic fetal brain MR volumes across different gestational weeks to construct longitudinally-complete source data. To capture domain-invariant information, we then perform Fourier decomposition to extract image content and style codes. Finally, we propose a novel Cycle-Consistent Domain Adaptation Network (C2DA-Net) to efficiently transfer the knowledge learned from high-quality isotropic volumes for accurate tissue segmentation of thick-slice scans. Our C2DA-Net can fully utilize a small set of annotated isotropic volumes to guide tissue segmentation on unannotated thick-slice scans. Extensive experiments on a large-scale dataset of 372 clinically acquired thick-slice MR scans demonstrate that our C2DA-Net achieves much better performance than cutting-edge methods quantitatively and qualitatively.

eess.IV↗

arXiv2023

Pulsars as candidates of LHAASO sources J2226+6057, J1908+0621 and J1825-1326: The leptonic origin

Recently, from 12 $γ$-ray Galactic sources, the LHAASO has detected ultrahigh-energy photons up to 1.4PeV. The $γ$-ray spectra of the sources J2226+6057, J1908+0621, J1825-1326 and the suggested origin pulsars near the sources have been published. In our previous work, we studied the hadronic $γ$-ray spectra of the sources J2226+6057, J1908+0621, J1825-1326 in terms of the Hertzian dipole model of pulsar. In this paper, we investigate the possibility of the leptonic origin of the $γ$-ray. We use the Hertzian dipole model to describe the pulsars around the sources. The electrons around the pulsars can be accelerated to PeV by the electromagnetic fields of pulsars. Under the assumption that the initial electrons are uniform distributed in a spherical shell between $10^{7}$ to $10^{9}$m around the pulsar, we obtain the energy distribution of electrons. The leptonic $γ$-ray spectra can be calculated through inverse Compton scattering processes. The leptonic $γ$-ray can roughly conform to the observation of LHAASO.

astro-ph.HE↗

arXiv2023

Gravitational waves from primordial scalar and tensor perturbations

We investigate the second order gravitational waves induced by the primordial scalar and tensor perturbations during radiation-dominated era. The explicit expressions of the power spectra of the second order GWs are presented. We calculate the energy density spectra of the second order GWs for a monochromatic primordial power spectra. For large $k$ $\left( k>k_* \right)$, the effects of the primordial tensor perturbation with tensor-to-scalar ratio $r=A_{h}/A_ζ=0.2$ lead to an around $50\% $ increase of the signal-to-noise ratio (SNR) for LISA observations.

astro-ph.CO↗

arXiv2023

Primordial black holes and third order scalar induced gravitational waves

The process of \acp{PBH} formation would be inevitably accompanied by \acp{SIGW}. This strong correlation between \acp{PBH} and \acp{SIGW} signals could be a promising approach to detecting \acp{PBH} in the upcoming \ac{GW} experiments, such as \ac{LISA}. We investigate the third order \acp{SIGW} during a \ac{RD} era in the case of a monochromatic primordial power spectrum $\mathcal{P}_ζ=A_ζk_*δ\left(k-k_*\right)$. For \ac{LISA} observations, the relations between \ac{SNR} and monochromatic primordial power spectrum are studied systematically. It shows that the effects of third order \acp{SIGW} extend the cutoff frequency from $2f_*$ to $3f_*$ and lead to about $200\%$ increase of the \ac{SNR} for frequency band from $10^{-5}$Hz to $1.6\times 10^{-3}$Hz corresponding to \acp{PBH} with mass range $4\times 10^{-12}M_{\odot} \sim 10^{-7}M_{\odot}$. We find that there exists a critical value $A_*=1.76\times 10^{-2}$ for the amplitude of the monochromatic primordial power spectra, such that when $A_ζ>A_*$, the energy density of third order \acp{SIGW} will be larger than the energy density of second order \acp{SIGW}.

astro-ph.CO↗

arXiv2022

Pulsars as candidates of LHAASO sources J2226+6057, J1908+0621 and J1825-1326

The LHAASO Collaboration has observed ultrahigh-energy photons up to $1.4$PeV from $12$ $γ$-ray Galactic sources. In particular, the $γ$-ray spectra of the sources J2226+6057, J1908+0621, J1825-1326 have been published. We investigate the possibility of suggested origin pulsars near the sources as the PeVatrons. The pulsar is described by a rotating magnetic dipole. Assuming protons are uniform distributed out of the light cylinders, we obtain the Lorentz distribution of proton energy spectrum. It is found that the protons around pulsar could be accelerated to PeV at short times. The hadronic $γ$-ray spectra of the suggested origin pulsars are in good agreement with the LHAASO observed $γ$-ray spectra of the sources J2226+6057, J1908+0621, J1825-1326.

astro-ph.HE↗

arXiv2022

Impact of the free-streaming neutrinos to the second order induced gravitational waves

The damping effect of the free-streaming neutrinos on the second order gravitational waves is investigated in detail. We solve the Boltzmann equation and give the anisotropic stress induced by neutrinos to second order. The first order tensor and its coupling with scalar perturbations induced gravitational waves are considered. We give the analytic equations of the damping kernel functions and finally obtain the energy density spectrum. The results show that the free-streaming neutrinos suppress the density spectrum significantly for low frequency gravitational waves and enlarge the logarithmic slope $n$ in the infrared region ($k \ll k_*$) of the spectrum. For the spectrum of $k_*\sim 10^{-7}$Hz, the damping effect in the range of $k<k_*$ is significant. The combined effect of the first and second order could reduce the amplitude by $30\%$ and make $n$ jump from $1.54$ to $1.63$ at $k\sim 10^{-9}$Hz, which may be probed by the pulsar timing arrays (PTA) in the future.

astro-ph.CO↗