arXiv subjects

Binod Bhattarai

Publications and source records attributed to Binod Bhattarai.

At least 37 records · Page 2Linked to original sources

arXiv2025

Surgical Vision World Model

Realistic and interactive surgical simulation has the potential to facilitate crucial applications, such as medical professional training and autonomous surgical agent training. In the natural visual domain, world models have enabled action-controlled data generation, demonstrating the potential to train autonomous agents in interactive simulated environments when large-scale real data acquisition is infeasible. However, such works in the surgical domain have been limited to simplified computer simulations, and lack realism. Furthermore, existing literature in world models has predominantly dealt with action-labeled data, limiting their applicability to real-world surgical data, where obtaining action annotation is prohibitively expensive. Inspired by the recent success of Genie in leveraging unlabeled video game data to infer latent actions and enable action-controlled data generation, we propose the first surgical vision world model. The proposed model can generate action-controllable surgical data and the architecture design is verified with extensive experiments on the unlabeled SurgToolLoc-2022 dataset. Codes and implementation details are available at https://github.com/bhattarailab/Surgical-Vision-World-Model

eess.IV↗

arXiv2025

NERO: Explainable Out-of-Distribution Detection with Neuron-level Relevance

Ensuring reliability is paramount in deep learning, particularly within the domain of medical imaging, where diagnostic decisions often hinge on model outputs. The capacity to separate out-of-distribution (OOD) samples has proven to be a valuable indicator of a model's reliability in research. In medical imaging, this is especially critical, as identifying OOD inputs can help flag potential anomalies that might otherwise go undetected. While many OOD detection methods rely on feature or logit space representations, recent works suggest these approaches may not fully capture OOD diversity. To address this, we propose a novel OOD scoring mechanism, called NERO, that leverages neuron-level relevance at the feature layer. Specifically, we cluster neuron-level relevance for each in-distribution (ID) class to form representative centroids and introduce a relevance distance metric to quantify a new sample's deviation from these centroids, enhancing OOD separability. Additionally, we refine performance by incorporating scaled relevance in the bias term and combining feature norms. Our framework also enables explainable OOD detection. We validate its effectiveness across multiple deep learning architectures on the gastrointestinal imaging benchmarks Kvasir and GastroVision, achieving improvements over state-of-the-art OOD detection methods.

cs.CV↗

arXiv2025

Estimating 2D Keypoints of Surgical Tools Using Vision-Language Models with Low-Rank Adaptation

This paper presents a novel pipeline for 2D keypoint estima- tion of surgical tools by leveraging Vision Language Models (VLMs) fine- tuned using a low rank adjusting (LoRA) technique. Unlike traditional Convolutional Neural Network (CNN) or Transformer-based approaches, which often suffer from overfitting in small-scale medical datasets, our method harnesses the generalization capabilities of pre-trained VLMs. We carefully design prompts to create an instruction-tuning dataset and use them to align visual features with semantic keypoint descriptions. Experimental results show that with only two epochs of fine tuning, the adapted VLM outperforms the baseline models, demonstrating the ef- fectiveness of LoRA in low-resource scenarios. This approach not only improves keypoint detection performance, but also paves the way for future work in 3D surgical hands and tools pose estimation.

cs.CV↗

arXiv2025

Effect of Data Augmentation on Conformal Prediction for Diabetic Retinopathy

The clinical deployment of deep learning models for high-stakes tasks such as diabetic retinopathy (DR) grading requires demonstrable reliability. While models achieve high accuracy, their clinical utility is limited by a lack of robust uncertainty quantification. Conformal prediction (CP) offers a distribution-free framework to generate prediction sets with statistical guarantees of coverage. However, the interaction between standard training practices like data augmentation and the validity of these guarantees is not well understood. In this study, we systematically investigate how different data augmentation strategies affect the performance of conformal predictors for DR grading. Using the DDR dataset, we evaluate two backbone architectures -- ResNet-50 and a Co-Scale Conv-Attentional Transformer (CoaT) -- trained under five augmentation regimes: no augmentation, standard geometric transforms, CLAHE, Mixup, and CutMix. We analyze the downstream effects on conformal metrics, including empirical coverage, average prediction set size, and correct efficiency. Our results demonstrate that sample-mixing strategies like Mixup and CutMix not only improve predictive accuracy but also yield more reliable and efficient uncertainty estimates. Conversely, methods like CLAHE can negatively impact model certainty. These findings highlight the need to co-design augmentation strategies with downstream uncertainty quantification in mind to build genuinely trustworthy AI systems for medical imaging.

cs.CV↗

arXiv2025

Addressing Bias in VLMs for Glaucoma Detection Without Protected Attribute Supervision

Vision-Language Models (VLMs) have achieved remarkable success on multimodal tasks such as image-text retrieval and zero-shot classification, yet they can exhibit demographic biases even when explicit protected attributes are absent during training. In this work, we focus on automated glaucoma screening from retinal fundus images, a critical application given that glaucoma is a leading cause of irreversible blindness and disproportionately affects underserved populations. Building on a reweighting-based contrastive learning framework, we introduce an attribute-agnostic debiasing method that (i) infers proxy subgroups via unsupervised clustering of image-image embeddings, (ii) computes gradient-similarity weights between the CLIP-style multimodal loss and a SimCLR-style image-pair contrastive loss, and (iii) applies these weights in a joint, top-$k$ weighted objective to upweight underperforming clusters. This label-free approach adaptively targets the hardest examples, thereby reducing subgroup disparities. We evaluate our method on the Harvard FairVLMed glaucoma subset, reporting Equalized Odds Distance (EOD), Equalized Subgroup AUC (ES AUC), and Groupwise AUC to demonstrate equitable performance across inferred demographic subgroups.

cs.CV↗

arXiv2025

The Open Cluster Chemical Abundances and Mapping Survey: VIII. Galactic Chemical Gradient and Azimuthal Analysis from SDSS/MWM DR19

The Open Cluster Chemical Abundances and Mapping (OCCAM) survey seeks to curate a large, comprehensive, uniform dataset of open clusters and member stars to constrain key Galactic parameters. This eighth entry from the OCCAM survey, based on the newly released SDSS-V/MWM Data Release 19 (DR19), has established a sample of 164 high quality open clusters that are used to constrain the radial and azimuthal gradients of the Milky Way. The DR19 cluster sample [Fe/H] abundances are roughly consistent with measurements from other large-scale spectroscopic surveys. However, the gradients we calculate deviate considerably for some elements. We find an overall linear Galactic radial [Fe/H] gradient of $-0.075 \pm 0.006$ dex kpc$^{-1}$ using the cluster's current Galactocentric Radius ($R_{GC}$) and a gradient of $-0.068 \pm 0.005$ dex kpc$^-1$ with respect to the cluster's guiding center radius. We do not find strong evidence for significant evolution of the differential element gradients ([X/Fe]) investigated here (O, Mg, Si, S, Ca, Ti, Cr, Mn, Fe, Co, Ni, Na, Al, K, Ce, Nd). For the first time using the OCCAM sample we have sufficient numbers of clusters to investigate Galactic azimuthal variations. In this work, we do find evidence of azimuthal variations in the measured radial abundance gradient in the Galactic disk using our open cluster sample.

astro-ph.GA↗

arXiv2025

Sloan Digital Sky Survey-V: Pioneering Panoptic Spectroscopy

The Sloan Digital Sky Survey-V (SDSS-V) is pioneering panoptic spectroscopy: it is the first all-sky, multi-epoch, optical-to-infrared spectroscopic survey. SDSS-V is mapping the sky with multi-object spectroscopy (MOS) at telescopes in both hemispheres (the 2.5-m Sloan Foundation Telescope at Apache Point Observatory and the 100-inch du Pont Telescope at Las Campanas Observatory), where 500 zonal robotic fiber positioners feed light from a wide-field focal plane to an optical (R$\sim 2000$, 500 fibers) and a near-infrared (R$\sim 22,000$, 300 fibers) spectrograph. In addition to these MOS capabilities, the survey is pioneering ultra wide-field ($\sim$ 4000~deg$^2$) integral field spectroscopy enabled by a new dedicated facility (LVM-I) at Las Campanas Observatory, where an integral field spectrograph (IFS) with 1801 lenslet-coupled fibers arranged in a 0.5 degree diameter hexagon feeds multiple R$\sim$4000 optical spectrographs that cover 3600-9800 angstroms. SDSS-V's hardware and multi-year survey strategy are designed to decode the chemo-dynamical history of the Milky Way Galaxy and tackle fundamental open issues in stellar physics in its Milky Way Mapper program, trace the growth physics of supermassive black holes in its Black Hole Mapper program, and understand the self-regulation mechanisms and the chemical enrichment of galactic ecosystems at the energy-injection scale in its Local Volume Mapper program. The survey is well-timed to multiply the scientific output from major all-sky space missions. The SDSS-V MOS programs began robotic operations in 2021; IFS observations began in 2023 with the completion of the LVM-I facility. SDSS-V builds upon decades of heritage of SDSS's pioneering advances in data analysis, collaboration spirit, infrastructure, and product deliverables in astronomy.

astro-ph.IM↗

arXiv2025

Hallucination-Aware Multimodal Benchmark for Gastrointestinal Image Analysis with Large Vision-Language Models

Vision-Language Models (VLMs) are becoming increasingly popular in the medical domain, bridging the gap between medical images and clinical language. Existing VLMs demonstrate an impressive ability to comprehend medical images and text queries to generate detailed, descriptive diagnostic medical reports. However, hallucination--the tendency to generate descriptions that are inconsistent with the visual content--remains a significant issue in VLMs, with particularly severe implications in the medical field. To facilitate VLM research on gastrointestinal (GI) image analysis and study hallucination, we curate a multimodal image-text GI dataset: Gut-VLM. This dataset is created using a two-stage pipeline: first, descriptive medical reports of Kvasir-v2 images are generated using ChatGPT, which introduces some hallucinated or incorrect texts. In the second stage, medical experts systematically review these reports, and identify and correct potential inaccuracies to ensure high-quality, clinically reliable annotations. Unlike traditional datasets that contain only descriptive texts, our dataset also features tags identifying hallucinated sentences and their corresponding corrections. A common approach to reducing hallucination in VLM is to finetune the model on a small-scale, problem-specific dataset. However, we take a different strategy using our dataset. Instead of finetuning the VLM solely for generating textual reports, we finetune it to detect and correct hallucinations, an approach we call hallucination-aware finetuning. Our results show that this approach is better than simply finetuning for descriptive report generation. Additionally, we conduct an extensive evaluation of state-of-the-art VLMs across several metrics, establishing a benchmark. GitHub Repo: https://github.com/bhattarailab/Hallucination-Aware-VLM.

cs.CV↗

arXiv2025

Federated Foundation Model for GI Endoscopy Images

Gastrointestinal (GI) endoscopy is essential in identifying GI tract abnormalities in order to detect diseases in their early stages and improve patient outcomes. Although deep learning has shown success in supporting GI diagnostics and decision-making, these models require curated datasets with labels that are expensive to acquire. Foundation models offer a promising solution by learning general-purpose representations, which can be finetuned for specific tasks, overcoming data scarcity. Developing foundation models for medical imaging holds significant potential, but the sensitive and protected nature of medical data presents unique challenges. Foundation model training typically requires extensive datasets, and while hospitals generate large volumes of data, privacy restrictions prevent direct data sharing, making foundation model training infeasible in most scenarios. In this work, we propose a FL framework for training foundation models for gastroendoscopy imaging, enabling data to remain within local hospital environments while contributing to a shared model. We explore several established FL algorithms, assessing their suitability for training foundation models without relying on task-specific labels, conducting experiments in both homogeneous and heterogeneous settings. We evaluate the trained foundation model on three critical downstream tasks--classification, detection, and segmentation--and demonstrate that it achieves improved performance across all tasks, highlighting the effectiveness of our approach in a federated, privacy-preserving setting.

cs.CV↗

arXiv2025

Multimodal Federated Learning With Missing Modalities through Feature Imputation Network

Multimodal federated learning holds immense potential for collaboratively training models from multiple sources without sharing raw data, addressing both data scarcity and privacy concerns, two key challenges in healthcare. A major challenge in training multimodal federated models in healthcare is the presence of missing modalities due to multiple reasons, including variations in clinical practice, cost and accessibility constraints, retrospective data collection, privacy concerns, and occasional technical or human errors. Previous methods typically rely on publicly available real datasets or synthetic data to compensate for missing modalities. However, obtaining real datasets for every disease is impractical, and training generative models to synthesize missing modalities is computationally expensive and prone to errors due to the high dimensionality of medical data. In this paper, we propose a novel, lightweight, low-dimensional feature translator to reconstruct bottleneck features of the missing modalities. Our experiments on three different datasets (MIMIC-CXR, NIH Open-I, and CheXpert), in both homogeneous and heterogeneous settings consistently improve the performance of competitive baselines. The code and implementation details are available at: https://github.com/bhattarailab/FedFeatGen

cs.LG↗

arXiv2025

Effect of gas accretion on $α$-element bimodality in Milky Way-mass galaxies in the FIRE-2 simulations

We analyse the stellar distributions on the [Fe/H]-[Mg/Fe] plane for 11 Milky Way-mass galaxies from the FIRE-2 cosmological baryonic zoom-in simulations. Alpha-element bimodality, in the form of two separate sequences on the [Fe/H]-[Mg/Fe] plane, is not a universal feature of disk galaxies. Five galaxies demonstrate double sequences with the $α$-enriched one being older and kinematically hotter, in qualitative agreement with the high-$α$ and low-$α$ populations in the Milky Way disk; three galaxies have unimodal distribution, two show weakly-bimodal features where low-$α$ sequence is visible only over a short range of metallicities, and one show strong bimodality with a different slope of high-$α$ population. We examine the galaxies' gas accretion history over the last 8 Gyr, when bimodal sequences emerge, and demonstrate that the presence of the low-$α$ sequence in the bimodal galaxies is related to the recent infall of metal-poor gas from the circumgalactic medium that joins the galaxy in the outskirts and induces significant growth of the gas disks compared to their non-bimodal counterparts. We also analyse the sources of the accreted gas and illustrate that both gas-rich mergers and smooth accretion of ambient gas can be the source of the accreted gas, and create slightly different bimodal patterns.

astro-ph.GA↗

arXiv2024

Prompt Augmentation for Self-supervised Text-guided Image Manipulation

Text-guided image editing finds applications in various creative and practical fields. While recent studies in image generation have advanced the field, they often struggle with the dual challenges of coherent image transformation and context preservation. In response, our work introduces prompt augmentation, a method amplifying a single input prompt into several target prompts, strengthening textual context and enabling localised image editing. Specifically, we use the augmented prompts to delineate the intended manipulation area. We propose a Contrastive Loss tailored to driving effective image editing by displacing edited areas and drawing preserved regions closer. Acknowledging the continuous nature of image manipulations, we further refine our approach by incorporating the similarity concept, creating a Soft Contrastive Loss. The new losses are incorporated to the diffusion model, demonstrating improved or competitive image editing results on public datasets and generated images over state-of-the-art approaches.

cs.CV↗

arXiv2024

NCDD: Nearest Centroid Distance Deficit for Out-Of-Distribution Detection in Gastrointestinal Vision

The integration of deep learning tools in gastrointestinal vision holds the potential for significant advancements in diagnosis, treatment, and overall patient care. A major challenge, however, is these tools' tendency to make overconfident predictions, even when encountering unseen or newly emerging disease patterns, undermining their reliability. We address this critical issue of reliability by framing it as an out-of-distribution (OOD) detection problem, where previously unseen and emerging diseases are identified as OOD examples. However, gastrointestinal images pose a unique challenge due to the overlapping feature representations between in- Distribution (ID) and OOD examples. Existing approaches often overlook this characteristic, as they are primarily developed for natural image datasets, where feature distinctions are more apparent. Despite the overlap, we hypothesize that the features of an in-distribution example will cluster closer to the centroids of their ground truth class, resulting in a shorter distance to the nearest centroid. In contrast, OOD examples maintain an equal distance from all class centroids. Based on this observation, we propose a novel nearest-centroid distance deficit (NCCD) score in the feature space for gastrointestinal OOD detection. Evaluations across multiple deep learning architectures and two publicly available benchmarks, Kvasir2 and Gastrovision, demonstrate the effectiveness of our approach compared to several state-of-the-art methods. The code and implementation details are publicly available at: https://github.com/bhattarailab/NCDD

cs.CV↗

arXiv2024

Guide3D: A Bi-planar X-ray Dataset for 3D Shape Reconstruction

Endovascular surgical tool reconstruction represents an important factor in advancing endovascular tool navigation, which is an important step in endovascular surgery. However, the lack of publicly available datasets significantly restricts the development and validation of novel machine learning approaches. Moreover, due to the need for specialized equipment such as biplanar scanners, most of the previous research employs monoplanar fluoroscopic technologies, hence only capturing the data from a single view and significantly limiting the reconstruction accuracy. To bridge this gap, we introduce Guide3D, a bi-planar X-ray dataset for 3D reconstruction. The dataset represents a collection of high resolution bi-planar, manually annotated fluoroscopic videos, captured in real-world settings. Validating our dataset within a simulated environment reflective of clinical settings confirms its applicability for real-world applications. Furthermore, we propose a new benchmark for guidewrite shape prediction, serving as a strong baseline for future work. Guide3D not only addresses an essential need by offering a platform for advancing segmentation and 3D reconstruction techniques but also aids the development of more accurate and efficient endovascular surgery interventions. Our project is available at https://airvlab.github.io/guide3d/.

eess.IV↗

arXiv2024

Active Label Refinement for Robust Training of Imbalanced Medical Image Classification Tasks in the Presence of High Label Noise

The robustness of supervised deep learning-based medical image classification is significantly undermined by label noise. Although several methods have been proposed to enhance classification performance in the presence of noisy labels, they face some challenges: 1) a struggle with class-imbalanced datasets, leading to the frequent overlooking of minority classes as noisy samples; 2) a singular focus on maximizing performance using noisy datasets, without incorporating experts-in-the-loop for actively cleaning the noisy labels. To mitigate these challenges, we propose a two-phase approach that combines Learning with Noisy Labels (LNL) and active learning. This approach not only improves the robustness of medical image classification in the presence of noisy labels, but also iteratively improves the quality of the dataset by relabeling the important incorrect labels, under a limited annotation budget. Furthermore, we introduce a novel Variance of Gradients approach in LNL phase, which complements the loss-based sample selection by also sampling under-represented samples. Using two imbalanced noisy medical classification datasets, we demonstrate that that our proposed technique is superior to its predecessors at handling class imbalance by not misidentifying clean samples from minority classes as mostly noisy samples.

cs.LG↗

arXiv2024

Dimension Mixer: Group Mixing of Input Dimensions for Efficient Function Approximation

The recent success of multiple neural architectures like CNNs, Transformers, and MLP-Mixers motivated us to look for similarities and differences between them. We found that these architectures can be interpreted through the lens of a general concept of dimension mixing. Research on coupling flows and the butterfly transform shows that partial and hierarchical signal mixing schemes are sufficient for efficient and expressive function approximation. In this work, we study group-wise sparse, non-linear, multi-layered and learnable mixing schemes of inputs and find that they are complementary to many standard neural architectures. Following our observations and drawing inspiration from the Fast Fourier Transform, we generalize Butterfly Structure to use non-linear mixer function allowing for MLP as mixing function called Butterfly MLP. We were also able to sparsely mix along sequence dimension for Transformer-based architectures called Butterfly Attention. Experiments on CIFAR and LRA datasets demonstrate that the proposed Non-Linear Butterfly Mixers are efficient and scale well when the host architectures are used as mixing function. Additionally, we propose Patch-Only MLP-Mixer for processing spatial 2D signals demonstrating a different dimension mixing strategy.

cs.LG↗

arXiv2024

CathAction: A Benchmark for Endovascular Intervention Understanding

Real-time visual feedback from catheterization analysis is crucial for enhancing surgical safety and efficiency during endovascular interventions. However, existing datasets are often limited to specific tasks, small scale, and lack the comprehensive annotations necessary for broader endovascular intervention understanding. To tackle these limitations, we introduce CathAction, a large-scale dataset for catheterization understanding. Our CathAction dataset encompasses approximately 500,000 annotated frames for catheterization action understanding and collision detection, and 25,000 ground truth masks for catheter and guidewire segmentation. For each task, we benchmark recent related works in the field. We further discuss the challenges of endovascular intentions compared to traditional computer vision tasks and point out open research questions. We hope that CathAction will facilitate the development of endovascular intervention understanding methods that can be applied to real-world applications. The dataset is available at https://airvlab.github.io/cathaction/.

cs.CV↗

arXiv2024

Strong Chemical Tagging in FIRE: Intra and Inter-Cluster Chemical Homogeneity in Open Clusters in Milky Way-like Galaxy Simulations

Open star clusters are the essential building blocks of the Galactic disk; "strong chemical tagging" - the premise that all star clusters can be reconstructed given chemistry information alone - is a driving force behind many current and upcoming large Galactic spectroscopic surveys. In this work, we characterize abundance patterns for 9 elements (C, N, O, Ne, Mg, Si, S, Ca, and Fe) in open clusters (OCs) in three galaxies (m12i, m12f, and m12m) from the Latte suite of FIRE-2 simulations to investigate if strong chemical tagging is possible in these simulations. We select young massive (>=10^(4.6) Msun) OCs formed in the last ~100 Myr and calculate the intra- and inter-cluster abundance scatter for these clusters. We compare these results with analogous calculations drawn from observations of OCs in the Milky Way. We find the intra-cluster scatter of the observations and simulations to be comparable. While the abundance scatter within each cluster is minimal (<0.020 dex), the mean abundance patterns of different clusters are not unique. We also calculate the chemical difference in intra- and inter-cluster star pairs and find it, in general, to be so small that it is difficult to distinguish between stars drawn from the same OC or from different OCs. Despite tracing three distinct nucleosynthetic families (core-collapse supernovae, white dwarf supernovae, and stellar winds), we conclude that these elemental abundances do not provide enough discriminating information to use strong chemical tagging for reliable OC membership.

astro-ph.GA↗