Just-in-time volumetric imaging: Learning 2D-3D thoracic motion models during image-guided interventions

Abstract

Despite two decades of innovation, cutting-edge bronchoscopic techniques still fail to yield clear diagnostic decisions in 30% of cases and miss malignancies in 20% of cases. Intraprocedural 3D imaging drastically improves both tool positioning and diagnostic yield. However, each 3D spin represents a static snapshot of an anatomy that is continuously evolving as the patient is ventilated throughout the procedure. Here, we propose just-in-time volumetric imaging, in which a patient-specific 2D--3D motion model is trained on pre-intervention scans before it is needed and then supplies volumetric images from single fluoroscopic projections during sampling. Given the inspiration-hold CBCT already acquired for navigation and an additional motion-inclusive CBCT, our optimization regime named VoxelMap-DualScan (VM-DS), solves a low-rank motion model by alternating least squares and then amortizes it into a projection encoder trained on synthesized projections with known motion, all within a 10-minute window. On eight held-out cases from the public DIR-Lab 4DCT dataset, VM-DS achieved a mean landmark target registration error of 2.8 mm, a structural similarity index of 0.95 and a peak signal-to-noise ratio of 37.6 dB. Accuracy was maintained when reducing the number of projections used for training from 227 to 57. Inference takes 1.5 ms per projection, making the framework suitable for real-time use during image-guided interventions. Code will be made available at https://github.com/Image-X-Institute/Voxelmap/tree/VM-DS