arXiv subjects

Nassir Navab

Publications and source records attributed to Nassir Navab.

At least 307 records · Page 17Linked to original sources

arXiv2022

Surgical Workflow Recognition: from Analysis of Challenges to Architectural Study

Algorithmic surgical workflow recognition is an ongoing research field and can be divided into laparoscopic (Internal) and operating room (External) analysis. So far many different works for the internal analysis have been proposed with the combination of a frame-level and an additional temporal model to address the temporal ambiguities between different workflow phases. For the External recognition task, Clip-level methods are in the focus of researchers targeting the local ambiguities present in the OR scene. In this work we evaluate combinations of different model architectures for the task of surgical workflow recognition to provide a fair comparison of the methods for both Internal and External analysis. We show that methods designed for the Internal analysis can be transferred to the external task with comparable performance gains for different architectures.

cs.CV↗

arXiv2022

Complementary Textures. A Novel Approach to Object Alignment in Mixed Reality

Alignment between real and virtual objects is a challenging task required for the deployment of Mixed Reality (MR) into manufacturing, medical, and construction applications. To face this challenge, a series of methods have been proposed. While many approaches use dynamic augmentations such as animations, arrows, or text to assist users, they require tracking the position of real objects. In contrast, when tracking of the real objects is not available or desired, alternative approaches use virtual replicas of real objects to allow for interactive, perceptual virtual-to-real, and/or real-to-virtual alignment. In these cases, the accuracy achieved strongly depends on the quality of the perceptual information provided to the user. This paper proposes a novel set of perceptual alignment concepts that go beyond the use of traditional visualization of virtual replicas, introducing the concept of COMPLEMENTARY TEXTURES to improve interactive alignment in MR applications. To showcase the advantages of using COMPLEMENTARY TEXTURES, we describe three different implementations that provide highly salient visual cues when misalignment is observed; or present semantic augmentations that, when combined with a real object, provide contextual information that can be used during the alignment process. The authors aim to open new paths for the community to explore rather than describing end-to-end solutions. The objective is to show the multitude of opportunities such concepts could provide for further research and development.

cs.HC↗

arXiv2022

DisPositioNet: Disentangled Pose and Identity in Semantic Image Manipulation

Graph representation of objects and their relations in a scene, known as a scene graph, provides a precise and discernible interface to manipulate a scene by modifying the nodes or the edges in the graph. Although existing works have shown promising results in modifying the placement and pose of objects, scene manipulation often leads to losing some visual characteristics like the appearance or identity of objects. In this work, we propose DisPositioNet, a model that learns a disentangled representation for each object for the task of image manipulation using scene graphs in a self-supervised manner. Our framework enables the disentanglement of the variational latent embeddings as well as the feature representation in the graph. In addition to producing more realistic images due to the decomposition of features like pose and identity, our method takes advantage of the probabilistic sampling in the intermediate features to generate more diverse images in object replacement or addition tasks. The results of our experiments show that disentangling the feature representations in the latent manifold of the model outperforms the previous works qualitatively and quantitatively on two public benchmarks. Project Page: https://scenegenie.github.io/DispositioNet/

cs.CV↗

arXiv2022

Improved Techniques for the Conditional Generative Augmentation of Clinical Audio Data

Data augmentation is a valuable tool for the design of deep learning systems to overcome data limitations and stabilize the training process. Especially in the medical domain, where the collection of large-scale data sets is challenging and expensive due to limited access to patient data, relevant environments, as well as strict regulations, community-curated large-scale public datasets, pretrained models, and advanced data augmentation methods are the main factors for developing reliable systems to improve patient care. However, for the development of medical acoustic sensing systems, an emerging field of research, the community lacks large-scale publicly available data sets and pretrained models. To address the problem of limited data, we propose a conditional generative adversarial neural network-based augmentation method which is able to synthesize mel spectrograms from a learned data distribution of a source data set. In contrast to previously proposed fully convolutional models, the proposed model implements residual Squeeze and Excitation modules in the generator architecture. We show that our method outperforms all classical audio augmentation techniques and previously published generative methods in terms of generated sample quality and a performance improvement of 2.84% of Macro F1-Score for a classifier trained on the augmented data set, an enhancement of $1.14\%$ in relation to previous work. By analyzing the correlation of intermediate feature spaces, we show that the residual Squeeze and Excitation modules help the model to reduce redundancy in the latent features. Therefore, the proposed model advances the state-of-the-art in the augmentation of clinical audio data and improves the data bottleneck for the design of clinical acoustic sensing systems.

cs.LG↗

arXiv2022

A new sensing paradigm for the vibroacoustic detection of pedicle screw loosening

There is an unmet clinical need for developing novel methods to complement and replace the current radiation-emitting imaging-based methods for the detection of loose pedicle screws as a complication after spinal fusion surgery which fail to identify a substantial amount of loose implants. In this work, we propose a new methodology and paradigm for the radiation-free, non-destructive, and easy-to-integrate detection of pedicle screw loosening based on vibroacoustic sensing. Furthermore, we propose a novel simulation technique for pedicle screw loosening, which is biomechanically validated. For the detection of a loose implant, we excite the vertebra of interest with a sine sweep vibration at the spinous process and attach a custom highly-sensitive piezo contact microphone to the screw head to capture the propagated vibration characteristics which are analyzed using a detection pipeline based on spectrogram features and a SE-ResNet-18. To validate the proposed approach, we conducted experiments using four human cadaveric lumbar spine specimens and evaluate our algorithm in a cross validation experiment. Our method reaches a sensitivity of 91.50 +- 6.58% and a specificity of 91.10 +- 2.27%. The proposed system shows great potentials for the development of alternative assessment methods for implant loosening based on vibroacoustic sensing.

physics.med-ph↗

arXiv2022

What can we learn about a generated image corrupting its latent representation?

Generative adversarial networks (GANs) offer an effective solution to the image-to-image translation problem, thereby allowing for new possibilities in medical imaging. They can translate images from one imaging modality to another at a low cost. For unpaired datasets, they rely mostly on cycle loss. Despite its effectiveness in learning the underlying data distribution, it can lead to a discrepancy between input and output data. The purpose of this work is to investigate the hypothesis that we can predict image quality based on its latent representation in the GANs bottleneck. We achieve this by corrupting the latent representation with noise and generating multiple outputs. The degree of differences between them is interpreted as the strength of the representation: the more robust the latent representation, the fewer changes in the output image the corruption causes. Our results demonstrate that our proposed method has the ability to i) predict uncertain parts of synthesized images, and ii) identify samples that may not be reliable for downstream tasks, e.g., liver segmentation task.

cs.CV↗

arXiv2022

Know your sensORs -- A Modality Study For Surgical Action Classification

The surgical operating room (OR) presents many opportunities for automation and optimization. Videos from various sources in the OR are becoming increasingly available. The medical community seeks to leverage this wealth of data to develop automated methods to advance interventional care, lower costs, and improve overall patient outcomes. Existing datasets from OR room cameras are thus far limited in size or modalities acquired, leaving it unclear which sensor modalities are best suited for tasks such as recognizing surgical action from videos. This study demonstrates that surgical action recognition performance can vary depending on the image modalities used. We perform a methodical analysis on several commonly available sensor modalities, presenting two fusion approaches that improve classification performance. The analyses are carried out on a set of multi-view RGB-D video recordings of 18 laparoscopic procedures.

cs.CV↗

arXiv2022

Precise Repositioning of Robotic Ultrasound: Improving Registration-based Motion Compensation using Ultrasound Confidence Optimization

Robotic ultrasound (US) imaging has been seen as a promising solution to overcome the limitations of free-hand US examinations, i.e., inter-operator variability. However, the fact that robotic US systems cannot react to subject movements during scans limits their clinical acceptance. Regarding human sonographers, they often react to patient movements by repositioning the probe or even restarting the acquisition, in particular for the scans of anatomies with long structures like limb arteries. To realize this characteristic, we proposed a vision-based system to monitor the subject's movement and automatically update the scan trajectory thus seamlessly obtaining a complete 3D image of the target anatomy. The motion monitoring module is developed using the segmented object masks from RGB images. Once the subject is moved, the robot will stop and recompute a suitable trajectory by registering the surface point clouds of the object obtained before and after the movement using the iterative closest point algorithm. Afterward, to ensure optimal contact conditions after repositioning US probe, a confidence-based fine-tuning process is used to avoid potential gaps between the probe and contact surface. Finally, the whole system is validated on a human-like arm phantom with an uneven surface, while the object segmentation network is also validated on volunteers. The results demonstrate that the presented system can react to object movements and reliably provide accurate 3D images.

cs.RO↗

arXiv2022

STTAR: Surgical Tool Tracking using off-the-shelf Augmented Reality Head-Mounted Displays

The use of Augmented Reality (AR) for navigation purposes has shown beneficial in assisting physicians during the performance of surgical procedures. These applications commonly require knowing the pose of surgical tools and patients to provide visual information that surgeons can use during the task performance. Existing medical-grade tracking systems use infrared cameras placed inside the Operating Room (OR) to identify retro-reflective markers attached to objects of interest and compute their pose. Some commercially available AR Head-Mounted Displays (HMDs) use similar cameras for self-localization, hand tracking, and estimating the objects' depth. This work presents a framework that uses the built-in cameras of AR HMDs to enable accurate tracking of retro-reflective markers, such as those used in surgical procedures, without the need to integrate any additional components. This framework is also capable of simultaneously tracking multiple tools. Our results show that the tracking and detection of the markers can be achieved with an accuracy of 0.09 +- 0.06 mm on lateral translation, 0.42 +- 0.32 mm on longitudinal translation, and 0.80 +- 0.39 deg for rotations around the vertical axis. Furthermore, to showcase the relevance of the proposed framework, we evaluate the system's performance in the context of surgical procedures. This use case was designed to replicate the scenarios of k-wire insertions in orthopedic procedures. For evaluation, two surgeons and one biomedical researcher were provided with visual navigation, each performing 21 injections. Results from this use case provide comparable accuracy to those reported in the literature for AR-based navigation procedures.

cs.RO↗

arXiv2022

CACTUSS: Common Anatomical CT-US Space for US examinations

Abdominal aortic aneurysm (AAA) is a vascular disease in which a section of the aorta enlarges, weakening its walls and potentially rupturing the vessel. Abdominal ultrasound has been utilized for diagnostics, but due to its limited image quality and operator dependency, CT scans are usually required for monitoring and treatment planning. Recently, abdominal CT datasets have been successfully utilized to train deep neural networks for automatic aorta segmentation. Knowledge gathered from this solved task could therefore be leveraged to improve US segmentation for AAA diagnosis and monitoring. To this end, we propose CACTUSS: a common anatomical CT-US space, which acts as a virtual bridge between CT and US modalities to enable automatic AAA screening sonography. CACTUSS makes use of publicly available labelled data to learn to segment based on an intermediary representation that inherits properties from both US and CT. We train a segmentation network in this new representation and employ an additional image-to-image translation network which enables our model to perform on real B-mode images. Quantitative comparisons against fully supervised methods demonstrate the capabilities of CACTUSS in terms of Dice Score and diagnostic metrics, showing that our method also meets the clinical requirements for AAA scanning and diagnosis.

eess.IV↗

arXiv2022

Towards Autonomous Atlas-based Ultrasound Acquisitions in Presence of Articulated Motion

Robotic ultrasound (US) imaging aims at overcoming some of the limitations of free-hand US examinations, e.g. difficulty in guaranteeing intra- and inter-operator repeatability. However, due to anatomical and physiological variations between patients and relative movement of anatomical substructures, it is challenging to robustly generate optimal trajectories to examine the anatomies of interest, in particular, when they comprise articulated joints. To address this challenge, this paper proposes a vision-based approach allowing autonomous robotic US limb scanning. To this end, an atlas MRI template of a human arm with annotated vascular structures is used to generate trajectories and register and project them onto patients' skin surfaces for robotic US acquisition. To effectively segment and accurately reconstruct the targeted 3D vessel, we make use of spatial continuity in consecutive US frames by incorporating channel attention modules into a U-Net-type neural network. The automatic trajectory generation method is evaluated on six volunteers with various articulated joint angles. In all cases, the system can successfully acquire the planned vascular structure on volunteers' limbs. For one volunteer the MRI scan was also available, which allows the evaluation of the average radius of the scanned artery from US images, resulting in a radius estimation ($1.2\pm0.05~mm$) comparable to the MRI ground truth ($1.2\pm0.04~mm$).

cs.RO↗

arXiv2022

Speckle2Speckle: Unsupervised Learning of Ultrasound Speckle Filtering Without Clean Data

In ultrasound imaging the appearance of homogeneous regions of tissue is subject to speckle, which for certain applications can make the detection of tissue irregularities difficult. To cope with this, it is common practice to apply speckle reduction filters to the images. Most conventional filtering techniques are fairly hand-crafted and often need to be finely tuned to the present hardware, imaging scheme and application. Learning based techniques on the other hand suffer from the need for a target image for training (in case of fully supervised techniques) or require narrow, complex physics-based models of the speckle appearance that might not apply in all cases. With this work we propose a deep-learning based method for speckle removal without these limitations. To enable this, we make use of realistic ultrasound simulation techniques that allow for instantiation of several independent speckle realizations that represent the exact same tissue, thus allowing for the application of image reconstruction techniques that work with pairs of differently corrupted data. Compared to two other state-of-the-art approaches (non-local means and the Optimized Bayesian non-local means filter) our method performs favorably in qualitative comparisons and quantitative evaluation, despite being trained on simulations alone, and is several orders of magnitude faster.

eess.IV↗

arXiv2022

CloudAttention: Efficient Multi-Scale Attention Scheme For 3D Point Cloud Learning

Processing 3D data efficiently has always been a challenge. Spatial operations on large-scale point clouds, stored as sparse data, require extra cost. Attracted by the success of transformers, researchers are using multi-head attention for vision tasks. However, attention calculations in transformers come with quadratic complexity in the number of inputs and miss spatial intuition on sets like point clouds. We redesign set transformers in this work and incorporate them into a hierarchical framework for shape classification and part and scene segmentation. We propose our local attention unit, which captures features in a spatial neighborhood. We also compute efficient and dynamic global cross attentions by leveraging sampling and grouping at each iteration. Finally, to mitigate the non-heterogeneity of point clouds, we propose an efficient Multi-Scale Tokenization (MST), which extracts scale-invariant tokens for attention operations. The proposed hierarchical model achieves state-of-the-art shape classification in mean accuracy and yields results on par with the previous segmentation methods while requiring significantly fewer computations. Our proposed architecture predicts segmentation labels with around half the latency and parameter count of the previous most efficient method with comparable performance. The code is available at https://github.com/YigeWang-WHU/CloudAttention.

cs.CV↗

arXiv2022

DA$^2$ Dataset: Toward Dexterity-Aware Dual-Arm Grasping

In this paper, we introduce DA$^2$, the first large-scale dual-arm dexterity-aware dataset for the generation of optimal bimanual grasping pairs for arbitrary large objects. The dataset contains about 9M pairs of parallel-jaw grasps, generated from more than 6000 objects and each labeled with various grasp dexterity measures. In addition, we propose an end-to-end dual-arm grasp evaluation model trained on the rendered scenes from this dataset. We utilize the evaluation model as our baseline to show the value of this novel and nontrivial dataset by both online analysis and real robot experiments. All data and related code will be open-sourced at https://sites.google.com/view/da2dataset.

cs.RO↗

arXiv2022

Spotlight on nerves: Portable multispectral optoacoustic imaging of peripheral nerve vascularization and morphology

Various morphological and functional parameters of peripheral nerves and their vascular supply are indicative of pathological changes due to injury or disease. Based on recent improvements in optoacoustic image quality, we explore the ability of multispectral optoacoustic tomography, in tandem with ultrasound imaging (OPUS), to investigate the vascular environment and morphology of peripheral nerves in vivo in a pilot study on healthy volunteers. We showcase the unique ability of optoacoustic imaging to visualize the vasa nervorum by observing intraneurial vessels in healthy nerves in vivo for the first time. In addition, we demonstrate that the label-free spectral optoacoustic contrast of the perfused connective tissue of peripheral nerves can be linked to the endogenous contrast of haemoglobin and collagen. We introduce metrics to analyze the composition of tissue based on its optoacoustic contrast and show that the high-resolution spectral contrast reveals specific differences between nervous tissue and reference tissue in the nerve's surrounding. We discuss how this showcased extraction of peripheral nerve characteristics using multispectral optoacoustic and ultrasound imaging can offer new insights into the pathophysiology of nerve damage and neuropathies, for example, in the context of diabetes.

eess.IV↗

arXiv2022

Exploiting Diversity of Unlabeled Data for Label-Efficient Semi-Supervised Active Learning

The availability of large labeled datasets is the key component for the success of deep learning. However, annotating labels on large datasets is generally time-consuming and expensive. Active learning is a research area that addresses the issues of expensive labeling by selecting the most important samples for labeling. Diversity-based sampling algorithms are known as integral components of representation-based approaches for active learning. In this paper, we introduce a new diversity-based initial dataset selection algorithm to select the most informative set of samples for initial labeling in the active learning setting. Self-supervised representation learning is used to consider the diversity of samples in the initial dataset selection algorithm. Also, we propose a novel active learning query strategy, which uses diversity-based sampling on consistency-based embeddings. By considering the consistency information with the diversity in the consistency-based embedding scheme, the proposed method could select more informative samples for labeling in the semi-supervised learning setting. Comparative experiments show that the proposed method achieves compelling results on CIFAR-10 and Caltech-101 datasets compared with previous active learning approaches by utilizing the diversity of unlabeled data.

cs.CV↗

arXiv2022

CheXplaining in Style: Counterfactual Explanations for Chest X-rays using StyleGAN

Deep learning models used in medical image analysis are prone to raising reliability concerns due to their black-box nature. To shed light on these black-box models, previous works predominantly focus on identifying the contribution of input features to the diagnosis, i.e., feature attribution. In this work, we explore counterfactual explanations to identify what patterns the models rely on for diagnosis. Specifically, we investigate the effect of changing features within chest X-rays on the classifier's output to understand its decision mechanism. We leverage a StyleGAN-based approach (StyleEx) to create counterfactual explanations for chest X-rays by manipulating specific latent directions in their latent space. In addition, we propose EigenFind to significantly reduce the computation time of generated explanations. We clinically evaluate the relevancy of our counterfactual explanations with the help of radiologists. Our code is publicly available.

eess.IV↗

arXiv2022

Shape-Aware Masking for Inpainting in Medical Imaging

Inpainting has recently been proposed as a successful deep learning technique for unsupervised medical image model discovery. The masks used for inpainting are generally independent of the dataset and are not tailored to perform on different given classes of anatomy. In this work, we introduce a method for generating shape-aware masks for inpainting, which aims at learning the statistical shape prior. We hypothesize that although the variation of masks improves the generalizability of inpainting models, the shape of the masks should follow the topology of the organs of interest. Hence, we propose an unsupervised guided masking approach based on an off-the-shelf inpainting model and a superpixel over-segmentation algorithm to generate a wide range of shape-dependent masks. Experimental results on abdominal MR image reconstruction show the superiority of our proposed masking method over standard methods using square-shaped or dataset of irregular shape masks.

eess.IV↗