arXiv subjects

Roarke Horstmeyer

Publications and source records attributed to Roarke Horstmeyer.

At least 37 records · Page 2Linked to original sources

arXiv2022

Transient motion classification through turbid volumes via parallelized single-photon detection and deep contrastive embedding

Fast noninvasive probing of spatially varying decorrelating events, such as cerebral blood flow beneath the human skull, is an essential task in various scientific and clinical settings. One of the primary optical techniques used is diffuse correlation spectroscopy (DCS), whose classical implementation uses a single or few single-photon detectors, resulting in poor spatial localization accuracy and relatively low temporal resolution. Here, we propose a technique termed Classifying Rapid decorrelation Events via Parallelized single photon dEtection (CREPE)}, a new form of DCS that can probe and classify different decorrelating movements hidden underneath turbid volume with high sensitivity using parallelized speckle detection from a $32\times32$ pixel SPAD array. We evaluate our setup by classifying different spatiotemporal-decorrelating patterns hidden beneath a 5mm tissue-like phantom made with rapidly decorrelating dynamic scattering media. Twelve multi-mode fibers are used to collect scattered light from different positions on the surface of the tissue phantom. To validate our setup, we generate perturbed decorrelation patterns by both a digital micromirror device (DMD) modulated at multi-kilo-hertz rates, as well as a vessel phantom containing flowing fluid. Along with a deep contrastive learning algorithm that outperforms classic unsupervised learning methods, we demonstrate our approach can accurately detect and classify different transient decorrelation events (happening in 0.1-0.4s) underneath turbid scattering media, without any data labeling. This has the potential to be applied to noninvasively monitor deep tissue motion patterns, for example identifying normal or abnormal cerebral blood flow events, at multi-Hertz rates within a compact and static detection probe.

eess.IV↗

arXiv2021

Roadmap on Wavefront Shaping and deep imaging in complex media

The last decade has seen the development of a wide set of tools, such as wavefront shaping, computational or fundamental methods, that allow to understand and control light propagation in a complex medium, such as biological tissues or multimode fibers. A vibrant and diverse community is now working on this field, that has revolutionized the prospect of diffraction-limited imaging at depth in tissues. This roadmap highlights several key aspects of this fast developing field, and some of the challenges and opportunities ahead.

physics.optics↗

arXiv2021

Quantitative Jones matrix imaging using vectorial Fourier ptychography

This paper presents a microscopic imaging technique that uses variable-angle illumination to recover the complex polarimetric properties of a specimen at high resolution and over a large field-of-view. The approach extends Fourier ptychography, which is a synthetic aperture-based imaging approach to improve resolution with phaseless measurements, to additionally account for the vectorial nature of light. After images are acquired using a standard microscope outfitted with an LED illumination array and two polarizers, our vectorial Fourier Ptychography (vFP) algorithm solves for the complex 2x2 Jones matrix of the anisotropic specimen of interest at each resolved spatial location. We introduce a new sequential Gauss-Newton-based solver that additionally jointly estimates and removes polarization-dependent imaging system aberrations. We demonstrate effective vFP performance by generating large-area (29 mm$^2$), high-resolution (1.24 $μ$m full-pitch) reconstructions of sample absorption, phase, orientation, diattenuation, and retardance for a variety of calibration samples and biological specimens.

physics.optics↗

arXiv2021

Increasing a microscope's effective field of view via overlapped imaging and machine learning

This work demonstrates a multi-lens microscopic imaging system that overlaps multiple independent fields of view on a single sensor for high-efficiency automated specimen analysis. Automatic detection, classification and counting of various morphological features of interest is now a crucial component of both biomedical research and disease diagnosis. While convolutional neural networks (CNNs) have dramatically improved the accuracy of counting cells and sub-cellular features from acquired digital image data, the overall throughput is still typically hindered by the limited space-bandwidth product (SBP) of conventional microscopes. Here, we show both in simulation and experiment that overlapped imaging and co-designed analysis software can achieve accurate detection of diagnostically-relevant features for several applications, including counting of white blood cells and the malaria parasite, leading to multi-fold increase in detection and processing throughput with minimal reduction in accuracy.

cs.CV↗

arXiv2021

Deep image prior for undersampling high-speed photoacoustic microscopy

Photoacoustic microscopy (PAM) is an emerging imaging method combining light and sound. However, limited by the laser's repetition rate, state-of-the-art high-speed PAM technology often sacrifices spatial sampling density (i.e., undersampling) for increased imaging speed over a large field-of-view. Deep learning (DL) methods have recently been used to improve sparsely sampled PAM images; however, these methods often require time-consuming pre-training and large training dataset with ground truth. Here, we propose the use of deep image prior (DIP) to improve the image quality of undersampled PAM images. Unlike other DL approaches, DIP requires neither pre-training nor fully-sampled ground truth, enabling its flexible and fast implementation on various imaging targets. Our results have demonstrated substantial improvement in PAM images with as few as 1.4$\%$ of the fully sampled pixels on high-speed PAM. Our approach outperforms interpolation, is competitive with pre-trained supervised DL method, and is readily translated to other high-speed, undersampling imaging modalities.

eess.IV↗

arXiv2020

Mesoscopic photogrammetry with an unstabilized phone camera

We present a feature-free photogrammetric technique that enables quantitative 3D mesoscopic (mm-scale height variation) imaging with tens-of-micron accuracy from sequences of images acquired by a smartphone at close range (several cm) under freehand motion without additional hardware. Our end-to-end, pixel-intensity-based approach jointly registers and stitches all the images by estimating a coaligned height map, which acts as a pixel-wise radial deformation field that orthorectifies each camera image to allow homographic registration. The height maps themselves are reparameterized as the output of an untrained encoder-decoder convolutional neural network (CNN) with the raw camera images as the input, which effectively removes many reconstruction artifacts. Our method also jointly estimates both the camera's dynamic 6D pose and its distortion using a nonparametric model, the latter of which is especially important in mesoscopic applications when using cameras not designed for imaging at short working distances, such as smartphone cameras. We also propose strategies for reducing computation time and memory, applicable to other multi-frame registration problems. Finally, we demonstrate our method using sequences of multi-megapixel images captured by an unstabilized smartphone on a variety of samples (e.g., painting brushstrokes, circuit board, seeds).

cs.CV↗

arXiv2020

Multi-element microscope optimization by a learned sensing network with composite physical layers

Standard microscopes offer a variety of settings to help improve the visibility of different specimens to the end microscope user. Increasingly, however, digital microscopes are used to capture images for automated interpretation by computer algorithms (e.g., for feature classification, detection or segmentation), often without any human involvement. In this work, we investigate an approach to jointly optimize multiple microscope settings, together with a classification network, for improved performance with such automated tasks. We explore the interplay between optimization of programmable illumination and pupil transmission, using experimentally imaged blood smears for automated malaria parasite detection, to show that multi-element "learned sensing" outperforms its single-element counterpart. While not necessarily ideal for human interpretation, the network's resulting low-resolution microscope images (20X-comparable) offer a machine learning network sufficient contrast to match the classification performance of corresponding high-resolution imagery (100X-comparable), pointing a path towards accurate automation over large fields-of-view.

eess.IV↗

arXiv2020

Reconstructing undersampled photoacoustic microscopy images using deep learning

One primary technical challenge in photoacoustic microscopy (PAM) is the necessary compromise between spatial resolution and imaging speed. In this study, we propose a novel application of deep learning principles to reconstruct undersampled PAM images and transcend the trade-off between spatial resolution and imaging speed. We compared various convolutional neural network (CNN) architectures, and selected a fully dense U-net (FD U-net) model that produced the best results. To mimic various undersampling conditions in practice, we artificially downsampled fully-sampled PAM images of mouse brain vasculature at different ratios. This allowed us to not only definitively establish the ground truth, but also train and test our deep learning model at various imaging conditions. Our results and numerical analysis have collectively demonstrated the robust performance of our model to reconstruct PAM images with as few as 2% of the original pixels, which may effectively shorten the imaging time without substantially sacrificing the image quality.

eess.IV↗

arXiv2020

Physics-enhanced machine learning for virtual fluorescence microscopy

This paper introduces a new method of data-driven microscope design for virtual fluorescence microscopy. Our results show that by including a model of illumination within the first layers of a deep convolutional neural network, it is possible to learn task-specific LED patterns that substantially improve the ability to infer fluorescence image information from unstained transmission microscopy images. We validated our method on two different experimental setups, with different magnifications and different sample types, to show a consistent improvement in performance as compared to conventional illumination methods. Additionally, to understand the importance of learned illumination on inference task, we varied the dynamic range of the fluorescent image targets (from one to seven bits), and showed that the margin of improvement for learned patterns increased with the information content of the target. This work demonstrates the power of programmable optical elements at enabling better machine learning algorithm performance and at providing physical insight into next generation of machine-controlled imaging systems.

eess.IV↗

arXiv2020

Towards an Intelligent Microscope: adaptively learned illumination for optimal sample classification

Recent machine learning techniques have dramatically changed how we process digital images. However, the way in which we capture images is still largely driven by human intuition and experience. This restriction is in part due to the many available degrees of freedom that alter the image acquisition process (lens focus, exposure, filtering, etc). Here we focus on one such degree of freedom - illumination within a microscope - which can drastically alter information captured by the image sensor. We present a reinforcement learning system that adaptively explores optimal patterns to illuminate specimens for immediate classification. The agent uses a recurrent latent space to encode a large set of variably-illuminated samples and illumination patterns. We train our agent using a reward that balances classification confidence with image acquisition cost. By synthesizing knowledge over multiple snapshots, the agent can classify on the basis of all previous images with higher accuracy than from naively illuminated images, thus demonstrating a smarter way to physically capture task-specific information.

eess.IV↗

arXiv2019

Diffraction tomography with a deep image prior

We present a tomographic imaging technique, termed Deep Prior Diffraction Tomography (DP-DT), to reconstruct the 3D refractive index (RI) of thick biological samples at high resolution from a sequence of low-resolution images collected under angularly varying illumination. DP-DT processes the multi-angle data using a phase retrieval algorithm that is extended by a deep image prior (DIP), which reparameterizes the 3D sample reconstruction with an untrained, deep generative 3D convolutional neural network (CNN). We show that DP-DT effectively addresses the missing cone problem, which otherwise degrades the resolution and quality of standard 3D reconstruction algorithms. As DP-DT does not require pre-captured data or pre-training, it is not biased towards any particular dataset. Hence, it is a general technique that can be applied to a wide variety of 3D samples, including scenarios in which large datasets for supervised training would be infeasible or expensive. We applied DP-DT to obtain 3D RI maps of bead phantoms and complex biological specimens, both in simulation and experiment, and show that DP-DT produces higher-quality results than standard regularization techniques. We further demonstrate the generality of DP-DT, using two different scattering models, the first Born and multi-slice models. Our results point to the potential benefits of DP-DT for other 3D imaging modalities, including X-ray computed tomography, magnetic resonance imaging, and electron microscopy.

eess.IV↗

arXiv2019

Artificial Neural Network with Physical Dynamic Metasurface Layer for Optimal Sensing

We address the fundamental question of how to optimally probe a scene with electromagnetic (EM) radiation to yield a maximum amount of information relevant to a particular task. Machine learning (ML) techniques have emerged as powerful tools to extract task-relevant information from a wide variety of EM measurements, ranging from optics to the microwave domain. However, given the ability to actively illuminate a particular scene with a programmable EM wavefront, it is often not clear what wavefronts optimally encode information for the task at hand (e.g., object detection, classification). Here, we show that by integrating a physical model of scene illumination and detection into a ML pipeline, we can jointly learn optimal sampling and measurement processing strategies for a given task. We consider in simulation the example of classifying objects using microwave radiation produced by dynamic metasurfaces. By integrating an analytical forward model describing the metamaterial elements as coupled dipoles into the ML pipeline, we jointly train analog model weights with digital neural network weights. The learned non-intuitive illumination settings yield a higher classification accuracy using fewer measurements. On the practical level, these results are highly relevant to emerging context-aware systems such as autonomous vehicles, touchless human-interactive devices or within smart health care, where strict time constraints place severe limits on measurement strategies. On the conceptual level, our work serves as a bridge between wavefront shaping and tunable metasurface design on the physical layer and ML techniques on the processing layer.

physics.app-ph↗

arXiv2017

Convolutional neural networks that teach microscopes how to image

Deep learning algorithms offer a powerful means to automatically analyze the content of medical images. However, many biological samples of interest are primarily transparent to visible light and contain features that are difficult to resolve with a standard optical microscope. Here, we use a convolutional neural network (CNN) not only to classify images, but also to optimize the physical layout of the imaging device itself. We increase the classification accuracy of a microscope's recorded images by merging an optical model of image formation into the pipeline of a CNN. The resulting network simultaneously determines an ideal illumination arrangement to highlight important sample features during image acquisition, along with a set of convolutional weights to classify the detected images post-capture. We demonstrate our joint optimization technique with an experimental microscope configuration that automatically identifies malaria-infected cells with 5-10% higher accuracy than standard and alternative microscope lighting designs.

cs.CV↗

arXiv2017

Scattering correlations of time-gated light

Manipulating the propagation of light through scattering media remains a major challenge for many applications, including astronomy, biomedical imaging and colloidal optics. Wavefront shaping is one of the most promising ways to mitigate scattering and focus through inhomogeneous samples. However, wavefront correction remains accurate over only a limited spatial extent within the scattering medium - a correlation range referred to as the optical memory effect. Here, by selecting only the weakly scattered light for wavefront shaping, we show that the addition of temporal degrees of freedom enhances this correlation range. We investigate spatial scattering correlations by digitally time-gating the early arriving light in the spectral domain. We demonstrate that the range of the translational memory effect for the early arriving light is increased almost fourfold, paving the way for a range of scattering media imaging applications.

physics.optics↗

arXiv2017

The generalized optical memory effect

The optical memory effect is a well-known type of wave correlation that is observed in coherent fields that scatter through thin and diffusive materials, like biological tissue. It is a fundamental physical property of scattering media that can be harnessed for deep-tissue microscopy or 'through-the-wall' imaging applications. Here we show that the optical memory effect is a special case of a far more general class of wave correlation. Our new theoretical framework explains how waves remain correlated over both space and angle when they are jointly shifted and tilted inside scattering media of arbitrary geometry. We experimentally demonstrate the existence of such coupled correlations and describe how they can be used to optimize the scanning range in adaptive optics microscopes.

physics.optics↗

arXiv2016

Subsampled Phase Retrieval for On-chip Lensless Holographic Video

On-chip holographic video is a convenient way to monitor biological samples simultaneously at high spatial resolution and over a wide field-of-view. However, due to the limited readout rate of digital detector arrays, one often faces a tradeoff between the per-frame pixel count and frame rate of the captured video. In this report, we propose a subsampled phase retrieval (SPR) algorithm to overcome the spatial-temporal trade-off in holographic video. Compared to traditional phase retrieval approaches, our SPR algorithm uses over an order of magnitude less pixel measurements while maintaining suitable reconstruction quality. We use an on-chip holographic video setup with pixel sub-sampling to experimentally demonstrate a factor of 5.5 increase in sensor frame rate while monitoring the in vivo movement of Peranema microorganisms.

physics.optics↗

arXiv2015

Diffraction tomography with Fourier ptychography

This article presents a method to perform diffraction tomography in a standard microscope that includes an LED array for illumination. After acquiring a sequence of intensity-only images of a thick sample, a ptychography-based reconstruction algorithm solves for its unknown complex index of refraction across three dimensions. The experimental microscope demonstrates a spatial resolution of 0.39 $μ$m and an axial resolution of 3.7 $μ$m at the Nyquist-Shannon sampling limit (0.54 $μ$m and 5.0 $μ$m at the Sparrow limit, respectively), across a total imaging volume of 2.2 mm $\times$ 2.2 mm $\times$ 110 $μ$m. Unlike competing methods, the 3D tomograms presented in this article are continuous, quantitative, and formed without the need for interferometry or any moving parts. Wide field-of-view reconstructions of thick biological specimens demonstrate potential applications in pathology and developmental biology.

physics.optics↗

arXiv2015

Toward Long Distance, Sub-diffraction Imaging Using Coherent Camera Arrays

In this work, we propose using camera arrays coupled with coherent illumination as an effective method of improving spatial resolution in long distance images by a factor of ten and beyond. Recent advances in ptychography have demonstrated that one can image beyond the diffraction limit of the objective lens in a microscope. We demonstrate a similar imaging system to image beyond the diffraction limit in long range imaging. We emulate a camera array with a single camera attached to an X-Y translation stage. We show that an appropriate phase retrieval based reconstruction algorithm can be used to effectively recover the lost high resolution details from the multiple low resolution acquired images. We analyze the effects of noise, required degree of image overlap, and the effect of increasing synthetic aperture size on the reconstructed image quality. We show that coherent camera arrays have the potential to greatly improve imaging performance. Our simulations show resolution gains of 10x and more are achievable. Furthermore, experimental results from our proof-of-concept systems show resolution gains of 4x-7x for real scenes. Finally, we introduce and analyze in simulation a new strategy to capture macroscopic Fourier Ptychography images in a single snapshot, albeit using a camera array.

cs.CV↗