Searcharxiv⌕ Search

arXiv subjects

Yoav Y. Schechner

Publications and source records attributed to Yoav Y. Schechner.

15 recordsLinked to original sources
arXiv2026

Telemetry is a Sensor: Opportunistic Wavefront Estimation for the James Webb Space Telescope

Space telescopes maintain optical alignment through periodic, resource-intensive wavefront calibration, leaving the optical state unobserved between corrections. We propose treating onboard engineering telemetry as an opportunistic wavefront sensor. Specifically, the high-cadence thermal and pointing signals already recorded by the James Webb Space Telescope may suffice to recover spatially resolved optical path difference at nanometer precision, without dedicated measurements. Our framework uses a two-stage gradient boosting regressor. It predicts optical path difference residuals for each mirror segment in a low-dimensional principal component basis. The model is trained and evaluated using data captured on-orbit during six months. Based on this limited data, optical path difference inference has high statistical significance in 13 out of 18 mirror segments. In 6 mirror segments, the model achieves explained variance above 50\%. These results demonstrate feasibility for telemetry-driven wavefront estimation, as a low-overhead sensing modality. The study suggests a potential to support continuous monitoring and calibration scheduling for large segmented observatories.

astro-ph.IM↗
arXiv2026

Fundamental Recovery Bounds for SPAD Signals under Stationary Flux

Single-photon avalanche diodes (SPADs) record light as a discrete stream of individual detections. The signal is stochastic. Its statistical structure depends on the sensor's operation mode: binary detection in fixed bins, timestamped detection in fixed bins, or free-running timestamped detection. We derive the likelihood score function for each of these three passive modes. From this single object, stem both fundamental limits of recovery (Cramer-Rao bounds) and practical recovery algorithms based on diffusion posterior sampling. The paper further generalizes fundamental limits to Bayesian Cramer-Rao lower bounds. This generalization makes use of a learned approximation of the score function of signal priors. In prior art, analyses and diffusion-based reconstruction for SPAD data have treated individual modes in isolation. Our unified treatment shows a qualitative high-flux gap between modes: binary counts saturate exponentially, while timestamped modes degrade only linearly. We further extend diffusion posterior sampling, previously restricted to binary SPAD data, to a full timestamped case using the suitable score function. We demonstrate experimentally that matching the score to the operation mode is beneficial for high-fidelity reconstruction. By tying the recovery bounds and diffusion to the score function, this work aims to establish a common foundation for both asking what is recoverable in single-photon sensing, and building methods that approach the bound.

cs.CV↗
arXiv2025

NnD: Diffusion-based Generation of Physically-Nonnegative Objects

Most natural objects have inherent complexity and variability. While some simple objects can be modeled from first principles, many real-world phenomena, such as cloud formation, require computationally expensive simulations that limit scalability. This work focuses on a class of physically meaningful, nonnegative objects that are computationally tractable but costly to simulate. To dramatically reduce computational costs, we propose nonnegative diffusion (NnD). This is a learned generative model using score based diffusion. It adapts annealed Langevin dynamics to enforce, by design, non-negativity throughout iterative scene generation and analysis (inference). NnD trains on high-quality physically simulated objects. Once trained, it can be used for generation and inference. We demonstrate generation of 3D volumetric clouds, comprising inherently nonnegative microphysical fields. Our generated clouds are consistent with cloud physics trends. They are effectively not distinguished as non-physical by expert perception.

cs.LG↗
arXiv2024

Learned 3D volumetric recovery of clouds and its uncertainty for climate analysis

Significant uncertainty in climate prediction and cloud physics is tied to observational gaps relating to shallow scattered clouds. Addressing these challenges requires remote sensing of their three-dimensional (3D) heterogeneous volumetric scattering content. This calls for passive scattering computed tomography (CT). We design a learning-based model (ProbCT) to achieve CT of such clouds, based on noisy multi-view spaceborne images. ProbCT infers - for the first time - the posterior probability distribution of the heterogeneous extinction coefficient, per 3D location. This yields arbitrary valuable statistics, e.g., the 3D field of the most probable extinction and its uncertainty. ProbCT uses a neural-field representation, making essentially real-time inference. ProbCT undergoes supervised training by a new labeled multi-class database of physics-based volumetric fields of clouds and their corresponding images. To improve out-of-distribution inference, we incorporate self-supervised learning through differential rendering. We demonstrate the approach in simulations and on real-world data, and indicate the relevance of 3D recovery and uncertainty to precipitation and renewable energy.

cs.CV↗
arXiv2023

Complex-valued Retrievals From Noisy Images Using Diffusion Models

In diverse microscopy modalities, sensors measure only real-valued intensities. Additionally, the sensor readouts are affected by Poissonian-distributed photon noise. Traditional restoration algorithms typically aim to minimize the mean squared error (MSE) between the original and recovered images. This often leads to blurry outcomes with poor perceptual quality. Recently, deep diffusion models (DDMs) have proven to be highly capable of sampling images from the a-posteriori probability of the sought variables, resulting in visually pleasing high-quality images. These models have mostly been suggested for real-valued images suffering from Gaussian noise. In this study, we generalize annealed Langevin Dynamics, a type of DDM, to tackle the fundamental challenges in optical imaging of complex-valued objects (and real images) affected by Poisson noise. We apply our algorithm to various optical scenarios, such as Fourier Ptychography, Phase Retrieval, and Poisson denoising. Our algorithm is evaluated on simulations and biological empirical data.

cs.CV↗
arXiv2021

Accelerating Inverse Rendering By Using a GPU and Reuse of Light Paths

Inverse rendering seeks to estimate scene characteristics from a set of data images. The dominant approach is based on differential rendering using Monte-Carlo. Algorithms as such usually rely on a forward model and use an iterative gradient method that requires sampling millions of light paths per iteration. This paper presents an efficient framework that speeds up existing inverse rendering algorithms. This is achieved by tailoring the iterative process of inverse rendering specifically to a GPU architecture. For this cause, we introduce two interleaved steps - Path Sorting and Path Recycling. Path Sorting allows the GPU to deal with light paths of the same size. Path Recycling allows the algorithm to use light paths from previous iterations to better utilize the information they encode. Together, these steps significantly speed up gradient optimization. In this paper, we give the theoretical background for Path Recycling. We demonstrate its efficiency for volumetric scattering tomography and reflectometry (surface reflections).

cs.CV↗
arXiv2021

What You Can Learn by Staring at a Blank Wall

We present a passive non-line-of-sight method that infers the number of people or activity of a person from the observation of a blank wall in an unknown room. Our technique analyzes complex imperceptible changes in indirect illumination in a video of the wall to reveal a signal that is correlated with motion in the hidden part of a scene. We use this signal to classify between zero, one, or two moving people, or the activity of a person in the hidden scene. We train two convolutional neural networks using data collected from 20 different scenes, and achieve an accuracy of $\approx94\%$ for both tasks in unseen test environments and real-time online settings. Unlike other passive non-line-of-sight methods, the technique does not rely on known occluders or controllable light sources, and generalizes to unknown rooms with no re-calibration. We analyze the generalization and robustness of our method with both real and synthetic data, and study the effect of the scene parameters on the signal quality.

cs.CV↗
arXiv2021

Image compression optimized for 3D reconstruction by utilizing deep neural networks

Computer vision tasks are often expected to be executed on compressed images. Classical image compression standards like JPEG 2000 are widely used. However, they do not account for the specific end-task at hand. Motivated by works on recurrent neural network (RNN)-based image compression and three-dimensional (3D) reconstruction, we propose unified network architectures to solve both tasks jointly. These joint models provide image compression tailored for the specific task of 3D reconstruction. Images compressed by our proposed models, yield 3D reconstruction performance superior as compared to using JPEG 2000 compression. Our models significantly extend the range of compression rates for which 3D reconstruction is possible. We also show that this can be done highly efficiently at almost no additional cost to obtain compression on top of the computation already required for performing the 3D reconstruction task.

eess.IV↗
arXiv2021

Advances in 3D scattering tomography of cloud micro-physics

We introduce new adjustments and advances in space-borne 3D volumetric scattering-tomography of cloud micro-physics. The micro-physical properties retrieved are the liquid water content and effective radius within a cloud. New adjustments include an advanced perspective polarization imager model, and the assumption of 3D variation of the effective radius. Under these assumptions, we advanced the retrieval to yield results that (compared to the simulated ground-truth) have smaller errors than the prior art. Elements of our advancement include initialization by a parametric horizontally-uniform micro-physical model. The parameters of this initialization are determined by a grid search of the cost function. Furthermore, we added viewpoints corresponding to single-scattering angles, where polarization yields enhanced sensitivity to the droplet micro-physics (i.e., the cloudbow region). In addition, we introduce an optional adjustment, in which optimization of the liquid water content and effective radius are separated to alternating periods. The suggested initialization model and additional advances have been evaluated by retrieval of a set of large-eddy simulation clouds.

eess.IV↗
arXiv2020

3D Scattering Tomography by Deep Learning with Architecture Tailored to Cloud Fields

We present 3DeepCT, a deep neural network for computed tomography, which performs 3D reconstruction of scattering volumes from multi-view images. Our architecture is dictated by the stationary nature of atmospheric cloud fields. The task of volumetric scattering tomography aims at recovering a volume from its 2D projections. This problem has been studied extensively, leading, to diverse inverse methods based on signal processing and physics models. However, such techniques are typically iterative, exhibiting high computational load and long convergence time. We show that 3DeepCT outperforms physics-based inverse scattering methods in term of accuracy as well as offering a significant orders of magnitude improvement in computational time. To further improve the recovery accuracy, we introduce a hybrid model that combines 3DeepCT and physics-based method. The resultant hybrid technique enjoys fast inference time and improved recovery performance.

eess.IV↗
arXiv2020

Spatiotemporal tomography based on scattered multiangular signals and its application for resolving evolving clouds using moving platforms

We derive computed tomography (CT) of a time-varying volumetric translucent object, using a small number of moving cameras. We particularly focus on passive scattering tomography, which is a non-linear problem. We demonstrate the approach on dynamic clouds, as clouds have a major effect on Earth's climate. State of the art scattering CT assumes a static object. Existing 4D CT methods rely on a linear image formation model and often on significant priors. In this paper, the angular and temporal sampling rates needed for a proper recovery are discussed. If these rates are used, the paper leads to a representation of the time-varying object, which simplifies 4D CT tomography. The task is achieved using gradient-based optimization. We demonstrate this in physics-based simulations and in an experiment that had yielded real-world data.

cs.CV↗
arXiv2020

Multi-view polarimetric scattering cloud tomography and retrieval of droplet size

Tomography aims to recover a three-dimensional (3D) density map of a medium or an object. In medical imaging, it is extensively used for diagnostics via X-ray computed tomography (CT). Optical diffusion tomography is an alternative to X-ray CT that uses multiply scattered light to deliver coarse density maps for soft tissues. We define and derive tomography of cloud droplet distributions via passive remote sensing. We use multi-view polarimetric images to fit a 3D polarized radiative transfer (RT) forward model. Our motivation is 3D volumetric probing of vertically-developed convectively-driven clouds that are ill-served by current methods in operational passive remote sensing. These techniques are based on strictly 1D RT modeling and applied to a single cloudy pixel, where cloud geometry is assumed to be that of a plane-parallel slab. Incident unpolarized sunlight, once scattered by cloud-droplets, changes its polarization state according to droplet size. Therefore, polarimetric measurements in the rainbow and glory angular regions can be used to infer the droplet size distribution. This work defines and derives a framework for a full 3D tomography of cloud droplets for both their mass concentration in space and their distribution across a range of sizes. This 3D retrieval of key microphysical properties is made tractable by our novel approach that involves a restructuring and differentiation of an open-source polarized 3D RT code to accommodate a special two-step optimization technique. Physically-realistic synthetic clouds are used to demonstrate the methodology with rigorous uncertainty quantification.

physics.comp-ph↗
arXiv2015

In-situ multi-scattering tomography

To recover the three dimensional (3D) volumetric distribution of matter in an object, images of the object are captured from multiple directions and locations. Using these images tomographic computations extract the distribution. In highly scattering media and constrained, natural irradiance, tomography must explicitly account for off-axis scattering. Furthermore, the tomographic model and recovery must function when imaging is done in-situ, as occurs in medical imaging and ground-based atmospheric sensing. We formulate tomography that handles arbitrary orders of scattering, using a monte-carlo model. Moreover, the model is highly parallelizable in our formulation. This enables large scale rendering and recovery of volumetric scenes having a large number of variables. We solve stability and conditioning problems that stem from radiative transfer (RT) modeling in-situ.

cs.CV↗
arXiv2015

The Next Best Underwater View

To image in high resolution large and occlusion-prone scenes, a camera must move above and around. Degradation of visibility due to geometric occlusions and distances is exacerbated by scattering, when the scene is in a participating medium. Moreover, underwater and in other media, artificial lighting is needed. Overall, data quality depends on the observed surface, medium and the time-varying poses of the camera and light source. This work proposes to optimize camera/light poses as they move, so that the surface is scanned efficiently and the descattered recovery has the highest quality. The work generalizes the next best view concept of robot vision to scattering media and cooperative movable lighting. It also extends descattering to platforms that move optimally. The optimization criterion is information gain, taken from information theory. We exploit the existence of a prior rough 3D model, since underwater such a model is routinely obtained using sonar. We demonstrate this principle in a scaled-down setup.

cs.CV↗