arXiv subjects

Wenzel Jakob

Publications and source records attributed to Wenzel Jakob.

15 recordsLinked to original sources

arXiv2026

Neural Particle Automata: Learning Self-Organizing Particle Dynamics

We introduce Neural Particle Automata (NPA), a Lagrangian generalization of Neural Cellular Automata (NCA) from static lattices to dynamic particle systems. Unlike classical Eulerian NCA where cells are pinned to pixels or voxels, NPA model each cell as a particle with a continuous position and internal state, both updated by a shared, learnable neural rule. This particle-based formulation yields clear individuation of cells, allows heterogeneous dynamics, and concentrates computation only on regions where activity is present. At the same time, particle systems pose challenges: neighborhoods are dynamic, and a naive implementation of local interactions scale quadratically with the number of particles. We address these challenges by replacing grid-based neighborhood perception with differentiable Smoothed Particle Hydrodynamics (SPH) operators backed by memory-efficient, CUDA-accelerated kernels, enabling scalable end-to-end training. Across tasks including morphogenesis, point-cloud classification, and particle-based texture synthesis, we show that NPA retain key NCA behaviors such as robustness and self-regeneration, while enabling new behaviors specific to particle systems. Together, these results position NPA as a compact neural model for learning self-organizing particle dynamics.

cs.NE↗

arXiv2025

MiGumi: Making Tightly Coupled Integral Joints Millable

Traditional integral wood joints, despite their strength, durability, and elegance, remain rare in modern workflows due to the cost and difficulty of manual fabrication. CNC milling offers a scalable alternative, but directly milling traditional joints often fails to produce functional results because milling induces geometric deviations, such as rounded inner corners, that alter the target geometries of the parts. Since joints rely on tightly fitting surfaces, such deviations introduce gaps or overlaps that undermine fit or block assembly. We propose to overcome this problem by (1) designing a language that represent millable geometry, and (2) co-optimizing part geometries to restore coupling. We introduce Millable Extrusion Geometry (MXG), a language for representing geometry as the outcome of milling operations performed with flat-end drill bits. MXG represents each operation as a subtractive extrusion volume defined by a tool direction and drill radius. This parameterization enables the modeling of artifact-free geometry under an idealized zero-radius drill bit, matching traditional joint designs. Increasing the radius then reveals milling-induced deviations, which compromise the integrity of the joint. To restore coupling, we formalize tight coupling in terms of both surface proximity and proximity constraints on the mill-bit paths associated with mating surfaces. We then derive two tractable, differentiable losses that enable efficient optimization of joint geometry. We evaluate our method on 30 traditional joint designs, demonstrating that it produces CNC-compatible, tightly fitting joints that approximates the original geometry. By reinterpreting traditional joints for CNC workflows, we continue the evolution of this heritage craft and help ensure its relevance in future making practices.

cs.GR↗

arXiv2025

MCMC: Bridging Rendering, Optimization and Generative AI

Generative artificial intelligence (AI) has made unprecedented advances in vision language models over the past two years. During the generative process, new samples (images) are generated from an unknown high-dimensional distribution. Markov Chain Monte Carlo (MCMC) methods are particularly effective in drawing samples from such complex, high-dimensional distributions. This makes MCMC methods an integral component for models like EBMs, ensuring accurate sample generation. Gradient-based optimization is at the core of modern generative models. The update step during the optimization forms a Markov chain where the new update depends only on the current state. This allows exploration of the parameter space in a memoryless manner, thus combining the benefits of gradient-based optimization and MCMC sampling. MCMC methods have shown an equally important role in physically based rendering where complex light paths are otherwise quite challenging to sample from simple importance sampling techniques. A lot of research is dedicated towards bringing physical realism to samples (images) generated from diffusion-based generative models in a data-driven manner, however, a unified framework connecting these techniques is still missing. In this course, we take the first steps toward understanding each of these components and exploring how MCMC could potentially serve as a bridge, linking these closely related areas of research. Our course aims to provide necessary theoretical and practical tools to guide students, researchers and practitioners towards the common goal of generative physically based rendering. All Jupyter notebooks with demonstrations associated to this tutorial can be found on the project webpage: https://sinbag.github.io/mcmc/

cs.GR↗

arXiv2025

Physically-Based Inverse Rendering Framework for PET Image Reconstruction

Differentiable rendering has been widely adopted in computer graphics as a powerful approach to inverse problems, enabling efficient gradient-based optimization by differentiating the image formation process with respect to millions of scene parameters. Inspired by this paradigm, we propose a physically-based inverse rendering (IR) framework, the first ever platform for PET image reconstruction using Dr.Jit, for PET image reconstruction. Our method integrates Monte Carlo sampling with an analytical projector in the forward rendering process to accurately model photon transport and physical process in the PET system. The emission image is iteratively optimized using voxel-wise gradients obtained via automatic differentiation, eliminating the need for manually derived update equations. The proposed framework was evaluated using both phantom studies and clinical brain PET data acquired from a Siemens Biograph mCT scanner. Implementing the Maximum Likelihood Expectation Maximization (MLEM) algorithm across both the CASToR toolkit and our IR framework, the IR reconstruction achieved a higher signal-to-noise ratio (SNR) and improved image quality compared to CASToR reconstructions. In clinical evaluation compared with the Siemens Biograph mCT platform, the IR reconstruction yielded higher hippocampal standardized uptake value ratios (SUVR) and gray-to-white matter ratios (GWR), indicating enhanced tissue contrast and the potential for more accurate tau localization and Braak staging in Alzheimer's disease assessment. The proposed IR framework offers a physically interpretable and extensible platform for high-fidelity PET image reconstruction, demonstrating strong performance in both phantom and real-world scenarios.

physics.med-ph↗

arXiv2025

Overprinting with Tomographic Volumetric Additive Manufacturing

Tomographic Volumetric Additive Manufacturing (TVAM) is a light-based 3D printing technique capable of producing centimeter-scale objects within seconds. A key challenge lies in the calculation of tomographic projection patterns under non-standard conditions, such as the presence of occlusions and materials with diverse optical properties, including varying refractive indices or scattering surfaces. This work demonstrates a broad range of overprinting scenarios, where new structures are directly printed onto or around pre-existing components made from different materials. Our simulations and experimental verifications perform overprinting of absorbing, refracting, reflecting and scattering elements in both round and square vials. All scenarios are optimized with our differentiable, physically based ray-optics approach using the open-source Dr.TVAM framework, delivering high-quality projections for both laser- and LED-based illuminations within minutes and lower-quality projections within seconds, exceeding existing open-source solutions in speed, flexibility, and quality.

physics.optics↗

arXiv2025

Neural Cellular Automata: From Cells to Pixels

Neural Cellular Automata (NCAs) are bio-inspired dynamical systems in which identical cells iteratively apply a learned local update rule to self-organize into complex patterns, exhibiting regeneration, robustness, and spontaneous dynamics. Despite their success in texture synthesis and morphogenesis, NCAs remain largely confined to low-resolution outputs. This limitation stems from (1) training time and memory requirements that grow quadratically with grid size, (2) the strictly local propagation of information that impedes long-range cell communication, and (3) the heavy compute demands of real-time inference at high resolution. In this work, we overcome this limitation by pairing an NCA that evolves on a coarse grid with a lightweight implicit decoder that maps cell states and local coordinates to appearance attributes, enabling the same model to render outputs at arbitrary resolution. Moreover, because both the decoder and NCA updates are local, inference remains highly parallelizable. To supervise high-resolution outputs efficiently, we introduce task-specific losses for morphogenesis (growth from a seed) and texture synthesis with minimal additional memory and computation overhead. Our experiments across 2D/3D grids and mesh domains demonstrate that our hybrid models produce high-resolution outputs in real-time, and preserve the characteristic self-organizing behavior of NCAs.

cs.CV↗

arXiv2025

Radiance Surfaces: Optimizing Surface Representations with a 5D Radiance Field Loss

We present a fast and simple technique to convert images into a radiance surface-based scene representation. Building on existing radiance volume reconstruction algorithms, we introduce a subtle yet impactful modification of the loss function requiring changes to only a few lines of code: instead of integrating the radiance field along rays and supervising the resulting images, we project the training images into the scene to directly supervise the spatio-directional radiance field. The primary outcome of this change is the complete removal of alpha blending and ray marching from the image formation model, instead moving these steps into the loss computation. In addition to promoting convergence to surfaces, this formulation assigns explicit semantic meaning to 2D subsets of the radiance field, turning them into well-defined radiance surfaces. We finally extract a level set from this representation, which results in a high-quality radiance surface model. Our method retains much of the speed and quality of the baseline algorithm. For instance, a suitably modified variant of Instant NGP maintains comparable computational efficiency, while achieving an average PSNR that is only 0.1 dB lower. Most importantly, our method generates explicit surfaces in place of an exponential volume, doing so with a level of simplicity not seen in prior work.

cs.GR↗

arXiv2024

Many-Worlds Inverse Rendering

Discontinuous visibility changes remain a major bottleneck when optimizing surfaces within a physically-based inverse renderer. Many previous works have proposed sophisticated algorithms and data structures to sample visibility silhouettes more efficiently. Our work presents another solution: instead of differentiating a tentative surface locally, we differentiate a volumetric perturbation of a surface. We refer this as a many-worlds representation because it models a non-interacting superposition of conflicting explanations (worlds) of the input dataset. Each world is optically isolated from others, leading to a new transport law that distinguishes our method from prior work based on exponential random media. The resulting Monte Carlo algorithm is simpler and more efficient than prior methods. We demonstrate that our method promotes rapid convergence, both in terms of the total iteration count and the cost per iteration.

cs.CV↗

arXiv2024

A Simple Approach to Differentiable Rendering of SDFs

We present a simple algorithm for differentiable rendering of surfaces represented by Signed Distance Fields (SDF), which makes it easy to integrate rendering into gradient-based optimization pipelines. To tackle visibility-related derivatives that make rendering non-differentiable, existing physically based differentiable rendering methods often rely on elaborate guiding data structures or reparameterization with a global impact on variance. In this article, we investigate an alternative that embraces nonzero bias in exchange for low variance and architectural simplicity. Our method expands the lower-dimensional boundary integral into a thin band that is easy to sample when the underlying surface is represented by an SDF. We demonstrate the performance and robustness of our formulation in end-to-end inverse rendering tasks, where it obtains results that are competitive with or superior to existing work.

cs.GR↗

arXiv2022

Solving Inverse PDE Problems using Grid-Free Monte Carlo Estimators

Modeling physical phenomena like heat transport and diffusion is crucially dependent on the numerical solution of partial differential equations (PDEs). A PDE solver finds the solution given coefficients and a boundary condition, whereas an inverse PDE solver goes the opposite way and reconstructs these inputs from an existing solution. In this article, we investigate techniques for solving inverse PDE problems using a gradient-based methodology. Conventional PDE solvers based on the finite element method require a domain meshing step that can be fragile and costly. Grid-free Monte Carlo methods instead stochastically sample paths using variations of the walk on spheres algorithm to construct an unbiased estimator of the solution. The uncanny similarity of these methods to physically-based rendering algorithms has been observed by several recent works. In the area of rendering, recent progress has led to the development of efficient unbiased derivative estimators. They solve an adjoint form of the problem and exploit arithmetic invertibility to compute gradients using a constant amount of memory and linear time complexity. Could these two lines of work be combined to compute cheap parametric derivatives of a grid-free PDE solver? We investigate this question and present preliminary results.

cs.GR↗

arXiv2022

Dr.Jit: A Just-In-Time Compiler for Differentiable Rendering

Dr$.$Jit is a new just-in-time compiler for physically based rendering and its derivative. Dr$.$Jit expedites research on these topics in two ways: first, it traces high-level simulation code (e.g., written in Python) and aggressively simplifies and specializes the resulting program representation, producing data-parallel kernels with state-of-the-art performance on CPUs and GPUs. Second, it simplifies the development of differentiable rendering algorithms. Efficient methods in this area turn the derivative of a simulation into a simulation of the derivative. Dr$.$Jit provides fine-grained control over the process of automatic differentiation to help with this transformation. Specialization is particularly helpful in the context of differentiation, since large parts of the simulation ultimately do not influence the computed gradients. Dr$.$Jit tracks data dependencies globally to find and remove redundant computation.

cs.GR↗

arXiv2021

Wide-Depth-Range 6D Object Pose Estimation in Space

6D pose estimation in space poses unique challenges that are not commonly encountered in the terrestrial setting. One of the most striking differences is the lack of atmospheric scattering, allowing objects to be visible from a great distance while complicating illumination conditions. Currently available benchmark datasets do not place a sufficient emphasis on this aspect and mostly depict the target in close proximity. Prior work tackling pose estimation under large scale variations relies on a two-stage approach to first estimate scale, followed by pose estimation on a resized image patch. We instead propose a single-stage hierarchical end-to-end trainable network that is more robust to scale variations. We demonstrate that it outperforms existing approaches not only on images synthesized to resemble images taken in space but also on standard benchmarks.

cs.CV↗

arXiv2018

$P_N$-Method for Multiple Scattering in Participating Media

Rendering highly scattering participating media using brute force path tracing is a challenge. The diffusion approximation reduces the problem to solving a simple linear partial differential equation. Flux-limited diffusion introduces non-linearities to improve the accuracy of the solution, especially in low optical depth media, but introduces several ad-hoc assumptions. Both methods are based on a spherical harmonics expansion of the radiance field that is truncated after the first order. In this paper, we investigate the open question of whether going to higher spherical harmonic orders provides a viable improvement to these two approaches. Increasing the order introduces a set of complex coupled partial differential equations (the $P_N$-equations), whose growing number make them difficult to work with at higher orders. We thus use a computer algebra framework for representing and manipulating the underlying mathematical equations, and use it to derive the real-valued $P_N$-equations for arbitrary orders. We further present a staggered-grid $P_N$-solver and generate its stencil code directly from the expression tree of the $P_N$-equations. Finally, we discuss how our method compares to prior work for various standard problems.

cs.GR↗

arXiv2017

Scratch iridescence: Wave-optical rendering of diffractive surface structure

The surface of metal, glass and plastic objects is often characterized by microscopic scratches caused by manufacturing and/or wear. A closer look onto such scratches reveals iridescent colors with a complex dependency on viewing and lighting conditions. The physics behind this phenomenon is well understood; it is caused by diffraction of the incident light by surface features on the order of the optical wavelength. Existing analytic models are able to reproduce spatially unresolved microstructure such as the iridescent appearance of compact disks and similar materials. Spatially resolved scratches, on the other hand, have proven elusive due to the highly complex wave-optical light transport simulations needed to account for their appearance. In this paper, we propose a wave-optical shading model based on non-paraxial scalar diffraction theory to render this class of effects. Our model expresses surface roughness as a collection of line segments. To shade a point on the surface, the individual diffraction patterns for contributing scratch segments are computed analytically and superimposed coherently. This provides natural transitions from localized glint-like iridescence to smooth BRDFs representing the superposition of many reflections at large viewing distances. We demonstrate that our model is capable of recreating the overall appearance as well as characteristic detail effects observed on real-world examples.

cs.GR↗

arXiv2017

Reversible Jump Metropolis Light Transport using Inverse Mappings

We study Markov Chain Monte Carlo (MCMC) methods operating in primary sample space and their interactions with multiple sampling techniques. We observe that incorporating the sampling technique into the state of the Markov Chain, as done in Multiplexed Metropolis Light Transport (MMLT), impedes the ability of the chain to properly explore the path space, as transitions between sampling techniques lead to disruptive alterations of path samples. To address this issue, we reformulate Multiplexed MLT in the Reversible Jump MCMC framework (RJMCMC) and introduce inverse sampling techniques that turn light paths into the random numbers that would produce them. This allows us to formulate a novel perturbation that can locally transition between sampling techniques without changing the geometry of the path, and we derive the correct acceptance probability using RJMCMC. We investigate how to generalize this concept to non-invertible sampling techniques commonly found in practice, and introduce probabilistic inverses that extend our perturbation to cover most sampling methods found in light transport simulations. Our theory reconciles the inverses with RJMCMC yielding an unbiased algorithm, which we call Reversible Jump MLT (RJMLT). We verify the correctness of our implementation in canonical and practical scenarios and demonstrate improved temporal coherence, decrease in structured artifacts, and faster convergence on a wide variety of scenes.

cs.GR↗