arXiv subjects

Joachim Weickert

Publications and source records attributed to Joachim Weickert.

At least 19 recordsLinked to original sources

arXiv2026

The Neural Echo: A Signal Processing Perspective for Understanding Neural Networks

We introduce the neural echo as a tool for understanding the behavior of neural networks. It generalizes the model-based concepts of impulse responses, diffusion echoes, and filter echoes to learning-based methods. It provides local, space-adaptive impulse responses and filter kernels for a neural network, its so-called echoes. These echoes depend on the input image and can be visualized to understand the learned dynamics of the network via an affine mapping. Neural echoes build a bridge from classical signal processing to modern explainable AI. They are very general and can be applied to both image-to-image and classification networks, with convolutional or fully connected structure, of feedforward or recurrent type, including modern transformer networks. Network differentiability is not required. In the differentiable case, neural echoes comprise concepts based on the network Jacobian, such as saliency maps and the analysis of adversarial perturbations, as special instances. As a simple blueprint to explain our framework, we derive neural echoes for the denoising convolutional neural network (DnCNN). Our experiments suggest that this network weights pixels based on their spatial and gray value distances. This not only clarifies its behavior, but also shows that it can reproduce key concepts of classical model-based denoisers such as bilateral filtering.

eess.SP↗

arXiv2026

Optimising Inpainting Data with Delaunay Averages

Inpainting-based image compression usually stores an optimised subset of all pixel locations and their colour values. In the decoding phase, the missing data are approximated via inpainting. Since the reconstruction quality depends critically on the selection of the stored data, we introduce a novel feature type: We store the vertex locations of a Delaunay triangulation together with the average colour values inside all triangles. We show that combining this feature type with homogeneous diffusion inpainting creates an elegant mathematical formulation with a positive definite linear system of equations. Even a simple solver such as the conjugate gradient method allows the handling of large images. To make our Delaunay averages maximally adaptive to the image, we develop an efficient data optimisation strategy specifically tailored to them. It incorporates ideas successfully used in the stippling literature. Experiments show that our approach outperforms the popular inpainting with optimised colour values by a large margin. Last but not least, we discover a favourable scaling behaviour: Doubling the image resolution allows us to halve the percentage of stored data while maintaining the quality level. This is attractive for compressing modern high-resolution images, where even data densities below 1 % yield appealing reconstructions.

eess.IV↗

arXiv2026

Optimizing Multiple Feature Types for Image Inpainting in the Linear and Nonlinear Setting

Inpainting-based compression stores a carefully optimized subset of the full image data and reconstructs the missing data by inpainting. The quality of these lossy codecs depends decisively on the stored data. So far, these data consist almost exclusively of pixel locations along with their grayscale or color values. In the present paper, we present a general theory and a practical framework that allows to incorporate arbitrary features which can be described by linear or nonlinear equations. This includes e.g. derivatives of arbitrary order or local integrals. Our features can be combined with linear or nonlinear inpainting operators. Moreover, we present an algorithm that automatically optimizes the location and the type of the selected feature. The approach of allowing different types of optimized features turns inpainting-based compression into a more general, versatile and powerful paradigm. Our experiments report a consistent quality gain when increasing the number of feature types from 1 to 5. With the same amount of stored data, the average peak signal-to-noise improvement is 2.76 dB for harmonic (homogeneous diffusion) inpainting, and 1.82 dB for edge-enhancing diffusion inpainting.

eess.IV↗

arXiv2026

The Filter Echo: A General Tool for Filter Visualisation

To select suitable filters for a task or to improve existing filters, a deep understanding of their inner workings is vital. Diffusion echoes, which are space-adaptive impulse responses, are useful to visualise the effect of nonlinear diffusion filters. However, they have received little attention in the literature. There may be two reasons for this: Firstly, the concept was introduced specifically for diffusion filters, which might appear too limited. Secondly, diffusion echoes have large storage requirements, which restricts their practicality. This work addresses both problems. We introduce the filter echo as a generalisation of the diffusion echo and use it for applications beyond adaptive smoothing, such as image inpainting, osmosis, and variational optic flow computation. We provide a framework to visualise and inspect echoes from various filters with different applications. Furthermore, we propose a compression approach for filter echoes, which reduces storage requirements by a factor of 20 to 100.

eess.IV↗

arXiv2024

Connecting Image Inpainting with Denoising in the Homogeneous Diffusion Setting

While local methods for image denoising and inpainting may use similar concepts, their connections have hardly been investigated so far. The goal of this work is to establish links between the two by focusing on the most foundational scenario on both sides: the homogeneous diffusion setting. To this end, we study a denoising by inpainting (DbI) framework: It averages multiple inpainting results from different noisy subsets. We derive equivalence results between DbI on shifted regular grids and homogeneous diffusion filtering in 1D via an explicit relation between the density and the diffusion time. We also provide an empirical extension to the 2-D case. We present experiments that confirm our theory and suggest that it can also be generalized to diffusions with non-homogeneous data or non-homogeneous diffusivities. More generally, our work demonstrates that the hardly explored idea of data adaptivity deserves more attention: It can be as powerful as some popular models with operator adaptivity.

eess.IV↗

arXiv2024

Efficient Parallel Data Optimization for Homogeneous Diffusion Inpainting of 4K Images

Homogeneous diffusion inpainting can reconstruct missing image areas with high quality from a sparse subset of known pixels, provided that their location as well as their gray or color values are well optimized. This property is exploited in inpainting-based image compression, which is a promising alternative to classical transform-based codecs such as JPEG and JPEG2000. However, optimizing the inpainting data is a challenging task. Current approaches are either fairly slow or do not produce high quality results. As a remedy we propose fast spatial and tonal optimization algorithms for homogeneous diffusion inpainting that efficiently utilize GPU parallelism, with a careful adaptation of some of the most successful numerical concepts. We propose a densification strategy using ideas from error-map dithering combined with a Delaunay triangulation for the spatial optimization. For the tonal optimization we design a domain decomposition solver that solves the corresponding normal equations in a matrix-free fashion and supplement it with a Voronoi-based initialization strategy. With our proposed methods we are able to generate high quality inpainting masks for homogeneous diffusion and optimized tonal values in a runtime that outperforms prior state-of-the-art by a wide margin.

eess.IV↗

arXiv2024

Image Compression with Isotropic and Anisotropic Shepard Inpainting

Inpainting-based codecs store sparse selected pixel data and decode by reconstructing the discarded image parts by inpainting. Successful codecs (coders and decoders) traditionally use inpainting operators that solve partial differential equations. This requires some numerical expertise if efficient implementations are necessary. Our goal is to investigate variants of Shepard inpainting as simple alternatives for inpainting-based compression. They can be implemented efficiently when we localise their weighting function. To turn them into viable codecs, we have to introduce novel extensions of classical Shepard interpolation that adapt successful ideas from previous codecs: Anisotropy allows direction-dependent inpainting, which improves reconstruction quality. Additionally, we incorporate data selection by subdivision as an efficient way to tailor the stored information to the image structure. On the encoding side, we introduce the novel concept of joint inpainting and prediction for isotropic Shepard codecs, where storage cost can be reduced based on intermediate inpainting results. In an ablation study, we show the usefulness of these individual contributions and demonstrate that they offer synergies which elevate the performance of Shepard inpainting to surprising levels. Our resulting approaches offer a more favourable trade-off between simplicity and quality than traditional inpainting-based codecs. Experiments show that they can outperform JPEG and JPEG2000 at high compression ratios.

eess.IV↗

arXiv2024

Neuroexplicit Diffusion Models for Inpainting of Optical Flow Fields

Deep learning has revolutionized the field of computer vision by introducing large scale neural networks with millions of parameters. Training these networks requires massive datasets and leads to intransparent models that can fail to generalize. At the other extreme, models designed from partial differential equations (PDEs) embed specialized domain knowledge into mathematical equations and usually rely on few manually chosen hyperparameters. This makes them transparent by construction and if designed and calibrated carefully, they can generalize well to unseen scenarios. In this paper, we show how to bring model- and data-driven approaches together by combining the explicit PDE-based approaches with convolutional neural networks to obtain the best of both worlds. We illustrate a joint architecture for the task of inpainting optical flow fields and show that the combination of model- and data-driven modeling leads to an effective architecture. Our model outperforms both fully explicit and fully data-driven baselines in terms of reconstruction quality, robustness and amount of required training data. Averaging the endpoint error across different mask densities, our method outperforms the explicit baselines by 11-27%, the GAN baseline by 47% and the Probabilisitic Diffusion baseline by 42%. With that, our method sets a new state of the art for inpainting of optical flow fields from random masks.

cs.CV↗

arXiv2024

Anisotropic Diffusion Stencils: From Simple Derivations over Stability Estimates to ResNet Implementations

Anisotropic diffusion processes with a diffusion tensor are important in image analysis, physics, and engineering. However, their numerical approximation has a strong impact on dissipative artefacts and deviations from rotation invariance. In this work, we study a large family of finite difference discretisations on a 3 x 3 stencil. We derive it by splitting 2-D anisotropic diffusion into four 1-D diffusions. The resulting stencil class involves one free parameter and covers a wide range of existing discretisations. It comprises the full stencil family of Weickert et al. (2013) and shows that their two parameters contain redundancy. Furthermore, we establish a bound on the spectral norm of the matrix corresponding to the stencil. This gives time step size limits that guarantee stability of an explicit scheme in the Euclidean norm. Our directional splitting also allows a very natural translation of the explicit scheme into ResNet blocks. Employing neural network libraries enables simple and highly efficient parallel implementations on GPUs.

math.NA↗

arXiv2024

Efficient Parallel Algorithms for Inpainting-Based Representations of 4K Images -- Part I: Homogeneous Diffusion Inpainting

In recent years inpainting-based compression methods have been shown to be a viable alternative to classical codecs such as JPEG and JPEG2000. Unlike transform-based codecs, which store coefficients in the transform domain, inpainting-based approaches store a small subset of the original image pixels and reconstruct the image from those by using a suitable inpainting operator. A good candidate for such an inpainting operator is homogeneous diffusion inpainting, as it is simple, theoretically well-motivated, and can achieve good reconstruction quality for optimized data. However, a major challenge has been to design fast solvers for homogeneous diffusion inpainting that scale to 4K image resolution ($3840 \times 2160$ pixels) and are real-time capable. We overcome this with a careful adaptation and fusion of two of the most efficient concept from numerical analysis: multigrid and domain decomposition. Our domain decomposition algorithm efficiently utilizes GPU parallelism by solving inpainting problems on small overlapping blocks. Unlike simple block decomposition strategies such as the ones in JPEG, our approach yields block artifact-free reconstructions. Furthermore, embedding domain decomposition in a full multigrid scheme provides global interactions and allows us to achieve optimal convergence by reducing both low- and high-frequency errors at the same rate. We are able to achieve 4K color image reconstruction at more than $60$ frames per second even from very sparse data - something which has been previously unfeasible.

eess.IV↗

arXiv2023

Regularised Diffusion-Shock Inpainting

We introduce regularised diffusion--shock (RDS) inpainting as a modification of diffusion--shock inpainting from our SSVM 2023 conference paper. RDS inpainting combines two carefully chosen components: homogeneous diffusion and coherence-enhancing shock filtering. It benefits from the complementary synergy of its building blocks: The shock term propagates edge data with perfect sharpness and directional accuracy over large distances due to its high degree of anisotropy. Homogeneous diffusion fills large areas efficiently. The second order equation underlying RDS inpainting inherits a maximum--minimum principle from its components, which is also fulfilled in the discrete case, in contrast to competing anisotropic methods. The regularisation addresses the largest drawback of the original model: It allows a drastic reduction in model parameters without any loss in quality. Furthermore, we extend RDS inpainting to vector-valued data. Our experiments show a performance that is comparable to or better than many inpainting methods based on partial differential equations and related integrodifferential models

eess.IV↗

arXiv2023

Efficient Neural Generation of 4K Masks for Homogeneous Diffusion Inpainting

With well-selected data, homogeneous diffusion inpainting can reconstruct images from sparse data with high quality. While 4K colour images of size 3840 x 2160 can already be inpainted in real time, optimising the known data for applications like image compression remains challenging: Widely used stochastic strategies can take days for a single 4K image. Recently, a first neural approach for this so-called mask optimisation problem offered high speed and good quality for small images. It trains a mask generation network with the help of a neural inpainting surrogate. However, these mask networks can only output masks for the resolution and mask density they were trained for. We solve these problems and enable mask optimisation for high-resolution images through a neuroexplicit coarse-to-fine strategy. Additionally, we improve the training and interpretability of mask networks by including a numerical inpainting solver directly into the network. This allows to generate masks for 4K images in around 0.6 seconds while exceeding the quality of stochastic methods on practically relevant densities. Compared to popular existing approaches, this is an acceleration of up to four orders of magnitude.

eess.IV↗

arXiv2023

Optimising Different Feature Types for Inpainting-based Image Representations

Inpainting-based image compression is a promising alternative to classical transform-based lossy codecs. Typically it stores a carefully selected subset of all pixel locations and their colour values. In the decoding phase the missing information is reconstructed by an inpainting process such as homogeneous diffusion inpainting. Optimising the stored data is the key for achieving good performance. A few heuristic approaches also advocate alternative feature types such as derivative data and construct dedicated inpainting concepts. However, one still lacks a general approach that allows to optimise and inpaint the data simultaneously w.r.t. a collection of different feature types, their locations, and their values. Our paper closes this gap. We introduce a generalised inpainting process that can handle arbitrary features which can be expressed as linear equality constraints. This includes e.g. colour values and derivatives of any order. We propose a fully automatic algorithm that aims at finding the optimal features from a given collection as well as their locations and their function values within a specified total feature density. Its performance is demonstrated with a novel set of features that also includes local averages. Our experiments show that it clearly outperforms the popular inpainting with optimised colour data with the same density.

eess.IV↗

arXiv2023

Diffusion-Shock Inpainting

We propose diffusion-shock (DS) inpainting as a hitherto unexplored integrodifferential equation for filling in missing structures in images. It combines two carefully chosen components that have proven their usefulness in different applications: homogeneous diffusion inpainting and coherence-enhancing shock filtering. DS inpainting enjoys the complementary synergy of its building blocks: It offers a high degree of anisotropy along an eigendirection of the structure tensor. This enables it to connect interrupted structures over large distances. Moreover, it benefits from the sharp edge structure generated by the shock filter, and it exploits the efficient filling-in effect of homogeneous diffusion. The second order equation that underlies DS inpainting inherits a continuous maximum-minimum principle from its constituents. In contrast to other attractive second order inpainting equations such as edge-enhancing anisotropic diffusion, we can guarantee this property also for the proposed discrete algorithm. Our experiments show a performance that is comparable to or better than many linear or nonlinear, isotropic or anisotropic processes of second or fourth order. They include homogeneous diffusion, biharmonic interpolation, TV inpainting, edge-enhancing anisotropic diffusion, the methods of Tschumperlé and of Bornemann and März, Cahn-Hilliard inpainting, and Euler's elastica.

eess.IV↗

arXiv2023

Image Blending with Osmosis

Image blending is an integral part of many multi-image applications such as panorama stitching or remote image acquisition processes. In such scenarios, multiple images are connected at predefined boundaries to form a larger image. A convincing transition between these boundaries may be challenging, since each image might have been acquired under different conditions or even by different devices. We propose the first blending approach based on osmosis filters. These drift-diffusion processes define an image evolution with a non-trivial steady state. For our blending purposes, we explore several ways to compose drift vector fields based on the derivatives of our input images. These vector fields guide the evolution such that the steady state yields a convincing blended result. Our method benefits from the well-founded theoretical results for osmosis, which include useful invariances under multiplicative changes of the colour values. Experiments on real-world data show that this yields better quality than traditional gradient domain blending, especially under challenging illumination conditions.

eess.IV↗

arXiv2023

Deep Spatial and Tonal Data Optimisation for Homogeneous Diffusion Inpainting

Diffusion-based inpainting can reconstruct missing image areas with high quality from sparse data, provided that their location and their values are well optimised. This is particularly useful for applications such as image compression, where the original image is known. Selecting the known data constitutes a challenging optimisation problem, that has so far been only investigated with model-based approaches. So far, these methods require a choice between either high quality or high speed since qualitatively convincing algorithms rely on many time-consuming inpaintings. We propose the first neural network architecture that allows fast optimisation of pixel positions and pixel values for homogeneous diffusion inpainting. During training, we combine two optimisation networks with a neural network-based surrogate solver for diffusion inpainting. This novel concept allows us to perform backpropagation based on inpainting results that approximate the solution of the inpainting equation. Without the need for a single inpainting during test time, our deep optimisation accelerates data selection by more than four orders of magnitude compared to common model-based approaches. This provides real-time performance with high quality results.

eess.IV↗

arXiv2023

CNN-based Euler's Elastica Inpainting with Deep Energy and Deep Image Prior

Euler's elastica constitute an appealing variational image inpainting model. It minimises an energy that involves the total variation as well as the level line curvature. These components are transparent and make it attractive for shape completion tasks. However, its gradient flow is a singular, anisotropic, and nonlinear PDE of fourth order, which is numerically challenging: It is difficult to find efficient algorithms that offer sharp edges and good rotation invariance. As a remedy, we design the first neural algorithm that simulates inpainting with Euler's Elastica. We use the deep energy concept which employs the variational energy as neural network loss. Furthermore, we pair it with a deep image prior where the network architecture itself acts as a prior. This yields better inpaintings by steering the optimisation trajectory closer to the desired solution. Our results are qualitatively on par with state-of-the-art algorithms on elastica-based shape completion. They combine good rotation invariance with sharp edges. Moreover, we benefit from the high efficiency and effortless parallelisation within a neural framework. Our neural elastica approach only requires 3x3 central difference stencils. It is thus much simpler than other well-performing algorithms for elastica inpainting. Last but not least, it is unsupervised as it requires no ground truth training data.

cs.CV↗

arXiv2022

Stabilised Inverse Flowline Evolution for Anisotropic Image Sharpening

The central limit theorem suggests Gaussian convolution as a generic blur model for images. Since Gaussian convolution is equivalent to homogeneous diffusion filtering, one way to deblur such images is to diffuse them backwards in time. However, backward diffusion is highly ill-posed. Thus, it requires stabilisation in the model as well as highly sophisticated numerical algorithms. Moreover, sharpening is often only desired across image edges but not along them, since it may cause very irregular contours. This creates the need to model a stabilised anisotropic backward evolution and to devise an appropriate numerical algorithm for this ill-posed process. We address both challenges. First we introduce stabilised inverse flowline evolution (SIFE) as an anisotropic image sharpening flow. Outside extrema, its partial differential equation (PDE) is backward parabolic in gradient direction. Interestingly, it is sufficient to stabilise it in extrema by imposing a zero flow there. We show that morphological derivatives - which are not common in the numerics of PDEs - are ideal for the numerical approximation of SIFE: They effortlessly approximate directional derivatives in gradient direction. Our scheme adapts one-sided morphological derivatives to the underlying image structure. It allows to progress in subpixel accuracy and enables us to prove stability properties. Our experiments show that SIFE allows nonflat steady states and outperforms other sharpening flows.

eess.IV↗