arXiv subjects

Joachim Weickert

Publications and source records attributed to Joachim Weickert.

At least 37 records · Page 2Linked to original sources

arXiv2022

Learning Sparse Masks for Diffusion-based Image Inpainting

Diffusion-based inpainting is a powerful tool for the reconstruction of images from sparse data. Its quality strongly depends on the choice of known data. Optimising their spatial location -- the inpainting mask -- is challenging. A commonly used tool for this task are stochastic optimisation strategies. However, they are slow as they compute multiple inpainting results. We provide a remedy in terms of a learned mask generation model. By emulating the complete inpainting pipeline with two networks for mask generation and neural surrogate inpainting, we obtain a model for highly efficient adaptive mask generation. Experiments indicate that our model can achieve competitive quality with an acceleration by as much as four orders of magnitude. Our findings serve as a basis for making diffusion-based inpainting more attractive for applications such as image compression, where fast encoding is highly desirable.

eess.IV↗

arXiv2022

Connections between Numerical Algorithms for PDEs and Neural Networks

We investigate numerous structural connections between numerical algorithms for partial differential equations (PDEs) and neural architectures. Our goal is to transfer the rich set of mathematical foundations from the world of PDEs to neural networks. Besides structural insights we provide concrete examples and experimental evaluations of the resulting architectures. Using the example of generalised nonlinear diffusion in 1D, we consider explicit schemes, acceleration strategies thereof, implicit schemes, and multigrid approaches. We connect these concepts to residual networks, recurrent neural networks, and U-net architectures. Our findings inspire a symmetric residual network design with provable stability guarantees and justify the effectiveness of skip connections in neural networks from a numerical perspective. Moreover, we present U-net architectures that implement multigrid techniques for learning efficient solutions of partial differential equation models, and motivate uncommon design choices such as trainable nonmonotone activation functions. Experimental evaluations show that the proposed architectures save half of the trainable parameters and can thus outperform standard ones with the same model complexity. Our considerations serve as a basis for explaining the success of popular neural architectures and provide a blueprint for developing new mathematically well-founded neural building blocks.

math.NA↗

arXiv2022

Designing Rotationally Invariant Neural Networks from PDEs and Variational Methods

Partial differential equation (PDE) models and their associated variational energy formulations are often rotationally invariant by design. This ensures that a rotation of the input results in a corresponding rotation of the output, which is desirable in applications such as image analysis. Convolutional neural networks (CNNs) do not share this property, and existing remedies are often complex. The goal of our paper is to investigate how diffusion and variational models achieve rotation invariance and transfer these ideas to neural networks. As a core novelty we propose activation functions which couple network channels by combining information from several oriented filters. This guarantees rotation invariance within the basic building blocks of the networks while still allowing for directional filtering. The resulting neural architectures are inherently rotationally invariant. With only a few small filters, they can achieve the same invariance as existing techniques which require a fine-grained sampling of orientations. Our findings help to translate diffusion and variational models into mathematically well-founded network architectures, and provide novel concepts for model-based CNN design.

cs.LG↗

arXiv2022

Domain Decomposition Algorithms for Real-time Homogeneous Diffusion Inpainting in 4K

Inpainting-based compression methods are qualitatively promising alternatives to transform-based codecs, but they suffer from the high computational cost of the inpainting step. This prevents them from being applicable to time-critical scenarios such as real-time inpainting of 4K images. As a remedy, we adapt state-of-the-art numerical algorithms of domain decomposition type to this problem. They decompose the image domain into multiple overlapping blocks that can be inpainted in parallel by means of modern GPUs. In contrast to classical block decompositions such as the ones in JPEG, the global inpainting problem is solved without creating block artefacts. We consider the popular homogeneous diffusion inpainting and supplement it with a multilevel version of an optimised restricted additive Schwarz (ORAS) method that solves the local problems with a conjugate gradient algorithm. This enables us to perform real-time inpainting of 4K colour images on contemporary GPUs, which is substantially more efficient than previous algorithms for diffusion-based inpainting.

eess.IV↗

arXiv2021

Sparse Inpainting with Smoothed Particle Hydrodynamics

Digital image inpainting refers to techniques used to reconstruct a damaged or incomplete image by exploiting available image information. The main goal of this work is to perform the image inpainting process from a set of sparsely distributed image samples with the Smoothed Particle Hydrodynamics (SPH) technique. As, in its naive formulation, the SPH technique is not even capable of reproducing constant functions, we modify the approach to obtain an approximation which can reproduce constant and linear functions. Furthermore, we examine the use of Voronoi tessellation for defining the necessary parameters in the SPH method as well as selecting optimally located image samples. In addition to this spatial optimization, optimization of data values is also implemented in order to further improve the results. Apart from a traditional Gaussian smoothing kernel, we assess the performance of other kernels on both random and spatially optimized masks. Since the use of isotropic smoothing kernels is not optimal in the presence of objects with a clear preferred orientation in the image, we also examine anisotropic smoothing kernels. Our final algorithm can compete with well-performing sparse inpainting techniques based on homogeneous or anisotropic diffusion processes as well as with exemplar-based approaches.

eess.IV↗

arXiv2021

A Systematic Evaluation of Coding Strategies for Sparse Binary Images

Inpainting-based compression represents images in terms of a sparse subset of its pixel data. Storing the carefully optimised positions of known data creates a lossless compression problem on sparse and often scattered binary images. This central issue is crucial for the performance of such codecs. Since it has only received little attention in the literature, we conduct the first systematic investigation of this problem so far. To this end, we first review and compare a wide range of existing methods from image compression and general purpose coding in terms of their coding efficiency and runtime. Afterwards, an ablation study enables us to identify and isolate the most useful components of existing methods. With context mixing, we combine those ingredients into new codecs that offer either better compression ratios or a more favourable trade-off between speed and performance.

eess.IV↗

arXiv2021

Efficient Data Optimisation for Harmonic Inpainting with Finite Elements

Harmonic inpainting with optimised data is very popular for inpainting-based image compression. We improve this approach in three important aspects. Firstly, we replace the standard finite differences discretisation by a finite element method with triangle elements. This does not only speed up inpainting and data selection, but even improves the reconstruction quality. Secondly, we propose highly efficient algorithms for spatial and tonal data optimisation that are several orders of magnitude faster than state-of-the-art methods. Last but not least, we show that our algorithms also allow working with very large images. This has previously been impractical due to the memory and runtime requirements of prior algorithms.

eess.IV↗

arXiv2021

Removing Multi-frame Gaussian Noise by Combining Patch-based Filters with Optical Flow

Patch-based approaches such as 3D block matching (BM3D) and non-local Bayes (NLB) are widely accepted filters for removing Gaussian noise from single-frame images. In this work, we propose three extensions for these filters when there exist multiple frames of the same scene. The first of them employs reference patches on every frame instead of a commonly used single reference frame method, thus utilizing the complete available information. The remaining two techniques use a separable spatio-temporal filter to reduce interactions between dissimilar regions, hence mitigating artifacts. In order to deal with non-registered datasets we combine all our extensions with robust optical flow computation. Two of our proposed multi-frame filters outperform existing extensions on most occasions by a significant margin while also being competitive with a state-of-the-art neural network-based technique. Moreover, one of these two strategies is the fastest among all due to its separable design.

eess.IV↗

arXiv2021

Learning Integrodifferential Models for Image Denoising

We introduce an integrodifferential extension of the edge-enhancing anisotropic diffusion model for image denoising. By accumulating weighted structural information on multiple scales, our model is the first to create anisotropy through multiscale integration. It follows the philosophy of combining the advantages of model-based and data-driven approaches within compact, insightful, and mathematically well-founded models with improved performance. We explore trained results of scale-adaptive weighting and contrast parameters to obtain an explicit modelling by smooth functions. This leads to a transparent model with only three parameters, without significantly decreasing its denoising performance. Experiments demonstrate that it outperforms its diffusion-based predecessors. We show that both multiscale information and anisotropy are crucial for its success.

eess.IV↗

arXiv2021

Translating Numerical Concepts for PDEs into Neural Architectures

We investigate what can be learned from translating numerical algorithms into neural networks. On the numerical side, we consider explicit, accelerated explicit, and implicit schemes for a general higher order nonlinear diffusion equation in 1D, as well as linear multigrid methods. On the neural network side, we identify corresponding concepts in terms of residual networks (ResNets), recurrent networks, and U-nets. These connections guarantee Euclidean stability of specific ResNets with a transposed convolution layer structure in each block. We present three numerical justifications for skip connections: as time discretisations in explicit schemes, as extrapolation mechanisms for accelerating those methods, and as recurrent connections in fixed point solvers for implicit schemes. Last but not least, we also motivate uncommon design choices such as nonmonotone activation functions. Our findings give a numerical perspective on the success of modern neural network architectures, and they provide design criteria for stable networks.

math.NA↗

arXiv2021

Inpainting-based Video Compression in FullHD

Compression methods based on inpainting are an evolving alternative to classical transform-based codecs for still images. Attempts to apply these ideas to video compression are rare, since reaching real-time performance is very challenging. Therefore, current approaches focus on simplified frame-by-frame reconstructions that ignore temporal redundancies. As a remedy, we propose a highly efficient, real-time capable prediction and correction approach that fully relies on partial differential equations (PDEs) in all steps of the codec: Dense variational optic flow fields yield accurate motion-compensated predictions, while homogeneous diffusion inpainting is applied for intra prediction. To compress residuals, we introduce a new highly efficient block-based variant of pseudodifferential inpainting. Our novel architecture outperforms other inpainting-based video codecs in terms of both quality and speed. For the first time in inpainting-based video compression, we can decompress FullHD (1080p) videos in real-time with a fully CPU-based implementation, outperforming previous approaches by roughly one order of magnitude.

eess.IV↗

arXiv2021

JPEG Meets PDE-based Image Compression

Inpainting-based image compression is emerging as a promising competitor to transform-based compression techniques. Its key idea is to reconstruct image information from only few known regions through inpainting. Specific partial differential equations (PDEs) such as edge-enhancing diffusion (EED) give high quality reconstructions of image structures with low or medium texture. Even though the strengths of PDE- and transform-based compression are complementary, they have rarely been combined within a hybrid codec. We propose to sparsify blocks of a JPEG compressed image and reconstruct them with EED inpainting. Our codec consistently outperforms JPEG and gives useful indications for successfully developing hybrid codecs further. Furthermore, our method is the first to choose regions rather than pixels as known data for PDE-based compression. It also gives novel insights into the importance of corner regions for EED-based codecs.

eess.IV↗

arXiv2021

Multi-frame Super-resolution from Noisy Data

Obtaining high resolution images from low resolution data with clipped noise is algorithmically challenging due to the ill-posed nature of the problem. So far such problems have hardly been tackled, and the few existing approaches use simplistic regularisers. We show the usefulness of two adaptive regularisers based on anisotropic diffusion ideas: Apart from evaluating the classical edge-enhancing anisotropic diffusion regulariser, we introduce a novel non-local one with one-sided differences and superior performance. It is termed sector diffusion. We combine it with all six variants of the classical super-resolution observational model that arise from permutations of its three operators for warping, blurring, and downsampling. Surprisingly, the evaluation in a practically relevant noisy scenario produces a different ranking than the one in the noise-free setting in our previous work (SSVM 2017).

cs.CV↗

arXiv2020

Robustness of Brain Tumor Segmentation

Purpose: The segmentation of brain tumors is one of the most active areas of medical image analysis. While current methods perform superhuman on benchmark data sets, their applicability in daily clinical practice has not been evaluated. In our work we investigate the generalization behavior of deep neural networks in this scenario. Approach: We evaluate the performance of three state-of-the-art methods, a basic U-net architecture and a cascadic Mumford-Shah approach. We also propose two simple modifications (which do not change the topology) to improve generalization performance. Results: In our experiments we show that a well-trained U-network shows the best generalization behavior and is sufficient to solve this segmentation problem. We illustrate why extensions of this model in a realistic scenario can be not only pointless but even harmful. Conclusions: We conclude from our experiments that the generalization performance of deep neural networks is severely limited in medical image analysis especially in the area of brain tumor segmentation. In our opinion, current topologies are optimized for the actual benchmark data set, but are not directly applicable in daily clinical practice.

eess.IV↗

arXiv2020

Compressing Colour Images with Joint Inpainting and Prediction

Inpainting-based codecs store sparse, quantised pixel data directly and decode by interpolating the discarded image parts. This interpolation can be used simultaneously for efficient coding by predicting pixel data to be stored. Such joint inpainting and prediction approaches yield good results with simple components such as regular grids and Shepard interpolation on grey value images, but they lack a dedicated mode for colour images. Therefore, we evaluate different approaches for inpainting-based colour compression. Inpainting operators are able to reconstruct a large range of colours from a small colour palette of the known pixels. We exploit this with a luma preference mode which uses higher sparsity in YCbCr colour channels than in the brightness channel. Furthermore, we propose the first full vector quantisation mode for an inpainting-based codec that stores only a small codebook of colours. Our experiments reveal that both colour extensions yield significant improvements.

eess.IV↗

arXiv2020

Object Segmentation Tracking from Generic Video Cues

We propose a light-weight variational framework for online tracking of object segmentations in videos based on optical flow and image boundaries. While high-end computer vision methods on this task rely on sequence specific training of dedicated CNN architectures, we show the potential of a variational model, based on generic video information from motion and color. Such cues are usually required for tasks such as robot navigation or grasp estimation. We leverage them directly for video object segmentation and thus provide accurate segmentations at potentially very low extra cost. Our simple method can provide competitive results compared to the costly CNN-based methods with parameter tuning. Furthermore, we show that our approach can be combined with state-of-the-art CNN-based segmentations in order to improve over their respective results. We evaluate our method on the datasets DAVIS 16,17 and SegTrack v2.

cs.CV↗

arXiv2020

PDE Evolutions for M-Smoothers in One, Two, and Three Dimensions

Local M-smoothers are interesting and important signal and image processing techniques with many connections to other methods. In our paper we derive a family of partial differential equations (PDEs) that result in one, two, and three dimensions as limiting processes from M-smoothers which are based on local order-$p$ means within a ball the radius of which tends to zero. The order $p$ may take any nonzero value $>-1$, allowing also negative values. In contrast to results from the literature, we show in the space-continuous case that mode filtering does not arise for $p \to 0$, but for $p \to -1$. Extending our filter class to $p$-values smaller than $-1$ allows to include e.g. the classical image sharpening flow of Gabor. The PDEs we derive in 1D, 2D, and 3D show large structural similarities. Since our PDE class is highly anisotropic and may contain backward parabolic operators, designing adequate numerical methods is difficult. We present an $L^\infty$-stable explicit finite difference scheme that satisfies a discrete maximum--minimum principle, offers excellent rotation invariance, and employs a splitting into four fractional steps to allow larger time step sizes. Although it approximates parabolic PDEs, it consequently benefits from stabilisation concepts from the numerics of hyperbolic PDEs. Our 2D experiments show that the PDEs for $p<1$ are of specific interest: Their backward parabolic term creates favourable sharpening properties, while they appear to maintain the strong shape simplification properties of mean curvature motion.

eess.IV↗

arXiv2020

Stable Backward Diffusion Models that Minimise Convex Energies

The inverse problem of backward diffusion is known to be ill-posed and highly unstable. Backward diffusion processes appear naturally in image enhancement and deblurring applications. It is therefore greatly desirable to establish a backward diffusion model which implements a smart stabilisation approach that can be used in combination with an easy to handle numerical scheme. So far, existing stabilisation strategies in literature require sophisticated numerics to solve the underlying initial value problem. We derive a class of space-discrete one-dimensional backward diffusion as gradient descent of energies where we gain stability by imposing range constraints. Interestingly, these energies are even convex. Furthermore, we establish a comprehensive theory for the time-continuous evolution and we show that stability carries over to a simple explicit time discretisation of our model. Finally, we confirm the stability and usefulness of our technique in experiments in which we enhance the contrast of digital greyscale and colour images.

math.NA↗