arXiv subjects

Markus Gross

Publications and source records attributed to Markus Gross.

At least 55 records · Page 3Linked to original sources

arXiv2023

Artist-Friendly Relightable and Animatable Neural Heads

An increasingly common approach for creating photo-realistic digital avatars is through the use of volumetric neural fields. The original neural radiance field (NeRF) allowed for impressive novel view synthesis of static heads when trained on a set of multi-view images, and follow up methods showed that these neural representations can be extended to dynamic avatars. Recently, new variants also surpassed the usual drawback of baked-in illumination in neural representations, showing that static neural avatars can be relit in any environment. In this work we simultaneously tackle both the motion and illumination problem, proposing a new method for relightable and animatable neural heads. Our method builds on a proven dynamic avatar approach based on a mixture of volumetric primitives, combined with a recently-proposed lightweight hardware setup for relightable neural fields, and includes a novel architecture that allows relighting dynamic neural avatars performing unseen expressions in any environment, even with nearfield illumination and viewpoints.

cs.CV↗

arXiv2023

Spatially Adaptive Cloth Regression with Implicit Neural Representations

The accurate representation of fine-detailed cloth wrinkles poses significant challenges in computer graphics. The inherently non-uniform structure of cloth wrinkles mandates the employment of intricate discretization strategies, which are frequently characterized by high computational demands and complex methodologies. Addressing this, the research introduced in this paper elucidates a novel anisotropic cloth regression technique that capitalizes on the potential of implicit neural representations of surfaces. Our first core contribution is an innovative mesh-free sampling approach, crafted to reduce the reliance on traditional mesh structures, thereby offering greater flexibility and accuracy in capturing fine cloth details. Our second contribution is a novel adversarial training scheme, which is designed meticulously to strike a harmonious balance between the sampling and simulation objectives. The adversarial approach ensures that the wrinkles are represented with high fidelity, while also maintaining computational efficiency. Our results showcase through various cloth-object interaction scenarios that our method, given the same memory constraints, consistently surpasses traditional discrete representations, particularly when modelling highly-detailed localized wrinkles.

cs.CV↗

arXiv2023

GroomGen: A High-Quality Generative Hair Model Using Hierarchical Latent Representations

Despite recent successes in hair acquisition that fits a high-dimensional hair model to a specific input subject, generative hair models, which establish general embedding spaces for encoding, editing, and sampling diverse hairstyles, are way less explored. In this paper, we present GroomGen, the first generative model designed for hair geometry composed of highly-detailed dense strands. Our approach is motivated by two key ideas. First, we construct hair latent spaces covering both individual strands and hairstyles. The latent spaces are compact, expressive, and well-constrained for high-quality and diverse sampling. Second, we adopt a hierarchical hair representation that parameterizes a complete hair model to three levels: single strands, sparse guide hairs, and complete dense hairs. This representation is critical to the compactness of latent spaces, the robustness of training, and the efficiency of inference. Based on this hierarchical latent representation, our proposed pipeline consists of a strand-VAE and a hairstyle-VAE that encode an individual strand and a set of guide hairs to their respective latent spaces, and a hybrid densification step that populates sparse guide hairs to a dense hair model. GroomGen not only enables novel hairstyle sampling and plausible hairstyle interpolation, but also supports interactive editing of complex hairstyles, or can serve as strong data-driven prior for hairstyle reconstruction from images. We demonstrate the superiority of our approach with qualitative examples of diverse sampled hairstyles and quantitative evaluation of generation quality regarding every single component and the entire pipeline.

cs.GR↗

arXiv2023

A Perceptual Shape Loss for Monocular 3D Face Reconstruction

Monocular 3D face reconstruction is a wide-spread topic, and existing approaches tackle the problem either through fast neural network inference or offline iterative reconstruction of face geometry. In either case carefully-designed energy functions are minimized, commonly including loss terms like a photometric loss, a landmark reprojection loss, and others. In this work we propose a new loss function for monocular face capture, inspired by how humans would perceive the quality of a 3D face reconstruction given a particular image. It is widely known that shading provides a strong indicator for 3D shape in the human visual system. As such, our new 'perceptual' shape loss aims to judge the quality of a 3D face estimate using only shading cues. Our loss is implemented as a discriminator-style neural network that takes an input face image and a shaded render of the geometry estimate, and then predicts a score that perceptually evaluates how well the shaded render matches the given image. This 'critic' network operates on the RGB image and geometry render alone, without requiring an estimate of the albedo or illumination in the scene. Furthermore, our loss operates entirely in image space and is thus agnostic to mesh topology. We show how our new perceptual shape loss can be combined with traditional energy terms for monocular 3D face optimization and deep neural network regression, improving upon current state-of-the-art results.

cs.CV↗

arXiv2023

Controllable Inversion of Black-Box Face Recognition Models via Diffusion

Face recognition models embed a face image into a low-dimensional identity vector containing abstract encodings of identity-specific facial features that allow individuals to be distinguished from one another. We tackle the challenging task of inverting the latent space of pre-trained face recognition models without full model access (i.e. black-box setting). A variety of methods have been proposed in literature for this task, but they have serious shortcomings such as a lack of realistic outputs and strong requirements for the data set and accessibility of the face recognition model. By analyzing the black-box inversion problem, we show that the conditional diffusion model loss naturally emerges and that we can effectively sample from the inverse distribution even without an identity-specific loss. Our method, named identity denoising diffusion probabilistic model (ID3PM), leverages the stochastic nature of the denoising diffusion process to produce high-quality, identity-preserving face images with various backgrounds, lighting, poses, and expressions. We demonstrate state-of-the-art performance in terms of identity preservation and diversity both qualitatively and quantitatively, and our method is the first black-box face recognition model inversion method that offers intuitive control over the generation process.

cs.CV↗

arXiv2022

Dynamics and steady states of a tracer particle in a confined critical fluid

The dynamics and the steady states of a point-like tracer particle immersed in a confined critical fluid are studied. The fluid is modeled field-theoretically in terms of an order parameter (concentration or density field) obeying dissipative or conservative equilibrium dynamics and (non-)symmetry-breaking boundary conditions. The tracer, which represents, e.g., a colloidal particle, interacts with the fluid by locally modifying its chemical potential or its correlations. The coupling between tracer and fluid gives rise to a nonlinear and non-Markovian tracer dynamics, which is investigated here analytically and via numerical simulations for a one-dimensional system. From the coupled Langevin equations for the tracer-fluid system we derive an effective Fokker-Planck equation for the tracer by means of adiabatic elimination as well as perturbation theory within a weak-coupling approximation. The effective tracer dynamics is found to be governed by a fluctuation-induced (Casimir) potential, a spatially dependent mobility, and a spatially dependent (multiplicative) noise, the characteristics of which depend on the interaction and the boundary conditions. The steady-state distribution of the tracer is typically inhomogeneous. Notably, when detailed balance is broken, the driving of the temporally correlated noise can induce an effective attraction of the tracer towards a boundary.

cond-mat.stat-mech↗

arXiv2022

Tracer particle in a confined correlated medium: an adiabatic elimination method

We present a simple and systematic procedure to determine the effective dynamics of a Brownian particle coupled to a rapidly fluctuating correlated medium, modeled as a scalar Gaussian field, under spatial confinement. The method allows us, in particular, to address the case in which the fluctuations of the medium are suppressed in the vicinity of the particle, as described by a quadratic coupling in the underlying Hamiltonian. As a consequence of the confinement of the correlated medium, the resulting effective Fokker-Planck equation features spatially dependent drift and diffusion coefficients. We apply our method to simplified fluid models of binary mixtures and microemulsions near criticality containing a colloidal particle, and we analyze the corrections to the stationary distribution of the particle position and the diffusion coefficient.

cond-mat.stat-mech↗

arXiv2022

Microdosing: Knowledge Distillation for GAN based Compression

Recently, significant progress has been made in learned image and video compression. In particular the usage of Generative Adversarial Networks has lead to impressive results in the low bit rate regime. However, the model size remains an important issue in current state-of-the-art proposals and existing solutions require significant computation effort on the decoding side. This limits their usage in realistic scenarios and the extension to video compression. In this paper, we demonstrate how to leverage knowledge distillation to obtain equally capable image decoders at a fraction of the original number of parameters. We investigate several aspects of our solution including sequence specialization with side information for image coding. Finally, we also show how to transfer the obtained benefits into the setting of video compression. Overall, this allows us to reduce the model size by a factor of 20 and to achieve 50% reduction in decoding time.

eess.IV↗

arXiv2021

Fluctuations of the critical Casimir force

The critical Casimir force (CCF) arises from confining fluctuations in a critical fluid and thus it is a fluctuating quantity itself. While the mean CCF is universal, its (static) variance has previously been found to depend on the microscopic details of the system which effectively set a large-momentum cutoff in the underlying field theory, rendering it potentially large. This raises the question how the properties of the force variance are reflected in experimentally observable quantities, such as the thickness of a wetting film or the position of a suspended colloidal particle. Here, based on a rigorous definition of the instantaneous force, we analyze static and dynamic correlations of the CCF for a conserved fluid in film geometry for various boundary conditions within the Gaussian approximation. We find that the dynamic correlation function of the CCF is independent of the momentum cutoff and decays algebraically in time. Within the Gaussian approximation, the associated exponent depends only on the dynamic universality class but not on the boundary conditions. We furthermore consider a fluid film, the thickness of which can fluctuate under the influence of the time-dependent CCF. The latter gives rise to an effective non-Markovian noise in the equation of motion of the film boundary and induces a distinct contribution to the position variance. Within the approximations used here, at short times, this contribution grows algebraically in time whereas, at long times, it saturates and contributes to the steady-state variance of the film thickness.

cond-mat.stat-mech↗

arXiv2021

DuctTake: Spatiotemporal Video Compositing

DuctTake is a system designed to enable practical compositing of multiple takes of a scene into a single video. Current industry solutions are based around object segmentation, a hard problem that requires extensive manual input and cleanup, making compositing an expensive part of the film-making process. Our method instead composites shots together by finding optimal spatiotemporal seams using motion-compensated 3D graph cuts through the video volume. We describe in detail the required components, decisions, and new techniques that together make a usable, interactive tool for compositing HD video, paying special attention to running time and performance of each section. We validate our approach by presenting a wide variety of examples and by comparing result quality and creation time to composites made by professional artists using current state-of-the-art tools.

cs.CV↗

arXiv2020

Blind Image Restoration with Flow Based Priors

Image restoration has seen great progress in the last years thanks to the advances in deep neural networks. Most of these existing techniques are trained using full supervision with suitable image pairs to tackle a specific degradation. However, in a blind setting with unknown degradations this is not possible and a good prior remains crucial. Recently, neural network based approaches have been proposed to model such priors by leveraging either denoising autoencoders or the implicit regularization captured by the neural network structure itself. In contrast to this, we propose using normalizing flows to model the distribution of the target content and to use this as a prior in a maximum a posteriori (MAP) formulation. By expressing the MAP optimization process in the latent space through the learned bijective mapping, we are able to obtain solutions through gradient descent. To the best of our knowledge, this is the first work that explores normalizing flows as prior in image enhancement problems. Furthermore, we present experimental results for a number of different degradations on data sets varying in complexity and show competitive results when comparing with the deep image prior approach.

eess.IV↗

arXiv2020

Lossy Image Compression with Normalizing Flows

Deep learning based image compression has recently witnessed exciting progress and in some cases even managed to surpass transform coding based approaches that have been established and refined over many decades. However, state-of-the-art solutions for deep image compression typically employ autoencoders which map the input to a lower dimensional latent space and thus irreversibly discard information already before quantization. Due to that, they inherently limit the range of quality levels that can be covered. In contrast, traditional approaches in image compression allow for a larger range of quality levels. Interestingly, they employ an invertible transformation before performing the quantization step which explicitly discards information. Inspired by this, we propose a deep image compression method that is able to go from low bit-rates to near lossless quality by leveraging normalizing flows to learn a bijective mapping from the image space to a latent representation. In addition to this, we demonstrate further advantages unique to our solution, such as the ability to maintain constant quality results through re-encoding, even when performed multiple times. To the best of our knowledge, this is the first work to explore the opportunities for leveraging normalizing flows for lossy image compression.

cs.CV↗

arXiv2020

Enriching Video Captions With Contextual Text

Understanding video content and generating caption with context is an important and challenging task. Unlike prior methods that typically attempt to generate generic video captions without context, our architecture contextualizes captioning by infusing extracted information from relevant text data. We propose an end-to-end sequence-to-sequence model which generates video captions based on visual input, and mines relevant knowledge such as names and locations from contextual text. In contrast to previous approaches, we do not preprocess the text further, and let the model directly learn to attend over it. Guided by the visual input, the model is able to copy words from the contextual text via a pointer-generator network, allowing to produce more specific video captions. We show competitive performance on the News Video Dataset and, through ablation studies, validate the efficacy of contextual video captioning as well as individual design choices in our model architecture.

cs.CV↗

arXiv2020

Shapley Value as Principled Metric for Structured Network Pruning

Structured pruning is a well-known technique to reduce the storage size and inference cost of neural networks. The usual pruning pipeline consists of ranking the network internal filters and activations with respect to their contributions to the network performance, removing the units with the lowest contribution, and fine-tuning the network to reduce the harm induced by pruning. Recent results showed that random pruning performs on par with other metrics, given enough fine-tuning resources. In this work, we show that this is not true on a low-data regime when fine-tuning is either not possible or not effective. In this case, reducing the harm caused by pruning becomes crucial to retain the performance of the network. First, we analyze the problem of estimating the contribution of hidden units with tools suggested by cooperative game theory and propose Shapley values as a principled ranking metric for this task. We compare with several alternatives proposed in the literature and discuss how Shapley values are theoretically preferable. Finally, we compare all ranking metrics on the challenging scenario of low-data pruning, where we demonstrate how Shapley values outperform other heuristics.

cs.LG↗

arXiv2020

Lagrangian Neural Style Transfer for Fluids

Artistically controlling the shape, motion and appearance of fluid simulations pose major challenges in visual effects production. In this paper, we present a neural style transfer approach from images to 3D fluids formulated in a Lagrangian viewpoint. Using particles for style transfer has unique benefits compared to grid-based techniques. Attributes are stored on the particles and hence are trivially transported by the particle motion. This intrinsically ensures temporal consistency of the optimized stylized structure and notably improves the resulting quality. Simultaneously, the expensive, recursive alignment of stylization velocity fields of grid approaches is unnecessary, reducing the computation time to less than an hour and rendering neural flow stylization practical in production settings. Moreover, the Lagrangian representation improves artistic control as it allows for multi-fluid stylization and consistent color transfer from images, and the generality of the method enables stylization of smoke and liquids likewise.

cs.GR↗

arXiv2019

Transport-Based Neural Style Transfer for Smoke Simulations

Artistically controlling fluids has always been a challenging task. Optimization techniques rely on approximating simulation states towards target velocity or density field configurations, which are often handcrafted by artists to indirectly control smoke dynamics. Patch synthesis techniques transfer image textures or simulation features to a target flow field. However, these are either limited to adding structural patterns or augmenting coarse flows with turbulent structures, and hence cannot capture the full spectrum of different styles and semantically complex structures. In this paper, we propose the first Transport-based Neural Style Transfer (TNST) algorithm for volumetric smoke data. Our method is able to transfer features from natural images to smoke simulations, enabling general content-aware manipulations ranging from simple patterns to intricate motifs. The proposed algorithm is physically inspired, since it computes the density transport from a source input smoke to a desired target configuration. Our transport-based approach allows direct control over the divergence of the stylization velocity field by optimizing incompressible and irrotational potentials that transport smoke towards stylization. Temporal consistency is ensured by transporting and aligning subsequent stylized velocities, and 3D reconstructions are computed by seamlessly merging stylizations from different camera viewpoints.

cs.GR↗

arXiv2019

Neural Importance Sampling

We propose to use deep neural networks for generating samples in Monte Carlo integration. Our work is based on non-linear independent components estimation (NICE), which we extend in numerous ways to improve performance and enable its application to integration problems. First, we introduce piecewise-polynomial coupling transforms that greatly increase the modeling power of individual coupling layers. Second, we propose to preprocess the inputs of neural networks using one-blob encoding, which stimulates localization of computation and improves inference. Third, we derive a gradient-descent-based optimization for the KL and the $χ^2$ divergence for the specific application of Monte Carlo integration with unnormalized stochastic estimates of the target distribution. Our approach enables fast and accurate inference and efficient sample generation independently of the dimensionality of the integration domain. We show its benefits on generating natural images and in two applications to light-transport simulation: first, we demonstrate learning of joint path-sampling densities in the primary sample space and importance sampling of multi-dimensional path prefixes thereof. Second, we use our technique to extract conditional directional densities driven by the product of incident illumination and the BSDF in the rendering equation, and we leverage the densities for path guiding. In all applications, our approach yields on-par or higher performance than competing techniques at equal sample count.

cs.LG↗

arXiv2019

Dynamics of the critical Casimir force for a conserved order parameter after a critical quench

Fluctuation-induced forces occur generically when long-ranged correlations (e.g., in fluids) are confined by external bodies. In classical systems, such correlations require specific conditions, e.g., a medium close to a critical point. On the other hand, long-ranged correlations appear more commonly in certain non-equilibrium systems with conservation laws. Consequently, a variety of non-equilibrium fluctuation phenomena, including fluctuation-induced forces, have been discovered and explored recently. Here, we address a long-standing problem of non-equilibrium critical Casimir forces emerging after a quench to the critical point in a confined fluid with order-parameter-conserving dynamics and non-symmetry-breaking boundary conditions. The interplay of inherent (critical) fluctuations and dynamical non-local effects (due to density conservation) gives rise to striking features, including correlation functions and forces exhibiting oscillatory time-dependences. Complex transient regimes arise, depending on initial conditions and the geometry of the confinement. Our findings pave the way for exploring a wealth of non-equilibrium processes in critical fluids (e.g., fluctuation-mediated self-assembly or aggregation). In certain regimes, our results are applicable to active matter.

cond-mat.stat-mech↗