arXiv subjects

Di Kang

Publications and source records attributed to Di Kang.

50 records · Page 3Linked to original sources

arXiv2021

Channelized Axial Attention for Semantic Segmentation -- Considering Channel Relation within Spatial Attention for Semantic Segmentation

Spatial and channel attentions, modelling the semantic interdependencies in spatial and channel dimensions respectively, have recently been widely used for semantic segmentation. However, computing spatial and channel attentions separately sometimes causes errors, especially for those difficult cases. In this paper, we propose Channelized Axial Attention (CAA) to seamlessly integrate channel attention and spatial attention into a single operation with negligible computation overhead. Specifically, we break down the dot-product operation of the spatial attention into two parts and insert channel relation in between, allowing for independently optimized channel attention on each spatial location. We further develop grouped vectorization, which allows our model to run with very little memory consumption without slowing down the running speed. Comparative experiments conducted on multiple benchmark datasets, including Cityscapes, PASCAL Context, and COCO-Stuff, demonstrate that our CAA outperforms many state-of-the-art segmentation models (including dual attention) on all tested datasets.

cs.CV↗

arXiv2021

Animatable Neural Radiance Fields from Monocular RGB Videos

We present animatable neural radiance fields (animatable NeRF) for detailed human avatar creation from monocular videos. Our approach extends neural radiance fields (NeRF) to the dynamic scenes with human movements via introducing explicit pose-guided deformation while learning the scene representation network. In particular, we estimate the human pose for each frame and learn a constant canonical space for the detailed human template, which enables natural shape deformation from the observation space to the canonical space under the explicit control of the pose parameters. To compensate for inaccurate pose estimation, we introduce the pose refinement strategy that updates the initial pose during the learning process, which not only helps to learn more accurate human reconstruction but also accelerates the convergence. In experiments we show that the proposed approach achieves 1) implicit human geometry and appearance reconstruction with high-quality details, 2) photo-realistic rendering of the human from novel views, and 3) animation of the human with novel poses.

cs.CV↗

arXiv2021

Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoders

Generating conversational gestures from speech audio is challenging due to the inherent one-to-many mapping between audio and body motions. Conventional CNNs/RNNs assume one-to-one mapping, and thus tend to predict the average of all possible target motions, resulting in plain/boring motions during inference. In order to overcome this problem, we propose a novel conditional variational autoencoder (VAE) that explicitly models one-to-many audio-to-motion mapping by splitting the cross-modal latent code into shared code and motion-specific code. The shared code mainly models the strong correlation between audio and motion (such as the synchronized audio and motion beats), while the motion-specific code captures diverse motion information independent of the audio. However, splitting the latent code into two parts poses training difficulties for the VAE model. A mapping network facilitating random sampling along with other techniques including relaxed motion loss, bicycle constraint, and diversity loss are designed to better train the VAE. Experiments on both 3D and 2D motion datasets verify that our method generates more realistic and diverse motions than state-of-the-art methods, quantitatively and qualitatively. Finally, we demonstrate that our method can be readily used to generate motion sequences with user-specified motion clips on the timeline. Code and more results are at https://jingli513.github.io/audio2gestures.

cs.CV↗

arXiv2021

High-Fidelity 3D Digital Human Head Creation from RGB-D Selfies

We present a fully automatic system that can produce high-fidelity, photo-realistic 3D digital human heads with a consumer RGB-D selfie camera. The system only needs the user to take a short selfie RGB-D video while rotating his/her head, and can produce a high quality head reconstruction in less than 30 seconds. Our main contribution is a new facial geometry modeling and reflectance synthesis procedure that significantly improves the state-of-the-art. Specifically, given the input video a two-stage frame selection procedure is first employed to select a few high-quality frames for reconstruction. Then a differentiable renderer based 3D Morphable Model (3DMM) fitting algorithm is applied to recover facial geometries from multiview RGB-D data, which takes advantages of a powerful 3DMM basis constructed with extensive data generation and perturbation. Our 3DMM has much larger expressive capacities than conventional 3DMM, allowing us to recover more accurate facial geometry using merely linear basis. For reflectance synthesis, we present a hybrid approach that combines parametric fitting and CNNs to synthesize high-resolution albedo/normal maps with realistic hair/pore/wrinkle details. Results show that our system can produce faithful 3D digital human faces with extremely realistic details. The main code and the newly constructed 3DMM basis is publicly available.

cs.CV↗

arXiv2021

Alignments of Triad Phases in 1D Burgers and 3D Navier-Stokes Flows

The goal of this study is to analyze the fine structure of nonlinear modal interactions in different 1D Burgers and 3D Navier-Stokes flows. This analysis is focused on preferential alignments characterizing the phases of Fourier modes participating in triadic interactions, which are key to determining the nature of energy fluxes between different scales. We develop novel diagnostic tools designed to probe the level of coherence among triadic interactions realizing different flow scenarios. We consider extreme 1D viscous Burgers flows and 3D Navier-Stokes flows which are complemented by singularity-forming inviscid Burgers flows as well as viscous Burgers flows and Navier-Stokes flows corresponding to generic turbulent and simple unimodal initial data, such as the Taylor-Green vortex. The main finding is that while the extreme viscous Burgers and Navier-Stokes flows reveal the same relative level of enstrophy amplification by nonlinear effects, this behaviour is realized via modal interactions with vastly different levels of coherence. In the viscous Burgers flows the flux-carrying triads have phase values which saturate the nonlinearity thereby maximizing the energy flux towards small scales. On the other hand, in 3D Navier-Stokes flows with the extreme initial data the energy flux to small scales is realized by a very small subset of helical triads. The second main finding concerns the role of initial coherence. Comparison of the flows resulting from the extreme and generic initial conditions shows striking similarities between these two types of flows, for the 1D viscous Burgers equation as well as the 3D Navier-Stokes equation.

physics.flu-dyn↗

arXiv2021

Model-based 3D Hand Reconstruction via Self-Supervised Learning

Reconstructing a 3D hand from a single-view RGB image is challenging due to various hand configurations and depth ambiguity. To reliably reconstruct a 3D hand from a monocular image, most state-of-the-art methods heavily rely on 3D annotations at the training stage, but obtaining 3D annotations is expensive. To alleviate reliance on labeled training data, we propose S2HAND, a self-supervised 3D hand reconstruction network that can jointly estimate pose, shape, texture, and the camera viewpoint. Specifically, we obtain geometric cues from the input image through easily accessible 2D detected keypoints. To learn an accurate hand reconstruction model from these noisy geometric cues, we utilize the consistency between 2D and 3D representations and propose a set of novel losses to rationalize outputs of the neural network. For the first time, we demonstrate the feasibility of training an accurate 3D hand reconstruction network without relying on manual annotations. Our experiments show that the proposed method achieves comparable performance with recent fully-supervised methods while using fewer supervision data.

cs.CV↗

arXiv2021

Joint Hand-object 3D Reconstruction from a Single Image with Cross-branch Feature Fusion

Accurate 3D reconstruction of the hand and object shape from a hand-object image is important for understanding human-object interaction as well as human daily activities. Different from bare hand pose estimation, hand-object interaction poses a strong constraint on both the hand and its manipulated object, which suggests that hand configuration may be crucial contextual information for the object, and vice versa. However, current approaches address this task by training a two-branch network to reconstruct the hand and object separately with little communication between the two branches. In this work, we propose to consider hand and object jointly in feature space and explore the reciprocity of the two branches. We extensively investigate cross-branch feature fusion architectures with MLP or LSTM units. Among the investigated architectures, a variant with LSTM units that enhances object feature with hand feature shows the best performance gain. Moreover, we employ an auxiliary depth estimation module to augment the input RGB image with the estimated depth map, which further improves the reconstruction accuracy. Experiments conducted on public datasets demonstrate that our approach significantly outperforms existing approaches in terms of the reconstruction accuracy of objects.

cs.CV↗

arXiv2020

Pricing the zero-coupon bond of the extended Cox-Ingersoll-Ross model using Malliavin calculus

In this paper, we price the zero-coupon bond of the extended Cox-Ingersoll-Ross model by a Dyson type formula established in one of the authors' paper Jin, Peng and Schelllhorn (2016) using Malliavin calculus. This formula provides a fast convergent series to represent the bond price, and it depends on the given drift and volatility of the interest rate process but not the instantaneous forward rate used in Maghsoodi (1996). This expression can be also regarded as a new solution to a class of Reccati equations with time-dependent coefficients.

math.PR↗

arXiv2020

Maximum Amplification of Enstrophy in 3D Navier-Stokes Flows

This investigation concerns a systematic search for potentially singular behavior in 3D Navier-Stokes flows. Enstrophy serves as a convenient indicator of the regularity of solutions to the Navier Stokes system --- as long as this quantity remains finite, the solutions are guaranteed to be smooth and satisfy the equations in the classical (pointwise) sense. However, there are no estimates available with finite a priori bounds on the growth of enstrophy and hence the regularity problem for the 3D Navier-Stokes system remains open. In order to quantify the maximum possible growth of enstrophy, we consider a family of PDE optimization problems in which initial conditions with prescribed enstrophy $\mathcal{E}_0$ are sought such that the enstrophy in the resulting Navier-Stokes flow is maximized at some time $T$. Such problems are solved computationally using a large-scale adjoint-based gradient approach derived in the continuous setting. By solving these problems for a broad range of values of $\mathcal{E}_0$ and $T$, we demonstrate that the maximum growth of enstrophy is in fact finite and scales in proportion to $\mathcal{E}_0^{3/2}$ as $\mathcal{E}_0$ becomes large. Thus, in such worst-case scenario the enstrophy still remains bounded for all times and there is no evidence for formation of singularity in finite time. We also analyze properties of the Navier-Stokes flows leading to the extreme enstrophy values and show that this behavior is realized by a series of vortex reconnection events.

physics.flu-dyn↗

arXiv2018

Crowd Counting by Adaptively Fusing Predictions from an Image Pyramid

Because of the powerful learning capability of deep neural networks, counting performance via density map estimation has improved significantly during the past several years. However, it is still very challenging due to severe occlusion, large scale variations, and perspective distortion. Scale variations (from image to image) coupled with perspective distortion (within one image) result in huge scale changes of the object size. Earlier methods based on convolutional neural networks (CNN) typically did not handle this scale variation explicitly, until Hydra-CNN and MCNN. MCNN uses three columns, each with different filter sizes, to extract features at different scales. In this paper, in contrast to using filters of different sizes, we utilize an image pyramid to deal with scale variations. It is more effective and efficient to resize the input fed into the network, as compared to using larger filter sizes. Secondly, we adaptively fuse the predictions from different scales (using adaptively changing per-pixel weights), which makes our method adapt to scale changes within an image. The adaptive fusing is achieved by generating an across-scale attention map, which softly selects a suitable scale for each pixel, followed by a 1x1 convolution. Extensive experiments on three popular datasets show very compelling results.

cs.CV↗

arXiv2018

Beyond Counting: Comparisons of Density Maps for Crowd Analysis Tasks - Counting, Detection, and Tracking

For crowded scenes, the accuracy of object-based computer vision methods declines when the images are low-resolution and objects have severe occlusions. Taking counting methods for example, almost all the recent state-of-the-art counting methods bypass explicit detection and adopt regression-based methods to directly count the objects of interest. Among regression-based methods, density map estimation, where the number of objects inside a subregion is the integral of the density map over that subregion, is especially promising because it preserves spatial information, which makes it useful for both counting and localization (detection and tracking). With the power of deep convolutional neural networks (CNNs) the counting performance has improved steadily. The goal of this paper is to evaluate density maps generated by density estimation methods on a variety of crowd analysis tasks, including counting, detection, and tracking. Most existing CNN methods produce density maps with resolution that is smaller than the original images, due to the downsample strides in the convolution/pooling operations. To produce an original-resolution density map, we also evaluate a classical CNN that uses a sliding window regressor to predict the density for every pixel in the image. We also consider a fully convolutional (FCNN) adaptation, with skip connections from lower convolutional layers to compensate for loss in spatial information during upsampling. In our experiments, we found that the lower-resolution density maps sometimes have better counting performance. In contrast, the original-resolution density maps improved localization tasks, such as detection and tracking, compared to bilinear upsampling the lower-resolution density maps. Finally, we also propose several metrics for measuring the quality of a density map, and relate them to experiment results on counting and localization.

cs.CV↗

arXiv2017

Marangoni effects on a thin liquid film coating a sphere with axial or radial thermal gradients

We study the time evolution of a thin liquid film coating the outer surface of a sphere in the presence of gravity, surface tension and thermal gradients. We derive the fourth-order nonlinear partial differential equation that models the thin film dynamics, including Marangoni terms arising from the dependence of surface tension on temperature. We consider two different imposed temperature distributions with axial or radial thermal gradients. We analyze the stability of a uniform coating under small perturbations and carry out numerical simulations in COMSOL for a range of parameter values. In the case of an axial temperature gradient, we find steady states with either uniform film thickness, or with the fluid accumulating at the bottom or near the top of the sphere, depending on the total volume of liquid in the film, dictating whether gravity or Marangoni effects dominate. In the case of a radial temperature gradient, a stability analysis reveals the most unstable non-axisymmetric modes on an initially uniform coating film.

cond-mat.soft↗

arXiv2017

Weak Solution of a Doubly Degenerate Parabolic Equation

This paper studies a fourth-order, nonlinear, doubly-degenerate parabolic equation derived from the thin film equation in spherical geometry. A regularization method is used to study the equation and several useful estimates are obtained. The main result of this paper is to prove the existence of a weak solution of the equation in a weighted Sobolev space.

math.AP↗

arXiv2016

Crowd Counting by Adapting Convolutional Neural Networks with Side Information

Computer vision tasks often have side information available that is helpful to solve the task. For example, for crowd counting, the camera perspective (e.g., camera angle and height) gives a clue about the appearance and scale of people in the scene. While side information has been shown to be useful for counting systems using traditional hand-crafted features, it has not been fully utilized in counting systems based on deep learning. In order to incorporate the available side information, we propose an adaptive convolutional neural network (ACNN), where the convolutional filter weights adapt to the current scene context via the side information. In particular, we model the filter weights as a low-dimensional manifold, parametrized by the side information, within the high-dimensional space of filter weights. With the help of side information and adaptive weights, the ACNN can disentangle the variations related to the side information, and extract discriminative features related to the current context. Since existing crowd counting datasets do not contain ground-truth side information, we collect a new dataset with the ground-truth camera angle and height as the side information. On experiments in crowd counting, the ACNN improves counting accuracy compared to a plain CNN with a similar number of parameters. We also apply ACNN to image deconvolution to show its potential effectiveness on other computer vision applications.

cs.CV↗