arXiv subjects

Xun Cao

Publications and source records attributed to Xun Cao.

At least 109 records · Page 6Linked to original sources

arXiv2021

Live Speech Portraits: Real-Time Photorealistic Talking-Head Animation

To the best of our knowledge, we first present a live system that generates personalized photorealistic talking-head animation only driven by audio signals at over 30 fps. Our system contains three stages. The first stage is a deep neural network that extracts deep audio features along with a manifold projection to project the features to the target person's speech space. In the second stage, we learn facial dynamics and motions from the projected audio features. The predicted motions include head poses and upper body motions, where the former is generated by an autoregressive probabilistic model which models the head pose distribution of the target person. Upper body motions are deduced from head poses. In the final stage, we generate conditional feature maps from previous predictions and send them with a candidate image set to an image-to-image translation network to synthesize photorealistic renderings. Our method generalizes well to wild audio and successfully synthesizes high-fidelity personalized facial details, e.g., wrinkles, teeth. Our method also allows explicit control of head poses. Extensive qualitative and quantitative evaluations, along with user studies, demonstrate the superiority of our method over state-of-the-art techniques.

cs.GR↗

arXiv2021

Detailed Avatar Recovery from Single Image

This paper presents a novel framework to recover \emph{detailed} avatar from a single image. It is a challenging task due to factors such as variations in human shapes, body poses, texture, and viewpoints. Prior methods typically attempt to recover the human body shape using a parametric-based template that lacks the surface details. As such resulting body shape appears to be without clothing. In this paper, we propose a novel learning-based framework that combines the robustness of the parametric model with the flexibility of free-form 3D deformation. We use the deep neural networks to refine the 3D shape in a Hierarchical Mesh Deformation (HMD) framework, utilizing the constraints from body joints, silhouettes, and per-pixel shading information. Our method can restore detailed human body shapes with complete textures beyond skinned models. Experiments demonstrate that our method has outperformed previous state-of-the-art approaches, achieving better accuracy in terms of both 2D IoU number and 3D metric distance.

cs.CV↗

arXiv2021

End-to-end Neural Video Coding Using a Compound Spatiotemporal Representation

Recent years have witnessed rapid advances in learnt video coding. Most algorithms have solely relied on the vector-based motion representation and resampling (e.g., optical flow based bilinear sampling) for exploiting the inter frame redundancy. In spite of the great success of adaptive kernel-based resampling (e.g., adaptive convolutions and deformable convolutions) in video prediction for uncompressed videos, integrating such approaches with rate-distortion optimization for inter frame coding has been less successful. Recognizing that each resampling solution offers unique advantages in regions with different motion and texture characteristics, we propose a hybrid motion compensation (HMC) method that adaptively combines the predictions generated by these two approaches. Specifically, we generate a compound spatiotemporal representation (CSTR) through a recurrent information aggregation (RIA) module using information from the current and multiple past frames. We further design a one-to-many decoder pipeline to generate multiple predictions from the CSTR, including vector-based resampling, adaptive kernel-based resampling, compensation mode selection maps and texture enhancements, and combines them adaptively to achieve more accurate inter prediction. Experiments show that our proposed inter coding system can provide better motion-compensated prediction and is more robust to occlusions and complex motions. Together with jointly trained intra coder and residual coder, the overall learnt hybrid coder yields the state-of-the-art coding efficiency in low-delay scenario, compared to the traditional H.264/AVC and H.265/HEVC, as well as recently published learning-based methods, in terms of both PSNR and MS-SSIM metrics.

eess.IV↗

arXiv2021

Audio-Driven Emotional Video Portraits

Despite previous success in generating audio-driven talking heads, most of the previous studies focus on the correlation between speech content and the mouth shape. Facial emotion, which is one of the most important features on natural human faces, is always neglected in their methods. In this work, we present Emotional Video Portraits (EVP), a system for synthesizing high-quality video portraits with vivid emotional dynamics driven by audios. Specifically, we propose the Cross-Reconstructed Emotion Disentanglement technique to decompose speech into two decoupled spaces, i.e., a duration-independent emotion space and a duration dependent content space. With the disentangled features, dynamic 2D emotional facial landmarks can be deduced. Then we propose the Target-Adaptive Face Synthesis technique to generate the final high-quality video portraits, by bridging the gap between the deduced landmarks and the natural head poses of target videos. Extensive experiments demonstrate the effectiveness of our method both qualitatively and quantitatively.

cs.CV↗

arXiv2020

Improving Multispectral Pedestrian Detection by Addressing Modality Imbalance Problems

Multispectral pedestrian detection is capable of adapting to insufficient illumination conditions by leveraging color-thermal modalities. On the other hand, it is still lacking of in-depth insights on how to fuse the two modalities effectively. Compared with traditional pedestrian detection, we find multispectral pedestrian detection suffers from modality imbalance problems which will hinder the optimization process of dual-modality network and depress the performance of detector. Inspired by this observation, we propose Modality Balance Network (MBNet) which facilitates the optimization process in a much more flexible and balanced manner. Firstly, we design a novel Differential Modality Aware Fusion (DMAF) module to make the two modalities complement each other. Secondly, an illumination aware feature alignment module selects complementary features according to the illumination conditions and aligns the two modality features adaptively. Extensive experimental results demonstrate MBNet outperforms the state-of-the-arts on both the challenging KAIST and CVC-14 multispectral pedestrian datasets in terms of the accuracy and the computational efficiency. Code is available at https://github.com/CalayZhou/MBNet.

cs.CV↗

arXiv2020

Neural Video Coding using Multiscale Motion Compensation and Spatiotemporal Context Model

Over the past two decades, traditional block-based video coding has made remarkable progress and spawned a series of well-known standards such as MPEG-4, H.264/AVC and H.265/HEVC. On the other hand, deep neural networks (DNNs) have shown their powerful capacity for visual content understanding, feature extraction and compact representation. Some previous works have explored the learnt video coding algorithms in an end-to-end manner, which show the great potential compared with traditional methods. In this paper, we propose an end-to-end deep neural video coding framework (NVC), which uses variational autoencoders (VAEs) with joint spatial and temporal prior aggregation (PA) to exploit the correlations in intra-frame pixels, inter-frame motions and inter-frame compensation residuals, respectively. Novel features of NVC include: 1) To estimate and compensate motion over a large range of magnitudes, we propose an unsupervised multiscale motion compensation network (MS-MCN) together with a pyramid decoder in the VAE for coding motion features that generates multiscale flow fields, 2) we design a novel adaptive spatiotemporal context model for efficient entropy coding for motion information, 3) we adopt nonlocal attention modules (NLAM) at the bottlenecks of the VAEs for implicit adaptive feature extraction and activation, leveraging its high transformation capacity and unequal weighting with joint global and local information, and 4) we introduce multi-module optimization and a multi-frame training strategy to minimize the temporal error propagation among P-frames. NVC is evaluated for the low-delay causal settings and compared with H.265/HEVC, H.264/AVC and the other learnt video compression methods following the common test conditions, demonstrating consistent gains across all popular test sequences for both PSNR and MS-SSIM distortion metrics.

eess.IV↗

arXiv2020

FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction

In this paper, we present a large-scale detailed 3D face dataset, FaceScape, and propose a novel algorithm that is able to predict elaborate riggable 3D face models from a single image input. FaceScape dataset provides 18,760 textured 3D faces, captured from 938 subjects and each with 20 specific expressions. The 3D models contain the pore-level facial geometry that is also processed to be topologically uniformed. These fine 3D facial models can be represented as a 3D morphable model for rough shapes and displacement maps for detailed geometry. Taking advantage of the large-scale and high-accuracy dataset, a novel algorithm is further proposed to learn the expression-specific dynamic details using a deep neural network. The learned relationship serves as the foundation of our 3D face prediction system from a single image input. Different than the previous methods, our predicted 3D models are riggable with highly detailed geometry under different expressions. The unprecedented dataset and code will be released to public for research purpose.

cs.CV↗

arXiv2020

Hyperspectral City V1.0 Dataset and Benchmark

This document introduces the background and the usage of the Hyperspectral City Dataset and the benchmark. The documentation first starts with the background and motivation of the dataset. Follow it, we briefly describe the method of collecting the dataset and the processing method from raw dataset to the final release dataset, specifically, the version 1.0. We also provide the detailed usage of the dataset and the evaluation metric for submitted the result for the 2019 Hyperspectral City Challenge.

cs.CV↗

arXiv2019

Phase-controllable growth of ultrathin 2D magnetic FeTe crystals

Two-dimensional (2D) magnets with intrinsic ferromagnetic/antiferromagnetic (FM/AFM) ordering are highly desirable for future spintronics devices. However, the synthesis of 2D magnetic crystals, especially the direct growth on SiO2/Si substrate, is just in its infancy. Here, we report a chemical vapor deposition (CVD)-based rational growth approach for the synthesis of ultrathin FeTe crystals with controlled structural and magnetic phases. By precisely optimizing the growth temperature (Tgrowth), FeTe nanoplates with either layered tetragonal or non-layered hexagonal phase can be controlled with high-quality. The two controllable phases lead to square and triangular morphologies with a thickness down to 3.6 and 2.8 nm, respectively. More importantly, transport measurements reveal that tetragonal FeTe is antiferromagnetic with a Neel temperature (TN) about 71.8 K, while hexagonal FeTe is ferromagnetic with a Curie temperature (TC) around 220 K. Theoretical calculations indicate that the ferromagnetic order in hexagonal FeTe is originated from a concomitant lattice distortion and the spin-lattice coupling. This study represents a major step forward in the CVD growth of 2D magnetic materials on SiO2/Si substrates and highlights on their potential applications in the future spintronic devices.

cond-mat.mtrl-sci↗

arXiv2019

Drone-based all-weather entanglement distribution

The quantum satellite is a cornerstone towards practical free-space quantum network and overcomes the photon loss over large distance. However, challenges still exist including real-time all-location coverage and multi-node construction, which may be complemented by the diversity of modern drones. Here we demonstrate the first drone-based entanglement distribution at all-weather conditions over 200 meters (test field limited), and the Clauser-Horne-Shimony-Holt S-parameter exceeds 2.49, within 35 kg take-off weight. With symmetric transmitter and receiver beam apertures and single-mode-fiber-coupling technology, such progress is ready for future quantum network with multi-node expansion. This network can be further integrated in picture-drone sizes for plug-and-play local-area coverage, or loaded onto high-altitude drones for wide-area coverage, which adds flexibility while connecting to the existing satellites and ground fiber-based quantum network.

quant-ph↗

arXiv2019

Detailed Human Shape Estimation from a Single Image by Hierarchical Mesh Deformation

This paper presents a novel framework to recover detailed human body shapes from a single image. It is a challenging task due to factors such as variations in human shapes, body poses, and viewpoints. Prior methods typically attempt to recover the human body shape using a parametric based template that lacks the surface details. As such the resulting body shape appears to be without clothing. In this paper, we propose a novel learning-based framework that combines the robustness of parametric model with the flexibility of free-form 3D deformation. We use the deep neural networks to refine the 3D shape in a Hierarchical Mesh Deformation (HMD) framework, utilizing the constraints from body joints, silhouettes, and per-pixel shading information. We are able to restore detailed human body shapes beyond skinned models. Experiments demonstrate that our method has outperformed previous state-of-the-art approaches, achieving better accuracy in terms of both 2D IoU number and 3D metric distance. The code is available in https://github.com/zhuhao-nju/hmd.git

cs.CV↗

arXiv2019

Non-local Attention Optimized Deep Image Compression

This paper proposes a novel Non-Local Attention Optimized Deep Image Compression (NLAIC) framework, which is built on top of the popular variational auto-encoder (VAE) structure. Our NLAIC framework embeds non-local operations in the encoders and decoders for both image and latent feature probability information (known as hyperprior) to capture both local and global correlations, and apply attention mechanism to generate masks that are used to weigh the features for the image and hyperprior, which implicitly adapt bit allocation for different features based on their importance. Furthermore, both hyperpriors and spatial-channel neighbors of the latent features are used to improve entropy coding. The proposed model outperforms the existing methods on Kodak dataset, including learned (e.g., Balle2019, Balle2018) and conventional (e.g., BPG, JPEG2000, JPEG) image compression methods, for both PSNR and MS-SSIM distortion metrics.

eess.IV↗

arXiv2019

Sputtered Spontaneously Nano-porous VO2-based Films via PTFE Self-Template: Localized Surface Plasmon Resonance Induced Robust Optical Performance for Solar Glazing Application

The PTFE (Teflon) has been selected as the self-template structural material in preparation of VO2 films using reactive magnetron sputtering systems and post annealing progress. Spontaneous random nano-porous structures of VO2 films growing on quartz glasses have been deliberately established via bottom-up processing through this novel and facile approach. The nano-porous VO2 films exhibit an excellent optical performance based on localized surface plasmon resonance (LSPR), with ultrahigh luminous transmittance (Tlum) up to 78.0% and the promoted solar modulation ability (ΔTsol) of 14.1%. Meanwhile, the ingenious microstructure of film provides an antireflection function from multiple perspectives in visible light, with the potential of the windshield on vehicles for smart solar modulation. The nano-porous films expand the practical application of thermochromic VO2 to a fire-new field, breaking the optical performance envelope of single-layer dense VO2 film away, and offering a universal method to prepare homogeneous nano-porous structures for thin films.

physics.app-ph↗

arXiv2019

Towards Commercializing Vanadium Dioxide Films: Investigation of the Impact of Different Interface on the Deterioration Process for Largely Extended Service Life

Long term stability is the most pressing issue that impedes commercialization of Vanadium Dioxide (VO2) based functional films, which show a gradual loss of relative phase transition performance, especially in humid conditions when serving as smart windows. Here, we investigated the impact of different interface on the deterioration process of VO2 films and proposed a novel encapsulation structure for largely extended service life. Hydrophobic and stable hafnium dioxide (HfO2) layers have been incorporated with VO2 films for encapsulated surfaces and cross sections. With modified thickness and structure of HfO2 layers, the degradation process of VO2 can be effectively suppressed. The proposed films can retain stable phase transition performances under high relative humidity (90%) and temperature (60 Celsius) over 100 days, which is equal to about 16 years in the real environment. Improving the stability of VO2 materials is a necessary step towards commercializing production of high performance films for long term use.

physics.app-ph↗

arXiv2019

Electrically Tunable Wafer-Sized Three-Dimensional Topological Insulator Thin Films Grown by Magnetron Sputtering

Three-dimensional (3D) topological insulators (TIs) are candidate materials for various electronic and spintronic devices due to their strong spin-orbit coupling and unique surface electronic structure. Rapid, low-cost preparation of large-area TI thin films compatible with conventional semiconductor technology is key to the practical applications of TIs. Here, we show that wafer-sized Bi2Te3 family TI and magnetic TI films with decent quality and well-controlled composition and properties can be prepared on amorphous SiO2/Si substrates by magnetron cosputtering. The SiO2/Si substrates enable us to electrically tune (Bi1-xSbx)2Te3 and Cr-doped (Bi1-xSbx)2Te3 TI films between p-type and n-type behavior and thus study the phenomena associated with topological surface states, such as the quantum anomalous Hall effect (QAHE). This work significantly facilitates the fabrication of TI-based devices for electronic and spintronic applications.

physics.app-ph↗

arXiv2018

View Extrapolation of Human Body from a Single Image

We study how to synthesize novel views of human body from a single image. Though recent deep learning based methods work well for rigid objects, they often fail on objects with large articulation, like human bodies. The core step of existing methods is to fit a map from the observable views to novel views by CNNs; however, the rich articulation modes of human body make it rather challenging for CNNs to memorize and interpolate the data well. To address the problem, we propose a novel deep learning based pipeline that explicitly estimates and leverages the geometry of the underlying human body. Our new pipeline is a composition of a shape estimation network and an image generation network, and at the interface a perspective transformation is applied to generate a forward flow for pixel value transportation. Our design is able to factor out the space of data variation and makes learning at each step much easier. Empirically, we show that the performance for pose-varying objects can be improved dramatically. Our method can also be applied on real data captured by 3D sensors, and the flow generated by our methods can be used for generating high quality results in higher resolution.

cs.CV↗

arXiv2018

Turing instability and Turing-Hopf bifurcation in diffusive Schnakenberg systems with gene expression time delay

For delayed reaction-diffusion Schnakenberg systems with Neumann boundary conditions, critical conditions for Turing instability are derived, which are necessary and sufficient. And existence conditions for Turing, Hopf and Turing-Hopf bifurcations are established. Normal forms truncated to order 3 at Turing-Hopf singularity of codimension 2, are derived. By investigating Turing-Hopf bifurcation, the parameter regions for the stability of a periodic solution, a pair of spatially inhomogeneous steady states and a pair of spatially inhomogeneous periodic solutions, are derived in $(τ,\varepsilon)$ parameter plane ($τ$ for time delay, $\varepsilon$ for diffusion rate). It is revealed that joint effects of diffusion and delay can lead to the occurrence of mixed spatial and temporal patterns. Moreover, it is also demonstrated that various spatially inhomogeneous patterns with different spatial frequencies can be achieved via changing the diffusion rate. And, the phenomenon that time delay may induce a failure of Turing instability observed by Gaffney and Monk (2006) are theoretically explained.

math.DS↗

arXiv2018

Multispectral Image Intrinsic Decomposition via Low Rank Constraint

Multispectral images contain many clues of surface characteristics of the objects, thus can be widely used in many computer vision tasks, e.g., recolorization and segmentation. However, due to the complex illumination and the geometry structure of natural scenes, the spectra curves of a same surface can look very different. In this paper, a Low Rank Multispectral Image Intrinsic Decomposition model (LRIID) is presented to decompose the shading and reflectance from a single multispectral image. We extend the Retinex model, which is proposed for RGB image intrinsic decomposition, for multispectral domain. Based on this, a low rank constraint is proposed to reduce the ill-posedness of the problem and make the algorithm solvable. A dataset of 12 images is given with the ground truth of shadings and reflectance, so that the objective evaluations can be conducted. The experiments demonstrate the effectiveness of proposed method.

cs.CV↗