arXiv subjects

Siwei Zhang

Publications and source records attributed to Siwei Zhang.

43 records · Page 3Linked to original sources

arXiv2021

Learning Motion Priors for 4D Human Body Capture in 3D Scenes

Recovering high-quality 3D human motion in complex scenes from monocular videos is important for many applications, ranging from AR/VR to robotics. However, capturing realistic human-scene interactions, while dealing with occlusions and partial views, is challenging; current approaches are still far from achieving compelling results. We address this problem by proposing LEMO: LEarning human MOtion priors for 4D human body capture. By leveraging the large-scale motion capture dataset AMASS, we introduce a novel motion smoothness prior, which strongly reduces the jitters exhibited by poses recovered over a sequence. Furthermore, to handle contacts and occlusions occurring frequently in body-scene interactions, we design a contact friction term and a contact-aware motion infiller obtained via per-instance self-supervised training. To prove the effectiveness of the proposed motion priors, we combine them into a novel pipeline for 4D human body capture in 3D scenes. With our pipeline, we demonstrate high-quality 4D human body capture, reconstructing smooth motions and physically plausible body-scene interactions. The code and data are available at https://sanweiliti.github.io/LEMO/LEMO.html.

cs.CV↗

arXiv2021

Neural Architecture Search as Sparse Supernet

This paper aims at enlarging the problem of Neural Architecture Search (NAS) from Single-Path and Multi-Path Search to automated Mixed-Path Search. In particular, we model the NAS problem as a sparse supernet using a new continuous architecture representation with a mixture of sparsity constraints. The sparse supernet enables us to automatically achieve sparsely-mixed paths upon a compact set of nodes. To optimize the proposed sparse supernet, we exploit a hierarchical accelerated proximal gradient algorithm within a bi-level optimization framework. Extensive experiments on Convolutional Neural Network and Recurrent Neural Network search demonstrate that the proposed method is capable of searching for compact, general and powerful neural architectures.

cs.CV↗

arXiv2020

Facial Emotion Recognition with Noisy Multi-task Annotations

Human emotions can be inferred from facial expressions. However, the annotations of facial expressions are often highly noisy in common emotion coding models, including categorical and dimensional ones. To reduce human labelling effort on multi-task labels, we introduce a new problem of facial emotion recognition with noisy multi-task annotations. For this new problem, we suggest a formulation from the point of joint distribution match view, which aims at learning more reliable correlations among raw facial images and multi-task labels, resulting in the reduction of noise influence. In our formulation, we exploit a new method to enable the emotion prediction and the joint distribution learning in a unified adversarial learning game. Evaluation throughout extensive experiments studies the real setups of the suggested new problem, as well as the clear superiority of the proposed method over the state-of-the-art competing methods on either the synthetic noisy labeled CIFAR-10 or practical noisy multi-task labeled RAF and AffectNet. The code is available at https://github.com/sanweiliti/noisyFER.

cs.CV↗

arXiv2020

PLACE: Proximity Learning of Articulation and Contact in 3D Environments

High fidelity digital 3D environments have been proposed in recent years, however, it remains extremely challenging to automatically equip such environment with realistic human bodies. Existing work utilizes images, depth or semantic maps to represent the scene, and parametric human models to represent 3D bodies. While being straightforward, their generated human-scene interactions are often lack of naturalness and physical plausibility. Our key observation is that humans interact with the world through body-scene contact. To synthesize realistic human-scene interactions, it is essential to effectively represent the physical contact and proximity between the body and the world. To that end, we propose a novel interaction generation method, named PLACE (Proximity Learning of Articulation and Contact in 3D Environments), which explicitly models the proximity between the human body and the 3D scene around it. Specifically, given a set of basis points on a scene mesh, we leverage a conditional variational autoencoder to synthesize the minimum distances from the basis points to the human body surface. The generated proximal relationship exhibits which region of the scene is in contact with the person. Furthermore, based on such synthesized proximity, we are able to effectively obtain expressive 3D human bodies that interact with the 3D scene naturally. Our perceptual study shows that PLACE significantly improves the state-of-the-art method, approaching the realism of real human-scene interaction. We believe our method makes an important step towards the fully automatic synthesis of realistic 3D human bodies in 3D scenes. The code and model are available for research at https://sanweiliti.github.io/PLACE/PLACE.html.

cs.CV↗

arXiv2019

One-shot Face Reenactment

To enable realistic shape (e.g. pose and expression) transfer, existing face reenactment methods rely on a set of target faces for learning subject-specific traits. However, in real-world scenario end-users often only have one target face at hand, rendering existing methods inapplicable. In this work, we bridge this gap by proposing a novel one-shot face reenactment learning framework. Our key insight is that the one-shot learner should be able to disentangle and compose appearance and shape information for effective modeling. Specifically, the target face appearance and the source face shape are first projected into latent spaces with their corresponding encoders. Then these two latent spaces are associated by learning a shared decoder that aggregates multi-level features to produce the final reenactment results. To further improve the synthesizing quality on mustache and hair regions, we additionally propose FusionNet which combines the strengths of our learned decoder and the traditional warping method. Extensive experiments show that our one-shot face reenactment system achieves superior transfer fidelity as well as identity preserving capability than alternatives. More remarkably, our approach trained with only one target image per subject achieves competitive results to those using a set of target images, demonstrating the practical merit of this work. Code, models and an additional set of reenacted faces have been publicly released at the project page.

cs.CV↗

arXiv2019

On the Potential of Multi-Mode Antennas for Direction-of-Arrival Estimation

In this paper, we show that a multi-mode antenna (MMA) is an interesting alternative to a conventional phased antenna array for direction-of-arrival (DoA) estimation. By MMA we mean a single physical radiator with multiple ports, which excite different characteristic modes. In contrast to phased arrays, a closed-form mathematical model of the antenna response, like a steering vector, is not straightforward to define for MMAs. Instead one has to rely on calibration measurement or electromagnetic field (EMF) simulation data, which is discrete. To perform DoA estimation, array interpolation technique (AIT) and wavefield modeling (WM) are suggested as methods with inherent interpolation capabilities, fully taking antenna nonidealities like mutual coupling into account. We present a non-coherent DoA estimator for low-cost receivers and show how coherent DoA estimation and joint DoA and polarization estimation can be performed with MMAs. Utilizing these methods, we assess the DoA estimation performance of an MMA prototype in simulations for both 2D and 3D cases. The results show that WM outperforms AIT for high SNR. Coherent estimation is superior to non-coherent, especially in 3D, because non-coherent suffers from estimation ambiguities. In conclusion, DoA estimation with a single MMA is feasible and accurate.

eess.SP↗

arXiv2018

Power-Based Direction-of-Arrival Estimation Using a Single Multi-Mode Antenna

Phased antenna arrays are widely used for direction-of-arrival (DoA) estimation. For low-cost applications, signal power or received signal strength indicator (RSSI) based approaches can be an alternative. However, they usually require multiple antennas, a single antenna that can be rotated, or switchable antenna beams. In this paper we show how a multi-mode antenna (MMA) can be used for power-based DoA estimation. Only a single MMA is needed and neither rotation nor switching of antenna beams is required. We derive an estimation scheme as well as theoretical bounds and validate them through simulations. It is found that power-based DoA estimation with an MMA is feasible and accurate.

cs.IT↗