arXiv subjects

Hao Zhu

Publications and source records attributed to Hao Zhu.

At least 253 records · Page 14Linked to original sources

arXiv2020

Speech2Video Synthesis with 3D Skeleton Regularization and Expressive Body Poses

In this paper, we propose a novel approach to convert given speech audio to a photo-realistic speaking video of a specific person, where the output video has synchronized, realistic, and expressive rich body dynamics. We achieve this by first generating 3D skeleton movements from the audio sequence using a recurrent neural network (RNN), and then synthesizing the output video via a conditional generative adversarial network (GAN). To make the skeleton movement realistic and expressive, we embed the knowledge of an articulated 3D human skeleton and a learned dictionary of personal speech iconic gestures into the generation process in both learning and testing pipelines. The former prevents the generation of unreasonable body distortion, while the later helps our model quickly learn meaningful body movement through a few recorded videos. To produce photo-realistic and high-resolution video with motion details, we propose to insert part attention mechanisms in the conditional GAN, where each detailed part, e.g. head and hand, is automatically zoomed in to have their own discriminators. To validate our approach, we collect a dataset with 20 high-quality videos from 1 male and 1 female model reading various documents under different topics. Compared with previous SoTA pipelines handling similar tasks, our approach achieves better results by a user study.

cs.CV↗

arXiv2020

The Return of Lexical Dependencies: Neural Lexicalized PCFGs

In this paper we demonstrate that $\textit{context free grammar (CFG) based methods for grammar induction benefit from modeling lexical dependencies}$. This contrasts to the most popular current methods for grammar induction, which focus on discovering $\textit{either}$ constituents $\textit{or}$ dependencies. Previous approaches to marry these two disparate syntactic formalisms (e.g. lexicalized PCFGs) have been plagued by sparsity, making them unsuitable for unsupervised grammar induction. However, in this work, we present novel neural models of lexicalized PCFGs which allow us to overcome sparsity problems and effectively induce both constituents and dependencies within a single model. Experiments demonstrate that this unified framework results in stronger results on both representations than achieved when modeling either formalism alone. Code is available at https://github.com/neulab/neural-lpcfg.

cs.CL↗

arXiv2020

Additively Homomorphical Encryption based Deep Neural Network for Asymmetrically Collaborative Machine Learning

The financial sector presents many opportunities to apply various machine learning techniques. Centralized machine learning creates a constraint which limits further applications in finance sectors. Data privacy is a fundamental challenge for a variety of finance and insurance applications that account on learning a model across different sections. In this paper, we define a new practical scheme of collaborative machine learning that one party owns data, but another party owns labels only, and term this \textbf{Asymmetrically Collaborative Machine Learning}. For this scheme, we propose a novel privacy-preserving architecture where two parties can collaboratively train a deep learning model efficiently while preserving the privacy of each party's data. More specifically, we decompose the forward propagation and backpropagation of the neural network into four different steps and propose a novel protocol to handle information leakage in these steps. Our extensive experiments on different datasets demonstrate not only stable training without accuracy loss, but also more than 100 times speedup compared with the state-of-the-art system.

cs.LG↗

arXiv2020

Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policies

Policy gradient (PG) methods are a widely used reinforcement learning methodology in many applications such as video games, autonomous driving, and robotics. In spite of its empirical success, a rigorous understanding of the global convergence of PG methods is lacking in the literature. In this work, we close the gap by viewing PG methods from a nonconvex optimization perspective. In particular, we propose a new variant of PG methods for infinite-horizon problems that uses a random rollout horizon for the Monte-Carlo estimation of the policy gradient. This method then yields an unbiased estimate of the policy gradient with bounded variance, which enables the tools from nonconvex optimization to be applied to establish global convergence. Employing this perspective, we first recover the convergence results with rates to the stationary-point policies in the literature. More interestingly, motivated by advances in nonconvex optimization, we modify the proposed PG method by introducing periodically enlarged stepsizes. The modified algorithm is shown to escape saddle points under mild assumptions on the reward and the policy parameterization. Under a further strict saddle points assumption, this result establishes convergence to essentially locally-optimal policies of the underlying problem, and thus bridges the gap in existing literature on the convergence of PG methods. Results from experiments on the inverted pendulum are then provided to corroborate our theory, namely, by slightly reshaping the reward function to satisfy our assumption, unfavorable saddle points can be avoided and better limit points can be attained. Intriguingly, this empirical finding justifies the benefit of reward-reshaping from a nonconvex optimization perspective.

math.OC↗

arXiv2020

Arbitrary Talking Face Generation via Attentional Audio-Visual Coherence Learning

Talking face generation aims to synthesize a face video with precise lip synchronization as well as a smooth transition of facial motion over the entire video via the given speech clip and facial image. Most existing methods mainly focus on either disentangling the information in a single image or learning temporal information between frames. However, cross-modality coherence between audio and video information has not been well addressed during synthesis. In this paper, we propose a novel arbitrary talking face generation framework by discovering the audio-visual coherence via the proposed Asymmetric Mutual Information Estimator (AMIE). In addition, we propose a Dynamic Attention (DA) block by selectively focusing the lip area of the input image during the training stage, to further enhance lip synchronization. Experimental results on benchmark LRW dataset and GRID dataset transcend the state-of-the-art methods on prevalent metrics with robust high-resolution synthesizing on gender and pose variations.

cs.CV↗

arXiv2020

Self-Supervised Human Depth Estimation from Monocular Videos

Previous methods on estimating detailed human depth often require supervised training with `ground truth' depth data. This paper presents a self-supervised method that can be trained on YouTube videos without known depth, which makes training data collection simple and improves the generalization of the learned network. The self-supervised learning is achieved by minimizing a photo-consistency loss, which is evaluated between a video frame and its neighboring frames warped according to the estimated depth and the 3D non-rigid motion of the human body. To solve this non-rigid motion, we first estimate a rough SMPL model at each video frame and compute the non-rigid body motion accordingly, which enables self-supervised learning on estimating the shape details. Experiments demonstrate that our method enjoys better generalization and performs much better on data in the wild.

cs.CV↗

arXiv2020

FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction

In this paper, we present a large-scale detailed 3D face dataset, FaceScape, and propose a novel algorithm that is able to predict elaborate riggable 3D face models from a single image input. FaceScape dataset provides 18,760 textured 3D faces, captured from 938 subjects and each with 20 specific expressions. The 3D models contain the pore-level facial geometry that is also processed to be topologically uniformed. These fine 3D facial models can be represented as a 3D morphable model for rough shapes and displacement maps for detailed geometry. Taking advantage of the large-scale and high-accuracy dataset, a novel algorithm is further proposed to learn the expression-specific dynamic details using a deep neural network. The learned relationship serves as the foundation of our 3D face prediction system from a single image input. Different than the previous methods, our predicted 3D models are riggable with highly detailed geometry under different expressions. The unprecedented dataset and code will be released to public for research purpose.

cs.CV↗

arXiv2020

SAPIEN: A SimulAted Part-based Interactive ENvironment

Building home assistant robots has long been a pursuit for vision and robotics researchers. To achieve this task, a simulated environment with physically realistic simulation, sufficient articulated objects, and transferability to the real robot is indispensable. Existing environments achieve these requirements for robotics simulation with different levels of simplification and focus. We take one step further in constructing an environment that supports household tasks for training robot learning algorithm. Our work, SAPIEN, is a realistic and physics-rich simulated environment that hosts a large-scale set for articulated objects. Our SAPIEN enables various robotic vision and interaction tasks that require detailed part-level understanding.We evaluate state-of-the-art vision algorithms for part detection and motion attribute recognition as well as demonstrate robotic interaction tasks using heuristic approaches and reinforcement learning algorithms. We hope that our SAPIEN can open a lot of research directions yet to be explored, including learning cognition through interaction, part motion discovery, and construction of robotics-ready simulated game environment.

cs.CV↗

arXiv2020

Dynamics of continuous maps induced on the space of probability measures

For a continuous self-map $f$ on a compact interval $I$ and the induced map $\hat f$ on the space $\mathcal{M}(I)$ of probability measures, we obtain a sharp condition to guarantee that $(I,f)$ is transitive if and only if $(\mathcal{M}(I),\hat f)$ is transitive. We also show that the sensitivity of $(I,f)$ is equivalent to that of $(\mathcal{M}(I),\hat f)$. We prove that $(\mathcal{M}(I),\hat f)$ must have infinite topological entropy for any transitive system $(I,f)$, while there exists a transitive non-autonomous system $(I,f_{0,\infty})$ such that $(\mathcal{M}(I),\hat f_{0,\infty})$ has zero topological entropy, where $f_{0,\infty}=\{f_n\}_{n=0}^\infty$ is a sequence of continuous self-maps on $I$. For a continuous self-map $f$ on a general compact metric space $X$, we show that chain transitivity of $(X, f)$ implies chain mixing of $(\mathcal{M}(X),\hat f)$, and we provide two counterexamples to demonstrate that the converse is not true. We confirm that shadowing of $(X,f)$ is not inherited by $(\mathcal{M}(X),\hat f)$ in general. For a non-autonomous system $(X,f_{0,\infty})$, we prove that if $(\mathcal{M}(X),\hat{f}_{0,\infty})$ is weak mixing of order $n$, then so is $(X,f_{0,\infty})$ for any $n\geq2$; while there exists $(X,f_{0,\infty})$ such that it is weak mixing of order $2$ but $(\mathcal{M} (X),\hat{f}_{0,\infty})$ is not. We then prove that Li-Yorke chaos (resp., distributional chaos) of $(X,f_{0,\infty})$ carries over to $(\mathcal{M}(X),\hat f_{0,\infty})$, and give an example to show that $(X,f)$ and $(\mathcal{M}(X),\hat f)$ may have no Li-Yorke pair simultaneously. We also prove that if $f_n$ is surjective for all $n\geq 0$, then chain mixing of $(\mathcal{M}(X),\hat f_{0,\infty})$ always holds true, and shadowing of $(\mathcal{M}(X),\hat f_{0,\infty})$ implies mixing of $(X, f_{0,\infty})$.

math.DS↗

arXiv2020

Transmission Switching Under Wind Uncertainty Using Linear Decision Rules

Increasing penetration of wind and renewable generation poses significant challenges to the power system operations and reliability. This paper considers the real-time optimal transmission switching (OTS) problem for determining the generation dispatch and network topology that can account for uncertain energy resources. To efficiently solve the resultant two-stage stochastic program, we propose a tractable linear decision rule (LDR) based approximation solution that can eliminate the uncertainty variables and lead to fixed number of constraints. The proposed LDR approach can guarantee feasibility, and significantly reduces the computational complexity of existing approaches that grows with the number of randomly generated samples of uncertainty. Numerical studies on IEEE test cases demonstrate the high approximation accuracy of the proposed LDR solution and its computational efficiency for real-time OTS implementations.

math.OC↗

arXiv2020

Deep Audio-Visual Learning: A Survey

Audio-visual learning, aimed at exploiting the relationship between audio and visual modalities, has drawn considerable attention since deep learning started to be used successfully. Researchers tend to leverage these two modalities either to improve the performance of previously considered single-modality tasks or to address new challenging problems. In this paper, we provide a comprehensive survey of recent audio-visual learning development. We divide the current audio-visual learning tasks into four different subfields: audio-visual separation and localization, audio-visual correspondence learning, audio-visual generation, and audio-visual representation learning. State-of-the-art methods as well as the remaining challenges of each subfield are further discussed. Finally, we summarize the commonly used datasets and performance metrics.

cs.CV↗

arXiv2019

A Dual-gate MoS2 Photodetector Based on Interface Coupling Effect

Two-dimensional (2D) transition metal dichalcogenides (TMDs) based photodetectors have shown great potential for the next generation optoelectronics. However, most of the reported MoS2 photodetectors function under the photogating effect originated from the charge-trap mechanism, which is difficult for quantitative control. Such devices generally suffer from a poor compromise between response speed and responsivity (R) and large dark current. Here, a dual-gated (DG) MoS2 phototransistor operating based on the interface coupling effect (ICE) is demonstrated. By simultaneously applying a negative top-gate voltage (VTG) and positive back-gate voltage (VBG) to the MoS2 channel, the photo-generated holes can be effectively trapped in the depleted region under TG. An ultrahigh R of ~1E5 A/W and detectivity (D*) of ~1E14 Jones have been achieved in several devices with different thickness under Pin of 53 uW/cm2 at VTG=-5 V. Moreover, the response time of the DG phototransistor can also be modulated based on the ICE. Based on these systematic measurements of MoS2 DG phototransistors, the results show that the ICE plays an important role in the modulation of photoelectric performances. Our results also pave the way for the future optoelectrical application of 2D TMDs materials and prompt for further investigation in the DG structured phototransistors.

physics.app-ph↗

arXiv2019

MoS$_2$ Dual-gate Transistors with Electrostatically Doped Contacts

Two-dimensional (2D) transition metal dichalcogenides (TMDs) such as molybdenum disulfide (MoS2) have been intensively investigated because of their exclusive physical properties for advanced electronics and optoelectronics. In the present work, we study the MoS2 transistor based on a novel tri-gate device architecture, with dual-gate (Dual-G) in the channel and the buried side-gate (Side-G) for the source/drain regions. All gates can be independently controlled without interference. For a MoS2 sheet with a thickness of 3.6 nm, the Schottky barrier (SB) and non-overlapped channel region can be effectively tuned by electrostatically doping the source/drain regions with Side-G. Thus, the extrinsic resistance can be effectively lowered, and a boost of the ON-state current can be achieved. Meanwhile, the channel control remains efficient under the Dual-G mode, with an ON-OFF current ratio of 3E7 and subthreshold swing of 83 mV/decade. The corresponding band diagram is also discussed to illustrate the device operation mechanism. This novel device structure opens up a new way toward fabrication of high-performance devices based on 2D-TMDs.

physics.app-ph↗

arXiv2019

S4G: Amodal Single-view Single-Shot SE(3) Grasp Detection in Cluttered Scenes

Grasping is among the most fundamental and long-lasting problems in robotics study. This paper studies the problem of 6-DoF(degree of freedom) grasping by a parallel gripper in a cluttered scene captured using a commodity depth sensor from a single viewpoint. We address the problem in a learning-based framework. At the high level, we rely on a single-shot grasp proposal network, trained with synthetic data and tested in real-world scenarios. Our single-shot neural network architecture can predict amodal grasp proposal efficiently and effectively. Our training data synthesis pipeline can generate scenes of complex object configuration and leverage an innovative gripper contact model to create dense and high-quality grasp annotations. Experiments in synthetic and real environments have demonstrated that the proposed approach can outperform state-of-the-arts by a large margin.

cs.RO↗

arXiv2019

FewRel 2.0: Towards More Challenging Few-Shot Relation Classification

We present FewRel 2.0, a more challenging task to investigate two aspects of few-shot relation classification models: (1) Can they adapt to a new domain with only a handful of instances? (2) Can they detect none-of-the-above (NOTA) relations? To construct FewRel 2.0, we build upon the FewRel dataset (Han et al., 2018) by adding a new test set in a quite different domain, and a NOTA relation choice. With the new dataset and extensive experimental analysis, we found (1) that the state-of-the-art few-shot relation classification models struggle on these two aspects, and (2) that the commonly-used techniques for domain adaptation and NOTA detection still cannot handle the two challenges well. Our research calls for more attention and further efforts to these two real-world issues. All details and resources about the dataset and baselines are released at https: //github.com/thunlp/fewrel.

cs.CL↗

arXiv2019

Enhancing the Spatio-Temporal Observability of Residential Loads

Enhancing the spatio-temporal observability of residential loads is crucial for achieving secure and efficient operations in distribution systems with increasing penetration of distributed energy resources (DERs). This paper presents a joint inference framework for residential loads by leveraging the real-time measurements from distribution-level sensors. Specifically, smart meter data is available for almost every load with unfortunately low temporal resolution, while distribution synchrophasor data is at very fast rates yet available at limited locations. By combining these two types of data with respective strengths, the problem is cast as a matrix recovery one with much less number of observations than unknowns. To improve the recovery performance, we introduce two regularization terms to promote a low rank plus sparse structure of the load matrix via a difference transformation. Accordingly, the recovery problem can be formulated as a convex optimization one which is efficiently solvable. Numerical tests using real residential load data demonstrate the effectiveness of our proposed approaches in identifying appliance activities and recovering the PV output profiles.

eess.SY↗

arXiv2019

Learned Point Cloud Geometry Compression

This paper presents a novel end-to-end Learned Point Cloud Geometry Compression (a.k.a., Learned-PCGC) framework, to efficiently compress the point cloud geometry (PCG) using deep neural networks (DNN) based variational autoencoders (VAE). In our approach, PCG is first voxelized, scaled and partitioned into non-overlapped 3D cubes, which is then fed into stacked 3D convolutions for compact latent feature and hyperprior generation. Hyperpriors are used to improve the conditional probability modeling of latent features. A weighted binary cross-entropy (WBCE) loss is applied in training while an adaptive thresholding is used in inference to remove unnecessary voxels and reduce the distortion. Objectively, our method exceeds the geometry-based point cloud compression (G-PCC) algorithm standardized by well-known Moving Picture Experts Group (MPEG) with a significant performance margin, e.g., at least 60% BD-Rate (Bjontegaard Delta Rate) gains, using common test datasets. Subjectively, our method has presented better visual quality with smoother surface reconstruction and appealing details, in comparison to all existing MPEG standard compliant PCC methods. Our method requires about 2.5MB parameters in total, which is a fairly small size for practical implementation, even on embedded platform. Additional ablation studies analyze a variety of aspects (e.g., cube size, kernels, etc) to explore the application potentials of our learned-PCGC.

cs.CV↗

arXiv2019

Secure Integration of Electric Vehicles with the Power Grid

This paper focuses on the secure integration of distributed energy resources (DERs), especially pluggable electric vehicles (EVs), with the power grid. We consider the vehicle-to-grid (V2G) system where EVs are connected to the power grid through an aggregator. In this paper, we propose a novel Cyber-Physical Anomaly Detection Engine that monitors system behavior and detects anomalies almost instantaneously. This detection engine ensures that the critical power grid component (viz.,aggregator)remains secure by monitoring(a)cyber messages for various state changes and data constraints along with (b)power data on the V2G cyber network using power measurements from sensors on the physical/power distribution network. Since the V2G system is time-sensitive, the anomaly detection engine also monitors the timing requirements of the protocol messages to enhance the safety of the aggregator. To the best of our knowledge, this is the first piece of work that combines(a)the EV charging/discharging protocols, the(b)cyber network and(c)power measurements from physical network to detect intrusions in the EV to power grid system.

eess.SY↗