arXiv subjects

Sheng Jin

Publications and source records attributed to Sheng Jin.

At least 91 records · Page 5Linked to original sources

arXiv2020

Whole-Body Human Pose Estimation in the Wild

This paper investigates the task of 2D human whole-body pose estimation, which aims to localize dense landmarks on the entire human body including face, hands, body, and feet. As existing datasets do not have whole-body annotations, previous methods have to assemble different deep models trained independently on different datasets of the human face, hand, and body, struggling with dataset biases and large model complexity. To fill in this blank, we introduce COCO-WholeBody which extends COCO dataset with whole-body annotations. To our best knowledge, it is the first benchmark that has manual annotations on the entire human body, including 133 dense landmarks with 68 on the face, 42 on hands and 23 on the body and feet. A single-network model, named ZoomNet, is devised to take into account the hierarchical structure of the full human body to solve the scale variation of different body parts of the same person. ZoomNet is able to significantly outperform existing methods on the proposed COCO-WholeBody dataset. Extensive experiments show that COCO-WholeBody not only can be used to train deep models from scratch for whole-body pose estimation but also can serve as a powerful pre-training dataset for many different tasks such as facial landmark detection and hand keypoint estimation. The dataset is publicly available at https://github.com/jin-s13/COCO-WholeBody.

cs.CV↗

arXiv2020

Differentiable Hierarchical Graph Grouping for Multi-Person Pose Estimation

Multi-person pose estimation is challenging because it localizes body keypoints for multiple persons simultaneously. Previous methods can be divided into two streams, i.e. top-down and bottom-up methods. The top-down methods localize keypoints after human detection, while the bottom-up methods localize keypoints directly and then cluster/group them for different persons, which are generally more efficient than top-down methods. However, in existing bottom-up methods, the keypoint grouping is usually solved independently from keypoint detection, making them not end-to-end trainable and have sub-optimal performance. In this paper, we investigate a new perspective of human part grouping and reformulate it as a graph clustering task. Especially, we propose a novel differentiable Hierarchical Graph Grouping (HGG) method to learn the graph grouping in bottom-up multi-person pose estimation task. Moreover, HGG is easily embedded into main-stream bottom-up methods. It takes human keypoint candidates as graph nodes and clusters keypoints in a multi-layer graph neural network model. The modules of HGG can be trained end-to-end with the keypoint detection network and is able to supervise the grouping process in a hierarchical manner. To improve the discrimination of the clustering, we add a set of edge discriminators and macro-node discriminators. Extensive experiments on both COCO and OCHuman datasets demonstrate that the proposed method improves the performance of bottom-up pose estimation methods.

cs.CV↗

arXiv2020

RL-Duet: Online Music Accompaniment Generation Using Deep Reinforcement Learning

This paper presents a deep reinforcement learning algorithm for online accompaniment generation, with potential for real-time interactive human-machine duet improvisation. Different from offline music generation and harmonization, online music accompaniment requires the algorithm to respond to human input and generate the machine counterpart in a sequential order. We cast this as a reinforcement learning problem, where the generation agent learns a policy to generate a musical note (action) based on previously generated context (state). The key of this algorithm is the well-functioning reward model. Instead of defining it using music composition rules, we learn this model from monophonic and polyphonic training data. This model considers the compatibility of the machine-generated note with both the machine-generated context and the human-generated context. Experiments show that this algorithm is able to respond to the human part and generate a melodic, harmonic and diverse machine part. Subjective evaluations on preferences show that the proposed algorithm generates music pieces of higher quality than the baseline method.

cs.LG↗

arXiv2019

HoMM: Higher-order Moment Matching for Unsupervised Domain Adaptation

Minimizing the discrepancy of feature distributions between different domains is one of the most promising directions in unsupervised domain adaptation. From the perspective of distribution matching, most existing discrepancy-based methods are designed to match the second-order or lower statistics, which however, have limited expression of statistical characteristic for non-Gaussian distributions. In this work, we explore the benefits of using higher-order statistics (mainly refer to third-order and fourth-order statistics) for domain matching. We propose a Higher-order Moment Matching (HoMM) method, and further extend the HoMM into reproducing kernel Hilbert spaces (RKHS). In particular, our proposed HoMM can perform arbitrary-order moment tensor matching, we show that the first-order HoMM is equivalent to Maximum Mean Discrepancy (MMD) and the second-order HoMM is equivalent to Correlation Alignment (CORAL). Moreover, the third-order and the fourth-order moment tensor matching are expected to perform comprehensive domain alignment as higher-order statistics can approximate more complex, non-Gaussian distributions. Besides, we also exploit the pseudo-labeled target samples to learn discriminative representations in the target domain, which further improves the transfer performance. Extensive experiments are conducted, showing that our proposed HoMM consistently outperforms the existing moment matching methods by a large margin. Codes are available at \url{https://github.com/chenchao666/HoMM-Master}

cs.CV↗

arXiv2019

SSAH: Semi-supervised Adversarial Deep Hashing with Self-paced Hard Sample Generation

Deep hashing methods have been proved to be effective and efficient for large-scale Web media search. The success of these data-driven methods largely depends on collecting sufficient labeled data, which is usually a crucial limitation in practical cases. The current solutions to this issue utilize Generative Adversarial Network (GAN) to augment data in semi-supervised learning. However, existing GAN-based methods treat image generations and hashing learning as two isolated processes, leading to generation ineffectiveness. Besides, most works fail to exploit the semantic information in unlabeled data. In this paper, we propose a novel Semi-supervised Self-pace Adversarial Hashing method, named SSAH to solve the above problems in a unified framework. The SSAH method consists of an adversarial network (A-Net) and a hashing network (H-Net). To improve the quality of generative images, first, the A-Net learns hard samples with multi-scale occlusions and multi-angle rotated deformations which compete against the learning of accurate hashing codes. Second, we design a novel self-paced hard generation policy to gradually increase the hashing difficulty of generated samples. To make use of the semantic information in unlabeled ones, we propose a semi-supervised consistent loss. The experimental results show that our method can significantly improve state-of-the-art models on both the widely-used hashing datasets and fine-grained datasets.

cs.CV↗

arXiv2019

TRB: A Novel Triplet Representation for Understanding 2D Human Body

Human pose and shape are two important components of 2D human body. However, how to efficiently represent both of them in images is still an open question. In this paper, we propose the Triplet Representation for Body (TRB) -- a compact 2D human body representation, with skeleton keypoints capturing human pose information and contour keypoints containing human shape information. TRB not only preserves the flexibility of skeleton keypoint representation, but also contains rich pose and human shape information. Therefore, it promises broader application areas, such as human shape editing and conditional image generation. We further introduce the challenging problem of TRB estimation, where joint learning of human pose and shape is required. We construct several large-scale TRB estimation datasets, based on popular 2D pose datasets: LSP, MPII, COCO. To effectively solve TRB estimation, we propose a two-branch network (TRB-net) with three novel techniques, namely X-structure (Xs), Directional Convolution (DC) and Pairwise Mapping (PM), to enforce multi-level message passing for joint feature learning. We evaluate our proposed TRB-net and several leading approaches on our proposed TRB datasets, and demonstrate the superiority of our method through extensive evaluations.

cs.CV↗

arXiv2019

New constraints on the dust and gas distribution in the LkCa 15 disk from ALMA

We search a large parameter space of the LkCa 15's disk density profile to fit its observed radial intensity profile of $^{12}$CO (J = 3-2) obtained from ALMA. The best-fit model within the parameter space has a disk mass of 0.1 $M_{\odot}$ (using an abundance ratio of $^{12}$CO/H$_2$ $=$ 1.4 $\times 10^{-4}$ in mass), an inner cavity of 45 AU in radius, an outer edge at $\sim$ 600 AU, and a disk surface density profile follows a power-law of the form $ρ_r \propto r^{-4}$. For the disk density profiles that can lead to a small reduced $χ^2$ of goodness-of-fit, we find that there is a clear linear correlation between the disk mass and the power-law index $γ$ in the equation of disk density profile. This suggests that the $^{12}$CO disk of LkCa 15 is optically thick and we can fit its $^{12}$CO radial intensity profile using either a lower disk mass with a smaller $γ$ or a higher disk mass with a bigger $γ$. By comparing the $^{12}$CO channel maps of the best-fit model with disk models with higher or lower masses, we find that a disk mass of $\sim$ 0.1 $M_{\odot}$ can best reproduce the observed morphology of the $^{12}$CO channel maps. The dust continuum map at 0.87 mm of the LkCa 15 disk shows an inner cavity of the similar size of the best-fit gas model, but its out edge is at $\sim$ 200 AU, much smaller than the fitted gas disk. Such a discrepancy between the outer edges of the gas and dust disks is consistent with dust drifting and trapping models.

astro-ph.EP↗

arXiv2019

Multi-person Articulated Tracking with Spatial and Temporal Embeddings

We propose a unified framework for multi-person pose estimation and tracking. Our framework consists of two main components,~\ie~SpatialNet and TemporalNet. The SpatialNet accomplishes body part detection and part-level data association in a single frame, while the TemporalNet groups human instances in consecutive frames into trajectories. Specifically, besides body part detection heatmaps, SpatialNet also predicts the Keypoint Embedding (KE) and Spatial Instance Embedding (SIE) for body part association. We model the grouping procedure into a differentiable Pose-Guided Grouping (PGG) module to make the whole part detection and grouping pipeline fully end-to-end trainable. TemporalNet extends spatial grouping of keypoints to temporal grouping of human instances. Given human proposals from two consecutive frames, TemporalNet exploits both appearance features encoded in Human Embedding (HE) and temporally consistent geometric features embodied in Temporal Instance Embedding (TIE) for robust tracking. Extensive experiments demonstrate the effectiveness of our proposed model. Remarkably, we demonstrate substantial improvements over the state-of-the-art pose tracking method from 65.4\% to 71.8\% Multi-Object Tracking Accuracy (MOTA) on the ICCV'17 PoseTrack Dataset.

cs.CV↗

arXiv2019

Deep Saliency Hashing

In recent years, hashing methods have been proved to be effective and efficient for the large-scale Web media search. However, the existing general hashing methods have limited discriminative power for describing fine-grained objects that share similar overall appearance but have subtle difference. To solve this problem, we for the first time introduce the attention mechanism to the learning of fine-grained hashing codes. Specifically, we propose a novel deep hashing model, named deep saliency hashing (DSaH), which automatically mines salient regions and learns semantic-preserving hashing codes simultaneously. DSaH is a two-step end-to-end model consisting of an attention network and a hashing network. Our loss function contains three basic components, including the semantic loss, the saliency loss, and the quantization loss. As the core of DSaH, the saliency loss guides the attention network to mine discriminative regions from pairs of images. We conduct extensive experiments on both fine-grained and general retrieval datasets for performance evaluation. Experimental results on fine-grained datasets, including Oxford Flowers-17, Stanford Dogs-120, and CUB Bird demonstrate that our DSaH performs the best for fine-grained retrieval task and beats the strongest competitor (DTQ) by approximately 10% on both Stanford Dogs-120 and CUB Bird. DSaH is also comparable to several state-of-the-art hashing methods on general datasets, including CIFAR-10 and NUS-WIDE.

cs.CV↗

arXiv2018

Unsupervised Semantic Deep Hashing

In recent years, deep hashing methods have been proved to be efficient since it employs convolutional neural network to learn features and hashing codes simultaneously. However, these methods are mostly supervised. In real-world application, it is a time-consuming and overloaded task for annotating a large number of images. In this paper, we propose a novel unsupervised deep hashing method for large-scale image retrieval. Our method, namely unsupervised semantic deep hashing (\textbf{USDH}), uses semantic information preserved in the CNN feature layer to guide the training of network. We enforce four criteria on hashing codes learning based on VGG-19 model: 1) preserving relevant information of feature space in hashing space; 2) minimizing quantization loss between binary-like codes and hashing codes; 3) improving the usage of each bit in hashing codes by using maximum information entropy, and 4) invariant to image rotation. Extensive experiments on CIFAR-10, NUSWIDE have demonstrated that \textbf{USDH} outperforms several state-of-the-art unsupervised hashing methods for image retrieval. We also conduct experiments on Oxford 17 datasets for fine-grained classification to verify its efficiency for other computer vision tasks.

cs.CV↗

arXiv2018

New constraints on turbulence and embedded planet mass in the HD 163296 disk from planet-disk hydrodynamic simulations

Recent Atacama Large Millimeter and Submillimeter Array (ALMA) observations of the protoplanetary disk around the Herbig Ae star HD 163296 revealed three depleted dust gaps at 60, 100 and 160 au in the 1.3 mm continuum as well as CO depletion in the middle and outer dust gaps. However, no CO depletion was found in the inner dust gap. To examine the planet--disk interaction model, we present results of two-dimensional two fluid (gas + dust) hydrodynamic simulations coupled with three-dimensional radiative transfer simulations. In order to fit the high gas-to-dust ratio of the first gap, we find the Shakura--Sunyaev viscosity parameter $α$ must be very small ($\lesssim 10^{-4}$) in the inner disk. On the other hand, a relatively large $α$ ($\sim 7.5\times 10^{-3}$) is required to reproduce the dust surface density in the outer disk. We interpret the variation of $α$ as an indicator of the transition from an inner dead zone to the outer magnetorotational instability (MRI) active zone. Within $\sim 100$ au, the HD 163296 disk's ionization level is low, and non-ideal magnetohydrodynamic (MHD) effects could suppress the MRI, so the disk can be largely laminar. The disk's ionization level gradually increases toward larger radii, and the outermost disk ($r > 300$ au) becomes turbulent due to MRI. Under this condition, we find that the observed dust continuum and CO gas line emissions can be reasonably fit by three half-Jovian-mass planets (0.46, 0.46 and 0.58 $M_\textrm{J}$) at 59, 105 and 160 au, respectively.

astro-ph.EP↗

arXiv2018

Hierarchical Heuristic Learning towards Effcient Norm Emergence

Social norms serve as an important mechanism to regulate the behaviors of agents and to facilitate coordination among them in multiagent systems. One important research question is how a norm can rapidly emerge through repeated local interaction within an agent society under different environments when their coordination space becomes large. To address this problem, we propose a Hierarchically Heuristic Learning Strategy (HHLS) under the hierarchical social learning framework, in which subordinate agents report their information to their supervisors, while supervisors can generate instructions (rules and suggestions) based on the information collected from their subordinates. Subordinate agents heuristically update their strategies based on both their own experience and the instructions from their supervisors. Extensive experiment evaluations show that HHLS can support the emergence of desirable social norms more efficiently and is applicable in a much wider range of multiagent interaction scenarios compared with previous work. We also investigate the effectiveness of HHLS by separating out the different components of the HHLS and evaluating the relative importance of those components. The influence of key related factors (e.g., hierarchical factors, non-hierarchical factors, fixed-strategy agents) are investigated as well.

cs.MA↗

arXiv2017

Compositional imprints in density-distance-time: a rocky composition for close-in low-mass exoplanets from the location of the valley of evaporation

We use an end-to-end model of planet formation, thermodynamic evolution, and atmospheric escape to investigate how the statistical imprints of evaporation depend on the bulk composition of planetary cores (rocky vs. icy). We find that the population-wide imprints like the location of the "evaporation valley" in the distance-radius plane and the corresponding bimodal radius distribution clearly differ depending on the bulk composition of the cores. Comparison with the observed position of the valley (Fulton et al. 2017) suggests that close-in low-mass Kepler planets have a predominately Earth-like rocky composition. Combined with the excess of period ratios outside of MMR, this suggests that low-mass Kepler planets formed inside of the water iceline, but still undergoing orbital migration. The core radius becomes visible for planets losing all primordial H/He. For planets in this "triangle of evaporation" in the distance-radius plane, the degeneracy in compositions is reduced. In the observed diagram, we identify a trend to more volatile-rich compositions with increasing radius (R/R_Earth<1.6 rocky; 1.6-3.0 ices and/or H/He; >3: H/He). The mass-density diagram contains important information about formation and evolution. Its characteristic broken V-shape reveals the transitions from solid planets to low-mass core-dominated planets with H/He and finally to gas-dominated giants. Evaporation causes density and orbital distance to be anti-correlated for low-mass planets, in contrast to giants, where closer-in planets are less dense, likely due to inflation. The temporal evolution of the statistical properties reported here will be of interest for the PLATO 2.0 mission which will observe the temporal dimension.

astro-ph.EP↗

arXiv2017

Terrestrial Planets Formation under Migration: the Systems near 4:2:1 Mean Motion Resonance

In this work, we extensively investigate the formation of near 4:2:1 mean motion resonances (MMRs) configuration by performing two sets of N-body simulations. We model the eccentricity damping, gas drag, type I and type II planetary migration of planetesimals, planetary embryos and giant planets in the first sets. For the simulations of giant planets with type II migration, the massive terrestrial planets, with a mass up to several Earth masses, are likely produced in the systems. We further show that by shepherding and/or scattering mechanisms through Jovian planet's type II migration, the terrestrial planets and giant planets in the systems can be evolved into a near 4:2:1 MMRs. Moreover, the models are applicable to the formation of Kepler-238 and 302 systems. In the second set, we study the 4:2:1 MMRs formation in the terrestrial planetary systems, where the planets undergo type I migration and eccentricity damping. By considering type I migration, $\sim$ 17.1\% of the simulations indicate that terrestrial planets are evolved into 4:2:1 MMRs. However, this probability should depend on the initial conditions of planets. Hence, we conclude that both type I and type II migration can play a crucial role in close-in terrestrial planet formation.

astro-ph.EP↗

arXiv2016

Long-Lived Dust Asymmetries at Dead Zone Edges in Protoplanetary Disks

A number of transition disks exhibit significant azimuthal asymmetries in thermal dust emission. One possible origin for these asymmetries is dust trapping in vortices formed at the edges of dead zones. We carry out high-resolution, two-dimensional hydrodynamic simulations of this scenario, including the effects of dust feedback. We find that, although feedback weakens the vortices and slows down the process of dust accumulation, the dust distribution in the disk can nonetheless remain asymmetric for many thousands of orbits. We show that even after $10^4$ orbits, or $2.5$ Myr when scaled to the parameters of Oph IRS 48 (a significant fraction of its age), the dust is not dispersed into an axisymmetric ring, in contrast to the case of a vortex formed by a planet. This is because accumulation of mass at the dead zone edge constantly replenishes the vortex, preventing it from being fully destroyed. We produce synthetic dust emission images using our simulation results. We find that multiple small clumps of dust may be distributed azimuthally. These clumps, if not resolved from one another, appear as a single large feature. A defining characteristic of a disk with a dead zone edge is that an asymmetric feature is accompanied by a ring of dust located about twice as far from the central star.

astro-ph.EP↗

arXiv2016

Modeling Dust Emission of HL Tau Disk Based on Planet-Disk Interactions

We use extensive global two-dimensional hydrodynamic disk gas+dust simulations with embedded planets, coupled with three dimensional radiative transfer calculations, to model the dust ring and gap structures in the HL Tau protoplanetary disk observed with the Atacama Large Millimeter/Submillimeter Array (ALMA). We include the self-gravity of disk gas and dust components and make reasonable choices of disk parameters, assuming an already settled dust distribution and no planet migration. We can obtain quite adequate fits to the observed dust emission using three planets with masses 0.35, 0.17, and 0.26 $M_{Jup}$ at 13.1, 33.0, and 68.6 AU, respectively. Implications for the planet formation as well as the limitations of this scenario are discussed.

astro-ph.EP↗

arXiv2014

Planetary population synthesis coupled with atmospheric escape: a statistical view of evaporation

We apply hydrodynamic evaporation models to different synthetic planet populations that were obtained from a planet formation code based on a core-accretion paradigm. We investigated the evolution of the planet populations using several evaporation models, which are distinguished by the driving force of the escape flow (X-ray or EUV), the heating efficiency in energy-limited evaporation regimes, or both. Although the mass distribution of the planet populations is barely affected by evaporation, the radius distribution clearly shows a break at approximately 2 $R_{\oplus}$. We find that evaporation can lead to a bimodal distribution of planetary sizes (Owen & Wu 2013) and to an "evaporation valley" running diagonally downwards in the orbital distance - planetary radius plane, separating bare cores from low-mass planet that have kept some primordial H/He. Furthermore, this bimodal distribution is related to the initial characteristics of the planetary populations because low-mass planetary cores can only accrete small primordial H/He envelopes and their envelope masses are proportional to their core masses. We also find that the population-wide effect of evaporation is not sensitive to the heating efficiency of energy-limited description. However, in two extreme cases, namely without evaporation or with a 100\% heating efficiency in an evaporation model, the final size distributions show significant differences; these two scenarios can be ruled out from the size distribution of $Kepler$ candidates.

astro-ph.EP↗

arXiv2011

Terrestrial Planet Formation in the Inclined Systems: Application to OGLE-2006-BLG-109L System

In this work, we extensively investigate the terrestrial planetary formation for the inclined planetary systems (considering the OGLE-2006-BLG-109L system as prototype) in the late stage. In the simulations, we show that the occurrence of terrestrial planets is quite common, in the final assembly stage. Moreover, we find that 40% of the runs finally occupy one planet in the habitable zone (HZ). On the other hand, the numerical results also indicate that the inner region of the planetesimal disk, ranging from $\sim 0.1$ to 0.3 AU, plays an important role in building up terrestrial planets. By examining all simulations, we note that the survivals are located either between 0.1$\sim$1.0 AU or beyond 7 AU, or at the 1:1 mean motion resonance of OGLE-2006-BLG-109Lb at $\sim$2.20 AU. The outcomes suggest that it may exist moderate possibility for the inclined systems to harbor terrestrial planets, even planets in the HZs.

astro-ph.EP↗