arXiv subjects

Hao Tang

Publications and source records attributed to Hao Tang.

At least 433 records · Page 24Linked to original sources

arXiv2021

Recurrent Mask Refinement for Few-Shot Medical Image Segmentation

Although having achieved great success in medical image segmentation, deep convolutional neural networks usually require a large dataset with manual annotations for training and are difficult to generalize to unseen classes. Few-shot learning has the potential to address these challenges by learning new classes from only a few labeled examples. In this work, we propose a new framework for few-shot medical image segmentation based on prototypical networks. Our innovation lies in the design of two key modules: 1) a context relation encoder (CRE) that uses correlation to capture local relation features between foreground and background regions; and 2) a recurrent mask refinement module that repeatedly uses the CRE and a prototypical network to recapture the change of context relationship and refine the segmentation mask iteratively. Experiments on two abdomen CT datasets and an abdomen MRI dataset show the proposed method obtains substantial improvement over the state-of-the-art methods by an average of 16.32%, 8.45% and 6.24% in terms of DSC, respectively. Code is publicly available.

cs.CV↗

arXiv2021

Cross-View Exocentric to Egocentric Video Synthesis

Cross-view video synthesis task seeks to generate video sequences of one view from another dramatically different view. In this paper, we investigate the exocentric (third-person) view to egocentric (first-person) view video generation task. This is challenging because egocentric view sometimes is remarkably different from the exocentric view. Thus, transforming the appearances across the two different views is a non-trivial task. Particularly, we propose a novel Bi-directional Spatial Temporal Attention Fusion Generative Adversarial Network (STA-GAN) to learn both spatial and temporal information to generate egocentric video sequences from the exocentric view. The proposed STA-GAN consists of three parts: temporal branch, spatial branch, and attention fusion. First, the temporal and spatial branches generate a sequence of fake frames and their corresponding features. The fake frames are generated in both downstream and upstream directions for both temporal and spatial branches. Next, the generated four different fake frames and their corresponding features (spatial and temporal branches in two directions) are fed into a novel multi-generation attention fusion module to produce the final video sequence. Meanwhile, we also propose a novel temporal and spatial dual-discriminator for more robust network optimization. Extensive experiments on the Side2Ego and Top2Ego datasets show that the proposed STA-GAN significantly outperforms the existing methods.

cs.CV↗

arXiv2021

Total Generate: Cycle in Cycle Generative Adversarial Networks for Generating Human Faces, Hands, Bodies, and Natural Scenes

We propose a novel and unified Cycle in Cycle Generative Adversarial Network (C2GAN) for generating human faces, hands, bodies, and natural scenes. Our proposed C2GAN is a cross-modal model exploring the joint exploitation of the input image data and guidance data in an interactive manner. C2GAN contains two different generators, i.e., an image-generation generator and a guidance-generation generator. Both generators are mutually connected and trained in an end-to-end fashion and explicitly form three cycled subnets, i.e., one image generation cycle and two guidance generation cycles. Each cycle aims at reconstructing the input domain and simultaneously produces a useful output involved in the generation of another cycle. In this way, the cycles constrain each other implicitly providing complementary information from both image and guidance modalities and bringing an extra supervision gradient across the cycles, facilitating a more robust optimization of the whole model. Extensive results on four guided image-to-image translation subtasks demonstrate that the proposed C2GAN is effective in generating more realistic images compared with state-of-the-art models. The code is available at https://github.com/Ha0Tang/C2GAN.

cs.CV↗

arXiv2021

Transformer-Based Source-Free Domain Adaptation

In this paper, we study the task of source-free domain adaptation (SFDA), where the source data are not available during target adaptation. Previous works on SFDA mainly focus on aligning the cross-domain distributions. However, they ignore the generalization ability of the pretrained source model, which largely influences the initial target outputs that are vital to the target adaptation stage. To address this, we make the interesting observation that the model accuracy is highly correlated with whether or not attention is focused on the objects in an image. To this end, we propose a generic and effective framework based on Transformer, named TransDA, for learning a generalized model for SFDA. Specifically, we apply the Transformer as the attention module and inject it into a convolutional network. By doing so, the model is encouraged to turn attention towards the object regions, which can effectively improve the model's generalization ability on the target domains. Moreover, a novel self-supervised knowledge distillation approach is proposed to adapt the Transformer with target pseudo-labels, thus further encouraging the network to focus on the object regions. Experiments on three domain adaptation tasks, including closed-set, partial-set, and open-set adaption, demonstrate that TransDA can greatly improve the adaptation accuracy and produce state-of-the-art results. The source code and trained models are available at https://github.com/ygjwd12345/TransDA.

cs.CV↗

arXiv2021

Balancing orbital effects and onsite Coulomb repulsion through Na modulations in NaxVO2

Vanadium oxides have been highly attractive for over half a century since the discovery of the metal insulator transition near room temperatures. Here NaxVO2 is studied through a systematic comparison with other layered sodium metal oxides with early 3d transition metals, first disclosing a unified evolution pattern of Na density waves through in situ XRD analysis. Combining ab-initio simulations and theoretical modelings, a sodium-modulated Peierls-like transition mechanism is then proposed for the bonding formation of metal ion dimers. More importantly, the unique trimer structure in NaxVO2 is shown to be very sensitive to the onsite Coulomb repulsion value, suggesting a delicate balance between strong electronic correlations and orbital effects that can be precisely modulated by both Na compositions and atomic stackings. This unveils a unique opportunity to design strongly correlated materials with tailored electronic transitions through electrochemical modulations and crystallographic designs, to elegantly balance various competition effects. We think the understanding will also help further elucidate complicated electronic behaviors in other vanadium oxide systems.

cond-mat.mtrl-sci↗

arXiv2021

Quantum Computation for Pricing the Collateralized Debt Obligations

Collateralized debt obligation (CDO) has been one of the most commonly used structured financial products and is intensively studied in quantitative finance. By setting the asset pool into different tranches, it effectively works out and redistributes credit risks and returns to meet the risk preferences for different tranche investors. The copula models of various kinds are normally used for pricing CDOs, and the Monte Carlo simulations are required to get their numerical solution. Here we implement two typical CDO models, the single-factor Gaussian copula model and Normal Inverse Gaussian copula model, and by applying the conditional independence approach, we manage to load each model of distribution in quantum circuits. We then apply quantum amplitude estimation as an alternative to Monte Carlo simulation for CDO pricing. We demonstrate the quantum computation results using IBM Qiskit. Our work addresses a useful task in finance instrument pricing, significantly broadening the application scope for quantum computing in finance.

q-fin.RM↗

arXiv2021

Geometric origins of topological insulation in twisted layered semiconductors

Twisted bilayers of two-dimensional (2D) materials are proving a fertile ground for investigating strongly correlated electron phases. This is because the moiré pattern introduced by the relative twist between layers introduces long-wavelength effective potentials which lead to electron localization. Here, we develop a generalized continuum model for the electronic structure of moiré patterns, based on first-principles calculations and tailored to capture the physics of twisted bilayer 2D semiconductors. We apply this model to a database of eighteen 2D crystals covering a range of atomic relaxation and electronic structure features. Many of these materials host topologically insulating (TI) moiré bands in a certain range of twist angles, which originate from the competition between triangular and hexagonal moiré patterns, tuned by the twist angle. The topological phases occur in the same range as the maximally flat moiré bands.

cond-mat.mtrl-sci↗

arXiv2020

Spatial Context-Aware Self-Attention Model For Multi-Organ Segmentation

Multi-organ segmentation is one of most successful applications of deep learning in medical image analysis. Deep convolutional neural nets (CNNs) have shown great promise in achieving clinically applicable image segmentation performance on CT or MRI images. State-of-the-art CNN segmentation models apply either 2D or 3D convolutions on input images, with pros and cons associated with each method: 2D convolution is fast, less memory-intensive but inadequate for extracting 3D contextual information from volumetric images, while the opposite is true for 3D convolution. To fit a 3D CNN model on CT or MRI images on commodity GPUs, one usually has to either downsample input images or use cropped local regions as inputs, which limits the utility of 3D models for multi-organ segmentation. In this work, we propose a new framework for combining 3D and 2D models, in which the segmentation is realized through high-resolution 2D convolutions, but guided by spatial contextual information extracted from a low-resolution 3D model. We implement a self-attention mechanism to control which 3D features should be used to guide 2D segmentation. Our model is light on memory usage but fully equipped to take 3D contextual information into account. Experiments on multiple organ segmentation datasets demonstrate that by taking advantage of both 2D and 3D models, our method is consistently outperforms existing 2D and 3D models in organ segmentation accuracy, while being able to directly take raw whole-volume image data as inputs.

eess.IV↗

arXiv2020

Towards Scale-Invariant Graph-related Problem Solving by Iterative Homogeneous Graph Neural Networks

Current graph neural networks (GNNs) lack generalizability with respect to scales (graph sizes, graph diameters, edge weights, etc..) when solving many graph analysis problems. Taking the perspective of synthesizing graph theory programs, we propose several extensions to address the issue. First, inspired by the dependency of the iteration number of common graph theory algorithms on graph size, we learn to terminate the message passing process in GNNs adaptively according to the computation progress. Second, inspired by the fact that many graph theory algorithms are homogeneous with respect to graph weights, we introduce homogeneous transformation layers that are universal homogeneous function approximators, to convert ordinary GNNs to be homogeneous. Experimentally, we show that our GNN can be trained from small-scale graphs but generalize well to large-scale graphs for a number of basic graph theory problems. It also shows generalizability for applications of multi-body physical simulation and image-based navigation problems.

cs.LG↗

arXiv2020

Refactoring Policy for Compositional Generalizability using Self-Supervised Object Proposals

We study how to learn a policy with compositional generalizability. We propose a two-stage framework, which refactorizes a high-reward teacher policy into a generalizable student policy with strong inductive bias. Particularly, we implement an object-centric GNN-based student policy, whose input objects are learned from images through self-supervised learning. Empirically, we evaluate our approach on four difficult tasks that require compositional generalizability, and achieve superior performance compared to baselines.

cs.CV↗

arXiv2020

A local-in-time theory for singular SDEs with applications to fluid models with transport noise

In this paper, we establish a local theory, i.e., existence, uniqueness and blow-up criterion, for a general family of singular SDEs in some Hilbert space. The key requirement is an approximation property that allows us to embed the singular drift and diffusion mappings into a hierarchy of regular mappings that are invariant with respect to the Hilbert space and enjoy a cancellation property. Various nonlinear models in fluid dynamics with transport noise belong to this type of singular SDEs. With a cancellation estimate for generalized Lie derivative operators, we can construct such regular approximations for cases involving the Lie derivative operators, or more generally, differential operators of order one with suitable coefficients. In particular, we apply the abstract theory to achieve novel local-in-time results for the stochastic two-component Camassa--Holm (CH) system and for the stochastic Córdoba-Córdoba-Fontelos (CCF) model.

math.PR↗

arXiv2020

On the stochastic Dullin-Gottwald-Holm equation: Global existence and wave-breaking phenomena

We consider a class of stochastic evolution equations that include in particular the stochastic Camassa--Holm equation. For the initial value problem on a torus, we first establish the local existence and uniqueness of pathwise solutions in the Sobolev spaces $H^s$ with $s>3/2$. Then we show that strong enough nonlinear noise can prevent blow-up almost surely. To analyze the effects of weaker noise, we consider a linearly multiplicative noise with non-autonomous pre-factor. Then, we formulate precise conditions on the inital data that lead to global existence of strong solutions or to blow-up. The blow-up occurs as wave breaking. For blow-up with positive probability, we derive lower bounds for these probabilities. Finally, the blow-up rate of these solutions is precisely analyzed.

math.AP↗

arXiv2020

Unified Generative Adversarial Networks for Controllable Image-to-Image Translation

We propose a unified Generative Adversarial Network (GAN) for controllable image-to-image translation, i.e., transferring an image from a source to a target domain guided by controllable structures. In addition to conditioning on a reference image, we show how the model can generate images conditioned on controllable structures, e.g., class labels, object keypoints, human skeletons, and scene semantic maps. The proposed model consists of a single generator and a discriminator taking a conditional image and the target controllable structure as input. In this way, the conditional image can provide appearance information and the controllable structure can provide the structure information for generating the target result. Moreover, our model learns the image-to-image mapping through three novel losses, i.e., color loss, controllable structure guided cycle-consistency loss, and controllable structure guided self-content preserving loss. Also, we present the Fréchet ResNet Distance (FRD) to evaluate the quality of the generated images. Experiments on two challenging image translation tasks, i.e., hand gesture-to-gesture translation and cross-view image translation, show that our model generates convincing results, and significantly outperforms other state-of-the-art methods on both tasks. Meanwhile, the proposed framework is a unified solution, thus it can be applied to solving other controllable structure guided image translation tasks such as landmark guided facial expression translation and keypoint guided person image generation. To the best of our knowledge, we are the first to make one GAN framework work on all such controllable structure guided image translation tasks. Code is available at https://github.com/Ha0Tang/GestureGAN.

cs.CV↗

arXiv2020

Dual Attention GANs for Semantic Image Synthesis

In this paper, we focus on the semantic image synthesis task that aims at transferring semantic label maps to photo-realistic images. Existing methods lack effective semantic constraints to preserve the semantic information and ignore the structural correlations in both spatial and channel dimensions, leading to unsatisfactory blurry and artifact-prone results. To address these limitations, we propose a novel Dual Attention GAN (DAGAN) to synthesize photo-realistic and semantically-consistent images with fine details from the input layouts without imposing extra training overhead or modifying the network architectures of existing methods. We also propose two novel modules, i.e., position-wise Spatial Attention Module (SAM) and scale-wise Channel Attention Module (CAM), to capture semantic structure attention in spatial and channel dimensions, respectively. Specifically, SAM selectively correlates the pixels at each position by a spatial attention map, leading to pixels with the same semantic label being related to each other regardless of their spatial distances. Meanwhile, CAM selectively emphasizes the scale-wise features at each channel by a channel attention map, which integrates associated features among all channel maps regardless of their scales. We finally sum the outputs of SAM and CAM to further improve feature representation. Extensive experiments on four challenging datasets show that DAGAN achieves remarkably better results than state-of-the-art methods, while using fewer model parameters. The source code and trained models are available at https://github.com/Ha0Tang/DAGAN.

cs.CV↗

arXiv2020

Bipartite Graph Reasoning GANs for Person Image Generation

We present a novel Bipartite Graph Reasoning GAN (BiGraphGAN) for the challenging person image generation task. The proposed graph generator mainly consists of two novel blocks that aim to model the pose-to-pose and pose-to-image relations, respectively. Specifically, the proposed Bipartite Graph Reasoning (BGR) block aims to reason the crossing long-range relations between the source pose and the target pose in a bipartite graph, which mitigates some challenges caused by pose deformation. Moreover, we propose a new Interaction-and-Aggregation (IA) block to effectively update and enhance the feature representation capability of both person's shape and appearance in an interactive way. Experiments on two challenging and public datasets, i.e., Market-1501 and DeepFashion, show the effectiveness of the proposed BiGraphGAN in terms of objective quantitative scores and subjective visual realness. The source code and trained models are available at https://github.com/Ha0Tang/BiGraphGAN.

cs.CV↗

arXiv2020

Audio-Visual Event Localization via Recursive Fusion by Joint Co-Attention

The major challenge in audio-visual event localization task lies in how to fuse information from multiple modalities effectively. Recent works have shown that attention mechanism is beneficial to the fusion process. In this paper, we propose a novel joint attention mechanism with multimodal fusion methods for audio-visual event localization. Particularly, we present a concise yet valid architecture that effectively learns representations from multiple modalities in a joint manner. Initially, visual features are combined with auditory features and then turned into joint representations. Next, we make use of the joint representations to attend to visual features and auditory features, respectively. With the help of this joint co-attention, new visual and auditory features are produced, and thus both features can enjoy the mutually improved benefits from each other. It is worth noting that the joint co-attention unit is recursive meaning that it can be performed multiple times for obtaining better joint representations progressively. Extensive experiments on the public AVE dataset have shown that the proposed method achieves significantly better results than the state-of-the-art methods.

cs.CV↗

arXiv2020

Dual In-painting Model for Unsupervised Gaze Correction and Animation in the Wild

In this paper we address the problem of unsupervised gaze correction in the wild, presenting a solution that works without the need for precise annotations of the gaze angle and the head pose. We have created a new dataset called CelebAGaze, which consists of two domains X, Y, where the eyes are either staring at the camera or somewhere else. Our method consists of three novel modules: the Gaze Correction module (GCM), the Gaze Animation module (GAM), and the Pretrained Autoencoder module (PAM). Specifically, GCM and GAM separately train a dual in-painting network using data from the domain $X$ for gaze correction and data from the domain $Y$ for gaze animation. Additionally, a Synthesis-As-Training method is proposed when training GAM to encourage the features encoded from the eye region to be correlated with the angle information, resulting in a gaze animation which can be achieved by interpolation in the latent space. To further preserve the identity information~(e.g., eye shape, iris color), we propose the PAM with an Autoencoder, which is based on Self-Supervised mirror learning where the bottleneck features are angle-invariant and which works as an extra input to the dual in-painting models. Extensive experiments validate the effectiveness of the proposed method for gaze correction and gaze animation in the wild and demonstrate the superiority of our approach in producing more compelling results than state-of-the-art baselines. Our code, the pretrained models and the supplementary material are available at: https://github.com/zhangqianhui/GazeAnimation.

cs.CV↗

arXiv2020

On a stochastic Camassa-Holm type equation with higher order nonlinearities

The subject of this paper is a generalized Camassa-Holm equation under random perturbation. We first establish local existence and uniqueness results as well as blow-up criteria for pathwise solutions in the Sobolev spaces $H^s$ with $s>3/2$. Then we analyze how noise affects the dependence of solutions on initial data. Even though the noise has some already known regularization effects, much less is known concerning the dependence on initial data. As a new concept we introduce the notion of stability of exiting times and construct an example showing that multiplicative noise (in Itô sense) cannot improve the stability of the exiting time, and simultaneously improve the continuity of the dependence on initial data. Finally, we obtain global existence theorems and estimate associated probabilities.

math.AP↗