arXiv subjects

Jianlong Fu

Publications and source records attributed to Jianlong Fu.

106 records · Page 6Linked to original sources

arXiv2019

Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting

High-quality image inpainting requires filling missing regions in a damaged image with plausible content. Existing works either fill the regions by copying image patches or generating semantically-coherent patches from region context, while neglect the fact that both visual and semantic plausibility are highly-demanded. In this paper, we propose a Pyramid-context ENcoder Network (PEN-Net) for image inpainting by deep generative models. The PEN-Net is built upon a U-Net structure, which can restore an image by encoding contextual semantics from full resolution input, and decoding the learned semantic features back into images. Specifically, we propose a pyramid-context encoder, which progressively learns region affinity by attention from a high-level semantic feature map and transfers the learned attention to the previous low-level feature map. As the missing content can be filled by attention transfer from deep to shallow in a pyramid fashion, both visual and semantic coherence for image inpainting can be ensured. We further propose a multi-scale decoder with deeply-supervised pyramid losses and an adversarial loss. Such a design not only results in fast convergence in training, but more realistic results in testing. Extensive experiments on various datasets show the superior performance of the proposed network

cs.CV↗

arXiv2019

Activitynet 2019 Task 3: Exploring Contexts for Dense Captioning Events in Videos

Contextual reasoning is essential to understand events in long untrimmed videos. In this work, we systematically explore different captioning models with various contexts for the dense-captioning events in video task, which aims to generate captions for different events in the untrimmed video. We propose five types of contexts as well as two categories of event captioning models, and evaluate their contributions for event captioning from both accuracy and diversity aspects. The proposed captioning models are plugged into our pipeline system for the dense video captioning challenge. The overall system achieves the state-of-the-art performance on the dense-captioning events in video task with 9.91 METEOR score on the challenge testing set.

cs.CV↗

arXiv2019

Looking for the Devil in the Details: Learning Trilinear Attention Sampling Network for Fine-grained Image Recognition

Learning subtle yet discriminative features (e.g., beak and eyes for a bird) plays a significant role in fine-grained image recognition. Existing attention-based approaches localize and amplify significant parts to learn fine-grained details, which often suffer from a limited number of parts and heavy computational cost. In this paper, we propose to learn such fine-grained features from hundreds of part proposals by Trilinear Attention Sampling Network (TASN) in an efficient teacher-student manner. Specifically, TASN consists of 1) a trilinear attention module, which generates attention maps by modeling the inter-channel relationships, 2) an attention-based sampler which highlights attended parts with high resolution, and 3) a feature distiller, which distills part features into a global one by weight sharing and feature preserving strategies. Extensive experiments verify that TASN yields the best performance under the same settings with the most competitive approaches, in iNaturalist-2017, CUB-Bird, and Stanford-Cars datasets.

cs.CV↗

arXiv2019

From Words to Sentences: A Progressive Learning Approach for Zero-resource Machine Translation with Visual Pivots

The neural machine translation model has suffered from the lack of large-scale parallel corpora. In contrast, we humans can learn multi-lingual translations even without parallel texts by referring our languages to the external world. To mimic such human learning behavior, we employ images as pivots to enable zero-resource translation learning. However, a picture tells a thousand words, which makes multi-lingual sentences pivoted by the same image noisy as mutual translations and thus hinders the translation model learning. In this work, we propose a progressive learning approach for image-pivoted zero-resource machine translation. Since words are less diverse when grounded in the image, we first learn word-level translation with image pivots, and then progress to learn the sentence-level translation by utilizing the learned word translation to suppress noises in image-pivoted multi-lingual sentences. Experimental results on two widely used image-pivot translation datasets, IAPR-TC12 and Multi30k, show that the proposed approach significantly outperforms other state-of-the-art methods.

cs.CL↗

arXiv2018

Properties and application of the SO(3) Majorana representation of spin: equivalence with the Jordan-Wigner transformation and exact $Z_{2}$ gauge theories for spin models

We explore the properties of the SO(3) Majorana representation of spin. Based on its non-local nature, it is shown that there is an equivalence between the SO(3) Majorana representation and the Jordan-Wigner transformation in one and two dimensions. From the relation between the SO(3) Majorana representation and one-dimensional Jordan-Wigner transformation, we show that application of the SO(3) Majorana representation usually results in $Z_{2}$ gauge structure. Based on lattice Chern-Simons gauge theory, it is shown that the anti-commuting link variables in the SO(3) Majorana representation make it equivalent to an operator form of compact $\text{U(1)}_{1}$ Chern-Simons Jordan-Wigner transformation in 2d. As examples of its application, we discuss two spin models, namely the quantum XY model on honeycomb lattice and the $90^{\circ}$ compass model on square lattice. It is shown that under the SO(3) Majorana representation both spin models can be exactly mapped into $Z_{2}$ gauge theory of spinons, with the standard form of $Z_{2}$ Gauss law constraint.

cond-mat.str-el↗

arXiv2018

Beyond Narrative Description: Generating Poetry from Images by Multi-Adversarial Training

Automatic generation of natural language from images has attracted extensive attention. In this paper, we take one step further to investigate generation of poetic language (with multiple lines) to an image for automatic poetry creation. This task involves multiple challenges, including discovering poetic clues from the image (e.g., hope from green), and generating poems to satisfy both relevance to the image and poeticness in language level. To solve the above challenges, we formulate the task of poem generation into two correlated sub-tasks by multi-adversarial training via policy gradient, through which the cross-modal relevance and poetic language style can be ensured. To extract poetic clues from images, we propose to learn a deep coupled visual-poetic embedding, in which the poetic representation from objects, sentiments and scenes in an image can be jointly learned. Two discriminative networks are further introduced to guide the poem generation, including a multi-modal discriminator and a poem-style discriminator. To facilitate the research, we have released two poem datasets by human annotators with two distinct properties: 1) the first human annotated image-to-poem pair dataset (with 8,292 pairs in total), and 2) to-date the largest public English poem corpus dataset (with 92,265 different poems in total). Extensive experiments are conducted with 8K images, among which 1.5K image are randomly picked for evaluation. Both objective and subjective evaluations show the superior performances against the state-of-the-art methods for poem generation from images. Turing test carried out with over 500 human subjects, among which 30 evaluators are poetry experts, demonstrates the effectiveness of our approach.

cs.CV↗

arXiv2018

Image Inspired Poetry Generation in XiaoIce

Vision is a common source of inspiration for poetry. The objects and the sentimental imprints that one perceives from an image may lead to various feelings depending on the reader. In this paper, we present a system of poetry generation from images to mimic the process. Given an image, we first extract a few keywords representing objects and sentiments perceived from the image. These keywords are then expanded to related ones based on their associations in human written poems. Finally, verses are generated gradually from the keywords using recurrent neural networks trained on existing poems. Our approach is evaluated by human assessors and compared to other generation baselines. The results show that our method can generate poems that are more artistic than the baseline methods. This is one of the few attempts to generate poetry from images. By deploying our proposed approach, XiaoIce has already generated more than 12 million poems for users since its release in July 2017. A book of its poems has been published by Cheers Publishing, which claimed that the book is the first-ever poetry collection written by an AI in human history.

cs.AI↗

arXiv2018

DA-GAN: Instance-level Image Translation by Deep Attention Generative Adversarial Networks (with Supplementary Materials)

Unsupervised image translation, which aims in translating two independent sets of images, is challenging in discovering the correct correspondences without paired data. Existing works build upon Generative Adversarial Network (GAN) such that the distribution of the translated images are indistinguishable from the distribution of the target set. However, such set-level constraints cannot learn the instance-level correspondences (e.g. aligned semantic parts in object configuration task). This limitation often results in false positives (e.g. geometric or semantic artifacts), and further leads to mode collapse problem. To address the above issues, we propose a novel framework for instance-level image translation by Deep Attention GAN (DA-GAN). Such a design enables DA-GAN to decompose the task of translating samples from two sets into translating instances in a highly-structured latent space. Specifically, we jointly learn a deep attention encoder, and the instancelevel correspondences could be consequently discovered through attending on the learned instance pairs. Therefore, the constraints could be exploited on both set-level and instance-level. Comparisons against several state-ofthe- arts demonstrate the superiority of our approach, and the broad application capability, e.g, pose morphing, data augmentation, etc., pushes the margin of domain translation problem.

cs.CV↗

arXiv2018

Majorana representations of spin and an alternative solution of the Kitaev honeycomb model

Based on the Dirac spinor representation of the SO(4) group, we discuss the relationship between three types of representation of spin in terms of Majorana fermions, namely the Kitaev representation, the SO(3) representation and the SO(4) chiral representation. Comparing the three types, we show that the Hilbert space of the SO(3) representation is different from the other two by requiring pairing of sites, but it has the advantage over the other two in that no unphysical states are involved. As an example of its application, we present a new alternative solution of the Kitaev honeycomb model. Our new solution involves no unphysical states which enables a systematic calculation of physical observables. Finally, we discuss an extension of the model to a more general exactly soluble $Z_{2}$ gauge theory interacting with complex fermions.

cond-mat.str-el↗

arXiv2018

Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions

Visual Question Answering (VQA) has attracted attention from both computer vision and natural language processing communities. Most existing approaches adopt the pipeline of representing an image via pre-trained CNNs, and then using the uninterpretable CNN features in conjunction with the question to predict the answer. Although such end-to-end models might report promising performance, they rarely provide any insight, apart from the answer, into the VQA process. In this work, we propose to break up the end-to-end VQA into two steps: explaining and reasoning, in an attempt towards a more explainable VQA by shedding light on the intermediate results between these two steps. To that end, we first extract attributes and generate descriptions as explanations for an image using pre-trained attribute detectors and image captioning models, respectively. Next, a reasoning module utilizes these explanations in place of the image to infer an answer to the question. The advantages of such a breakdown include: (1) the attributes and captions can reflect what the system extracts from the image, thus can provide some explanations for the predicted answer; (2) these intermediate results can help us identify the inabilities of both the image understanding part and the answer inference part when the predicted answer is wrong. We conduct extensive experiments on a popular VQA dataset and dissect all results according to several measurements of the explanation quality. Our system achieves comparable performance with the state-of-the-art, yet with added benefits of explainability and the inherent ability to further improve with higher quality explanations.

cs.CV↗

arXiv2017

Self-view Grounding Given a Narrated 360° Video

Narrated 360° videos are typically provided in many touring scenarios to mimic real-world experience. However, previous work has shown that smart assistance (i.e., providing visual guidance) can significantly help users to follow the Normal Field of View (NFoV) corresponding to the narrative. In this project, we aim at automatically grounding the NFoVs of a 360° video given subtitles of the narrative (referred to as "NFoV-grounding"). We propose a novel Visual Grounding Model (VGM) to implicitly and efficiently predict the NFoVs given the video content and subtitles. Specifically, at each frame, we efficiently encode the panorama into feature map of candidate NFoVs using a Convolutional Neural Network (CNN) and the subtitles to the same hidden space using an RNN with Gated Recurrent Units (GRU). Then, we apply soft-attention on candidate NFoVs to trigger sentence decoder aiming to minimize the reconstruct loss between the generated and given sentence. Finally, we obtain the NFoV as the candidate NFoV with the maximum attention without any human supervision. To train VGM more robustly, we also generate a reverse sentence conditioning on one minus the soft-attention such that the attention focuses on candidate NFoVs less relevant to the given sentence. The negative log reconstruction loss of the reverse sentence (referred to as "irrelevant loss") is jointly minimized to encourage the reverse sentence to be different from the given sentence. To evaluate our method, we collect the first narrated 360° videos dataset and achieve state-of-the-art NFoV-grounding performance.

cs.CV↗

arXiv2017

Show, Adapt and Tell: Adversarial Training of Cross-domain Image Captioner

Impressive image captioning results are achieved in domains with plenty of training image and sentence pairs (e.g., MSCOCO). However, transferring to a target domain with significant domain shifts but no paired training data (referred to as cross-domain image captioning) remains largely unexplored. We propose a novel adversarial training procedure to leverage unpaired data in the target domain. Two critic networks are introduced to guide the captioner, namely domain critic and multi-modal critic. The domain critic assesses whether the generated sentences are indistinguishable from sentences in the target domain. The multi-modal critic assesses whether an image and its generated sentence are a valid pair. During training, the critics and captioner act as adversaries -- captioner aims to generate indistinguishable sentences, whereas critics aim at distinguishing them. The assessment improves the captioner through policy gradient updates. During inference, we further propose a novel critic-based planning method to select high-quality sentences without additional supervision (e.g., tags). To evaluate, we use MSCOCO as the source domain and four other datasets (CUB-200-2011, Oxford-102, TGIF, and Flickr30k) as the target domains. Our method consistently performs well on all datasets. In particular, on CUB-200-2011, we achieve 21.8% CIDEr-D improvement after adaptation. Utilizing critics during inference further gives another 4.5% boost.

cs.CV↗

arXiv2017

Fingerprints of quantum spin ice in Raman scattering

We develop a theory of the dynamical response of a minimal model of quantum spin ice (QSI) by means of inelastic light scattering. In particular, we are interested in the Raman response of the fractionalized U(1) spin liquid realized in the XXZ QSI. We show that the low-energy Raman intensity is dominated by spinon and gauge fluctuations. We find that the Raman response in the QSI state of that model appears only in the $T_{2g}$ polarization channel. We show that the Raman intensity profile displays a broad continuum from the spinons and coupled spinon and gauge fluctuations, and a low-energy peak arising entirely from gauge fluctuations. Both features originate from the exotic interaction between photon and the fractionalized excitations of QSI. Our theoretical results suggest that inelastic Raman scattering can in principle serve as a promising experimental probe of the nature of a U(1) spin liquid in QSI.

cond-mat.str-el↗

arXiv2016

Storytelling of Photo Stream with Bidirectional Multi-thread Recurrent Neural Network

Visual storytelling aims to generate human-level narrative language (i.e., a natural paragraph with multiple sentences) from a photo streams. A typical photo story consists of a global timeline with multi-thread local storylines, where each storyline occurs in one different scene. Such complex structure leads to large content gaps at scene transitions between consecutive photos. Most existing image/video captioning methods can only achieve limited performance, because the units in traditional recurrent neural networks (RNN) tend to "forget" the previous state when the visual sequence is inconsistent. In this paper, we propose a novel visual storytelling approach with Bidirectional Multi-thread Recurrent Neural Network (BMRNN). First, based on the mined local storylines, a skip gated recurrent unit (sGRU) with delay control is proposed to maintain longer range visual information. Second, by using sGRU as basic units, the BMRNN is trained to align the local storylines into the global sequential timeline. Third, a new training scheme with a storyline-constrained objective function is proposed by jointly considering both global and local matches. Experiments on three standard storytelling datasets show that the BMRNN model outperforms the state-of-the-art methods.

cs.CV↗

arXiv2015

Application of Mythen Detector In-situ XRD Study on The Thermal Expansion Behavior of Metal Indium

A Mythen detector has been equipped at the beamline 4B9A of Beijing Synchrotron Radiation Facility, which can be used for in-situ real-time measurement of X-ray diffraction (XRD) full profiles. In this paper, the thermal expansion behavior of metal indium has been studied by using the in-situ XRD technique with the Mythen detector. The indium film was heated from 30 to 160 °C with a heating rate of 2 °C/min. The in-situ XRD full-profiles were collected with a rate of one profile per 10 seconds. Rietveld refinement was used to extract the structural parameters. The results demonstrate that the thermal expansion of metal indium is nonlinear especially when the sample temperature was close to its melting point (156.5 °C). The expansion of a-axis and the contraction of c-axis of the tetragonal unit cell of metallic indium can be well described by biquadratic and cubic polynomials, respectively. The tetragonal unit cell presents a tendency to become cubic one with the increase of temperature but without detectable phase change. This study is not only beneficial to the application of metal indium, but also exhibits the capacity of in-situ time-resolved XRD experiments at the X-ray diffraction station of BSRF.

cond-mat.mtrl-sci↗

arXiv2013

Transition-metal distribution in kagome antiferromagnet CoCu3(OH)6Cl2 revealed by resonant x-ray diffraction

The distribution of chemically similar transition-metal ions is a fundamental issue in the study of herbertsmithite-type kagome antiferromagnets. Using synchrotron radiation, we have performed resonant powder x-ray diffractions on newly synthesized CoCu3(OH)6Cl2, which provide an exact distribution of transition-metal ions in the frustrated antiferromagnet. Both magnetic susceptibility and specific heat measurements are quantitatively consistent with the occupation fractions determined by resonant x-ray diffraction. The distribution of transition-metal ions and residual magnetic entropy suggest a novel low temperature (T < 4 K) magnetism, where the interlayer triangular spins undergo a spin-glass freezing while the kagome spins still keep highly frustrated.

cond-mat.str-el↗