arXiv subjects

Jialin Wu

Publications and source records attributed to Jialin Wu.

48 records · Page 3Linked to original sources

arXiv2020

Improving VQA and its Explanations \\ by Comparing Competing Explanations

Most recent state-of-the-art Visual Question Answering (VQA) systems are opaque black boxes that are only trained to fit the answer distribution given the question and visual content. As a result, these systems frequently take shortcuts, focusing on simple visual concepts or question priors. This phenomenon becomes more problematic as the questions become complex that requires more reasoning and commonsense knowledge. To address this issue, we present a novel framework that uses explanations for competing answers to help VQA systems select the correct answer. By training on human textual explanations, our framework builds better representations for the questions and visual content, and then reweights confidences in the answer candidates using either generated or retrieved explanations from the training set. We evaluate our framework on the VQA-X dataset, which has more difficult questions with human explanations, achieving new state-of-the-art results on both VQA and its explanations.

cs.CV↗

arXiv2020

Hidden State Guidance: Improving Image Captioning using An Image Conditioned Autoencoder

Most RNN-based image captioning models receive supervision on the output words to mimic human captions. Therefore, the hidden states can only receive noisy gradient signals via layers of back-propagation through time, leading to less accurate generated captions. Consequently, we propose a novel framework, Hidden State Guidance (HSG), that matches the hidden states in the caption decoder to those in a teacher decoder trained on an easier task of autoencoding the captions conditioned on the image. During training with the REINFORCE algorithm, the conventional rewards are sentence-based evaluation metrics equally distributed to each generated word, no matter their relevance. HSG provides a word-level reward that helps the model learn better hidden representations. Experimental results demonstrate that HSG clearly outperforms various state-of-the-art caption decoders using either raw images or detected objects as inputs.

cs.CV↗

arXiv2020

Generating Question Relevant Captions to Aid Visual Question Answering

Visual question answering (VQA) and image captioning require a shared body of general knowledge connecting language and vision. We present a novel approach to improve VQA performance that exploits this connection by jointly generating captions that are targeted to help answer a specific visual question. The model is trained using an existing caption dataset by automatically determining question-relevant captions using an online gradient-based method. Experimental results on the VQA v2 challenge demonstrates that our approach obtains state-of-the-art VQA performance (e.g. 68.4% on the Test-standard set using a single model) by simultaneously generating question-relevant captions.

cs.CV↗

arXiv2019

Self-Critical Reasoning for Robust Visual Question Answering

Visual Question Answering (VQA) deep-learning systems tend to capture superficial statistical correlations in the training data because of strong language priors and fail to generalize to test data with a significantly different question-answer (QA) distribution. To address this issue, we introduce a self-critical training objective that ensures that visual explanations of correct answers match the most influential image regions more than other competitive answer candidates. The influential regions are either determined from human visual/textual explanations or automatically from just significant words in the question and answer. We evaluate our approach on the VQA generalization task using the VQA-CP dataset, achieving a new state-of-the-art i.e., 49.5% using textual explanations and 48.5% using automatically annotated regions.

cs.CV↗

arXiv2019

Dynamic Filtering with Large Sampling Field for ConvNets

We propose a dynamic filtering strategy with large sampling field for ConvNets (LS-DFN), where the position-specific kernels learn from not only the identical position but also multiple sampled neighbor regions. During sampling, residual learning is introduced to ease training and an attention mechanism is applied to fuse features from different samples. Such multiple samples enlarge the kernels' receptive fields significantly without requiring more parameters. While LS-DFN inherits the advantages of DFN, namely avoiding feature map blurring by position-wise kernels while keeping translation invariance, it also efficiently alleviates the overfitting issue caused by much more parameters than normal CNNs. Our model is efficient and can be trained end-to-end via standard back-propagation. We demonstrate the merits of our LS-DFN on both sparse and dense prediction tasks involving object detection, semantic segmentation, and flow estimation. Our results show LS-DFN enjoys stronger recognition abilities in object detection and semantic segmentation tasks on VOC benchmark and sharper responses in flow estimation on FlyingChairs dataset compared to strong baselines.

cs.CV↗

arXiv2019

Faithful Multimodal Explanation for Visual Question Answering

AI systems' ability to explain their reasoning is critical to their utility and trustworthiness. Deep neural networks have enabled significant progress on many challenging problems such as visual question answering (VQA). However, most of them are opaque black boxes with limited explanatory capability. This paper presents a novel approach to developing a high-performing VQA system that can elucidate its answers with integrated textual and visual explanations that faithfully reflect important aspects of its underlying reasoning while capturing the style of comprehensible human explanations. Extensive experimental evaluation demonstrates the advantages of this approach compared to competing methods with both automatic evaluation metrics and human evaluation metrics.

cs.CL↗

arXiv2018

Image Score: How to Select Useful Samples

There has long been debates on how we could interpret neural networks and understand the decisions our models make. Specifically, why deep neural networks tend to be error-prone when dealing with samples that output low softmax scores. We present an efficient approach to measure the confidence of decision-making steps by statistically investigating each unit's contribution to that decision. Instead of focusing on how the models react on datasets, we study the datasets themselves given a pre-trained model. Our approach is capable of assigning a score to each sample within a dataset that measures the frequency of occurrence of that sample's chain of activation. We demonstrate with experiments that our method could select useful samples to improve deep neural networks in a semi-supervised leaning setting.

cs.CV↗

arXiv2018

Joint Image Captioning and Question Answering

Answering visual questions need acquire daily common knowledge and model the semantic connection among different parts in images, which is too difficult for VQA systems to learn from images with the only supervision from answers. Meanwhile, image captioning systems with beam search strategy tend to generate similar captions and fail to diversely describe images. To address the aforementioned issues, we present a system to have these two tasks compensate with each other, which is capable of jointly producing image captions and answering visual questions. In particular, we utilize question and image features to generate question-related captions and use the generated captions as additional features to provide new knowledge to the VQA system. For image captioning, our system attains more informative results in term of the relative improvements on VQA tasks as well as competitive results using automated metrics. Applying our system to the VQA tasks, our results on VQA v2 dataset achieve 65.8% using generated captions and 69.1% using annotated captions in validation set and 68.4% in the test-standard set. Further, an ensemble of 10 models results in 69.7% in the test-standard split.

cs.CL↗

arXiv2016

Action Recognition with Joint Attention on Multi-Level Deep Features

We propose a novel deep supervised neural network for the task of action recognition in videos, which implicitly takes advantage of visual tracking and shares the robustness of both deep Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN). In our method, a multi-branch model is proposed to suppress noise from background jitters. Specifically, we firstly extract multi-level deep features from deep CNNs and feed them into 3d-convolutional network. After that we feed those feature cubes into our novel joint LSTM module to predict labels and to generate attention regularization. We evaluate our model on two challenging datasets: UCF101 and HMDB51. The results show that our model achieves the state-of-art by only using convolutional features.

cs.CV↗

arXiv2007

Fixed point of second virial coefficients in the glass transition

Classical thermodynamic theory still holds true in subsystem that is a percolation connected by 8 orders of self-similar 2-body-3-body coupling clusters. The fixed point, $B_2^* \equiv 3/8$, for the clusters of different size, existing in reduced second Virial coefficients has been proved by scaling theory in percolation field. It is shown that, if $B_2^* \equiv 3/8$ is combined with $B_3^* \equiv 5/8$, the potentials of 2-body-3-body coupling clusters, in critical local cluster growth phase transition, balance the kinetic energy in the glass transition. It is also proved that the glass transition corresponds to the regime in which the chemical potentials in all subsystems hold zero.

math-ph↗

arXiv2007

Fixed point of self-similar Lennard-Jones potentials in the glass transition

The existence of potential energy landscape in the glass transition has been theoretically proved using the recursive equation of reinforce-restraint of self-similar 8 orders of Lennard-Jones (L-J) potential fluctuations. The stability condition for fluctuation reinforce-restraint is just the Lindemann ratio that is exactly deduced as 0.1047in this paper. The origin and transfer of interface excitation comes of the balance between self-similar L-J potential fluctuation and geometric phase potential fluctuation, which also gives rise to a new attractive potential of -17/16ε_i, lower than the potential well energy -ε_i of i-th order of L-J potential, in the self-similar mean field of mean fields of different sizes. The delocalization energy of two-body is exactly the transfer energy of excited interface, and the delocalization path is along 8 orders of geodesics in topological analyses.

math-ph↗

arXiv2006

Eight orders of dynamical clusters and hard-spheres in the glass transition

The nature may be disclosed that the glass transition is only determined by the intrinsic 8 orders of instant 2-D mosaic geometric structures, without any presupposition and relevant parameter. An interface excited state on the geometric structures comes from the additional Lindemann distance increment, which is a vector with 8 orders of relaxation times, 8 orders of additional restoring force moment (ARFM), quantized energy and extra volume. Each order of anharmonic ARFM gives rise to an additional position-asymmetry on a 2-D projection plane of a reference particle, thus, in removing additional position-asymmetry, the 8 orders of 2-D clusters and hard-spheres accompanied with the 4 excited interface relaxations of the reference particle have been illustrated. Dynamical behavior comes of the slow inverse energy cascade to generate 8 orders of clusters, to thaw a solid-domain, and the fast cascade to relax tension and rearrange structure. This model provides a unified mechanism to interpret hard-sphere, compacting cluster, free volume, cage, jamming behaviors, geometrical frustration, reptation, Ising model, breaking solid lattice, percolation, cooperative migration and orientation, critical entanglement chain length and structure rearrangements. It also directly deduces a series of quantitative values for the average energy of cooperative migration in one direction, localized energy independent of temperature and the activation energy to break solid lattice. In a flexible polymer system, there are all 320 different interface excited states that have the same quantized excited energy but different interaction times, relaxation times and phases. The quantized excited energy is about 6.4 k = 0.55meV.

cond-mat.dis-nn↗