arXiv subjects

He Zhao

Publications and source records attributed to He Zhao.

At least 127 records · Page 7Linked to original sources

arXiv2022

Leveraging Cross Feedback of User and Item Embeddings with Attention for Variational Autoencoder based Collaborative Filtering

Matrix factorization (MF) has been widely applied to collaborative filtering in recommendation systems. Its Bayesian variants can derive posterior distributions of user and item embeddings, and are more robust to sparse ratings. However, the Bayesian methods are restricted by their update rules for the posterior parameters due to the conjugacy of the priors and the likelihood. Variational autoencoders (VAE) can address this issue by capturing complex mappings between the posterior parameters and the data. However, current research on VAEs for collaborative filtering only considers the mappings based on the explicit data information while the implicit embedding information is overlooked. In this paper, we first derive evidence lower bounds (ELBO) for Bayesian MF models from two viewpoints: user-oriented and item-oriented. Based on the ELBOs, we propose a VAE-based Bayesian MF framework. It leverages not only the data but also the embedding information to approximate the user-item joint distribution. As suggested by the ELBOs, the approximation is iterative with cross feedback of user and item embeddings into each other's encoders. More specifically, user embeddings sampled at the previous iteration are fed to the item-side encoders to estimate the posterior parameters for the item embeddings at the current iteration, and vice versa. The estimation also attends to the cross-fed embeddings to further exploit useful information. The decoder then reconstructs the data via the matrix factorization over the currently re-sampled user and item embeddings.

cs.LG↗

arXiv2022

Sports Video Analysis on Large-Scale Data

This paper investigates the modeling of automated machine description on sports video, which has seen much progress recently. Nevertheless, state-of-the-art approaches fall quite short of capturing how human experts analyze sports scenes. There are several major reasons: (1) The used dataset is collected from non-official providers, which naturally creates a gap between models trained on those datasets and real-world applications; (2) previously proposed methods require extensive annotation efforts (i.e., player and ball segmentation at pixel level) on localizing useful visual features to yield acceptable results; (3) very few public datasets are available. In this paper, we propose a novel large-scale NBA dataset for Sports Video Analysis (NSVA) with a focus on captioning, to address the above challenges. We also design a unified approach to process raw videos into a stack of meaningful features with minimum labelling efforts, showing that cross modeling on such features using a transformer architecture leads to strong performance. In addition, we demonstrate the broad application of NSVA by addressing two additional tasks, namely fine-grained sports action recognition and salient player identification. Code and dataset are available at https://github.com/jackwu502/NSVA.

cs.CV↗

arXiv2022

Learning to Re-weight Examples with Optimal Transport for Imbalanced Classification

Imbalanced data pose challenges for deep learning based classification models. One of the most widely-used approaches for tackling imbalanced data is re-weighting, where training samples are associated with different weights in the loss function. Most of existing re-weighting approaches treat the example weights as the learnable parameter and optimize the weights on the meta set, entailing expensive bilevel optimization. In this paper, we propose a novel re-weighting method based on optimal transport (OT) from a distributional point of view. Specifically, we view the training set as an imbalanced distribution over its samples, which is transported by OT to a balanced distribution obtained from the meta set. The weights of the training samples are the probability mass of the imbalanced distribution and learned by minimizing the OT distance between the two distributions. Compared with existing methods, our proposed one disengages the dependence of the weight learning on the concerned classifier at each iteration. Experiments on image, text and point cloud datasets demonstrate that our proposed re-weighting method has excellent performance, achieving state-of-the-art results in many cases and providing a promising tool for addressing the imbalanced classification issue.

cs.LG↗

arXiv2022

Neural Topic Model via Optimal Transport

Recently, Neural Topic Models (NTMs) inspired by variational autoencoders have obtained increasingly research interest due to their promising results on text analysis. However, it is usually hard for existing NTMs to achieve good document representation and coherent/diverse topics at the same time. Moreover, they often degrade their performance severely on short documents. The requirement of reparameterisation could also comprise their training quality and model flexibility. To address these shortcomings, we present a new neural topic model via the theory of optimal transport (OT). Specifically, we propose to learn the topic distribution of a document by directly minimising its OT distance to the document's word distributions. Importantly, the cost matrix of the OT distance models the weights between topics and words, which is constructed by the distances between topics and words in an embedding space. Our proposed model can be trained efficiently with a differentiable loss. Extensive experiments show that our framework significantly outperforms the state-of-the-art NTMs on discovering more coherent and diverse topics and deriving better document representations for both regular and short texts.

cs.IR↗

arXiv2022

Spin-polarized imaging of the antiferromagnetic structure and field-tunable bound states in kagome magnet FeSn

Kagome metals are as an exciting playground for the explorations of novel phenomena at the intersection of topology, electron correlations and magnetism. The family of FeSn-based kagome magnets in particular attracted a lot of attention for simplicity of the layered crystal structure and tunable topological electronic band structure. Despite a significant progress in understanding their bulk properties, surface electronic and magnetic structures are yet to be fully explored in many of these systems. In this work, we focus on a prototypical kagome metal FeSn. Using a combination of spin-averaged and spin-polarized scanning tunneling microscopy, we provide the first atomic-scale visualization of the layered antiferromagnetic structure at the surface of FeSn. In contrast to the field-tunable electronic structure of cousin material Fe3Sn2 that is a ferromagnet, we find that electronic density-of-states of FeSn is robust to the application of external magnetic field. Interestingly, despite the field-insensitive electronic band structure, FeSn exhibits bounds states tied to specific impurities with large effective moments that strongly couple to the magnetic field. Our experiments provide microscopic insights necessary for theoretical modeling of FeSn and serve as a spring board for spin-polarized measurements of topological magnets in general.

cond-mat.mtrl-sci↗

arXiv2022

P3IV: Probabilistic Procedure Planning from Instructional Videos with Weak Supervision

In this paper, we study the problem of procedure planning in instructional videos. Here, an agent must produce a plausible sequence of actions that can transform the environment from a given start to a desired goal state. When learning procedure planning from instructional videos, most recent work leverages intermediate visual observations as supervision, which requires expensive annotation efforts to localize precisely all the instructional steps in training videos. In contrast, we remove the need for expensive temporal video annotations and propose a weakly supervised approach by learning from natural language instructions. Our model is based on a transformer equipped with a memory module, which maps the start and goal observations to a sequence of plausible actions. Furthermore, we augment our model with a probabilistic generative module to capture the uncertainty inherent to procedure planning, an aspect largely overlooked by previous work. We evaluate our model on three datasets and show our weaklysupervised approach outperforms previous fully supervised state-of-the-art models on multiple metrics.

cs.CV↗

arXiv2022

Manipulation of Dirac band curvature and momentum-dependent g-factor in a kagome magnet YMn6Sn6

The Zeeman effect describes the energy change of an atomic quantum state in magnetic field. The magnitude and the direction of this change depend on the dimensionless Lande g-factor. In quantum solids, the response of the Bloch electron states to the magnetic field also exhibits the Zeeman effect with an effective g-factor that was theoretically predicted to be dependent on the momentum. While typically negligible in many ordinary solids, the momentum-dependent variation of the g-factor is theorized to be substantially enhanced in many topological and magnetic systems. However, the momentum-dependence of the g-factor is notoriously difficult to extract and it is yet to be directly experimentally measured. In this work, we report the experimental discovery of a strongly momentum-dependent g-factor in a kagome magnet YMn6Sn6. Using spectroscopic-imaging scanning tunneling microscopy, we map the evolution of a massive Dirac band in the vicinity of the Fermi level as a function of magnetic field. We find that electronic states at different lattice momenta exhibit markedly different Zeeman energy shifts, giving rise to an anomalous g-factor that peaks around the Dirac point. Our work provides the first momentum-resolved visualization of Dirac band curvature manipulation by magnetic field, which should in principle be highly relevant to other topological kagome magnets.

cond-mat.str-el↗

arXiv2022

Representing Mixtures of Word Embeddings with Mixtures of Topic Embeddings

A topic model is often formulated as a generative model that explains how each word of a document is generated given a set of topics and document-specific topic proportions. It is focused on capturing the word co-occurrences in a document and hence often suffers from poor performance in analyzing short documents. In addition, its parameter estimation often relies on approximate posterior inference that is either not scalable or suffers from large approximation error. This paper introduces a new topic-modeling framework where each document is viewed as a set of word embedding vectors and each topic is modeled as an embedding vector in the same embedding space. Embedding the words and topics in the same vector space, we define a method to measure the semantic difference between the embedding vectors of the words of a document and these of the topics, and optimize the topic embeddings to minimize the expected difference over all documents. Experiments on text analysis demonstrate that the proposed method, which is amenable to mini-batch stochastic gradient descent based optimization and hence scalable to big corpora, provides competitive performance in discovering more coherent and diverse topics and extracting better document representations.

cs.LG↗

arXiv2022

A Unified Wasserstein Distributional Robustness Framework for Adversarial Training

It is well-known that deep neural networks (DNNs) are susceptible to adversarial attacks, exposing a severe fragility of deep learning systems. As the result, adversarial training (AT) method, by incorporating adversarial examples during training, represents a natural and effective approach to strengthen the robustness of a DNN-based classifier. However, most AT-based methods, notably PGD-AT and TRADES, typically seek a pointwise adversary that generates the worst-case adversarial example by independently perturbing each data sample, as a way to "probe" the vulnerability of the classifier. Arguably, there are unexplored benefits in considering such adversarial effects from an entire distribution. To this end, this paper presents a unified framework that connects Wasserstein distributional robustness with current state-of-the-art AT methods. We introduce a new Wasserstein cost function and a new series of risk functions, with which we show that standard AT methods are special cases of their counterparts in our framework. This connection leads to an intuitive relaxation and generalization of existing AT methods and facilitates the development of a new family of distributional robustness AT-based algorithms. Extensive experiments show that our distributional robustness AT algorithms robustify further their standard AT counterparts in various settings.

cs.LG↗

arXiv2022

Learning to Attack with Fewer Pixels: A Probabilistic Post-hoc Framework for Refining Arbitrary Dense Adversarial Attacks

Deep neural network image classifiers are reported to be susceptible to adversarial evasion attacks, which use carefully crafted images created to mislead a classifier. Many adversarial attacks belong to the category of dense attacks, which generate adversarial examples by perturbing all the pixels of a natural image. To generate sparse perturbations, sparse attacks have been recently developed, which are usually independent attacks derived by modifying a dense attack's algorithm with sparsity regularisations, resulting in reduced attack efficiency. In this paper, we aim to tackle this task from a different perspective. We select the most effective perturbations from the ones generated from a dense attack, based on the fact we find that a considerable amount of the perturbations on an image generated by dense attacks may contribute little to attacking a classifier. Accordingly, we propose a probabilistic post-hoc framework that refines given dense attacks by significantly reducing the number of perturbed pixels but keeping their attack power, trained with mutual information maximisation. Given an arbitrary dense attack, the proposed model enjoys appealing compatibility for making its adversarial images more realistic and less detectable with fewer perturbations. Moreover, our framework performs adversarial attacks much faster than existing sparse attacks.

cs.CV↗

arXiv2022

Dust Mass Associated with the Supernova Remnant IC 443 when Emission Meets Extinction

The dust mass of the well-known supernova remnant (SNR) IC 443 is estimated from both the infrared emission and the visual extinction. With photometry to the images taken by \emph{Spitzer}, \emph{WISE}, \emph{IRAS}, \emph{AKARI} and \emph{Planck}, the spectral energy distribution (SED) of the dust is obtained after subtracting the synchrotron radiation and considering the spectral line emission. The dust mass is derived from fitting the SED by a two-component model, which results in a warm component of the temperature of $\sim$ 53 K and the mass of 0.1 $M_\odot$, and a cold component of the temperature of $\sim 17$ K and the mass of 46 $M_\odot$. On the other hand, the dust mass is derived to be $\sim$ 66 $M_\odot$ from the visual extinction of IC 443 which is identified from the 3D Bayestar extinction map and its coincidence with the infrared emission morphology. Roughly the dust mass derived from the infrared emission and the extinction agree mutually. However, the dust mass derived from the infrared emission can be adjusted to be more consistent with that from the extinction by using different dust opacity property or considering optically thick radiation. In addition, the distribution of dust temperature and mass is analyzed by fitting the SED pixel by pixel.

astro-ph.HE↗

arXiv2022

Improving Ensemble Robustness by Collaboratively Promoting and Demoting Adversarial Robustness

Ensemble-based adversarial training is a principled approach to achieve robustness against adversarial attacks. An important technique of this approach is to control the transferability of adversarial examples among ensemble members. We propose in this work a simple yet effective strategy to collaborate among committee models of an ensemble model. This is achieved via the secure and insecure sets defined for each model member on a given sample, hence help us to quantify and regularize the transferability. Consequently, our proposed framework provides the flexibility to reduce the adversarial transferability as well as to promote the diversity of ensemble members, which are two crucial factors for better robustness in our ensemble approach. We conduct extensive and comprehensive experiments to demonstrate that our proposed method outperforms the state-of-the-art ensemble baselines, at the same time can detect a wide range of adversarial examples with a nearly perfect accuracy. Our code is available at: https://github.com/tuananhbui89/Crossing-Collaborative-Ensemble.

cs.CV↗

arXiv2022

MED-TEX: Transferring and Explaining Knowledge with Less Data from Pretrained Medical Imaging Models

Deep learning methods usually require a large amount of training data and lack interpretability. In this paper, we propose a novel knowledge distillation and model interpretation framework for medical image classification that jointly solves the above two issues. Specifically, to address the data-hungry issue, a small student model is learned with less data by distilling knowledge from a cumbersome pretrained teacher model. To interpret the teacher model and assist the learning of the student, an explainer module is introduced to highlight the regions of an input that are important for the predictions of the teacher model. Furthermore, the joint framework is trained by a principled way derived from the information-theoretic perspective. Our framework outperforms on the knowledge distillation and model interpretation tasks compared to state-of-the-art methods on a fundus dataset.

cs.CV↗

arXiv2021

The Dust Mass of Supernova Remnants in M31

The dust temperature and mass of the supernova remnants (SNRs) in M31 are estimated by fitting the infrared spectral energy distribution calculated from the images in the Spitzer/IRAC4 and MIPS24, Herschel/PACS70, 100, 160, and Herschel/SPIRE250, 350$μ$m band. Twenty SNRs with relatively reliable photometry exhibit an average dust temperature of $20.1^{+1.8}_{-1.5}$K, which is higher than the surrounding and indicating the heating effect of supernova explosion. The dust mass of these SNRs ranges from about 100 to 800$ M_{\odot}$, much bigger than the SNRs in the Milky Way. On the other hand, this yields the dust surface density of $0.10^{+0.07}_{-0.04}{ M_{\odot} \rm pc^{-2}}$, about half of the surrounding area, which implies that about half dust in the SNRs is destroyed by the supernova explosion. The dust temperature, the radius, and thus the dust mass all demonstrate that the studied SNRs are old and very likely in the snowplow or even fade away phase because of the limitation by the far distance and observation resolution of M31, and the results can serve as a reference to the final effect of supernova explosion on the surrounding dust.

astro-ph.GA↗

arXiv2021

Understanding and Achieving Efficient Robustness with Adversarial Supervised Contrastive Learning

Contrastive learning (CL) has recently emerged as an effective approach to learning representation in a range of downstream tasks. Central to this approach is the selection of positive (similar) and negative (dissimilar) sets to provide the model the opportunity to `contrast' between data and class representation in the latent space. In this paper, we investigate CL for improving model robustness using adversarial samples. We first designed and performed a comprehensive study to understand how adversarial vulnerability behaves in the latent space. Based on this empirical evidence, we propose an effective and efficient supervised contrastive learning to achieve model robustness against adversarial attacks. Moreover, we propose a new sample selection strategy that optimizes the positive/negative sets by removing redundancy and improving correlation with the anchor. Extensive experiments show that our Adversarial Supervised Contrastive Learning (ASCL) approach achieves comparable performance with the state-of-the-art defenses while significantly outperforms other CL-based defense methods by using only $42.8\%$ positives and $6.3\%$ negatives.

cs.LG↗

arXiv2021

Neural Attention-Aware Hierarchical Topic Model

Neural topic models (NTMs) apply deep neural networks to topic modelling. Despite their success, NTMs generally ignore two important aspects: (1) only document-level word count information is utilized for the training, while more fine-grained sentence-level information is ignored, and (2) external semantic knowledge regarding documents, sentences and words are not exploited for the training. To address these issues, we propose a variational autoencoder (VAE) NTM model that jointly reconstructs the sentence and document word counts using combinations of bag-of-words (BoW) topical embeddings and pre-trained semantic embeddings. The pre-trained embeddings are first transformed into a common latent topical space to align their semantics with the BoW embeddings. Our model also features hierarchical KL divergence to leverage embeddings of each document to regularize those of their sentences, thereby paying more attention to semantically relevant sentences. Both quantitative and qualitative experiments have shown the efficacy of our model in 1) lowering the reconstruction errors at both the sentence and document levels, and 2) discovering more coherent topics from real-world datasets.

cs.CL↗

arXiv2021

The Extinction and Distance of the MBM Molecular Clouds at High Galactic Latitude

Based on the accurate color excess $E_{\rm G_{BP},G_{RP}}$ of more than 4 million stars and $E_{\rm NUV,G_{BP}}$ of more than 1 million stars from \citet{2021ApJS..254...38S}, the distance and the extinction of the molecular clouds in the MBM catalog at $|b|>20^{\circ}$ are studied in combination with the distance measurement of \emph{Gaia}/EDR3. The distance as well as the color excess is determined for 66 molecular clouds. The color excess ratio $E_{\rm G_{BP},G_{RP}}/E_{\rm NUV,G_{BP}}$ is derived for 39 of them, which is obviously larger and implies more small particles at smaller extinction. In addition, the scale height of the dust disk is found to be about 100 pc and becomes large at the anticenter direction due to the disk flaring.

astro-ph.GA↗

arXiv2021

Review of Video Predictive Understanding: Early Action Recognition and Future Action Prediction

Video predictive understanding encompasses a wide range of efforts that are concerned with the anticipation of the unobserved future from the current as well as historical video observations. Action prediction is a major sub-area of video predictive understanding and is the focus of this review. This sub-area has two major subdivisions: early action recognition and future action prediction. Early action recognition is concerned with recognizing an ongoing action as soon as possible. Future action prediction is concerned with the anticipation of actions that follow those previously observed. In either case, the \textbf{\textit{causal}} relationship between the past, current, and potential future information is the main focus. Various mathematical tools such as Markov Chains, Gaussian Processes, Auto-Regressive modeling, and Bayesian recursive filtering are widely adopted jointly with computer vision techniques for these two tasks. However, these approaches face challenges such as the curse of dimensionality, poor generalization, and constraints from domain-specific knowledge. Recently, structures that rely on deep convolutional neural networks and recurrent neural networks have been extensively proposed for improving the performance of existing vision tasks, in general, and action prediction tasks, in particular. However, they have their own shortcomings, \eg reliance on massive training data and lack of strong theoretical underpinnings. In this survey, we start by introducing the major sub-areas of the broad area of video predictive understanding, which recently have received intensive attention and proven to have practical value. Next, a thorough review of various early action recognition and future action prediction algorithms are provided with suitably organized divisions. Finally, we conclude our discussion with future research directions.

cs.CV↗