arXiv subjects

Tian Han

Publications and source records attributed to Tian Han.

At least 37 records · Page 2Linked to original sources

arXiv2023

Molecule Design by Latent Space Energy-Based Modeling and Gradual Distribution Shifting

Generation of molecules with desired chemical and biological properties such as high drug-likeness, high binding affinity to target proteins, is critical for drug discovery. In this paper, we propose a probabilistic generative model to capture the joint distribution of molecules and their properties. Our model assumes an energy-based model (EBM) in the latent space. Conditional on the latent vector, the molecule and its properties are modeled by a molecule generation model and a property regression model respectively. To search for molecules with desired properties, we propose a sampling with gradual distribution shifting (SGDS) algorithm, so that after learning the model initially on the training data of existing molecules and their properties, the proposed algorithm gradually shifts the model distribution towards the region supported by molecules with desired values of properties. Our experiments show that our method achieves very strong performances on various molecule design tasks.

q-bio.BM↗

arXiv2022

Learning Sparse Latent Representations for Generator Model

Sparsity is a desirable attribute. It can lead to more efficient and more effective representations compared to the dense model. Meanwhile, learning sparse latent representations has been a challenging problem in the field of computer vision and machine learning due to its complexity. In this paper, we present a new unsupervised learning method to enforce sparsity on the latent space for the generator model with a gradually sparsified spike and slab distribution as our prior. Our model consists of only one top-down generator network that maps the latent variable to the observed data. Latent variables can be inferred following generator posterior direction using non-persistent gradient based method. Spike and Slab regularization in the inference step can push non-informative latent dimensions towards zero to induce sparsity. Extensive experiments show the model can preserve majority of the information from original images with sparse representations while demonstrating improved results compared to other existing methods. We observe that our model can learn disentangled semantics and increase explainability of the latent codes while boosting the robustness in the task of classification and denoising.

cs.CV↗

arXiv2022

Adaptive Multi-stage Density Ratio Estimation for Learning Latent Space Energy-based Model

This paper studies the fundamental problem of learning energy-based model (EBM) in the latent space of the generator model. Learning such prior model typically requires running costly Markov Chain Monte Carlo (MCMC). Instead, we propose to use noise contrastive estimation (NCE) to discriminatively learn the EBM through density ratio estimation between the latent prior density and latent posterior density. However, the NCE typically fails to accurately estimate such density ratio given large gap between two densities. To effectively tackle this issue and learn more expressive prior models, we develop the adaptive multi-stage density ratio estimation which breaks the estimation into multiple stages and learn different stages of density ratio sequentially and adaptively. The latent prior model can be gradually learned using ratio estimated in previous stage so that the final latent space EBM prior can be naturally formed by product of ratios in different stages. The proposed method enables informative and much sharper prior than existing baselines, and can be trained efficiently. Our experiments demonstrate strong performances in image generation and reconstruction as well as anomaly detection.

cs.LG↗

arXiv2022

A Modified Union Bound on Symbol Error Probability for Fading Channels

In this paper, we propose a new upper bound on the error probability performance of maximum-likelihood (ML) detection. The proposed approach provides a much tighter upper bound when compared to the traditionally used union bound, especially when the number of pairwise error probabilities (PEPs) is large. In fact, the proposed approach tightens the union bound by first assuming that a detection error always occurs in a deep fading event where the channel gain is lower than a certain threshold. A minimisation is then taken with respect to the gain threshold in order to make the upper bound as tight as possible. We also prove that the objective function has a single minimiser under several general assumptions so that the minimiser can be easily found using optimisation algorithms. The expression of the new upper bound under correlated Rayleigh fading channels is derived and several analytical and numerical examples are provided to show the performance of the proposed bound.

cs.IT↗

arXiv2022

Combined Radar and Communications with Phase-Modulated Frequency Permutations

This paper focuses on the combined radar and communications problem and conducts a thorough analytical investigation on the effect of phase and frequency change on the communication and sensing functionality. First, we consider the classical stepped frequency radar waveform and modulate data using M-ary phase shift keying (MPSK). Two important analytical tools in radar waveform design, namely the ambiguity function (AF) and the Fisher information matrix (FIM) are derived, based on which, we make the important conclusion that MPSK modulation has a negligible effect on radar local accuracy. Next, we extend the analysis to incorporate frequency permutations and propose a new signalling scheme in which the mapping between incoming data and waveforms is performed based on an efficient combinatorial transform called the Lehmer code. We also provide an efficient communications receiver based on the Hungarian algorithm. From the communications perspective, we consider the optimal maximum likelihood (ML) detector and derive the union bound and nearest neighbour approximation on the block error probability. From the radar sensing perspective, we discuss the broader structure of the waveform based on the AF derivation and quantify the radar local accuracy based on the FIM. Extensive numerical examples are provided to illustrate the accuracy of our results.

cs.IT↗

arXiv2021

Context-aware Health Event Prediction via Transition Functions on Dynamic Disease Graphs

With the wide application of electronic health records (EHR) in healthcare facilities, health event prediction with deep learning has gained more and more attention. A common feature of EHR data used for deep-learning-based predictions is historical diagnoses. Existing work mainly regards a diagnosis as an independent disease and does not consider clinical relations among diseases in a visit. Many machine learning approaches assume disease representations are static in different visits of a patient. However, in real practice, multiple diseases that are frequently diagnosed at the same time reflect hidden patterns that are conducive to prognosis. Moreover, the development of a disease is not static since some diseases can emerge or disappear and show various symptoms in different visits of a patient. To effectively utilize this combinational disease information and explore the dynamics of diseases, we propose a novel context-aware learning framework using transition functions on dynamic disease graphs. Specifically, we construct a global disease co-occurrence graph with multiple node properties for disease combinations. We design dynamic subgraphs for each patient's visit to leverage global and local contexts. We further define three diagnosis roles in each visit based on the variation of node properties to model disease transition processes. Experimental results on two real-world EHR datasets show that the proposed model outperforms state of the art in predicting health events.

cs.LG↗

arXiv2021

Learning from the Tangram to Solve Mini Visual Tasks

Current pre-training methods in computer vision focus on natural images in the daily-life context. However, abstract diagrams such as icons and symbols are common and important in the real world. This work is inspired by Tangram, a game that requires replicating an abstract pattern from seven dissected shapes. By recording human experience in solving tangram puzzles, we present the Tangram dataset and show that a pre-trained neural model on the Tangram helps solve some mini visual tasks based on low-resolution vision. Extensive experiments demonstrate that our proposed method generates intelligent solutions for aesthetic tasks such as folding clothes and evaluating room layouts. The pre-trained feature extractor can facilitate the convergence of few-shot learning tasks on human handwriting and improve the accuracy in identifying icons by their contours. The Tangram dataset is available at https://github.com/yizhouzhao/Tangram.

cs.CV↗

arXiv2021

GenMotion: Data-driven Motion Generators for Real-time Animation Synthesis

With the recent success of deep learning algorithms, many researchers have focused on generative models for human motion animation. However, the research community lacks a platform for training and benchmarking various algorithms, and the animation industry needs a toolkit for implementing advanced motion synthesizing techniques. To facilitate the study of deep motion synthesis methods for skeleton-based human animation and their potential applications in practical animation making, we introduce \genmotion: a library that provides unified pipelines for data loading, model training, and animation sampling with various deep learning algorithms. Besides, by combining Python coding in the animation software \genmotion\ can assist animators in creating real-time 3D character animation. Source code is available at https://github.com/realvcla/GenMotion/.

cs.GR↗

arXiv2021

Frequency Permutations for Joint Radar and Communications

This paper presents a new joint radar and communication technique based on the classical stepped frequency radar waveform. The randomization in the waveform, which is achieved by using permutations of the sequence of frequency tones, is utilized for data transmission. A new signaling scheme is proposed in which the mapping between incoming data and waveforms is performed based on an efficient combinatorial transform called the Lehmer code. Considering the optimum maximum likelihood (ML) detection, the union bound and the nearest neighbour approximation on the communication block error probability is derived for communication in an additive white Gaussian noise (AWGN) channel. The results are further extended to incorporate the Rician fading channel model, of which the Rayleigh fading channel model is presented as a special case. Furthermore, an efficient communication receiver implementation is discussed based on the Hungarian algorithm which achieves optimum performance with much less operational complexity when compared to an exhaustive search. From the radar perspective, two key analytical tools, namely, the ambiguity function (AF) and the Fisher information matrix are derived. Furthermore, accurate approximations to the Cramer-Rao lower bounds (CRLBs) on the delay and Doppler estimation errors are derived based on which the range and velocity estimation accuracy of the waveform is analysed. Numerical examples are used to highlight the accuracy of the analysis and to illustrate the performance of the proposed waveform.

cs.IT↗

arXiv2021

Proceedings of ICML 2021 Workshop on Theoretic Foundation, Criticism, and Application Trend of Explainable AI

This is the Proceedings of ICML 2021 Workshop on Theoretic Foundation, Criticism, and Application Trend of Explainable AI. Deep neural networks (DNNs) have undoubtedly brought great success to a wide range of applications in computer vision, computational linguistics, and AI. However, foundational principles underlying the DNNs' success and their resilience to adversarial attacks are still largely missing. Interpreting and theorizing the internal mechanisms of DNNs becomes a compelling yet controversial topic. This workshop pays a special interest in theoretic foundations, limitations, and new application trends in the scope of XAI. These issues reflect new bottlenecks in the future development of XAI.

cs.LG↗

arXiv2021

STAR: Sparse Transformer-based Action Recognition

The cognitive system for human action and behavior has evolved into a deep learning regime, and especially the advent of Graph Convolution Networks has transformed the field in recent years. However, previous works have mainly focused on over-parameterized and complex models based on dense graph convolution networks, resulting in low efficiency in training and inference. Meanwhile, the Transformer architecture-based model has not yet been well explored for cognitive application in human action and behavior estimation. This work proposes a novel skeleton-based human action recognition model with sparse attention on the spatial dimension and segmented linear attention on the temporal dimension of data. Our model can also process the variable length of video clips grouped as a single batch. Experiments show that our model can achieve comparable performance while utilizing much less trainable parameters and achieve high speed in training and inference. Experiments show that our model achieves 4~18x speedup and 1/7~1/15 model size compared with the baseline models at competitive accuracy.

cs.CV↗

arXiv2021

Generative Text Modeling through Short Run Inference

Latent variable models for text, when trained successfully, accurately model the data distribution and capture global semantic and syntactic features of sentences. The prominent approach to train such models is variational autoencoders (VAE). It is nevertheless challenging to train and often results in a trivial local optimum where the latent variable is ignored and its posterior collapses into the prior, an issue known as posterior collapse. Various techniques have been proposed to mitigate this issue. Most of them focus on improving the inference model to yield latent codes of higher quality. The present work proposes a short run dynamics for inference. It is initialized from the prior distribution of the latent variable and then runs a small number (e.g., 20) of Langevin dynamics steps guided by its posterior distribution. The major advantage of our method is that it does not require a separate inference model or assume simple geometry of the posterior distribution, thus rendering an automatic, natural and flexible inference engine. We show that the models trained with short run dynamics more accurately model the data, compared to strong language model and VAE baselines, and exhibit no sign of posterior collapse. Analyses of the latent space show that interpolation in the latent space is able to generate coherent sentences with smooth transition and demonstrate improved classification over strong baselines with latent features from unsupervised pretraining. These results together expose a well-structured latent space of our generative model.

cs.LG↗

arXiv2020

Learning Latent Space Energy-Based Prior Model

We propose to learn energy-based model (EBM) in the latent space of a generator model, so that the EBM serves as a prior model that stands on the top-down network of the generator model. Both the latent space EBM and the top-down network can be learned jointly by maximum likelihood, which involves short-run MCMC sampling from both the prior and posterior distributions of the latent vector. Due to the low dimensionality of the latent space and the expressiveness of the top-down network, a simple EBM in latent space can capture regularities in the data effectively, and MCMC sampling in latent space is efficient and mixes well. We show that the learned model exhibits strong performances in terms of image and text generation and anomaly detection. The one-page code can be found in supplementary materials.

stat.ML↗

arXiv2020

Learning Latent Space Energy-Based Prior Model for Molecule Generation

Deep generative models have recently been applied to molecule design. If the molecules are encoded in linear SMILES strings, modeling becomes convenient. However, models relying on string representations tend to generate invalid samples and duplicates. Prior work addressed these issues by building models on chemically-valid fragments or explicitly enforcing chemical rules in the generation process. We argue that an expressive model is sufficient to implicitly and automatically learn the complicated chemical rules from the data, even if molecules are encoded in simple character-level SMILES strings. We propose to learn latent space energy-based prior model with SMILES representation for molecule modeling. Our experiments show that our method is able to generate molecules with validity and uniqueness competitive with state-of-the-art models. Interestingly, generated molecules have structural and chemical features whose distributions almost perfectly match those of the real molecules.

cs.LG↗

arXiv2020

Semi-supervised Learning by Latent Space Energy-Based Model of Symbol-Vector Coupling

This paper proposes a latent space energy-based prior model for semi-supervised learning. The model stands on a generator network that maps a latent vector to the observed example. The energy term of the prior model couples the latent vector and a symbolic one-hot vector, so that classification can be based on the latent vector inferred from the observed example. In our learning method, the symbol-vector coupling, the generator network and the inference network are learned jointly. Our method is applicable to semi-supervised learning in various data domains such as image, text, and tabular data. Our experiments demonstrate that our method performs well on semi-supervised learning tasks.

cs.LG↗

arXiv2020

Learning Multi-layer Latent Variable Model via Variational Optimization of Short Run MCMC for Approximate Inference

This paper studies the fundamental problem of learning deep generative models that consist of multiple layers of latent variables organized in top-down architectures. Such models have high expressivity and allow for learning hierarchical representations. Learning such a generative model requires inferring the latent variables for each training example based on the posterior distribution of these latent variables. The inference typically requires Markov chain Monte Caro (MCMC) that can be time consuming. In this paper, we propose to use noise initialized non-persistent short run MCMC, such as finite step Langevin dynamics initialized from the prior distribution of the latent variables, as an approximate inference engine, where the step size of the Langevin dynamics is variationally optimized by minimizing the Kullback-Leibler divergence between the distribution produced by the short run MCMC and the posterior distribution. Our experiments show that the proposed method outperforms variational auto-encoder (VAE) in terms of reconstruction error and synthesis quality. The advantage of the proposed method is that it is simple and automatic without the need to design an inference model.

stat.ML↗

arXiv2020

Joint Training of Variational Auto-Encoder and Latent Energy-Based Model

This paper proposes a joint training method to learn both the variational auto-encoder (VAE) and the latent energy-based model (EBM). The joint training of VAE and latent EBM are based on an objective function that consists of three Kullback-Leibler divergences between three joint distributions on the latent vector and the image, and the objective function is of an elegant symmetric and anti-symmetric form of divergence triangle that seamlessly integrates variational and adversarial learning. In this joint training scheme, the latent EBM serves as a critic of the generator model, while the generator model and the inference model in VAE serve as the approximate synthesis sampler and inference sampler of the latent EBM. Our experiments show that the joint training greatly improves the synthesis quality of the VAE. It also enables learning of an energy function that is capable of detecting out of sample examples for anomaly detection.

cs.CV↗

arXiv2019

On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Models

This study investigates the effects of Markov chain Monte Carlo (MCMC) sampling in unsupervised Maximum Likelihood (ML) learning. Our attention is restricted to the family of unnormalized probability densities for which the negative log density (or energy function) is a ConvNet. We find that many of the techniques used to stabilize training in previous studies are not necessary. ML learning with a ConvNet potential requires only a few hyper-parameters and no regularization. Using this minimal framework, we identify a variety of ML learning outcomes that depend solely on the implementation of MCMC sampling. On one hand, we show that it is easy to train an energy-based model which can sample realistic images with short-run Langevin. ML can be effective and stable even when MCMC samples have much higher energy than true steady-state samples throughout training. Based on this insight, we introduce an ML method with purely noise-initialized MCMC, high-quality short-run synthesis, and the same budget as ML with informative MCMC initialization such as CD or PCD. Unlike previous models, our energy model can obtain realistic high-diversity samples from a noise signal after training. On the other hand, ConvNet potentials learned with non-convergent MCMC do not have a valid steady-state and cannot be considered approximate unnormalized densities of the training data because long-run MCMC samples differ greatly from observed images. We show that it is much harder to train a ConvNet potential to learn a steady-state over realistic images. To our knowledge, long-run MCMC samples of all previous models lose the realism of short-run samples. With correct tuning of Langevin noise, we train the first ConvNet potentials for which long-run and steady-state MCMC samples are realistic images.

stat.ML↗