Geetakrishnasai Gunapati

arXiv2022

Continuous Depth Recurrent Neural Differential Equations

Recurrent neural networks (RNNs) have brought a lot of advancements in sequence labeling tasks and sequence data. However, their effectiveness is limited when the observations in the sequence are irregularly sampled, where the observations arrive at irregular time intervals. To address this, continuous time variants of the RNNs were introduced based on neural ordinary differential equations (NODE). They learn a better representation of the data using the continuous transformation of hidden states over time, taking into account the time interval between the observations. However, they are still limited in their capability as they use the discrete transformations and a fixed discrete number of layers (depth) over an input in the sequence to produce the output observation. We intend to address this limitation by proposing RNNs based on differential equations which model continuous transformations over both depth and time to predict an output for a given input in the sequence. Specifically, we propose continuous depth recurrent neural differential equations (CDR-NDE) which generalizes RNN models by continuously evolving the hidden states in both the temporal and depth dimensions. CDR-NDE considers two separate differential equations over each of these dimensions and models the evolution in the temporal and depth directions alternatively. We also propose the CDR-NDE-heat model based on partial differential equations which treats the computation of hidden states as solving a heat equation over time. We demonstrate the effectiveness of the proposed models by comparing against the state-of-the-art RNN models on real world sequence labeling problems and data.

cs.LG↗

arXiv2021

Variational Inference as an alternative to MCMC for parameter estimation and model selection

Most applications of Bayesian Inference for parameter estimation and model selection in astrophysics involve the use of Monte Carlo techniques such as Markov Chain Monte Carlo (MCMC) and nested sampling. However, these techniques are time consuming and their convergence to the posterior could be difficult to determine. In this work, we advocate Variational inference as an alternative to solve the above problems, and demonstrate its usefulness for parameter estimation and model selection in Astrophysics. Variational inference converts the inference problem into an optimization problem by approximating the posterior from a known family of distributions and using Kullback-Leibler divergence to characterize the difference. It takes advantage of fast optimization techniques, which make it ideal to deal with large datasets and makes it trivial to parallelize on a multicore platform. We also derive a new approximate evidence estimation based on variational posterior, and importance sampling technique called posterior weighted importance sampling for the calculation of evidence (PWISE), which is useful to perform Bayesian model selection. As a proof of principle, we apply variational inference to five different problems in astrophysics, where Monte Carlo techniques were previously used. These include assessment of significance of annual modulation in the COSINE-100 dark matter experiment, measuring exoplanet orbital parameters from radial velocity data, tests of periodicities in measurements of Newton's constant $G$, assessing the significance of a turnover in the spectral lag data of GRB 160625B and estimating the mass of a galaxy cluster using weak gravitational lensing. We find that variational inference is much faster than MCMC and nested sampling techniques for most of these problems while providing competitive results. All our analysis codes have been made publicly available.

astro-ph.IM↗