arXiv subjects

Sourish Das

Publications and source records attributed to Sourish Das.

27 records · Page 2Linked to original sources

arXiv2018

Regularization and Variable Selection with Copula Prior

In this work, we show that under specific choices of the copula, the lasso, elastic net, and $g$-prior are particular cases of `copula prior,' for regularization and variable selection method. We present `lasso with Gauss copula prior' and `lasso with t-copula prior.' The simulation study and real-world data for regression, classification, and large time-series data show that the `copula prior' often outperforms the lasso and elastic net while having a comparable sparsity of representation. Also, the copula prior encourages a grouping effect. The strongly correlated predictors tend to be in or out of the model collectively under the copula prior. The `copula prior' is a generic method, which can be used to define the new prior distribution. The application of copulas in modeling prior distribution for Bayesian methodology has not been explored much. We present the resampling-based optimization procedure to handle big data with copula prior.

stat.ME↗

arXiv2018

Characterization of catastrophic instabilities: Market crashes as paradigm

Catastrophic events, though rare, do occur and when they occur, they have devastating effects. It is, therefore, of utmost importance to understand the complexity of the underlying dynamics and signatures of catastrophic events, such as market crashes. For deeper understanding, we choose the US and Japanese markets from 1985 onward, and study the evolution of the cross-correlation structures of stock return matrices and their eigenspectra over different short time-intervals or "epochs". A slight non-linear distortion is applied to the correlation matrix computed for any epoch, leading to the emerging spectrum of eigenvalues. The statistical properties of the emerging spectrum display: (i) the shape of the emerging spectrum reflects the market instability, (ii) the smallest eigenvalue may be able to statistically distinguish the nature of a market turbulence or crisis -- internal instability or external shock, and (iii) the time-lagged smallest eigenvalue has a statistically significant correlation with the mean market cross-correlation. The smallest eigenvalue seems to indicate that the financial market has become more turbulent in a similar way as the mean does. Yet we show features of the smallest eigenvalue of the emerging spectrum that distinguish different types of market instabilities related to internal or external causes. Based on the paradigmatic character of financial time series for other complex systems, the capacity of the emerging spectrum to understand the nature of instability may be a new feature, which can be broadly applied.

q-fin.GN↗

arXiv2017

Clustering Mixed Datasets Using Homogeneity Analysis with Applications to Big Data

Datasets with a mixture of numerical and categorical attributes are routinely encountered in many application domains. In this work we examine an approach to clustering such datasets using homogeneity analysis. Homogeneity analysis determines a euclidean representation of the data. This can be analyzed by leveraging the large body of tools and techniques for data with a euclidean representation. Experiments conducted as part of this study suggest that this approach can be useful in the analysis and exploration of big datasets with a mixture of numerical and categorical attributes.

stat.ML↗

arXiv2017

Big Data Classification Using Augmented Decision Trees

We present an algorithm for classification tasks on big data. Experiments conducted as part of this study indicate that the algorithm can be as accurate as ensemble methods such as random forests or gradient boosted trees. Unlike ensemble methods, the models produced by the algorithm can be easily interpreted. The algorithm is based on a divide and conquer strategy and consists of two steps. The first step consists of using a decision tree to segment the large dataset. By construction, decision trees attempt to create homogeneous class distributions in their leaf nodes. However, non-homogeneous leaf nodes are usually produced. The second step of the algorithm consists of using a suitable classifier to determine the class labels for the non-homogeneous leaf nodes. The decision tree segment provides a coarse segment profile while the leaf level classifier can provide information about the attributes that affect the label within a segment.

stat.ML↗

arXiv2017

Fast Gaussian Process Regression for Big Data

Gaussian Processes are widely used for regression tasks. A known limitation in the application of Gaussian Processes to regression tasks is that the computation of the solution requires performing a matrix inversion. The solution also requires the storage of a large matrix in memory. These factors restrict the application of Gaussian Process regression to small and moderate size data sets. We present an algorithm that combines estimates from models developed using subsets of the data obtained in a manner similar to the bootstrap. The sample size is a critical parameter for this algorithm. Guidelines for reasonable choices of algorithm parameters, based on detailed experimental study, are provided. Various techniques have been proposed to scale Gaussian Processes to large scale regression tasks. The most appropriate choice depends on the problem context. The proposed method is most appropriate for problems where an additive model works well and the response depends on a small number of features. The minimax rate of convergence for such problems is attractive and we can build effective models with a small subset of the data. The Stochastic Variational Gaussian Process and the Sparse Gaussian Process are also appropriate choices for such problems. These methods pick a subset of data based on theoretical considerations. The proposed algorithm uses bagging and random sampling. Results from experiments conducted as part of this study indicate that the algorithm presented in this work can be as effective as these methods. Model stacking can be used to combine the model developed with the proposed method with models from other methods for large scale regression such as Gradient Boosted Trees. This can yield performance gains.

cs.LG↗

arXiv2017

Big Data Regression Using Tree Based Segmentation

Scaling regression to large datasets is a common problem in many application areas. We propose a two step approach to scaling regression to large datasets. Using a regression tree (CART) to segment the large dataset constitutes the first step of this approach. The second step of this approach is to develop a suitable regression model for each segment. Since segment sizes are not very large, we have the ability to apply sophisticated regression techniques if required. A nice feature of this two step approach is that it can yield models that have good explanatory power as well as good predictive performance. Ensemble methods like Gradient Boosted Trees can offer excellent predictive performance but may not provide interpretable models. In the experiments reported in this study, we found that the predictive performance of the proposed approach matched the predictive performance of Gradient Boosted Trees.

stat.ML↗

arXiv2017

A Statistical Machine Learning Approach to Yield Curve Forecasting

Yield curve forecasting is an important problem in finance. In this work we explore the use of Gaussian Processes in conjunction with a dynamic modeling strategy, much like the Kalman Filter, to model the yield curve. Gaussian Processes have been successfully applied to model functional data in a variety of applications. A Gaussian Process is used to model the yield curve. The hyper-parameters of the Gaussian Process model are updated as the algorithm receives yield curve data. Yield curve data is typically available as a time series with a frequency of one day. We compare existing methods to forecast the yield curve with the proposed method. The results of this study showed that while a competing method (a multivariate time series method) performed well in forecasting the yields at the short term structure region of the yield curve, Gaussian Processes perform well in the medium and long term structure regions of the yield curve. Accuracy in the long term structure region of the yield curve has important practical implications. The Gaussian Process framework yields uncertainty and probability estimates directly in contrast to other competing methods. Analysts are frequently interested in this information. In this study the proposed method has been applied to yield curve forecasting, however it can be applied to model high frequency time series data or data streams in other domains.

stat.ML↗

arXiv2016

Understanding Sea Ice Melting via Functional Data Analysis

In this article, we considered the problem of sea ice cover is melting. Considering the `satellite passive microwave remote sensing data' as functional data, we studied daily observation of sea ice cover of each year as a smooth continuous function of time. We investigated the mean function for the sea ice area for following decades and computed the corresponding $95\%$ bootstrap confidence interval for the both Arctic and Antarctic Oceans. We found the mean function for the sea ice area dropped statistically significantly in recent decades for the Arctic Ocean. However, no such statistical evidence was found for the Antarctic ocean. Essentially, the mean function for sea ice area in the Antarctic Ocean is unchanged. Additional evidence of the melting of sea ice area in the Arctic Ocean is provided by three types of phase curve (namely, Area vs. Velocity, Area vs. Acceleration, and Velocity Vs. Acceleration). In the Arctic Ocean, during the summer, the current decades is observing the size of the sea ice area about $30\%$ less, than what it used to be during the first decade. In this article, we have taken a distribution-free approach for our analysis, except the data generating process, belongs to the Hilbert space.

stat.AP↗

arXiv2015

Regularizing Portfolio Risk Analysis: A Bayesian Approach

It is important for a portfolio manager to estimate and analyze recent portfolio volatility to keep the portfolio's risk within limit. Though the number of financial instruments in the portfolio can be very large, sometimes more than thousands, daily returns considered for analysis are only for a month or even less. In this case rank of portfolio covariance matrix is less than full, hence solution is not unique. It is typically known as the ``ill-posed" problem. In this paper we discuss a Bayesian approach to regularize the problem. One of the additional advantages of this approach is to analyze the source of risk by estimating the probability of positive `conditional contribution to total risk' (CCTR). Each source's CCTR would sum up to the portfolio's total volatility risk. Existing methods only estimate CCTR of a source, and does not estimate the probability of CCTR to be significantly greater (or less) than zero. This paper presents Bayesian methodology to do so. We use a parallelizable and easy to use Monte Carlo (MC) approach to achieve our objective. Estimation of various risk measures, such as Value at Risk and Expected Shortfall, becomes a by-product of this Monte-Carlo approach.

q-fin.ST↗