SEARCH · Searcharxiv

Results for “stat.OT”

Search indexed arXiv papers on artificial intelligence, large language models, computer vision and robotics. Read source abstracts and follow links to arXiv.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 289 records · Page 16Linked to original sources

arXiv2017

MOLIERE: Automatic Biomedical Hypothesis Generation System

Hypothesis generation is becoming a crucial time-saving technique which allows biomedical researchers to quickly discover implicit connections between important concepts. Typically, these systems operate on domain-specific fractions of public medical data. MOLIERE, in contrast, utilizes information from over 24.5 million documents. At the heart of our approach lies a multi-modal and multi-relational network of biomedical objects extracted from several heterogeneous datasets from the National Center for Biotechnology Information (NCBI). These objects include but are not limited to scientific papers, keywords, genes, proteins, diseases, and diagnoses. We model hypotheses using Latent Dirichlet Allocation applied on abstracts found near shortest paths discovered within this network, and demonstrate the effectiveness of MOLIERE by performing hypothesis generation on historical data. Our network, implementation, and resulting data are all publicly available for the broad scientific community.

cs.IR↗

arXiv2017

Social Big Data Analytics of Consumer Choices: A Two Sided Online Platform Perspective

This dissertation examines three distinct big data analytics problems related to the social aspects of consumers' choices. The main goal of this line of research is to help two sided platform firms to target their marketing policies given the great heterogeneity among their customers. In three essays, I combined structural modeling and machine learning approaches to first understand customers' responses to intrinsic and extrinsic factors, using unique data sets I scraped from the web, and then explore methods to optimize two sided platforms' firms' reactions accordingly. The first essay examines "social learning" in the mobile app store context, controlling for intrinsic value of hedonic and utilitarian mobile apps, price, advertising, and number of options available. The second essay investigates bidders' anticipated winner and loser regret in the context of the eBay online auction platform. Using a large data set from eBay and empirical Bayesian estimation method, I quantify the bidders' anticipation of regret in various product categories, and investigate the role of experience in explaining the bidders' regret and learning behaviors. The third essay investigates the effects of Gamification incentive mechanisms in an online platform for user generated content. I use an ensemble method over LDA, mixed normal and k-mean clustering methods to segment users into competitors, collaborators, achievers, explorers and uninterested users. These findings help the Gamification platform to target its users. The simulation counterfactual analysis suggests that a two sided platform can increase the number of user contributions, by making earning badges more difficult.

stat.OT↗

arXiv2017

J.B.S. Haldane Could Have Done Better

In a review on the contribution of J.B.S. Haldane to the development of the Bayes factor hypothesis test (arXiv:1511.08180), Etz and Wagenmakers focus on Haldane's proposition of a mixture prior in a genetic example (Haldane 1932, A note on inverse probability. Mathematical Proceedings of the Cambridge Philosophical Society, 28, 55-61.). As Haldane never followed up on these ideas, it is difficult to gauge his motivation and intentions. I argue that contrary to Haldane's stated intention of replacing flat priors with more reasonable assumptions, he actually chose in this example an unreasonable flat prior. Considering the information available to Haldane, I derive a superior prior and compare to Haldane's flat prior. Haldane's main intent with his article seems to have been to explore the different parameter regions of the binomial and the conjugate beta. Furthermore, I agree with Etz and Wagenmakers that Haldane serendipitously adopted a mixture prior comprising a point mass and smooth distribution in his genetic example.

stat.OT↗

arXiv2017

Bootstrap with Clustering in Two or More Dimensions

We propose a bootstrap procedure for data that may exhibit clustering in two or more dimensions. We use insights from the theory of generalized U-statistics to analyze the large-sample properties of statistics that are sample averages from the observations pooled across clusters. The asymptotic distribution of these statistics may be non-standard if there is no clustering in means. We show that the proposed bootstrap procedure is (a) point-wise consistent for any fixed data-generating process (DGP), (b) uniformly consistent if we exclude the case of clustering without clustering in means, and (c) provides refinements for any DGP such that the limiting distribution is Gaussian.

stat.ME↗

arXiv2018

Estimating the Probability that a Function Observed with Noise is Convex

Consider a real-valued function that can only be observed with stochastic noise at a finite set of design points within a Euclidean space. We wish to determine whether there exists a convex function that goes through the true function values at the design points. We develop an asymptotically consistent Bayesian sequential sampling procedure that estimates the posterior probability of this being true. In each iteration, the posterior probability is estimated using Monte Carlo simulation. We offer three variance reduction methods -- change of measure, acceptance-rejection, and conditional Monte Carlo. Numerical experiments suggest that the conditional Monte Carlo method should be preferred.

stat.OT↗

arXiv2024

Gaussian spatial regression using the spmoran package: case study examples

This study demonstrates how to use the "spmoran" package implementing scalable spatial regression models for Gaussian and non-Gaussian data. Implemented models include spatially varying coefficient models, models with group effects, spatial unconditional quantile regression model, and low rank spatial econometric models. All of these models are estimated in a computationally efficient manner for large samples. Moran eigenvectors are used to an approximate Gaussian process (GP) modeling that is interpretable in terms of the Moran coefficient. The GP is used for modeling the spatial processes in residuals and regression coefficients. The sample codes are available from https://github.com/dmuraka/spmoran. While this vignette mainly focuses on Gaussian data modeling, another vignette focusing on non-Gaussian data including count regression is also available from the same GitHub page.

stat.OT↗

arXiv2017

Generalised Reichenbachian Common Cause Systems

The principle of the common cause claims that if an improbable coincidence has occurred, there must exist a common cause. This is generally taken to mean that positive correlations between non-causally related events should disappear when conditioning on the action of some underlying common cause. The extended interpretation of the principle, by contrast, urges that common causes should be called for in order to explain positive deviations between the estimated correlation of two events and the expected value of their correlation. The aim of this paper is to provide the extended reading of the principle with a general probabilistic model, capturing the simultaneous action of a system of multiple common causes. To this end, two distinct models are elaborated, and the necessary and sufficient conditions for their existence are determined.

stat.OT↗

arXiv2017

Data-Mining Research in Education

As an interdisciplinary discipline, data mining (DM) is popular in education area especially when examining students' learning performances. It focuses on analyzing educational related data to develop models for improving learners' learning experiences and enhancing institutional effectiveness. Therefore, DM does help education institutions provide high-quality education for its learners. Applying data mining in education also known as educational data mining (EDM), which enables to better understand how students learn and identify how improve educational outcomes. Present paper is designed to justify the capabilities of data mining approaches in the filed of education. The latest trends on EDM research are introduced in this review. Several specific algorithms, methods, applications and gaps in the current literature and future insights are discussed here.

stat.OT↗

arXiv2017

What is the best fractional derivative to fit data?

The aim of this work is to show, based on concrete data observation, that the choice of the fractional derivative when modelling a problem is relevant for the accuracy of a method. Using the least squares fitting technique, we determine the order of the fractional differential equation that better describes the experimental data, for different types of fractional derivatives.

stat.OT↗

arXiv2017

Nonparametric estimation of the conditional distribution at regression boundary points

Nonparametric regression is a standard statistical tool with increased importance in the Big Data era. Boundary points pose additional difficulties but local polynomial regression can be used to alleviate them. Local linear regression, for example, is easy to implement and performs quite well both at interior as well as boundary points. Estimating the conditional distribution function and/or the quantile function at a given regressor point is immediate via standard kernel methods but problems ensue if local linear methods are to be used. In particular, the distribution function estimator is not guaranteed to be monotone increasing, and the quantile curves can "cross". In the paper at hand, a simple method of correcting the local linear distribution estimator for monotonicity is proposed, and its good performance is demonstrated via simulations and real data examples.

stat.OT↗

arXiv2017

Rethinking probabilistic prediction in the wake of the 2016 U.S. presidential election

To many statisticians and citizens, the outcome of the most recent U.S. presidential election represents a failure of data-driven methods on the grandest scale. This impression has led to much debate and discussion about how the election predictions went awry -- Were the polls inaccurate? Were the models wrong? Did we misinterpret the probabilities? -- and how they went right -- Perhaps the analyses were correct even though the predictions were wrong, that's just the nature of probabilistic forecasting. With this in mind, we analyze the election outcome with respect to a core set of effectiveness principles. Regardless of whether and how the election predictions were right or wrong, we argue that they were ineffective in conveying the extent to which the data was informative of the outcome and the level of uncertainty in making these assessments. Among other things, our analysis sheds light on the shortcomings of the classical interpretations of probability and its communication to consumers in the form of predictions. We present here an alternative approach, based on a notion of validity, which offers two immediate insights for predictive inference. First, the predictions are more conservative, arguably more realistic, and come with certain guarantees on the probability of an erroneous prediction. Second, our approach easily and naturally reflects the (possibly substantial) uncertainty about the model by outputting plausibilities instead of probabilities. Had these simple steps been taken by the popular prediction outlets, the election outcome may not have been so shocking.

stat.OT↗

arXiv2017

Automated Diagnosis of Epilepsy Employing Multifractal Detrended Fluctuation Analysis Based Features

This contribution reports an application of MultiFractal Detrended Fluctuation Analysis, MFDFA based novel feature extraction technique for automated detection of epilepsy. In fractal geometry, Multifractal Detrended Fluctuation Analysis MFDFA is a popular technique to examine the self-similarity of a nonlinear, chaotic and noisy time series. In the present research work, EEG signals representing healthy, interictal (seizure free) and ictal activities (seizure) are acquired from an existing available database. The acquired EEG signals of different states are at first analyzed using MFDFA. To requisite the time series singularity quantification at local and global scales, a novel set of fourteen different features. Suitable feature ranking employing students t-test has been done to select the most statistically significant features which are henceforth being used as inputs to a support vector machines (SVM) classifier for the classification of different EEG signals. Eight different classification problems have been presented in this paper and it has been observed that the overall classification accuracy using MFDFA based features are reasonably satisfactory for all classification problems. The performance of the proposed method are also found to be quite commensurable and in some cases even better when compared with the results published in existing literature studied on the similar data set.

cs.CV↗

arXiv2017

A Mathematically Sensible Explanation of the Concept of Statistical Population

In statistics education, the concept of population is widely felt hard to grasp, as a result of vague explanations in textbooks. Some textbook authors therefore chose not to mention it. This paper offers a new explanation by proposing a new theoretical framework of population and sampling, which aims to achieve high mathematical sensibleness. In the explanation, the term population is given clear definition, and the relationship between simple random sampling and iid random variables are examined mathematically.

stat.OT↗

arXiv2017

A Tutorial on Kernel Density Estimation and Recent Advances

This tutorial provides a gentle introduction to kernel density estimation (KDE) and recent advances regarding confidence bands and geometric/topological features. We begin with a discussion of basic properties of KDE: the convergence rate under various metrics, density derivative estimation, and bandwidth selection. Then, we introduce common approaches to the construction of confidence intervals/bands, and we discuss how to handle bias. Next, we talk about recent advances in the inference of geometric and topological features of a density function using KDE. Finally, we illustrate how one can use KDE to estimate a cumulative distribution function and a receiver operating characteristic curve. We provide R implementations related to this tutorial at the end.

stat.ME↗

arXiv2018

Classical and bayesian componentwise predictors for non-compact correlated ARH(1) processes

A special class of standard Gaussian Autoregressive Hilbertian processes of order one (Gaussian ARH(1) processes), with bounded linear autocorrelation operator, which does not satisfy the usual Hilbert-Schmidt assumption, is considered. To compensate the slow decay of the diagonal coefficients of the autocorrelation operator, a faster decay velocity of the eigenvalues of the trace autocovariance operator of the innovation process is assumed. As usual, the eigenvectors of the autocovariance operator of the ARH(1) process are considered for projection, since, here, they are assumed to be known. Diagonal componentwise classical and bayesian estimation of the autocorrelation operator is studied for prediction. The asymptotic efficiency and equivalence of both estimators is proved, as well as of their associated componentwise ARH(1) plugin predictors. A simulation study is undertaken to illustrate the theoretical results derived.

stat.AP↗

arXiv2017

Remark On Variance Bounds

It is shown that the formula for the variance of combined series yields surprisingly simple proofs of some well known variance bounds.

stat.OT↗

arXiv2017

Information vs. Uncertainty as the Foundation for a Science of Environmental Modeling

Information accounting provides a better foundation for hypothesis testing than does uncertainty quantification. A quantitative account of science is derived under this perspective that alleviates the need for epistemic bridge principles, solves the problem of ad hoc falsification criteria, and deals with verisimilitude by facilitating a general approach to process-level diagnostics. Our argument is that the well-known inconsistencies of both Bayesian and classical statistical hypothesis tests are due to the fact that probability theory is an insufficient logic of science. Information theory, as an extension of probability theory, is required to provide a complete logic on which to base quantitative theories of empirical learning. The organizing question in this case becomes not whether our theories or models are more or less true, or about how much uncertainty is associated with a particular model, but instead whether there is any information available from experimental data that might allow us to improve the model. This becomes a formal hypothesis test, provides a theory of model diagnostics, and suggests a new approach to building dynamical systems models.

stat.OT↗

arXiv2017

Semiparametric estimation in the normal variance-mean mixture model

In this paper we study the problem of statistical inference on the parameters of the semiparametric variance-mean mixtures. This class of mixtures has recently become rather popular in statistical and financial modelling. We design a semiparametric estimation procedure that first estimates the mean of the underlying normal distribution and then recovers nonparametrically the density of the corresponding mixing distribution. We illustrate the performance of our procedure on simulated and real data.

stat.OT↗