SEARCH · Searcharxiv

Results for “stat.OT”

Search indexed arXiv papers on artificial intelligence, large language models, computer vision and robotics. Read source abstracts and follow links to arXiv.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 685 records · Page 38Linked to original sources

arXiv2021

ROBustness In Network (robin): an R package for Comparison and Validation of communities

In network analysis, many community detection algorithms have been developed, however, their implementation leaves unaddressed the question of the statistical validation of the results. Here we present robin(ROBustness In Network), an R package to assess the robustness of the community structure of a network found by one or more methods to give indications about their reliability. The procedure initially detects if the community structure found by a set of algorithms is statistically significant and then compares two selected detection algorithms on the same graph to choose the one that better fits the network of interest. We demonstrate the use of our package on the American College Football benchmark dataset.

stat.CO↗

arXiv2021

De-carbonization of global energy use during the COVID-19 pandemic

The COVID-19 pandemic has disrupted human activities, leading to unprecedented decreases in both global energy demand and GHG emissions. Yet a little known that there is also a low carbon shift of the global energy system in 2020. Here, using the near-real-time data on energy-related GHG emissions from 30 countries (about 70% of global power generation), we show that the pandemic caused an unprecedented de-carbonization of global power system, representing by a dramatic decrease in the carbon intensity of power sector that reached a historical low of 414.9 tCO2eq/GWh in 2020. Moreover, the share of energy derived from renewable and low-carbon sources (nuclear, hydro-energy, wind, solar, geothermal, and biomass) exceeded that from coal and oil for the first time in history in May of 2020. The decrease in global net energy demand (-1.3% in the first half of 2020 relative to the average of the period in 2016-2019) masks a large down-regulation of fossil-fuel-burning power plants supply (-6.1%) coincident with a surge of low-carbon sources (+6.2%). Concomitant changes in the diurnal cycle of electricity demand also favored low-carbon generators, including a flattening of the morning ramp, a lower midday peak, and delays in both the morning and midday load peaks in most countries. However, emission intensities in the power sector have since rebounded in many countries, and a key question for climate mitigation is thus to what extent countries can achieve and maintain lower, pandemic-level carbon intensities of electricity as part of a green recovery.

physics.ao-ph↗

arXiv2022

COVIDHunter: An Accurate, Flexible, and Environment-Aware Open-Source COVID-19 Outbreak Simulation Model

Background: Early detection and isolation of COVID-19 patients are essential for successful implementation of mitigation strategies and eventually curbing the disease spread. With a limited number of daily COVID-19 tests performed in every country, simulating the COVID-19 spread along with the potential effect of each mitigation strategy currently remains one of the most effective ways in managing the healthcare system and guiding policy-makers. Methods: We introduce COVIDHunter, a flexible and accurate COVID-19 outbreak simulation model that evaluates the current mitigation measures that are applied to a region and provides suggestions on what strength the upcoming mitigation measure should be. The key idea of COVIDHunter is to quantify the spread of COVID-19 in a geographical region by simulating the average number of new infections caused by an infected person considering the effect of external factors, such as environmental conditions (e.g., climate, temperature, humidity) and mitigation measures. Results: Using Switzerland as a case study, COVIDHunter estimates that if the policy-makers relax the mitigation measures by 50% for 30 days then both the daily capacity need for hospital beds and daily number of deaths increase exponentially by an average of 5.1x, who may occupy ICU beds and ventilators for a period of time. Unlike existing models, the COVIDHunter model accurately monitors and predicts the daily number of cases, hospitalizations, and deaths due to COVID-19. Our model is flexible to configure and simple to modify for modeling different scenarios under different environmental conditions and mitigation measures. Availability: We release the source code of the COVIDHunter implementation at https://github.com/CMU- SAFARI/COVIDHunter and show how to flexibly configure our model for any scenario and easily extend it for different measures and conditions than we account for.

q-bio.PE↗

arXiv2021

A Text Mining Discovery of Similarities and Dissimilarities Among Sacred Scriptures

The careful examination of sacred texts gives valuable insights into human psychology, different ideas regarding the organization of societies as well as into terms like truth and God. To improve and deepen our understanding of sacred texts, their comparison, and their separation is crucial. For this purpose, we use our data set has nine sacred scriptures. This work deals with the separation of the Quran, the Asian scriptures Tao-Te-Ching, the Buddhism, the Yogasutras, and the Upanishads as well as the four books from the Bible, namely the Book of Proverbs, the Book of Ecclesiastes, the Book of Ecclesiasticus, and the Book of Wisdom. These scriptures are analyzed based on the natural language processing NLP creating the mathematical representation of the corpus in terms of frequencies called document term matrix (DTM). After this analysis, machine learning methods like supervised and unsupervised learning are applied to perform classification. Here we use the Multinomial Naive Bayes (MNB), the Super Vector Machine (SVM), the Random Forest (RF), and the K-nearest Neighbors (KNN). We obtain that among these methods MNB is able to predict the class of a sacred text with an accuracy of about 85.84 %.

stat.OT↗

arXiv2021

Making the most of your day: online learning for optimal allocation of time

We study online learning for optimal allocation when the resource to be allocated is time. %Examples of possible applications include job scheduling for a computing server, a driver filling a day with rides, a landlord renting an estate, etc. An agent receives task proposals sequentially according to a Poisson process and can either accept or reject a proposed task. If she accepts the proposal, she is busy for the duration of the task and obtains a reward that depends on the task duration. If she rejects it, she remains on hold until a new task proposal arrives. We study the regret incurred by the agent, first when she knows her reward function but does not know the distribution of the task duration, and then when she does not know her reward function, either. This natural setting bears similarities with contextual (one-armed) bandits, but with the crucial difference that the normalized reward associated to a context depends on the whole distribution of contexts.

stat.ML↗

arXiv2021

Big Data meets Causal Survey Research: Understanding Nonresponse in the Recruitment of a Mixed-mode Online Panel

Survey scientists increasingly face the problem of high-dimensionality in their research as digitization makes it much easier to construct high-dimensional (or "big") data sets through tools such as online surveys and mobile applications. Machine learning methods are able to handle such data, and they have been successfully applied to solve \emph{predictive} problems. However, in many situations, survey statisticians want to learn about \emph{causal} relationships to draw conclusions and be able to transfer the findings of one survey to another. Standard machine learning methods provide biased estimates of such relationships. We introduce into survey statistics the double machine learning approach, which gives approximately unbiased estimators of causal parameters, and show how it can be used to analyze survey nonresponse in a high-dimensional panel setting.

stat.ME↗

arXiv2025

Taylor's Theorem and Mean Value Theorem for Random Functions and Random Variables

This study addresses the often-overlooked issue of measurability at intermediate points when applying Taylor's theorems to random functions and random vectors (e.g., likelihood functions with respect to estimators) in statistics. Classical Taylor-related theorems were originally developed for deterministic settings. Consequently, they do not directly extend to stochastic functions and variables and do not inherently guarantee the measurability of intermediate points. In statistical contexts, applying these theorems without properly accounting for randomness can lead to analyses that lack well-defined probabilistic interpretations. Elementary approaches, such as pointwise constructions, are insufficient for handling random quantities and establishing measurable intermediate points. Moreover, some statistical literature has implicitly disregarded this issue, often neglecting the stochastic nature of the problem and assuming that intermediate points are measurable. To address this gap, we develop multivariate Taylor's and mean value theorems tailored for random functions and random variables under mild assumptions. We provide illustrative examples demonstrating the applicability of our results to commonly used statistical methods, including maximum likelihood estimation, $M$-estimation, and profile estimation. Our findings contribute a rigorous foundation for the applications of Taylor expansions in statistics.

stat.OT↗

arXiv2021

Optimal Transport of Information

We study the general problem of Bayesian persuasion (optimal information design) with continuous actions and continuous state space in arbitrary dimensions. First, we show that with a finite signal space, the optimal information design is always given by a partition. Second, we take the limit of an infinite signal space and characterize the solution in terms of a Monge-Kantorovich optimal transport problem with an endogenous information transport cost. We use our novel approach to: 1. Derive necessary and sufficient conditions for optimality based on Bregman divergences for non-convex functions. 2. Compute exact bounds for the Hausdorff dimension of the support of an optimal policy. 3. Derive a non-linear, second-order partial differential equation whose solutions correspond to regular optimal policies. We illustrate the power of our approach by providing explicit solutions to several non-linear, multidimensional Bayesian persuasion problems.

econ.GN↗

arXiv2022

Valid Instrumental Variables Selection Methods using Negative Control Outcomes and Constructing Efficient Estimator

In observational studies, instrumental variable (IV) methods are commonly applied when there exists some unmeasured covariates. In Mendelian Randomization (MR), constructing an allele score by using many single nucleotide polymorphisms (SNPs) is often implemented; however, there are risks estimating biased causal effects by including some invalid IVs. Invalid IVs are candidates of IVs associated with some unobserved variables. To solve this problem, we propose a novel strategy in this paper: using Negative Control Outcomes (NCOs) as auxiliary variables. By using NCOs, we can essentialy select only valid IVs and exclude invalid IVs without any information of IV candidates. We also propose the new two-step estimating procedure and prove the semiparametric efficiency. We demonstrate the superior performance of the proposed estimator compared with existing estimators via simulation studies.

stat.ME↗

arXiv2021

Bridging Breiman's Brook: From Algorithmic Modeling to Statistical Learning

In 2001, Leo Breiman wrote of a divide between "data modeling" and "algorithmic modeling" cultures. Twenty years later this division feels far more ephemeral, both in terms of assigning individuals to camps, and in terms of intellectual boundaries. We argue that this is largely due to the "data modelers" incorporating algorithmic methods into their toolbox, particularly driven by recent developments in the statistical understanding of Breiman's own Random Forest methods. While this can be simplistically described as "Breiman won", these same developments also expose the limitations of the prediction-first philosophy that he espoused, making careful statistical analysis all the more important. This paper outlines these exciting recent developments in the random forest literature which, in our view, occurred as a result of a necessary blending of the two ways of thinking Breiman originally described. We also ask what areas statistics and statisticians might currently overlook.

stat.OT↗

arXiv2021

Constructing Evacuation Evolution Patterns and Decisions Using Mobile Device Location Data: A Case Study of Hurricane Irma

Understanding individuals' behavior during hurricane evacuation is of paramount importance for local, state, and government agencies hoping to be prepared for natural disasters. Complexities involved with human decision-making procedures and lack of data for such disasters are the main reasons that make hurricane evacuation studies challenging. In this paper, we utilized a large mobile phone Location-Based Services (LBS) data to construct the evacuation pattern during the landfall of Hurricane Irma. By employing our proposed framework on more than 11 billion mobile phone location sightings, we were able to capture the evacuation decision of 807,623 smartphone users who were living within the state of Florida. We studied users' evacuation decisions, departure and reentry date distribution, and destination choice. In addition to these decisions, we empirically examined the influence of evacuation order and low-lying residential areas on individuals' evacuation decisions. Our analysis revealed that 57.92% of people living in mandatory evacuation zones evacuated their residences while this ratio was 32.98% and 33.68% for people living in areas with no evacuation order and voluntary evacuation order, respectively. Moreover, our analysis revealed the importance of the individuals' mobility behavior in modeling the evacuation decision choice. Historical mobility behavior information such as number of trips taken by each individual and the spatial area covered by individuals' location trajectory estimated significant in our choice model and improve the overall accuracy of the model significantly.

cs.LG↗

arXiv2021

Online High-Dimensional Change-Point Detection using Topological Data Analysis

Topological Data Analysis (TDA) is a rapidly growing field, which studies methods for learning underlying topological structures present in complex data representations. TDA methods have found recent success in extracting useful geometric structures for a wide range of applications, including protein classification, neuroscience, and time-series analysis. However, in many such applications, one is also interested in sequentially detecting changes in this topological structure. We propose a new method called Persistence Diagram based Change-Point (PD-CP), which tackles this problem by integrating the widely-used persistence diagrams in TDA with recent developments in nonparametric change-point detection. The key novelty in PD-CP is that it leverages the distribution of points on persistence diagrams for online detection of topological changes. We demonstrate the effectiveness of PD-CP in an application to solar flare monitoring.

stat.ME↗

arXiv2021

Minimizing Information Leakage of Abrupt Changes in Stochastic Systems

This work investigates the problem of analyzing privacy of abrupt changes for general Markov processes. These processes may be affected by changes, or exogenous signals, that need to remain private. Privacy refers to the disclosure of information of these changes through observations of the underlying Markov chain. In contrast to previous work on privacy, we study the problem for an online sequence of data. We use theoretical tools from optimal detection theory to motivate a definition of online privacy based on the average amount of information per observation of the stochastic system in consideration. Two cases are considered: the full-information case, where the eavesdropper measures all but the signals that indicate a change, and the limited-information case, where the eavesdropper only measures the state of the Markov process. For both cases, we provide ways to derive privacy upper-bounds and compute policies that attain a higher privacy level. It turns out that the problem of computing privacy-aware policies is concave, and we conclude with some examples and numerical simulations for both cases.

eess.SY↗

arXiv2021

Health and Demographic Surveillance Systems and the 2030 Agenda: Sustainable Development Goals

The health and demographic surveillance system (HDSS) is an old method for intensively monitoring a population to assess the effects of healthcare or other population-level interventions - often clinical trials. The strengths of HDSS include very detailed descriptions of whole populations with frequent updates. This often provides long time series of accurate population and health indicators for the HDSS study population. The primary weakness of HDSS is that the data describe only the HDSS study population and cannot be generalized beyond that. The 2030 agenda is the ecosystem of activities - many including population-level monitoring - that relate to the United Nations (UN) Sustainable Development Goals (SDG). With respect to the 2030 agenda, HDSS can contribute by: continuing to conduct cause-and-effect studies; contributing to data triangulation or amalgamation initiatives; characterizing the bias in and calibrating 'big data'; and contributing more to the rapid training of data-oriented professionals, especially in the population and health fields.

stat.OT↗

arXiv2021

visTree: Visualization of Subgroups for a Decision Tree

Decision trees are flexible prediction models which are constructed to quantify outcome-covariate relationships and characterize relevant population subgroups. However, the standard graphical representation of fitted decision trees highlights individual split points, and hence is suboptimal for visualizing defined subgroups. In this paper, we present a novel visual representation of decision trees which shifts the primary focus to characterizing subgroups, both in terms of their defining covariates and their outcome distribution. We implement our method in the \texttt{visTree} package, which builds on the toolkit and infrastructure provided by the \texttt{partykit} package and enables the visualization to be applied to varied decision trees. Individual functions are demonstrated using data from the Box Lunch study [French et al., 2014], a randomized trial to evaluate the effect of exposure to different lunch sizes on energy intake and body weight among working adults.

stat.AP↗

arXiv2021

Measuring Reciprocity in a Directed Preferential Attachment Network

Empirical studies show that online social networks have not only in- and out-degree distributions with Pareto-like tails but also a high proportion of reciprocal edges. A classical directed preferential attachment (PA) model generates in- and out-degree distribution with power-law tails, but theoretical properties of the reciprocity feature in this model have not yet been studied. We derive the asymptotic results on the number of reciprocal edges between two fixed nodes, as well as the proportion of reciprocal edges in the entire PA network. We see that with certain choices of parameters, the proportion of reciprocal edges in a directed PA network is close to 0, which differs from the empirical observation. This points out one potential problem of fitting a classical PA model to a given network dataset with high reciprocity and indicates alternative models need to be considered.

physics.soc-ph↗

arXiv2021

Two-sided confidence interval of a binomial proportion: how to choose?

Introduction: estimation of confidence intervals (CIs) of binomial proportions has been reviewed more than once but the directional interpretation, distinguishing the overestimation from the underestimation, was neglected while the sample size and theoretical proportion variances from experiment to experiment have not been formally taken in account. Herein, we define and apply new evaluation criteria, then give recommendations for the practical use of these CIs. Materials & methods: Google Scholar was used for bibliographic research. Evaluation criteria were (i) one-sided conditional errors, (ii) one-sided local average errors assuming a random theoretical proportion and (iii) expected half-widths of CIs. Results: Wald's CI did not control any of the risks, even when the expected number of successes reached 32. The likelihood ratio CI had a better balance than the logistic Wald CI. The Clopper-Pearson mid-P CI controlled well one-sided local average errors whereas the simple Clopper-Pearson CI was strictly conservative on both one-sided conditional errors. The percentile and basic bootstrap CIs had the same bias order as Wald's CI whereas the studentized CIs and BCa, modified for discrete bootstrap distributions, were less biased but not as efficient as the parametric methods. The half-widths of CIs mirrored local average errors. Conclusion: we recommend using the Clopper-Pearson mid-P CI for the estimation of a proportion except for observed-theoretical proportion comparison under controlled experimental conditions in which the Clopper-Pearson CI may be better.

stat.OT↗

arXiv2021

A unified approach for covariance matrix estimation under Stein loss

In this paper, we address the problem of estimating a covariance matrix of a multivariate Gaussian distribution, relative to a Stein loss function, from a decision theoretic point of view. We investigate the case where the covariance matrix is invertible and the case when it is non--invertible in a unified approach.

math.ST↗