arXiv subjects
Stephen E. Fienberg
Publications and source records attributed to Stephen E. Fienberg.
Bayesian Models and Methods in Public Policy and Government Settings
Starting with the neo-Bayesian revival of the 1950s, many statisticians argued that it was inappropriate to use Bayesian methods, and in particular subjective Bayesian methods in governmental and public policy settings because of their reliance upon prior distributions. But the Bayesian framework often provides the primary way to respond to questions raised in these settings and the numbers and diversity of Bayesian applications have grown dramatically in recent years. Through a series of examples, both historical and recent, we argue that Bayesian approaches with formal and informal assessments of priors AND likelihood functions are well accepted and should become the norm in public settings. Our examples include census-taking and small area estimation, US election night forecasting, studies reported to the US Food and Drug Administration, assessing global climate change, and measuring potential declines in disability among the elderly.
Discussion of "Network routing in a dynamic environment"
Discussion of "Network routing in a dynamic environment" by N.D. Singpurwalla [arXiv:1107.4852]
Exploring the Consequences of IED Deployment with a Generalized Linear Model Implementation of the Canadian Traveller Problem
The deployment of improvised explosive devices (IEDs) along major roadways has been a favoured strategy of insurgents in recent war zones, both for the ability to cause damage to targets along roadways at minimal cost, but also as a means of controlling the flow of traffic and causing additional expense to opposing forces. Among other related approaches (which we discuss), the adversarial problem has an analogue in the Canadian Traveller Problem, wherein a stretch of road is blocked with some independent probability, and the state of the road is only discovered once the traveller reaches one of the intersections that bound this stretch of road. We discuss the implementation of ideas from social network analysis, namely the notion of "betweenness centrality", and how this can be adapted to the notion of deployment of IEDs with the aid of Generalized Linear Models (GLMs): namely, how we can model the probability of an IED deployment in terms of the increased effort due to Canadian betweenness, how we can include expert judgement on the probability of a deployment, and how we can extend the approach to estimation and updating over several time steps.
Introduction to papers on the modeling and analysis of network data---II
Introduction to papers on the modeling and analysis of network data---II
Introduction to papers on the modeling and analysis of network data
Introduction to papers on the modeling and analysis of network data
User Interest and Interaction Structure in Online Forums
We present a new similarity measure tailored to posts in an online forum. Our measure takes into account all the available information about user interest and interaction --- the content of posts, the threads in the forum, and the author of the posts. We use this post similarity to build a similarity between users, based on principal coordinate analysis. This allows easy visualization of the user activity as well. Similarity between users has numerous applications, such as clustering or classification. We show that including the author of a post in the post similarity has a smoothing effect on principal coordinate projections. We demonstrate our method on real data drawn from an internal corporate forum, and compare our results to those given by a standard document classification method. We conclude our method gives a more detailed picture of both the local and global network structure.
Algebraic statistics for a directed random graph model with reciprocation
The p_1 model is a directed random graph model used to describe dyadic interactions in a social network in terms of effects due to differential attraction (popularity) and expansiveness, as well as an additional effect due to reciprocation. In this article we carry out an algebraic statistics analysis of this model. We show that the p_1 model is a toric model specified by a multi-homogeneous ideal. We conduct an extensive study of the Markov bases for p_1 models that incorporate explicitly the constraint arising from multi-homogeneity. Our results are directly relevant to the estimation and conditional goodness-of-fit testing problems in p_1 models.
On the Geometry of Discrete Exponential Families with Application to Exponential Random Graph Models
There has been an explosion of interest in statistical models for analyzing network data, and considerable interest in the class of exponential random graph (ERG) models, especially in connection with difficulties in computing maximum likelihood estimates. The issues associated with these difficulties relate to the broader structure of discrete exponential families. This paper re-examines the issues in two parts. First we consider the closure of $k$-dimensional exponential families of distribution with discrete base measure and polyhedral convex support $\mathrm{P}$. We show that the normal fan of $\mathrm{P}$ is a geometric object that plays a fundamental role in deriving the statistical and geometric properties of the corresponding extended exponential families. We discuss its relevance to maximum likelihood estimation, both from a theoretical and computational standpoint. Second, we apply our results to the analysis of ERG models. In particular, by means of a detailed example, we provide some characterization of the properties of ERG models, and, in particular, of certain behaviors of ERG models known as degeneracy.
Sequential category aggregation and partitioning approaches for multi-way contingency tables based on survey and census data
Large contingency tables arise in many contexts but especially in the collection of survey and census data by government statistical agencies. Because the vast majority of the variables in this context have a large number of categories, agencies and users need a systematic way of constructing tables which are summaries of such contingency tables. We propose such an approach in this paper by finding members of a class of restricted log-linear models which maximize the likelihood of the data and use this to find a parsimonious means of representing the table. In contrast with more standard approaches for model search in hierarchical log-linear models (HLLM), our procedure systematically reduces the number of categories of the variables. Through a series of examples, we illustrate the extent to which it can preserve the interaction structure found with HLLMs and be used as a data simplification procedure prior to HLL modeling. A feature of the procedure is that it can easily be applied to many tables with millions of cells, providing a new way of summarizing large data sets in many disciplines. The focus is on information and description rather than statistical testing. The procedure may treat each variable in the table in different ways, preserving full detail, treating it as fully nominal, or preserving ordinality.
The Early Statistical Years: 1947--1967 A Conversation with Howard Raiffa
Howard Raiffa earned his bachelor's degree in mathematics, his master's degree in statistics and his Ph.D. in mathematics at the University of Michigan. Since 1957, Raiffa has been a member of the faculty at Harvard University, where he is now the Frank P. Ramsey Chair in Managerial Economics (Emeritus) in the Graduate School of Business Administration and the Kennedy School of Government. A pioneer in the creation of the field known as decision analysis, his research interests span statistical decision theory, game theory, behavioral decision theory, risk analysis and negotiation analysis. Raiffa has supervised more than 90 doctoral dissertations and written 11 books. His new book is Negotiation Analysis: The Science and Art of Collaborative Decision Making. Another book, Smart Choices, co-authored with his former doctoral students John Hammond and Ralph Keeney, was the CPR (formerly known as the Center for Public Resources) Institute for Dispute Resolution Book of the Year in 1998. Raiffa helped to create the International Institute for Applied Systems Analysis and he later became its first Director, serving in that capacity from 1972 to 1975. His many honors and awards include the Distinguished Contribution Award from the Society of Risk Analysis; the Frank P. Ramsey Medal for outstanding contributions to the field of decision analysis from the Operations Research Society of America; and the Melamed Prize from the University of Chicago Business School for The Art and Science of Negotiation. He earned a Gold Medal from the International Association for Conflict Management and a Lifetime Achievement Award from the CPR Institute for Dispute Resolution. He holds honorary doctor's degrees from Carnegie Mellon University, the University of Michigan, Northwestern University, Ben Gurion University of the Negev and Harvard University. The latter was awarded in 2002.
Editorial: Statistics and "The lost tomb of Jesus"
What makes a problem suitable for statistical analysis? Are historical and religious questions addressable using statistical calculations? Such issues have long been debated in the statistical community and statisticians and others have used historical information and texts to analyze such questions as the economics of slavery, the authorship of the Federalist Papers and the question of the existence of God. But what about historical and religious attributions associated with information gathered from archeological finds? In 1980, a construction crew working in the Jerusalem neighborhood of East Talpiot stumbled upon a crypt. Archaeologists from the Israel Antiquities Authority came to the scene and found 10 limestone burial boxes, known as ossuaries, in the crypt. Six of these had inscriptions. The remains found in the ossuaries were reburied, as required by Jewish religious tradition, and the ossuaries were catalogued and stored in a warehouse. The inscriptions on the ossuaries were catalogued and published by Rahmani (1994) and by Kloner (1996) but there reports did not receive widespread public attention. Fast forward to March 2007, when a television ``docudrama'' aired on The Discovery Channel entitled ``The Lost Tomb of Jesus'' touched off a public and religious controversy--one only need think about the title to see why there might be a controversy! The program, and a simultaneously published book [Jacobovici and Pellegrino (2007)], described the ``rediscovery'' of the East Talpiot archeological find and they presented interpretations of the ossuary inscriptions from a number of perspectives. Among these was a statistical calculation attributed to the statistician Andrey Feuerverger: ``that the odds that all six names would appear together in one tomb are 1 in 600, calculated conservatively--or possibly even as much as one in one million.''
Describing disability through individual-level mixture models for multivariate binary data
Data on functional disability are of widespread policy interest in the United States, especially with respect to planning for Medicare and Social Security for a growing population of elderly adults. We consider an extract of functional disability data from the National Long Term Care Survey (NLTCS) and attempt to develop disability profiles using variations of the Grade of Membership (GoM) model. We first describe GoM as an individual-level mixture model that allows individuals to have partial membership in several mixture components simultaneously. We then prove the equivalence between individual-level and population-level mixture models, and use this property to develop a Markov Chain Monte Carlo algorithm for Bayesian estimation of the model. We use our approach to analyze functional disability data from the NLTCS.
Editorial: Statistics and forensic science
Forensic science is usually taken to mean the application of a broad spectrum of scientific tools to answer questions of interest to the legal system. Despite such popular television series as CSI: Crime Scene Investigation and its spinoffs--CSI: Miami and CSI: New York--on which the forensic scientists use the latest high-tech scientific tools to identify the perpetrator of a crime and always in under an hour, forensic science is under assault, in the public media, popular magazines [Talbot (2007), Toobin (2007)] and in the scientific literature [Kennedy (2003), Saks and Koehler (2005)]. Ironically, this growing controversy over forensic science has occurred precisely at the time that DNA evidence has become the ``gold standard'' in the courts, leading to the overturning of hundreds of convictions many of which were based on clearly less credible forensic evidence, including eyewitness testimony [Berger (2006)].
William Kruskal: My Scholarly and Scientific Model
Discussion of ``The William Kruskal Legacy: 1919--2005'' by Stephen E. Fienberg, Stephen M. Stigler and Judith M. Tanur [arXiv:0710.5063]
The William Kruskal Legacy: 1919--2005
William Kruskal (Bill) was a distinguished statistician who spent virtually his entire professional career at the University of Chicago, and who had a lasting impact on the Institute of Mathematical Statistics and on the field of statistics more broadly, as well as on many who came in contact with him. Bill passed away last April following an extended illness, and on May 19, 2005, the University of Chicago held a memorial service at which several of Bill's colleagues and collaborators spoke along with members of his family and other friends. This biography and the accompanying commentaries derive in part from brief presentations on that occasion, along with recollections and input from several others. Bill was known personally to most of an older generation of statisticians as an editor and as an intellectual and professional leader. In 1994, Statistical Science published an interview by Sandy Zabell (Vol. 9, 285--303) in which Bill looked back on selected events in his professional life. One of the purposes of the present biography and accompanying commentaries is to reintroduce him to old friends and to introduce him for the first time to new generations of statisticians who never had an opportunity to interact with him and to fall under his influence.
A statistical approach to simultaneous mapping and localization for mobile robots
Mobile robots require basic information to navigate through an environment: they need to know where they are (localization) and they need to know where they are going. For the latter, robots need a map of the environment. Using sensors of a variety of forms, robots gather information as they move through an environment in order to build a map. In this paper we present a novel sampling algorithm to solving the simultaneous mapping and localization (SLAM) problem in indoor environments. We approach the problem from a Bayesian statistics perspective. The data correspond to a set of range finder and odometer measurements, obtained at discrete time instants. We focus on the estimation of the posterior distribution over the space of possible maps given the data. By exploiting different factorizations of this distribution, we derive three sampling algorithms based on importance sampling. We illustrate the results of our approach by testing the algorithms with two real data sets obtained through robot navigation inside office buildings at Carnegie Mellon University and the Pontificia Universidad Catolica de Chile.
Comment: Complex Causal Questions Require Careful Model Formulation: Discussion of Rubin on Experiments with "Censoring" Due to Death
Comment on Complex Causal Questions Require Careful Model Formulation: Discussion of Rubin on Experiments with ``Censoring'' Due to Death [math.ST/0612783]