arXiv subjects

Aaron Clauset

Publications and source records attributed to Aaron Clauset.

70 records · Page 4Linked to original sources

arXiv2009

Power-law distributions in empirical data

Power-law distributions occur in many situations of scientific interest and have significant consequences for our understanding of natural and man-made phenomena. Unfortunately, the detection and characterization of power laws is complicated by the large fluctuations that occur in the tail of the distribution -- the part of the distribution representing large but rare events -- and by the difficulty of identifying the range over which power-law behavior holds. Commonly used methods for analyzing power-law data, such as least-squares fitting, can produce substantially inaccurate estimates of parameters for power-law distributions, and even in cases where such methods return accurate answers they are still unsatisfactory because they give no indication of whether the data obey a power law at all. Here we present a principled statistical framework for discerning and quantifying power-law behavior in empirical data. Our approach combines maximum-likelihood fitting methods with goodness-of-fit tests based on the Kolmogorov-Smirnov statistic and likelihood ratios. We evaluate the effectiveness of the approach with tests on synthetic data and give critical comparisons to previous approaches. We also apply the proposed methods to twenty-four real-world data sets from a range of different disciplines, each of which has been conjectured to follow a power-law distribution. In some cases we find these conjectures to be consistent with the data while in others the power law is ruled out.

physics.data-an↗

arXiv2009

Evolutionary Model of Species Body Mass Diversification

We present a quantitative model for the biological evolution of species body masses within large groups of related species, e.g., terrestrial mammals, in which body mass M evolves according to branching (speciation), multiplicative diffusion, and an extinction probability that increases logarithmically with mass. We describe this evolution in terms of a convection-diffusion-reaction equation for ln M. The steady-state behavior is in good agreement with empirical data on recent terrestrial mammals, and the time-dependent behavior also agrees with data on extinct mammal species between 95 - 50 Myr ago.

q-bio.PE↗

arXiv2009

Comment on Yu et al., "High Quality Binary Protein Interaction Map of the Yeast Interactome Network." Science 322, 104 (2008)

We test the claim by Yu et al. -- presented in Science 322, 104 (2008) -- that the degree distribution of the yeast (Saccharomyces cerevisiae) protein-interaction network is best approximated by a power law. Yu et al. consider three versions of this network. In all three cases, however, we find the most likely power-law model of the data is distinct from and incompatible with the one given by Yu et al. Only one network admits good statistical support for any power law, and in that case, the power law explains only the distribution of the upper 10% of node degrees. These results imply that there is considerably more structure present in the yeast interactome than suggested by Yu et al., and that these networks should probably not be called "scale free."

q-bio.MN↗

arXiv2009

The evolution and distribution of species body size

The distribution of species body size within taxonomic groups exhibits a heavy right-tail extending over many orders of magnitude, where most species are significantly larger than the smallest species. We provide a simple model of cladogenetic diffusion over evolutionary time that omits explicit mechanisms for inter-specific competition and other microevolutionary processes yet fully explains the shape of this distribution. We estimate the model's parameters from fossil data and find that it robustly reproduces the distribution of 4002 mammal species from the late Quaternary. The observed fit suggests that the asymmetric distribution arises from a fundamental tradeoff between the short-term selective advantages (Cope's rule) and long-term selective risks of increased species body size, in the presence of a taxon-specific lower limit on body size.

q-bio.PE↗

arXiv2008

Hierarchical structure and the prediction of missing links in networks

Networks have in recent years emerged as an invaluable tool for describing and quantifying complex systems in many branches of science. Recent studies suggest that networks often exhibit hierarchical organization, where vertices divide into groups that further subdivide into groups of groups, and so forth over multiple scales. In many cases these groups are found to correspond to known functional units, such as ecological niches in food webs, modules in biochemical networks (protein interaction networks, metabolic networks, or genetic regulatory networks), or communities in social networks. Here we present a general technique for inferring hierarchical structure from network data and demonstrate that the existence of hierarchy can simultaneously explain and quantitatively reproduce many commonly observed topological properties of networks, such as right-skewed degree distributions, high clustering coefficients, and short path lengths. We further show that knowledge of hierarchical structure can be used to predict missing connections in partially known networks with high accuracy, and for more general network structures than competing techniques. Taken together, our results suggest that hierarchy is a central organizing principle of complex networks, capable of offering insight into many network phenomena.

stat.ML↗

arXiv2008

How many species have mass M?

Within large taxonomic assemblages, the number of species with adult body mass M is characterized by a broad but asymmetric distribution, with the largest mass being orders of magnitude larger than the typical mass. This canonical shape can be explained by cladogenetic diffusion that is bounded below by a hard limit on viable species mass and above by extinction risks that increase weakly with mass. Here we introduce and analytically solve a simplified cladogenetic diffusion model. When appropriately parameterized, the diffusion-reaction equation predicts mass distributions that are in good agreement with data on 4002 terrestrial mammal from the late Quaternary and 8617 extant bird species. Under this model, we show that a specific tradeoff between the strength of within-lineage drift toward larger masses (Cope's rule) and the increased risk of extinction from increased mass is necessary to produce realistic mass distributions for both taxa. We then make several predictions about the evolution of avian species masses.

q-bio.PE↗

arXiv2007

On the Frequency of Severe Terrorist Events

In the spirit of Richardson's original (1948) study of the statistics of deadly conflicts, we study the frequency and severity of terrorist attacks worldwide since 1968. We show that these events are uniformly characterized by the phenomenon of scale invariance, i.e., the frequency scales as an inverse power of the severity, P(x) ~ x^-alpha. We find that this property is a robust feature of terrorism, persisting when we control for economic development of the target country, the type of weapon used, and even for short time-scales. Further, we show that the center of the distribution oscillates slightly with a period of roughly tau ~ 13 years, that there exist significant temporal correlations in the frequency of severe events, and that current models of event incidence cannot account for these variations or the scale invariance property of global terrorism. Finally, we describe a simple toy model for the generation of these statistics, and briefly discuss its implications.

physics.soc-ph↗

arXiv2006

Structural Inference of Hierarchies in Networks

One property of networks that has received comparatively little attention is hierarchy, i.e., the property of having vertices that cluster together in groups, which then join to form groups of groups, and so forth, up through all levels of organization in the network. Here, we give a precise definition of hierarchical structure, give a generic model for generating arbitrary hierarchical structure in a random graph, and describe a statistically principled way to learn the set of hierarchical features that most plausibly explain a particular real-world network. By applying this approach to two example networks, we demonstrate its advantages for the interpretation of network data, the annotation of graphs with edge, vertex and community properties, and the generation of generic null models for further hypothesis testing.

physics.soc-ph↗

arXiv2006

On the Bias of Traceroute Sampling; or, Power-law Degree Distributions in Regular Graphs

Understanding the structure of the Internet graph is a crucial step for building accurate network models and designing efficient algorithms for Internet applications. Yet, obtaining its graph structure is a surprisingly difficult task, as edges cannot be explicitly queried. Instead, empirical studies rely on traceroutes to build what are essentially single-source, all-destinations, shortest-path trees. These trees only sample a fraction of the network's edges, and a recent paper by Lakhina et al. found empirically that the resuting sample is intrinsically biased. For instance, the observed degree distribution under traceroute sampling exhibits a power law even when the underlying degree distribution is Poisson. In this paper, we study the bias of traceroute sampling systematically, and, for a very general class of underlying degree distributions, calculate the likely observed distributions explicitly. To do this, we use a continuous-time realization of the process of exposing the BFS tree of a random graph with a given degree distribution, calculate the expected degree distribution of the tree, and show that it is sharply concentrated. As example applications of our machinery, we show how traceroute sampling finds power-law degree distributions in both delta-regular and Poisson-distributed random graphs. Thus, our work puts the observations of Lakhina et al. on a rigorous footing, and extends them to nearly arbitrary degree distributions.

cond-mat.dis-nn↗

arXiv2006

Scale Invariance in Road Networks

We study the topological and geographic structure of the national road networks of the United States, England and Denmark. By transforming these networks into their dual representation, where roads are vertices and an edge connects two vertices if the corresponding roads ever intersect, we show that they exhibit both topological and geographic scale invariance. That is, we show that for sufficiently large geographic areas, the dual degree distribution follows a power law with exponent 2.2 < alpha < 2.4, and that journeys, regardless of their length, have a largely identical structure. To explain these properties, we introduce and analyze a simple fractal model of road placement that reproduces the observed structure, and suggests a testable connection between the scaling exponent alpha and the fractal dimensions governing the placement of roads and intersections.

physics.soc-ph↗

arXiv2005

Scale Invariance in Global Terrorism

Traditional analyses of international terrorism have not sought to explain the emergence of rare but extremely severe events. Using the tools of extremal statistics to analyze the set of terrorist attacks worldwide between 1968 and 2004, as compiled by the National Memorial Institute for the Prevention of Terrorism (MIPT), we find that the relationship between the frequency and severity of terrorist attacks exhibits the ``scale-free'' property with an exponent of close to two. This property is robust, even when we restrict our analysis to events from a single type of weapon or events within major industrialized nations. We also find that the distribution of event sizes has changed very little over the past 37 years, suggesting that scale invariance is an inherent feature of global terrorism.

physics.soc-ph↗

arXiv2005

Finding local community structure in networks

Although the inference of global community structure in networks has recently become a topic of great interest in the physics community, all such algorithms require that the graph be completely known. Here, we define both a measure of local community structure and an algorithm that infers the hierarchy of communities that enclose a given vertex by exploring the graph one vertex at a time. This algorithm runs in time O(d*k^2) for general graphs when $d$ is the mean degree and k is the number of vertices to be explored. For graphs where exploring a new vertex is time-consuming, the running time is linear, O(k). We show that on computer-generated graphs this technique compares favorably to algorithms that require global knowledge. We also use this algorithm to extract meaningful local clustering information in the large recommender network of an online retailer and show the existence of mesoscopic structure.

physics.data-an↗

arXiv2004

Finding community structure in very large networks

The discovery and analysis of community structure in networks is a topic of considerable recent interest within the physics community, but most methods proposed so far are unsuitable for very large networks because of their computational cost. Here we present a hierarchical agglomeration algorithm for detecting community structure which is faster than many competing algorithms: its running time on a network with n vertices and m edges is O(m d log n) where d is the depth of the dendrogram describing the community structure. Many real-world networks are sparse and hierarchical, with m ~ n and d ~ log n, in which case our algorithm runs in essentially linear time, O(n log^2 n). As an example of the application of this algorithm we use it to analyze a network of items for sale on the web-site of a large online retailer, items in the network being linked if they are frequently purchased by the same buyer. The network has more than 400,000 vertices and 2 million edges. We show that our algorithm can extract meaningful communities from this network, revealing large-scale patterns present in the purchasing habits of customers.

cond-mat.stat-mech↗

arXiv2004

Why Mapping the Internet is Hard

Despite great effort spent measuring topological features of large networks like the Internet, it was recently argued that sampling based on taking paths through the network (e.g., traceroutes) introduces a fundamental bias in the observed degree distribution. We examine this bias analytically and experimentally. For classic random graphs with mean degree c, we show analytically that traceroute sampling gives an observed degree distribution P(k) ~ 1/k for k < c, even though the underlying degree distribution is Poisson. For graphs whose degree distributions have power-law tails P(k) ~ k^-alpha, the accuracy of traceroute sampling is highly sensitive to the population of low-degree vertices. In particular, when the graph has a large excess (i.e., many more edges than vertices), traceroute sampling can significantly misestimate alpha.

cond-mat.dis-nn↗

arXiv2004

Traceroute sampling makes random graphs appear to have power law degree distributions

The topology of the Internet has typically been measured by sampling traceroutes, which are roughly shortest paths from sources to destinations. The resulting measurements have been used to infer that the Internet's degree distribution is scale-free; however, many of these measurements have relied on sampling traceroutes from a small number of sources. It was recently argued that sampling in this way can introduce a fundamental bias in the degree distribution, for instance, causing random (Erdos-Renyi) graphs to appear to have power law degree distributions. We explain this phenomenon analytically using differential equations to model the growth of a breadth-first tree in a random graph G(n,p=c/n) of average degree c, and show that sampling from a single source gives an apparent power law degree distribution P(k) ~ 1/k for k < c.

cond-mat.dis-nn↗

arXiv2003

How Do Networks Become Navigable?

Networks created and maintained by social processes, such as the human friendship network and the World Wide Web, appear to exhibit the property of navigability: namely, not only do short paths exist between any pair of nodes, but such paths can easily be found using only local information. It has been shown that for networks with an underlying metric, algorithms using only local information perform extremely well if there is a power-law distribution of link lengths. However, it is not clear why or how real networks might develop this distribution. In this paper we define a decentralized ``rewiring'' process, inspired by surfers on the Web, in which each surfer attempts to travel from their home page to a random destination, and updates the outgoing link from their home page if this journey takes too long. We show that this process does indeed cause the link length distribution to converge to a power law, achieving a routing time of O(log^2 n) on networks of size n. We also study finite-size effects on the optimal exponent, and show that it converges polylogarithmically slowly as the lattice size goes to infinity.

cond-mat↗