arXiv subjects

Stilian Stoev

Publications and source records attributed to Stilian Stoev.

27 records · Page 2Linked to original sources

arXiv2016

AMON: An Open Source Architecture for Online Monitoring, Statistical Analysis and Forensics of Multi-gigabit Streams

The Internet, as a global system of interconnected networks, carries an extensive array of information resources and services. Key requirements include good quality-of-service and protection of the infrastructure from nefarious activity (e.g. distributed denial of service--DDoS--attacks). Network monitoring is essential to network engineering, capacity planning and prevention / mitigation of threats. We develop an open source architecture, AMON (All-packet MONitor), for online monitoring and analysis of multi-gigabit network streams. It leverages the high-performance packet monitor PF RING and is readily deployable on commodity hardware. AMON examines all packets, partitions traffic into sub-streams by using rapid hashing and computes certain real-time data products. The resulting data structures provide views of the intensity and connectivity structure of network traffic at the time-scale of routing. The proposed integrated framework includes modules for the identification of heavy-hitters as well as for visualization and statistical detection at the time-of-onset of high impact events such as DDoS. This allows operators to quickly visualize and diagnose attacks, and limit offline and time consuming post-mortem analysis. We demonstrate our system in the context of real-world attack incidents, and validate it against state-of-the-art alternatives. AMON has been deployed and is currently processing 10Gbps+ live Internet traffic at Merit Network. It is extensible and allows the addition of further statistical and filtering modules for real-time forensics.

cs.NI↗

arXiv2016

Stochastic integral representations and classification of sum- and max-infinitely divisible processes

Introduced is the notion of minimality for spectral representations of sum- and max-infinitely divisible processes and it is shown that the minimal spectral representation on a Borel space exists and is unique. This fact is used to show that a stationary, stochastically continuous, sum- or max-i.d. random process on $\mathbb{R}^d$ can be generated by a measure-preserving flow on a $σ$-finite Borel measure space and that this flow is unique. This development makes it possible to extend the classification program of Rosiński (Ann. Probab. 23 (1995) 1163-1187) with a unified treatment of both sum- and max-infinitely divisible processes. As a particular case, a characterization of stationary, stochastically continuous, union-infinitely divisible random measurable subsets of $\mathbb{R}^d$ is obtained. Introduced and classified are several new max-i.d. random field models including fields of Penrose type and fields associated to Poisson line processes.

math.PR↗

arXiv2015

Probabilities of concurrent extremes

The statistical modelling of spatial extremes has recently made major advances. Much of its focus so far has been on the modelling of the magnitudes of extreme events but little attention has been paid on the timing of extremes. To address this gap, this paper introduces the notion of extremal concurrence. Suppose that one measures precipitation at several synoptic stations over multiple days. We say that extremes are concurrent if the maximum precipitation over time at each station is achieved simultaneously, e.g., on a single day. Under general conditions, we show that the finite sample concurrence probability converges to an asymptotic quantity, deemed extremal concurrence probability. Using Palm calculus, we establish general expressions for the extremal concurrence probability through the max-stable process emerging in the limit of the componentwise maxima of the sample. Explicit forms of the extremal concurrence probabilities are obtained for various max-stable models and several estimators are introduced. In particular, we prove that the pairwise extremal concurrence probability for max-stable vectors is precisely equal to the Kendall's $τ$. The estimators are evaluated by using simulations and applied to study the concurrence patterns of temperature extremes in the United States. The results demonstrate that concurrence probability can provide a powerful new perspective and tools for the analysis of the spatial structure and impact of extremes.

math.ST↗

arXiv2014

Hashing Pursuit for Online Identification of Heavy-Hitters in High-Speed Network Streams

Distributed Denial of Service (DDoS) attacks have become more prominent recently, both in frequency of occurrence, as well as magnitude. Such attacks render key Internet resources unavailable and disrupt its normal operation. It is therefore of paramount importance to quickly identify malicious Internet activity. The DDoS threat model includes characteristics such as: (i) heavy-hitters that transmit large volumes of traffic towards "victims", (ii) persistent-hitters that send traffic, not necessarily large, to specific destinations to be used as attack facilitators, (iii) host and port scanning for compiling lists of un-secure servers to be used as attack amplifiers, etc. This conglomeration of problems motivates the development of space/time efficient summaries of data traffic streams that can be used to identify heavy-hitters associated with the above attack vectors. This paper presents a hashing-based framework and fast algorithms that take into account the large-dimensionality of the incoming network stream and can be employed to quickly identify the culprits. The algorithms and data structures proposed provide a synopsis of the network stream that is not taxing to fast-memory, and can be efficiently implemented in hardware due to simple bit-wise operations. The methods are evaluated using real-world Internet data from a large academic network.

cs.CR↗

arXiv2014

Implicit Extremes and Implicit Max-Stable Laws

Let $X_1,...,X_n$ be iid random vectors and $f\ge 0$ be a non-negative function. Let also $k(n) = {\rm Argmax}_{i=1,...,n} f(X_i)$. We are interested in the distribution of $X_{k(n)}$ and their limit theorems. In other words, what is the distribution the random vector where a function of its components is extreme? This question is motivated by a kind of inverse problem where one wants to determine the extremal behavior of $X$ when only explicitly observing $f(X)$. We shall refer to such types of results as to implicit extremes. It turns out that, as in the usual case of explicit extremes, all limit implicit extreme value laws are implicit max-stable. We characterize the regularly varying implicit max-stable laws in terms of their spectral and stochastic representations. We also establish the asymptotic behavior of implicit order statistics relative to a given homogeneous loss and conclude with several examples drawing connections to prior work involving regular variation on general cones.

math.ST↗

arXiv2013

Fast Approximation Algorithms for Near-optimal Large-scale Network Monitoring

We study the problem of optimal traffic prediction and monitoring in large-scale networks. Our goal is to determine which subset of K links to monitor in order to "best" predict the traffic on the remaining links in the network. We consider several optimality criteria. This can be formulated as a combinatorial optimization problem, belonging to the family of subset selection problems. Similar NP-hard problems arise in statistics, machine learning and signal processing. Some include subset selection for regression, variable selection, and sparse approximation. Exact solutions are computationally prohibitive. We present both new heuristics as well as new efficient algorithms implementing the classical greedy heuristic - commonly used to tackle such combinatorial problems. Our approach exploits connections to principal component analysis (PCA), and yields new types of performance lower bounds which do not require submodularity of the objective functions. We show that an ensemble method applied to our new randomized heuristic algorithm, often outperforms the classical greedy heuristic in practice. We evaluate our algorithms under several large-scale networks, including real life networks.

cs.DS↗

arXiv2013

Efficient Approximation Algorithms for Optimal Large-scale Network Monitoring

The growing amount of applications that generate vast amount of data in short time scales render the problem of partial monitoring, coupled with prediction, a rather fundamental one. We study the aforementioned canonical problem under the context of large-scale monitoring of communication networks. We consider the problem of selecting the "best" subset of links so as to optimally predict the quantity of interest at the remaining ones. This is a well know NP-hard problem, and algorithms seeking the exact solution are prohibitively expensive. We present a number of approximation algorithms that: 1) their computational complexity gains a significant improvement over existing greedy algorithms; 2) exploit the geometry of principal component analysis, which also helps us establish theoretical bounds on the prediction error; 3) are amenable for randomized implementation and execution in parallel or distributed fashion, a process that often yields the exact solution. The new algorithms are demonstrated and evaluated using real-world network data.

cs.DS↗

arXiv2013

CRPS M-estimation for max-stable models

Max-stable random fields provide canonical models for the dependence of multivariate extremes. Inference with such models has been challenging due to the lack of tractable likelihoods. In contrast, the finite dimensional cumulative distribution functions (CDFs) are often readily available and natural to work with. Motivated by this fact, in this work we develop an M-estimation framework for max-stable models based on the continuous ranked probability score (CRPS) of multivariate CDFs. We start by establishing conditions for the consistency and asymptotic normality of the CRPS-based estimators in a general context. We then implement them in the max-stable setting and provide readily computable expressions for their asymptotic covariance matrices. The resulting point and asymptotic confidence interval estimates are illustrated over popular simulated models. They enjoy accurate coverages and offer an alternative to composite likelihood based methods.

stat.ME↗

arXiv2013

A State-Space Approach for Optimal Traffic Monitoring via Network Flow Sampling

The robustness and integrity of IP networks require efficient tools for traffic monitoring and analysis, which scale well with traffic volume and network size. We address the problem of optimal large-scale flow monitoring of computer networks under resource constraints. We propose a stochastic optimization framework where traffic measurements are done by exploiting the spatial (across network links) and temporal relationship of traffic flows. Specifically, given the network topology, the state-space characterization of network flows and sampling constraints at each monitoring station, we seek an optimal packet sampling strategy that yields the best traffic volume estimation for all flows of the network. The optimal sampling design is the result of a concave minimization problem; then, Kalman filtering is employed to yield a sequence of traffic estimates for each network flow. We evaluate our algorithm using real-world Internet2 data.

eess.SY↗