arXiv subjects

Matthew Roughan

Publications and source records attributed to Matthew Roughan.

At least 37 records · Page 2Linked to original sources

arXiv2021

High Performance Out-of-sample Embedding Techniques for Multidimensional Scaling

The recent rapid growth of the dimension of many datasets means that many approaches to dimension reduction (DR) have gained significant attention. High-performance DR algorithms are required to make data analysis feasible for big and fast data sets. However, many traditional DR techniques are challenged by truly large data sets. In particular multidimensional scaling (MDS) does not scale well. MDS is a popular group of DR techniques because it can perform DR on data where the only input is a dissimilarity function. However, common approaches are at least quadratic in memory and computation and, hence, prohibitive for large-scale data. We propose an out-of-sample embedding (OSE) solution to extend the MDS algorithm for large-scale data utilising the embedding of only a subset of the given data. We present two OSE techniques: the first based on an optimisation approach and the second based on a neural network model. With a minor trade-off in the approximation, the out-of-sample techniques can process large-scale data with reasonable computation and memory requirements. While both methods perform well, the neural network model outperforms the optimisation approach of the OSE solution in terms of efficiency. OSE has the dual benefit that it allows fast DR on streaming datasets as well as static databases.

cs.LG↗

arXiv2021

Em-K Indexing for Approximate Query Matching in Large-scale ER

Accurate and efficient entity resolution (ER) is a significant challenge in many data mining and analysis projects requiring integrating and processing massive data collections. It is becoming increasingly important in real-world applications to develop ER solutions that produce prompt responses for entity queries on large-scale databases. Some of these applications demand entity query matching against large-scale reference databases within a short time. We define this as the query matching problem in ER in this work. Indexing or blocking techniques reduce the search space and execution time in the ER process. However, approximate indexing techniques that scale to very large-scale datasets remain open to research. In this paper, we investigate the query matching problem in ER to propose an indexing method suitable for approximate and efficient query matching. We first use spatial mappings to embed records in a multidimensional Euclidean space that preserves the domain-specific similarity. Among the various mapping techniques, we choose multidimensional scaling. Then using a Kd-tree and the nearest neighbour search, the method returns a block of records that includes potential matches for a query. Our method can process queries against a large-scale dataset using only a fraction of the data $L$ (given the dataset size is $N$), with a $O(L^2)$ complexity where $L \ll N$. The experiments conducted on several datasets showed the effectiveness of the proposed method.

cs.DB↗

arXiv2021

Differential Entropy Rate Characterisations of Long Range Dependent Processes

A quantity of interest to characterise continuous-valued stochastic processes is the differential entropy rate. The rate of convergence of many properties of LRD processes is slower than might be expected, based on the intuition for conventional processes, e.g. Markov processes. Is this also true of the entropy rate? In this paper we consider the properties of the differential entropy rate of stochastic processes that have an autocorrelation function that decays as a power law. We show that power law decaying processes with similar autocorrelation and spectral density functions, Fractional Gaussian Noise and ARFIMA(0,d,0), have different entropic properties, particularly for negatively correlated parameterisations. Then we provide an equivalence between the mutual information between past and future and the differential excess entropy for stationary Gaussian processes, showing the finiteness of this quantity is the boundary between long and short range dependence. Finally, we analyse the convergence of the conditional entropy to the differential entropy rate and show that for short range dependence that the rate of convergence is of the order $O(n^{-1})$, but it is slower for long range dependent processes and depends on the Hurst parameter.

cs.IT↗

arXiv2021

Convergence of Conditional Entropy for Long Range Dependent Markov Chains

In this paper we consider the convergence of the conditional entropy to the entropy rate for Markov chains. Convergence of certain statistics of long range dependent processes, such as the sample mean, is slow. It has been shown in Carpio and Daley \cite{carpio2007long} that the convergence of the $n$-step transition probabilities to the stationary distribution is slow, without quantifying the convergence rate. We prove that the slow convergence also applies to convergence to an information-theoretic measure, the entropy rate, by showing that the convergence rate is equivalent to the convergence rate of the $n$-step transition probabilities to the stationary distribution, which is equivalent to the Markov chain mixing time problem. Then we quantify this convergence rate, and show that it is $O(n^{2H-2})$, where $n$ is the number of steps of the Markov chain and $H$ is the Hurst parameter. Finally, we show that due to this slow convergence, the mutual information between past and future is infinite if and only if the Markov chain is long range dependent. This is a discrete analogue of characterisations which have been shown for other long range dependent processes.

math.PR↗

arXiv2021

Popularity and Centrality in Spotify Networks: Critical transitions in eigenvector centrality

The modern age of digital music access has increased the availability of data about music consumption and creation, facilitating the large-scale analysis of the complex networks that connect music together. Data about user streaming behaviour, and the musical collaboration networks are particularly important with new data-driven recommendation systems. Without thorough analysis, such collaboration graphs can lead to false or misleading conclusions. Here we present a new collaboration network of artists from the online music streaming service Spotify, and demonstrate a critical change in the eigenvector centrality of artists, as low popularity artists are removed. The critical change in centrality, from classical artists to rap artists, demonstrates deeper structural properties of the network. A Social Group Centrality model is presented to simulate this critical transition behaviour, and switching between dominant eigenvectors is observed. This model presents a novel investigation of the effect of popularity bias on how centrality and importance are measured, and provides a new tool for examining such flaws in networks.

cs.SI↗

arXiv2021

NPD Entropy: A Non-Parametric Differential Entropy Rate Estimator

The estimation of entropy rates for stationary discrete-valued stochastic processes is a well studied problem in information theory. However, estimating the entropy rate for stationary continuous-valued stochastic processes has not received as much attention. In fact, many current techniques are not able to accurately estimate or characterise the complexity of the differential entropy rate for strongly correlated processes, such as Fractional Gaussian Noise and ARFIMA(0,d,0). To the point that some cannot even detect the trend of the entropy rate, e.g. when it increases/decreases, maximum, or asymptotic trends, as a function of their Hurst parameter. However, a recently developed technique provides accurate estimates at a high computational cost. In this paper, we define a robust technique for non-parametrically estimating the differential entropy rate of a continuous valued stochastic process from observed data, by making an explicit link between the differential entropy rate and the Shannon entropy rate of a quantised version of the original data. Estimation is performed by a Shannon entropy rate estimator, and then converted to a differential entropy rate estimate. We show that this technique inherits many important statistical properties from the Shannon entropy rate estimator. The estimator is able to provide better estimates than the defined relative measures and much quicker estimates than known absolute measures, for strongly correlated processes. Finally, we analyse the complexity of the estimation technique and test the robustness to non-stationarity, and show that none of the current techniques are robust to non-stationarity, even if they are robust to strong correlations.

cs.IT↗

arXiv2020

A new non-negative distribution with both finite and infinite support

The Tukey-$λ$ distribution has interesting properties including (i) for some parameters values it has finite support, and for others infinite support, and (ii) it can mimic several other distributions such that parameter estimation for the Tukey distribution is a method for identifying an appropriate class of distribution to model a set of data. The Tukey-$λ$ is, however, symmetric. Here we define a new class of {\em non-negative} distribution with similar properties to the Tukey-$λ$ distribution. As with the Tukey-$λ$ distribution, our distribution is defined in terms of its quantile function, which in this case is given by the polylogarithm function. We show the support of the distribution to be the Riemann zeta function (when finite), and we provide a closed form for the expectation, provide simple means to calculate the CDF and PDF, and show that it has relationships to the uniform, exponential, inverse beta and extreme-value distributions.

math.ST↗

arXiv2020

The Polylogarithm Function in Julia

The polylogarithm function is one of the constellation of important mathematical functions. It has a long history, and many connections to other special functions and series, and many applications, for instance in statistical physics. However, the practical aspects of its numerical evaluation have not received the type of comprehensive treatments lavished on its siblings. Only a handful of formal publications consider the evaluation of the function, and most focus on a specific domain and/or presume arbitrary precision arithmetic will be used. And very little of the literature contains any formal validation of numerical performance. In this paper we present an algorithm for calculating polylogarithms for both complex parameter and argument and evaluate it thoroughly in comparison to the arbitrary precision implementation in Mathematica. The implementation was created in a new scientific computing language Julia, which is ideal for the purpose, but also allows us to write the code in a simple, natural manner so as to make it easy to port the implementation to other such languages.

math.NA↗

arXiv2020

Simulating Name-like Vectors for Testing Large-scale Entity Resolution

Accurate and efficient entity resolution (ER) has been a problem in data analysis and data mining projects for decades. In our work, we are interested in developing ER methods to handle big data. Good public datasets are restricted in this area and usually small in size. Simulation is one technique for generating datasets for testing. Existing simulation tools have problems of complexity, scalability and limitations of resampling. We address these problems by introducing a better way of simulating testing data for big data ER. Our proposed simulation model is simple, inexpensive and fast. We focus on avoiding the detail-level simulation of records using a simple vector representation. In this paper, we will discuss how to simulate simple vectors that approximate the properties of names (commonly used as identification keys).

stat.ME↗

arXiv2020

Likelihood-based inference for modelling packet transit from thinned flow summaries

The substantial growth of network traffic speed and volume presents practical challenges to network data analysis. Packet thinning and flow aggregation protocols such as NetFlow reduce the size of datasets by providing structured data summaries, but conversely this impedes statistical inference. Methods which aim to model patterns of traffic propagation typically do not account for the packet thinning and summarisation process into the analysis, and are often simplistic, e.g.~method-of-moments. As a result, they can be of limited practical use. We introduce a likelihood-based analysis which fully incorporates packet thinning and NetFlow summarisation into the analysis. As a result, inferences can be made for models on the level of individual packets while only observing thinned flow summary information. We establish consistency of the resulting maximum likelihood estimator, derive bounds on the volume of traffic which should be observed to achieve required levels of estimator accuracy, and identify an ideal family of models. The robust performance of the estimator is examined through simulated analyses and an application on a publicly available trace dataset containing over 36m packets over a 1 minute period.

stat.AP↗

arXiv2019

Counting Candy Crush Configurations

A k-stable c-coloured Candy Crush grid is a weak proper c-colouring of a particular type of k-uniform hypergraph. In this paper we introduce a fully polynomial randomised approximation scheme (FPRAS) which counts the number of k-stable c-coloured Candy Crush grids of a given size (m, n) for certain values of c and k. We implemented this algorithm on Matlab, and found that in a Candy Crush grid with7 available colours there are approximately 4.3*10^61 3-stable colourings. (Note that, typical Candy Crush games are played with 6 colours and our FPRAS is not guaranteed to work in expected polynomial time with k= 3 and c= 6.) We also discuss the applicability of this FPRAS to the problem of counting the number of weak c-colourings of other, more general hypergraphs.

math.CO↗

arXiv2019

Bayesian inference of network structure from information cascades

Contagion processes are strongly linked to the network structures on which they propagate, and learning these structures is essential for understanding and intervention on complex network processes such as epidemics and (mis)information propagation. However, using contagion data to infer network structure is a challenging inverse problem. In particular, it is imperative to have appropriate measures of uncertainty in network structure estimates, however these are largely ignored in most machine-learning approaches. We present a probabilistic framework that uses samples from the distribution of networks that are compatible with the dynamics observed to produce network and uncertainty estimates. We demonstrate the method using the well known independent cascade model to sample from the distribution of networks P(G) conditioned on the observation of a set of infections C. We evaluate the accuracy of the method by using the marginal probabilities of each edge in the distribution, and show the bene ts of quantifying uncertainty to improve estimates and understanding, particularly with small amounts of data.

cs.SI↗

arXiv2019

How the Avengers assemble: Ecological modelling of effective cast sizes for movies

The number of characters in a movie is an interesting feature. However, it is non-trivial to measure directly. Naive metrics such as the number of credited characters vary wildly. Here, we show that a metric based on the notion of "ecological diversity" as expressed through a Shannon-entropy based metric can characterise the number of characters in a movie, and is useful in taxonomic classification. We also show how the metric can be generalised using Jensen-Shannon divergence to provide a measure of the similarity of characters appearing in different movies, for instance of use in recommender systems, e.g., Netflix. We apply our measures to the Marvel Cinematic Universe (MCU), and show what they teach us about this highly successful franchise of movies. In particular, these measures provide a useful predictor of "success" for films in the MCU, as well as a natural means to understand the relationships between the stories in the overall film arc.

cs.SI↗

arXiv2019

ForestFirewalls: Getting Firewall Configuration Right in Critical Networks (Technical Report)

Firewall configuration is critical, yet often conducted manually with inevitable errors, leaving networks vulnerable to cyber attack [40]. The impact of misconfigured firewalls can be catastrophic in Supervisory Control and Data Acquisition (SCADA) networks. These networks control the distributed assets of industrial systems such as power generation and water distribution systems. Automation can make designing firewall configurations less tedious and their deployment more reliable. In this paper, we propose ForestFirewalls, a high-level approach to configuring SCADA firewalls. Our goals are three-fold. We aim to: first, decouple implementation details from security policy design by abstracting the former; second, simplify policy design; and third, provide automated checks, pre and post-deployment, to guarantee configuration accuracy. We achieve these goals by automating the implementation of a policy to a network and by auto-validating each stage of the configuration process. We test our approach on a real SCADA network to demonstrate its effectiveness.

cs.CR↗

arXiv2019

Verifying and Monitoring IoTs Network Behavior using MUD Profiles

IoT devices are increasingly being implicated in cyber-attacks, raising community concern about the risks they pose to critical infrastructure, corporations, and citizens. In order to reduce this risk, the IETF is pushing IoT vendors to develop formal specifications of the intended purpose of their IoT devices, in the form of a Manufacturer Usage Description (MUD), so that their network behavior in any operating environment can be locked down and verified rigorously. This paper aims to assist IoT manufacturers in developing and verifying MUD profiles, while also helping adopters of these devices to ensure they are compatible with their organizational policies and track devices network behavior based on their MUD profile. Our first contribution is to develop a tool that takes the traffic trace of an arbitrary IoT device as input and automatically generates the MUD profile for it. We contribute our tool as open source, apply it to 28 consumer IoT devices, and highlight insights and challenges encountered in the process. Our second contribution is to apply a formal semantic framework that not only validates a given MUD profile for consistency, but also checks its compatibility with a given organizational policy. We apply our framework to representative organizations and selected devices, to demonstrate how MUD can reduce the effort needed for IoT acceptance testing. Finally, we show how operators can dynamically identify IoT devices using known MUD profiles and monitor their behavioral changes on their network.

cs.NI↗

arXiv2018

The one comparing narrative social network extraction techniques

Analysing narratives through their social networks is an expanding field in quantitative literary studies. Manually extracting a social network from any narrative can be time consuming, so automatic extraction methods of varying complexity have been developed. However, the effect of different extraction methods on the analysis is unknown. Here we model and compare three extraction methods for social networks in narratives: manual extraction, co-occurrence automated extraction and automated extraction using machine learning. Although the manual extraction method produces more precise results in the network analysis, it is much more time consuming and the automatic extraction methods yield comparable conclusions for density, centrality measures and edge weights. Our results provide evidence that social networks extracted automatically are reliable for many analyses. We also describe which aspects of analysis are not reliable with such a social network. We anticipate that our findings will make it easier to analyse more narratives, which help us improve our understanding of how stories are written and evolve, and how people interact with each other.

cs.SI↗

arXiv2018

Generating Connected Random Graphs

Sampling random graphs is essential in many applications, and often algorithms use Markov chain Monte Carlo methods to sample uniformly from the space of graphs. However, often there is a need to sample graphs with some property that we are unable, or it is too inefficient, to sample using standard approaches. In this paper, we are interested in sampling graphs from a conditional ensemble of the underlying graph model. We present an algorithm to generate samples from an ensemble of connected random graphs using a Metropolis-Hastings framework. The algorithm extends to a general framework for sampling from a known distribution of graphs, conditioned on a desired property. We demonstrate the method to generate connected spatially embedded random graphs, specifically the well known Waxman network, and illustrate the convergence and practicalities of the algorithm.

cs.SI↗

arXiv2018

Surreal Birthdays and Their Arithmetic

Surreal numbers are created recursively, with the "birthday" being the depth of the recursion. Birthday arithmetic describes how birthdays of surreal numbers are transformed by standard arithemetic operations. This paper shows that birthday arithmetic follows simple rules.

math.HO↗