SearcharxivSearch

arXiv subjects

Dave Cliff

Publications and source records attributed to Dave Cliff.

At least 19 recordsLinked to original sources

Tournament versus Circulant: On Simulating 7-Species Evolutionary Spatial Cyclic Games with Ablated Predator-Prey Networks as Models of Biodiversity

Computer simulations of minimal population-dynamics models have long been used to explore questions in ecosystems coexistence and species biodiversity, via simple agent-based models of three interacting species, referred to as $R$, $P$, and $S$, and where individual agents compete with one another in predator/prey contests that are determined by the cyclic dominance rules of the Rock-Paper-Scissors game. Recent publications have explored the dynamics of five-species models, based on the Rock-Paper-Scissors-Lizard-Spock (RPSLS) game. A 2022 paper by Zhong et al. reported simulation studies of species coexistence in spatial RPSLS systems in which one or more directed edges are ablated from the five-vertex tournament digraph defining the RPSLS game: Zhong et al. showed that the ablation of a single inter-species interaction can lead to a collapse in biodiversity. In this paper I present first results from simulation studies of evolutionary spatial cyclic games where there are seven species, but where each species is still in its own local five-species RPSLS-like interaction network: the dominance networks I use for this are a subset of the $n$-node $k$-regular circulant digraphs $D(n,\Omega)$ for odd-numbered $n$ and $|\Omega|=2$. My results indicate that Zhong et al.'s results are due to the specific fully-connected tournament dominance network used in their RPSLS model: when other, equally realistic, RPSLS-style dominance networks are used instead, no such sudden collapse in biodiversity occurs. The Python source-code used for the work reported here is freely available on GitHub.

q-bio.PE

Never Mind The No-Ops: Faster and Less Volatile Simulation Modelling of Co-Evolutionary Species Interactions via Spatial Cyclic Games

Issues in co-evolutionary population dynamics have long been studied via computationally intensive simulations of minimally simple agent-based models, known as Evolutionary Spatial Cyclic Games (ESCGs), involving multiple interacting biological species in which each agent has its own unique spatial location in a cell on a regular lattice, and can move from cell to cell over time. Many papers have been published exploring the dynamics of ESCGs where competitive inter-species predator/prey relationships are modelled via the cyclic game Rock-Paper-Scissors (RPS) for three species, or Rock-Paper-Scissors-Lizard-Spock (RPSLS) for five. At the core of these simulations is the "Elementary Step" (ES), in which one or two agents are chosen at random to either compete to the death, or to reproduce, or to move location. ESCG studies typically involve executing trillions of ESs and hence the computational efficiency of the core ES algorithm is a key concern. In this paper I demonstrate that the de facto standard "Original ES" (OES) algorithm is computationally inefficient both in time and in space due to the implicit execution of many "no-op" commands (i.e., commands that do nothing) and because at steady state large numbers of cells can be empty, and yet empty cells serve no purpose. I present a new "Revised ES" (RES) algorithm which eliminates these inefficiencies, and I show empirically that ESCGs with RES exhibit qualitatively the same characteristics as those with OES, and are also markedly more stable. The more stable dynamics of RES-based simulations means that they can be run with smaller lattices than when using OES, leading to reductions in total simulation times of 85% or more. Python source code developed for the experiments reported here is freely available on GitHub.

q-bio.PE

On Long-Term Species Coexistence in Five-Species Evolutionary Spatial Cyclic Games with Ablated and Non-Ablated Dominance Networks

I present a replication and, to some extent, a refutation of key results published by Zhong, Zhang, Li, Dai, & Yang in their 2022 paper "Species coexistence in spatial cyclic game of five species" (Chaos, Solitons and Fractals, 156: 111806), where ecosystem species coexistence was explored via simulation studies of the evolutionary spatial cyclic game (ESCG) Rock-Paper-Scissors-Lizard-Spock (RPSLS) with certain predator-prey relationships removed from the game's "interaction structure", i.e. with specific arcs ablated in the ESCG's dominance network, and with the ESCG run for 100,000 Monte Carlo Steps (MCS) to identify its asymptotic behaviors. I replicate the results presented by Zhong et al. for interaction structures with one, two, three, and four arcs ablated from the dominance network. I then empirically demonstrate that the dynamics of the RPSLS ESCG have sufficiently long time constants that the true asymptotic outcomes can often only be identified after running the ablated ESCG for 10,000,000MCS or longer, and that the true long-term outcomes can be markedly less diverse than those reported by Zhong et al. as asymptotic. Finally I demonstrate that, when run for sufficiently many MCS, the original unablated RPSLS system exhibits essentially the same asymptotic outcomes as the ablated RPSLS systems, and in this sense the only causal effect of the ablations is to alter the time required for the system to converge to the long-term asymptotic states that the unablated system eventually settles to anyhow.

q-bio.PE

XGBoost Learning of Dynamic Wager Placement for In-Play Betting on an Agent-Based Model of a Sports Betting Exchange

We present first results from the use of XGBoost, a highly effective machine learning (ML) method, within the Bristol Betting Exchange (BBE), an open-source agent-based model (ABM) designed to simulate a contemporary sports-betting exchange with in-play betting during track-racing events such as horse races. We use the BBE ABM and its array of minimally-simple bettor-agents as a synthetic data generator which feeds into our XGBoost ML system, with the intention that XGBoost discovers profitable dynamic betting strategies by learning from the more profitable bets made by the BBE bettor-agents. After this XGBoost training, which results in one or more decision trees, a bettor-agent with a betting strategy determined by the XGBoost-learned decision tree(s) is added to the BBE ABM and made to bet on a sequence of races under various conditions and betting-market scenarios, with profitability serving as the primary metric of comparison and evaluation. Our initial findings presented here show that XGBoost trained in this way can indeed learn profitable betting strategies, and can generalise to learn strategies that outperform each of the set of strategies used for creation of the training data. To foster further research and enhancements, the complete version of our extended BBE, including the XGBoost integration, has been made freely available as an open-source release on GitHub.

cs.LG

Exploring Coevolutionary Dynamics of Competitive Arms-Races Between Infinitely Diverse Heterogenous Adaptive Automated Trader-Agents

We report on a series of experiments in which we study the coevolutionary "arms-race" dynamics among groups of agents that engage in adaptive automated trading in an accurate model of contemporary financial markets. At any one time, every trader in the market is trying to make as much profit as possible given the current distribution of different other trading strategies that it finds itself pitched against in the market; but the distribution of trading strategies and their observable behaviors is constantly changing, and changes in any one trader are driven to some extent by the changes in all the others. Prior studies of coevolutionary dynamics in markets have concentrated on systems where traders can choose one of a small number of fixed pure strategies, and can change their choice occasionally, thereby giving a market with a discrete phase-space, made up of a finite set of possible system states. Here we present first results from two independent sets of experiments, where we use minimal-intelligence trading-agents but in which the space of possible strategies is continuous and hence infinite. Our work reveals that by taking only a small step in the direction of increased realism we move immediately into high-dimensional phase-spaces, which then present difficulties in visualising and understanding the coevolutionary dynamics unfolding within the system. We conclude that further research is required to establish better analytic tools for monitoring activity and progress in co-adapting markets. We have released relevant Python code as open-source on GitHub, to enable others to continue this work.

cs.CE

Implementing the BBE Agent-Based Model of a Sports-Betting Exchange

We describe three independent implementations of a new agent-based model (ABM) that simulates a contemporary sports-betting exchange, such as those offered commercially by companies including Betfair, Smarkets, and Betdaq. The motivation for constructing this ABM, which is known as the Bristol Betting Exchange (BBE), is so that it can serve as a synthetic data generator, producing large volumes of data that can be used to develop and test new betting strategies via advanced data analytics and machine learning techniques. Betting exchanges act as online platforms on which bettors can find willing counterparties to a bet, and they do this in a way that is directly comparable to the manner in which electronic financial exchanges, such as major stock markets, act as platforms that allow traders to find willing counterparties to buy from or sell to: the platform aggregates and anonymises orders from multiple participants, showing a summary of the market that is updated in real-time. In the first instance, BBE is aimed primarily at producing synthetic data for in-play betting (also known as in-race or in-game betting) where bettors can place bets on the outcome of a track-race event, such as a horse race, after the race has started and for as long as the race is underway, with betting only ceasing when the race ends. The rationale for, and design of, BBE has been described in detail in a previous paper that we summarise here, before discussing our comparative results which contrast a single-threaded implementation in Python, a multi-threaded implementation in Python, and an implementation where Python header-code calls simulations of the track-racing events written in OpenCL that execute on a 640-core GPU -- this runs approximately 1000 times faster than the single-threaded Python. Our source-code for BBE is freely available on GitHub.

cs.CE

BBE: Simulating the Microstructural Dynamics of an In-Play Betting Exchange via Agent-Based Modelling

I describe the rationale for, and design of, an agent-based simulation model of a contemporary online sports-betting exchange: such exchanges, closely related to the exchange mechanisms at the heart of major financial markets, have revolutionized the gambling industry in the past 20 years, but gathering sufficiently large quantities of rich and temporally high-resolution data from real exchanges - i.e., the sort of data that is needed in large quantities for Deep Learning - is often very expensive, and sometimes simply impossible; this creates a need for a plausibly realistic synthetic data generator, which is what this simulation now provides. The simulator, named the "Bristol Betting Exchange" (BBE), is intended as a common platform, a data-source and experimental test-bed, for researchers studying the application of AI and machine learning (ML) techniques to issues arising in betting exchanges; and, as far as I have been able to determine, BBE is the first of its kind: a free open-source agent-based simulation model consisting not only of a sports-betting exchange, but also a minimal simulation model of racetrack sporting events (e.g., horse-races or car-races) about which bets may be made, and a population of simulated bettors who each form their own private evaluation of odds and place bets on the exchange before and - crucially - during the race itself (i.e., so-called "in-play" betting) and whose betting opinions change second-by-second as each race event unfolds. BBE is offered as a proof-of-concept system that enables the generation of large high-resolution data-sets for automated discovery or improvement of profitable strategies for betting on sporting events via the application of AI/ML and advanced data analytics techniques. This paper offers an extensive survey of relevant literature and explains the motivation and design of BBE, and presents brief illustrative results.

cs.MA

Parameterised-Response Zero-Intelligence Traders

I introduce PRZI (Parameterised-Response Zero Intelligence), a new form of zero-intelligence trader intended for use in simulation studies of the dynamics of continuous double auction markets. Like Gode & Sunder's classic ZIC trader, PRZI generates quote-prices from a random distribution over some specified domain of allowable quote-prices. Unlike ZIC, which uses a uniform distribution to generate prices, the probability distribution in a PRZI trader is parameterised in such a way that its probability mass function (PMF) is determined by a real-valued control variable s in the range [-1.0, +1.0] that determines the _strategy_ for that trader. When s=0, a PRZI trader is identical to ZIC, with a uniform PMF; but when |s|=~1 the PRZI trader's PMF becomes maximally skewed to one extreme or the other of the price-range, thereby making its quote-prices more or less urgent, biasing the quote-price distribution toward or away from the trader's limit-price. To explore the co-evolutionary dynamics of populations of PRZI traders that dynamically adapt their strategies, I show results from long-term market experiments in which each trader uses a simple stochastic hill-climber algorithm to repeatedly evaluate alternative s-values and choose the most profitable at any given time. In these experiments the profitability of any particular s-value may be non-stationary because the profitability of one trader's strategy at any one time can depend on the mix of strategies being played by the other traders at that time, which are each themselves continuously adapting. Results from these market experiments demonstrate that the population of traders' strategies can exhibit rich dynamics, with periods of stability lasting over hundreds of thousands of trader interactions interspersed by occasional periods of change. Python source-code for the work reported here has been made publicly available on GitHub.

q-fin.TR

Time Matters: Exploring the Effects of Urgency and Reaction Speed in Automated Traders

We consider issues of time in automated trading strategies in simulated financial markets containing a single exchange with public limit order book and continuous double auction matching. In particular, we explore two effects: (i) reaction speed - the time taken for trading strategies to calculate a response to market events; and (ii) trading urgency - the sensitivity of trading strategies to approaching deadlines. Much of the literature on trading agents focuses on optimising pricing strategies only and ignores the effects of time, while real-world markets continue to experience a race to zero latency, as automated trading systems compete to quickly access information and act in the market ahead of others. We demonstrate that modelling reaction speed can significantly alter previously published results, with simple strategies such as SHVR outperforming more complex adaptive algorithms such as AA. We also show that adding a pace parameter to ZIP traders (ZIP-Pace, or ZIPP) can create a sense of urgency that significantly improves profitability.

cs.MA

Market Impact in Trader-Agents: Adding Multi-Level Order-Flow Imbalance-Sensitivity to Automated Trading Systems

Financial markets populated by human traders often exhibit "market impact", where the traders' quote-prices move in the direction of anticipated change, before any transaction has taken place, as an immediate reaction to the arrival of a large (i.e., "block") buy or sell order in the market: e.g., traders in the market know that a block buy order will push the price up, and so they immediately adjust their quote-prices upwards. Most major financial markets now involve many "robot traders", autonomous adaptive software agents, rather than humans. This paper explores how to give such trader-agents a reliable anticipatory sensitivity to block orders, such that markets populated entirely by robot traders also show market-impact effects. In a 2019 publication Church & Cliff presented initial results from a simple deterministic robot trader, ISHV, which exhibits this market impact effect via monitoring a metric of imbalance between supply and demand in the market. The novel contributions of our paper are: (a) we critique the methods used by Church & Cliff, revealing them to be weak, and argue that a more robust measure of imbalance is required; (b) we argue for the use of multi-level order-flow imbalance (MLOFI: Xu et al., 2019) as a better basis for imbalance-sensitive robot trader-agents; and (c) we demonstrate the use of the more robust MLOFI measure in extending ISHV, and also the well-known AA and ZIP trading-agent algorithms (which have both been previously shown to consistently outperform human traders). We demonstrate that the new imbalance-sensitive trader-agents introduced here do exhibit market impact effects, and hence are better-suited to operating in markets where impact is a factor of concern or interest, but do not suffer the weaknesses of the methods used by Church & Cliff. The source-code for our work reported here is freely available on GitHub.

q-fin.TR

Exploring Narrative Economics: An Agent-Based-Modeling Platform that Integrates Automated Traders with Opinion Dynamics

In seeking to explain aspects of real-world economies that defy easy understanding when analysed via conventional means, Nobel Laureate Robert Shiller has since 2017 introduced and developed the idea of Narrative Economics, where observable economic factors such as the dynamics of prices in asset markets are explained largely as a consequence of the narratives (i.e., the stories) heard, told, and believed by participants in those markets. Shiller argues that otherwise irrational and difficult-to-explain behaviors, such as investors participating in highly volatile cryptocurrency markets, are best explained and understood in narrative terms: people invest because they believe, because they have a heartfelt opinions, about the future prospects of the asset, and they tell to themselves and others stories (narratives) about those beliefs and opinions. In this paper we describe what is, to the best of our knowledge, the first ever agent-based modelling platform that allows for the study of issues in narrative economics. We have created this by integrating and synthesizing research in two previously separate fields: opinion dynamics (OD), and agent-based computational economics (ACE) in the form of minimally-intelligent trader-agents operating in accurately modelled financial markets. We show here for the first time how long-established models in OD and in ACE can be brought together to enable the experimental study of issues in narrative economics, and we present initial results from our system. The program-code for our simulation platform has been released as freely-available open-source software on GitHub, to enable other researchers to replicate and extend our work

q-fin.TR

Methods Matter: A Trading Agent with No Intelligence Routinely Outperforms AI-Based Traders

There's a long tradition of research using computational intelligence (methods from artificial intelligence (AI) and machine learning (ML)), to automatically discover, implement, and fine-tune strategies for autonomous adaptive automated trading in financial markets, with a sequence of research papers on this topic published at AI conferences such as IJCAI and in journals such as Artificial Intelligence: we show here that this strand of research has taken a number of methodological mis-steps and that actually some of the reportedly best-performing public-domain AI/ML trading strategies can routinely be out-performed by extremely simple trading strategies that involve no AI or ML at all. The results that we highlight here could easily have been revealed at the time that the relevant key papers were published, more than a decade ago, but the accepted methodology at the time of those publications involved a somewhat minimal approach to experimental evaluation of trader-agents, making claims on the basis of a few thousand test-sessions of the trader-agent in a small number of market scenarios. In this paper we present results from exhaustive testing over wide ranges of parameter values, using parallel cloud-computing facilities, where we conduct millions of tests and thereby create much richer data from which firmer conclusions can be drawn. We show that the best public-domain AI/ML traders in the published literature can be routinely outperformed by a "sub-zero-intelligence" trading strategy that at face value appears to be so simple as to be financially ruinous, but which interacts with the market in such a way that in practice it is more profitable than the well-known AI/ML strategies from the research literature. That such a simple strategy can outperform established AI/ML-based strategies is a sign that perhaps the AI/ML trading strategies were good answers to the wrong question.

cs.CE

Automated Creation of a High-Performing Algorithmic Trader via Deep Learning on Level-2 Limit Order Book Data

We present results demonstrating that an appropriately configured deep learning neural network (DLNN) can automatically learn to be a high-performing algorithmic trading system, operating purely from training-data inputs generated by passive observation of an existing successful trader T. That is, we can point our black-box DLNN system at trader T and successfully have it learn from T's trading activity, such that it trades at least as well as T. Our system, called DeepTrader, takes inputs derived from Level-2 market data, i.e. the market's Limit Order Book (LOB) or Ladder for a tradeable asset. Unusually, DeepTrader makes no explicit prediction of future prices. Instead, we train it purely on input-output pairs where in each pair the input is a snapshot S of Level-2 LOB data taken at the time when T issued a quote Q (i.e. a bid or an ask order) to the market; and DeepTrader's desired output is to produce Q when it is shown S. That is, we train our DLNN by showing it the LOB data S that T saw at the time when T issued quote Q, and in doing so our system comes to behave like T, acting as an algorithmic trader issuing specific quotes in response to specific LOB conditions. We train DeepTrader on large numbers of these S/Q snapshot/quote pairs, and then test it in a variety of market scenarios, evaluating it against other algorithmic trading systems in the public-domain literature, including two that have repeatedly been shown to outperform human traders. Our results demonstrate that DeepTrader learns to match or outperform such existing algorithmic trading systems. We analyse the successful DeepTrader network to identify what features it is relying on, and which features can be ignored. We propose that our methods can in principle create an explainable copy of an arbitrary trader T via "black-box" deep learning methods.

q-fin.TR

Which Trading Agent is Best? Using a Threaded Parallel Simulation of a Financial Market Changes the Pecking-Order

This paper presents novel results generated from a new simulation model of a contemporary financial market, that cast serious doubt on the previously widely accepted view of the relative performance of various well-known public-domain automated-trading algorithms. Various public-domain trading algorithms have been proposed over the past 25 years in a kind of arms-race, where each new trading algorithm was compared to the previous best, thereby establishing a "pecking order", i.e. a partially-ordered dominance hierarchy from best to worst of the various trading algorithms. Many of these algorithms were developed and tested using simple minimal simulations of financial markets that only weakly approximated the fact that real markets involve many different trading systems operating asynchronously and in parallel. In this paper we use BSE, a public-domain market simulator, to run a set of experiments generating benchmark results from several well-known trading algorithms. BSE incorporates a very simple time-sliced approach to simulating parallelism, which has obvious known weaknesses. We then alter and extend BSE to make it threaded, so that different trader algorithms operate asynchronously and in parallel: we call this simulator Threaded-BSE (TBSE). We then re-run the trader experiments on TBSE and compare the TBSE results to our earlier benchmark results from BSE. Our comparison shows that the dominance hierarchy in our more realistic experiments is different from the one given by the original simple simulator. We conclude that simulated parallelism matters a lot, and that earlier results from simple simulations comparing different trader algorithms are no longer to be entirely trusted.

q-fin.TR

Automated Composition of Picture-Synched Music Soundtracks for Movies

We describe the implementation of and early results from a system that automatically composes picture-synched musical soundtracks for videos and movies. We use the phrase "picture-synched" to mean that the structure of the automatically composed music is determined by visual events in the input movie, i.e. the final music is synchronised to visual events and features such as cut transitions or within-shot key-frame events. Our system combines automated video analysis and computer-generated music-composition techniques to create unique soundtracks in response to the video input, and can be thought of as an initial step in creating a computerised replacement for a human composer writing music to fit the picture-locked edit of a movie. Working only from the video information in the movie, key features are extracted from the input video, using video analysis techniques, which are then fed into a machine-learning-based music generation tool, to compose a piece of music from scratch. The resulting soundtrack is tied to video features, such as scene transition markers and scene-level energy values, and is unique to the input video. Although the system we describe here is only a preliminary proof-of-concept, user evaluations of the output of the system have been positive.

cs.MM

Adaptive-Aggressive Traders Don't Dominate

For more than a decade Vytelingum's Adaptive-Aggressive (AA) algorithm has been recognized as the best-performing automated auction-market trading-agent strategy currently known in the AI/Agents literature; in this paper, we demonstrate that it is in fact routinely outperformed by another algorithm when exhaustively tested across a sufficiently wide range of market scenarios. The novel step taken here is to use large-scale compute facilities to brute-force exhaustively evaluate AA in a variety of market environments based on those used for testing it in the original publications. Our results show that even in these simple environments AA is consistently out-performed by IBM's GDX algorithm, first published in 2002. We summarize here results from more than one million market simulation experiments, orders of magnitude more testing than was reported in the original publications that first introduced AA. A 2019 ICAART paper by Cliff claimed that AA's failings were revealed by testing it in more realistic experiments, with conditions closer to those found in real financial markets, but here we demonstrate that even in the simple experiment conditions that were used in the original AA papers, exhaustive testing shows AA to be outperformed by GDX. We close this paper with a discussion of the methodological implications of our work: any results from previous papers where any one trading algorithm is claimed to be superior to others on the basis of only a few thousand trials are probably best treated with some suspicion now. The rise of cloud computing means that the compute-power necessary to subject trading algorithms to millions of trials over a wide range of conditions is readily available at reasonable cost: we should make use of this; exhaustive testing such as is shown here should be the norm in future evaluations and comparisons of new trading algorithms.

q-fin.TR

A Cloud-Native Globally Distributed Financial Exchange Simulator for Studying Real-World Trading-Latency Issues at Planetary Scale

We describe a new public-domain open-source simulator of an electronic financial exchange, and of the traders that interact with the exchange, which is a truly distributed and cloud-native system that been designed to run on widely available commercial cloud-computing services, and in which various components can be placed in specified geographic regions around the world, thereby enabling the study of planetary-scale latencies in contemporary automated trading systems. Our simulator allows an exchange server to be launched in the cloud, specifying a particular geographic zone for the cloud hosting service; automated-trading clients which attach to the exchange can then also be launched in the cloud, in the same geographic zone and/or in different zones anywhere else on the planet, and those clients are then subject to the real-world latencies introduced by planetary-scale cloud communication interconnections. In this paper we describe the design and implementation of our simulator, called DBSE, which is based on a previous public-domain simulator, extended in ways that are partly inspired by the architecture of the real-world Jane Street Exchange. DBSE relies fundamentally on UDP and TCP network communications protocols and implements a subset of the FIX de facto standard protocol for financial information exchange. We show results from an example in which the exchange server is remotely launched on a cloud facility located in London (UK), with trader clients running in Ohio (USA) and Sydney (Australia). We close with discussion of how our simulator could be further used to study planetary-scale latency arbitrage in financial markets.

cs.CY

Deep Learning can Replicate Adaptive Traders in a Limit-Order-Book Financial Market

We report successful results from using deep learning neural networks (DLNNs) to learn, purely by observation, the behavior of profitable traders in an electronic market closely modelled on the limit-order-book (LOB) market mechanisms that are commonly found in the real-world global financial markets for equities (stocks & shares), currencies, bonds, commodities, and derivatives. Successful real human traders, and advanced automated algorithmic trading systems, learn from experience and adapt over time as market conditions change; our DLNN learns to copy this adaptive trading behavior. A novel aspect of our work is that we do not involve the conventional approach of attempting to predict time-series of prices of tradeable securities. Instead, we collect large volumes of training data by observing only the quotes issued by a successful sales-trader in the market, details of the orders that trader is executing, and the data available on the LOB (as would usually be provided by a centralized exchange) over the period that the trader is active. In this paper we demonstrate that suitably configured DLNNs can learn to replicate the trading behavior of a successful adaptive automated trader, an algorithmic system previously demonstrated to outperform human traders. We also demonstrate that DLNNs can learn to perform better (i.e., more profitably) than the trader that provided the training data. We believe that this is the first ever demonstration that DLNNs can successfully replicate a human-like, or super-human, adaptive trader operating in a realistic emulation of a real-world financial market. Our results can be considered as proof-of-concept that a DLNN could, in principle, observe the actions of a human trader in a real financial market and over time learn to trade equally as well as that human trader, and possibly better.

cs.CE