arXiv subjects

Ross Anderson

Publications and source records attributed to Ross Anderson.

At least 37 records · Page 2Linked to original sources

arXiv2022

Chat Control or Child Protection?

Ian Levy and Crispin Robinson's position paper "Thoughts on child safety on commodity platforms" is to be welcomed for extending the scope of the debate about the extent to which child safety concerns justify legal limits to online privacy. Their paper's context is the laws proposed in both the UK and the EU to give the authorities the power to undermine end-to-end cryptography in online communications services, with a justification of preventing and detecting of child abuse and terrorist recruitment. Both jurisdictions plan to make it easier to get service firms to take down a range of illegal material from their servers; but they also propose to mandate client-side scanning - not just for known illegal images, but for text messages indicative of sexual grooming or terrorist recruitment. In this initial response, I raise technical issues about the capabilities of the technologies the authorities propose to mandate, and a deeper strategic issue: that we should view the child safety debate from the perspective of children at risk of violence, rather than from that of the security and intelligence agencies and the firms that sell surveillance software. The debate on terrorism similarly needs to be grounded in the context in which young people are radicalised. Both political violence and violence against children tend to be politicised and as a result are often poorly policed. Effective policing, particularly of crimes embedded in wicked social problems, must be locally led and involve multiple stakeholders; the idea of using 'artificial intelligence' to replace police officers, social workers and teachers is just the sort of magical thinking that leads to bad policy. The debate must also be conducted within the boundary conditions set by human rights and privacy law, and to be pragmatic must also consider reasonable police priorities.

cs.CY↗

arXiv2022

Talking Trojan: Analyzing an Industry-Wide Disclosure

While vulnerability research often focuses on technical findings and post-public release industrial response, we provide an analysis of the rest of the story: the coordinated disclosure process from discovery through public release. The industry-wide 'Trojan Source' vulnerability which affected most compilers, interpreters, code editors, and code repositories provided an interesting natural experiment, enabling us to compare responses by firms versus nonprofits and by firms that managed their own response versus firms that outsourced it. We document the interaction with bug bounty programs, government disclosure assistance, academic peer review, and press coverage, among other topics. We compare the response to an attack on source code with the response to a comparable attack on NLP systems employing machine-learning techniques. We conclude with recommendations to improve the global coordinated disclosure system.

cs.CR↗

arXiv2022

Constrained Discrete Black-Box Optimization using Mixed-Integer Programming

Discrete black-box optimization problems are challenging for model-based optimization (MBO) algorithms, such as Bayesian optimization, due to the size of the search space and the need to satisfy combinatorial constraints. In particular, these methods require repeatedly solving a complex discrete global optimization problem in the inner loop, where popular heuristic inner-loop solvers introduce approximations and are difficult to adapt to combinatorial constraints. In response, we propose NN+MILP, a general discrete MBO framework using piecewise-linear neural networks as surrogate models and mixed-integer linear programming (MILP) to optimize the acquisition function. MILP provides optimality guarantees and a versatile declarative language for domain-specific constraints. We test our approach on a range of unconstrained and constrained problems, including DNA binding, constrained binary quadratic problems from the MINLPLib benchmark, and the NAS-Bench-101 neural architecture search benchmark. NN+MILP surpasses or matches the performance of black-box algorithms tailored to the constraints at hand, with global optimization of the acquisition problem running in a few minutes using only standard software packages and hardware.

math.OC↗

arXiv2022

Attack of the Clones: Measuring the Maintainability, Originality and Security of Bitcoin 'Forks' in the Wild

Since Bitcoin appeared in 2009, over 6,000 different cryptocurrency projects have followed. The cryptocurrency world may be the only technology where a massive number of competitors offer similar services yet claim unique benefits, including scalability, fast transactions, and security. But are these projects really offering unique features and significant enhancements over their competitors? To answer this question, we conducted a large-scale empirical analysis of code maintenance activities, originality and security across 592 crypto projects. We found that about half of these projects have not been updated for the last six months; over two years, about three-quarters of them disappeared, or were reported as scams or inactive. We also investigated whether 11 security vulnerabilities patched in Bitcoin were also patched in other projects. We found that about 80% of 510 C-language-based cryptocurrency projects have at least one unpatched vulnerability, and the mean time taken to fix the vulnerability is 237.8 days. Among those 510 altcoins, we found that at least 157 altcoins are likely to have been forked from Bitcoin, about a third of them containing only slight changes from the Bitcoin version from which they were forked. As case studies, we did a deep dive into 20 altcoins (e.g., Litecoin, FujiCoin, and Feathercoin) similar to the version of Bitcoin used for the fork. About half of them did not make any technically meaningful change - failing to comply with the promises (e.g., about using Proof of Stake) made in their whitepapers.

cs.CR↗

arXiv2021

Bad Characters: Imperceptible NLP Attacks

Several years of research have shown that machine-learning systems are vulnerable to adversarial examples, both in theory and in practice. Until now, such attacks have primarily targeted visual models, exploiting the gap between human and machine perception. Although text-based models have also been attacked with adversarial examples, such attacks struggled to preserve semantic meaning and indistinguishability. In this paper, we explore a large class of adversarial examples that can be used to attack text-based models in a black-box setting without making any human-perceptible visual modification to inputs. We use encoding-specific perturbations that are imperceptible to the human eye to manipulate the outputs of a wide range of Natural Language Processing (NLP) systems from neural machine-translation pipelines to web search engines. We find that with a single imperceptible encoding injection -- representing one invisible character, homoglyph, reordering, or deletion -- an attacker can significantly reduce the performance of vulnerable models, and with three injections most models can be functionally broken. Our attacks work against currently-deployed commercial systems, including those produced by Microsoft and Google, in addition to open source models published by Facebook, IBM, and HuggingFace. This novel series of attacks presents a significant threat to many language processing systems: an attacker can affect systems in a targeted manner without any assumptions about the underlying model. We conclude that text-based NLP systems require careful input sanitization, just like conventional applications, and that given such systems are now being deployed rapidly at scale, the urgent attention of architects and operators is required.

cs.CL↗

arXiv2021

Bugs in our Pockets: The Risks of Client-Side Scanning

Our increasing reliance on digital technology for personal, economic, and government affairs has made it essential to secure the communications and devices of private citizens, businesses, and governments. This has led to pervasive use of cryptography across society. Despite its evident advantages, law enforcement and national security agencies have argued that the spread of cryptography has hindered access to evidence and intelligence. Some in industry and government now advocate a new technology to access targeted data: client-side scanning (CSS). Instead of weakening encryption or providing law enforcement with backdoor keys to decrypt communications, CSS would enable on-device analysis of data in the clear. If targeted information were detected, its existence and, potentially, its source, would be revealed to the agencies; otherwise, little or no information would leave the client device. Its proponents claim that CSS is a solution to the encryption versus public safety debate: it offers privacy -- in the sense of unimpeded end-to-end encryption -- and the ability to successfully investigate serious crime. In this report, we argue that CSS neither guarantees efficacious crime prevention nor prevents surveillance. Indeed, the effect is the opposite. CSS by its nature creates serious security and privacy risks for all society while the assistance it can provide for law enforcement is at best problematic. There are multiple ways in which client-side scanning can fail, can be evaded, and can be abused.

cs.CR↗

arXiv2021

Manipulating SGD with Data Ordering Attacks

Machine learning is vulnerable to a wide variety of attacks. It is now well understood that by changing the underlying data distribution, an adversary can poison the model trained with it or introduce backdoors. In this paper we present a novel class of training-time attacks that require no changes to the underlying dataset or model architecture, but instead only change the order in which data are supplied to the model. In particular, we find that the attacker can either prevent the model from learning, or poison it to learn behaviours specified by the attacker. Furthermore, we find that even a single adversarially-ordered epoch can be enough to slow down model learning, or even to reset all of the learning progress. Indeed, the attacks presented here are not specific to the model or dataset, but rather target the stochastic nature of modern learning procedures. We extensively evaluate our attacks on computer vision and natural language benchmarks to find that the adversary can disrupt model training and even introduce backdoors.

cs.LG↗

arXiv2021

Markpainting: Adversarial Machine Learning meets Inpainting

Inpainting is a learned interpolation technique that is based on generative modeling and used to populate masked or missing pieces in an image; it has wide applications in picture editing and retouching. Recently, inpainting started being used for watermark removal, raising concerns. In this paper we study how to manipulate it using our markpainting technique. First, we show how an image owner with access to an inpainting model can augment their image in such a way that any attempt to edit it using that model will add arbitrary visible information. We find that we can target multiple different models simultaneously with our technique. This can be designed to reconstitute a watermark if the editor had been trying to remove it. Second, we show that our markpainting technique is transferable to models that have different architectures or were trained on different datasets, so watermarks created using it are difficult for adversaries to remove. Markpainting is novel and can be used as a manipulation alarm that becomes visible in the event of inpainting.

cs.LG↗

arXiv2021

Sponge Examples: Energy-Latency Attacks on Neural Networks

The high energy costs of neural network training and inference led to the use of acceleration hardware such as GPUs and TPUs. While this enabled us to train large-scale neural networks in datacenters and deploy them on edge devices, the focus so far is on average-case performance. In this work, we introduce a novel threat vector against neural networks whose energy consumption or decision latency are critical. We show how adversaries can exploit carefully crafted $\boldsymbol{sponge}~\boldsymbol{examples}$, which are inputs designed to maximise energy consumption and latency. We mount two variants of this attack on established vision and language models, increasing energy consumption by a factor of 10 to 200. Our attacks can also be used to delay decisions where a network has critical real-time performance, such as in perception for autonomous vehicles. We demonstrate the portability of our malicious inputs across CPUs and a variety of hardware accelerator chips including GPUs, and an ASIC simulator. We conclude by proposing a defense strategy which mitigates our attack by shifting the analysis of energy consumption in hardware from an average-case to a worst-case perspective.

cs.LG↗

arXiv2020

Hey Alexa what did I just type? Decoding smartphone sounds with a voice assistant

Voice assistants are now ubiquitous and listen in on our everyday lives. Ever since they became commercially available, privacy advocates worried that the data they collect can be abused: might private conversations be extracted by third parties? In this paper we show that privacy threats go beyond spoken conversations and include sensitive data typed on nearby smartphones. Using two different smartphones and a tablet we demonstrate that the attacker can extract PIN codes and text messages from recordings collected by a voice assistant located up to half a meter away. This shows that remote keyboard-inference attacks are not limited to physical keyboards but extend to virtual keyboards too. As our homes become full of always-on microphones, we need to work through the implications.

cs.CR↗

arXiv2020

Nudge Attacks on Point-Cloud DNNs

The wide adaption of 3D point-cloud data in safety-critical applications such as autonomous driving makes adversarial samples a real threat. Existing adversarial attacks on point clouds achieve high success rates but modify a large number of points, which is usually difficult to do in real-life scenarios. In this paper, we explore a family of attacks that only perturb a few points of an input point cloud, and name them nudge attacks. We demonstrate that nudge attacks can successfully flip the results of modern point-cloud DNNs. We present two variants, gradient-based and decision-based, showing their effectiveness in white-box and grey-box scenarios. Our extensive experiments show nudge attacks are effective at generating both targeted and untargeted adversarial point clouds, by changing a few points or even a single point from the entire point-cloud input. We find that with a single point we can reliably thwart predictions in 12--80% of cases, whereas 10 points allow us to further increase this to 37--95%. Finally, we discuss the possible defenses against such attacks, and explore their limitations.

cs.CR↗

arXiv2020

The Convex Relaxation Barrier, Revisited: Tightened Single-Neuron Relaxations for Neural Network Verification

We improve the effectiveness of propagation- and linear-optimization-based neural network verification algorithms with a new tightened convex relaxation for ReLU neurons. Unlike previous single-neuron relaxations which focus only on the univariate input space of the ReLU, our method considers the multivariate input space of the affine pre-activation function preceding the ReLU. Using results from submodularity and convex geometry, we derive an explicit description of the tightest possible convex relaxation when this multivariate input is over a box domain. We show that our convex relaxation is significantly stronger than the commonly used univariate-input relaxation which has been proposed as a natural convex relaxation barrier for verification. While our description of the relaxation may require an exponential number of inequalities, we show that they can be separated in linear time and hence can be efficiently incorporated into optimization algorithms on an as-needed basis. Based on this novel relaxation, we design two polynomial-time algorithms for neural network verification: a linear-programming-based algorithm that leverages the full power of our relaxation, and a fast propagation algorithm that generalizes existing approaches. In both cases, we show that for a modest increase in computational effort, our strengthened relaxation enables us to verify a significantly larger number of instances compared to similar algorithms.

cs.LG↗

arXiv2020

Reinforcement Learning with Combinatorial Actions: An Application to Vehicle Routing

Value-function-based methods have long played an important role in reinforcement learning. However, finding the best next action given a value function of arbitrary complexity is nontrivial when the action space is too large for enumeration. We develop a framework for value-function-based deep reinforcement learning with a combinatorial action space, in which the action selection problem is explicitly formulated as a mixed-integer optimization problem. As a motivating example, we present an application of this framework to the capacitated vehicle routing problem (CVRP), a combinatorial optimization problem in which a set of locations must be covered by a single vehicle with limited capacity. On each instance, we model an action as the construction of a single route, and consider a deterministic policy which is improved through a simple policy iteration algorithm. Our approach is competitive with other reinforcement learning methods and achieves an average gap of 1.7% with state-of-the-art OR methods on standard library instances of medium size.

cs.LG↗

arXiv2020

BatNet: Data transmission between smartphones over ultrasound

In this paper, we present BatNet, a data transmission mechanism using ultrasound signals over the built-in speakers and microphones of smartphones. Using phase shift keying with an 8-point constellation and frequencies between 20--24kHz, it can transmit data at over 600bit/s up to 6m. The target application is a censorship-resistant mesh network. We also evaluated it for Covid contact tracing but concluded that in this application ultrasonic communications do not appear to offer enough advantage over Bluetooth Low Energy to be worth further development.

cs.CR↗

arXiv2020

Snitches Get Stitches: On The Difficulty of Whistleblowing

One of the most critical security protocol problems for humans is when you are betraying a trust, perhaps for some higher purpose, and the world can turn against you if you're caught. In this short paper, we report on efforts to enable whistleblowers to leak sensitive documents to journalists more safely. Following a survey of cases where whistleblowers were discovered due to operational or technological issues, we propose a game-theoretic model capturing the power dynamics involved in whistleblowing. We find that the whistleblower is often at the mercy of motivations and abilities of others. We identify specific areas where technology may be used to mitigate the whistleblower's risk. However we warn against technical solutionism: the main constraints are often institutional.

cs.CY↗

arXiv2020

To compress or not to compress: Understanding the Interactions between Adversarial Attacks and Neural Network Compression

As deep neural networks (DNNs) become widely used, pruned and quantised models are becoming ubiquitous on edge devices; such compressed DNNs are popular for lowering computational requirements. Meanwhile, recent studies show that adversarial samples can be effective at making DNNs misclassify. We, therefore, investigate the extent to which adversarial samples are transferable between uncompressed and compressed DNNs. We find that adversarial samples remain transferable for both pruned and quantised models. For pruning, the adversarial samples generated from heavily pruned models remain effective on uncompressed models. For quantisation, we find the transferability of adversarial samples is highly sensitive to integer precision.

cs.CR↗

arXiv2020

Tendrils of Crime: Visualizing the Diffusion of Stolen Bitcoins

The first six months of 2018 saw cryptocurrency thefts of $761 million, and the technology is also the latest and greatest tool for money laundering. This increase in crime has caused both researchers and law enforcement to look for ways to trace criminal proceeds. Although tracing algorithms have improved recently, they still yield an enormous amount of data of which very few datapoints are relevant or interesting to investigators, let alone ordinary bitcoin owners interested in provenance. In this work we describe efforts to visualize relevant data on a blockchain. To accomplish this we come up with a graphical model to represent the stolen coins and then implement this using a variety of visualization techniques.

cs.CY↗

arXiv2020

CAQL: Continuous Action Q-Learning

Value-based reinforcement learning (RL) methods like Q-learning have shown success in a variety of domains. One challenge in applying Q-learning to continuous-action RL problems, however, is the continuous action maximization (max-Q) required for optimal Bellman backup. In this work, we develop CAQL, a (class of) algorithm(s) for continuous-action Q-learning that can use several plug-and-play optimizers for the max-Q problem. Leveraging recent optimization results for deep neural networks, we show that max-Q can be solved optimally using mixed-integer programming (MIP). When the Q-function representation has sufficient power, MIP-based optimization gives rise to better policies and is more robust than approximate methods (e.g., gradient ascent, cross-entropy search). We further develop several techniques to accelerate inference in CAQL, which despite their approximate nature, perform well. We compare CAQL with state-of-the-art RL algorithms on benchmark continuous-control problems that have different degrees of action constraints and show that CAQL outperforms policy-based methods in heavily constrained environments, often dramatically.

cs.LG↗