arXiv subjects

Alessio Russo

Publications and source records attributed to Alessio Russo.

31 records · Page 2Linked to original sources

arXiv2023

Conformal Off-Policy Evaluation in Markov Decision Processes

Reinforcement Learning aims at identifying and evaluating efficient control policies from data. In many real-world applications, the learner is not allowed to experiment and cannot gather data in an online manner (this is the case when experimenting is expensive, risky or unethical). For such applications, the reward of a given policy (the target policy) must be estimated using historical data gathered under a different policy (the behavior policy). Most methods for this learning task, referred to as Off-Policy Evaluation (OPE), do not come with accuracy and certainty guarantees. We present a novel OPE method based on Conformal Prediction that outputs an interval containing the true reward of the target policy with a prescribed level of certainty. The main challenge in OPE stems from the distribution shift due to the discrepancies between the target and the behavior policies. We propose and empirically evaluate different ways to deal with this shift. Some of these methods yield conformalized intervals with reduced length compared to existing approaches, while maintaining the same certainty level.

cs.LG↗

arXiv2023

Model Based Residual Policy Learning with Applications to Antenna Control

Non-differentiable controllers and rule-based policies are widely used for controlling real systems such as telecommunication networks and robots. Specifically, parameters of mobile network base station antennas can be dynamically configured by these policies to improve users coverage and quality of service. Motivated by the antenna tilt control problem, we introduce Model-Based Residual Policy Learning (MBRPL), a practical reinforcement learning (RL) method. MBRPL enhances existing policies through a model-based approach, leading to improved sample efficiency and a decreased number of interactions with the actual environment when compared to off-the-shelf RL methods.To the best of our knowledge, this is the first paper that examines a model-based approach for antenna control. Experimental results reveal that our method delivers strong initial performance while improving sample efficiency over previous RL methods, which is one step towards deploying these algorithms in real networks.

cs.LG↗

arXiv2023

What influences occupants' behavior in residential buildings: An experimental study on window operation in the KTH Live-In Lab

Window-opening and window-closing behaviors play an important role in indoor environmental conditions and therefore have an impact on building energy efficiency. On the other hand, the same environmental conditions drive occupants to interact with windows. Understanding this mutual relationship of interaction between occupants and the residential building is thus crucial to improve energy efficiency without disregarding occupants' comfort. This paper investigates the influence of physical environmental variables (i.e., indoor and outside climate parameters) and categorical variables (i.e., time of the day) on occupants' behavior patterns related to window operation, utilizing a multivariate logistic regression analysis. The data considered in this study are collected during winter months, when the effect on the energy consumption of the window operation is the highest, at a Swedish residential building, the KTH Live-In Lab, accommodating four occupants in separate studio apartments. Although all the occupants seem to share a sensitivity to some common factors, such as air quality and time of the day, we can also observe individual variability with respect to the most significant drivers influencing window operation behaviors.

eess.SY↗

arXiv2023

Analysis and Detectability of Offline Data Poisoning Attacks on Linear Dynamical Systems

In recent years, there has been a growing interest in the effects of data poisoning attacks on data-driven control methods. Poisoning attacks are well-known to the Machine Learning community, which, however, make use of assumptions, such as cross-sample independence, that in general do not hold for linear dynamical systems. Consequently, these systems require different attack and detection methods than those developed for supervised learning problems in the i.i.d.\ setting. Since most data-driven control algorithms make use of the least-squares estimator, we study how poisoning impacts the least-squares estimate through the lens of statistical testing, and question in what way data poisoning attacks can be detected. We establish under which conditions the set of models compatible with the data includes the true model of the system, and we analyze different poisoning strategies for the attacker. On the basis of the arguments hereby presented, we propose a stealthy data poisoning attack on the least-squares estimator that can escape classical statistical tests, and conclude by showing the efficiency of the proposed attack.

eess.SY↗

arXiv2022

On the Sample Complexity of Representation Learning in Multi-task Bandits with Global and Local structure

We investigate the sample complexity of learning the optimal arm for multi-task bandit problems. Arms consist of two components: one that is shared across tasks (that we call representation) and one that is task-specific (that we call predictor). The objective is to learn the optimal (representation, predictor)-pair for each task, under the assumption that the optimal representation is common to all tasks. Within this framework, efficient learning algorithms should transfer knowledge across tasks. We consider the best-arm identification problem for a fixed confidence, where, in each round, the learner actively selects both a task, and an arm, and observes the corresponding reward. We derive instance-specific sample complexity lower bounds satisfied by any $(δ_G,δ_H)$-PAC algorithm (such an algorithm identifies the best representation with probability at least $1-δ_G$, and the best predictor for a task with probability at least $1-δ_H$). We devise an algorithm OSRL-SC whose sample complexity approaches the lower bound, and scales at most as $H(G\log(1/δ_G)+ X\log(1/δ_H))$, with $X,G,H$ being, respectively, the number of tasks, representations and predictors. By comparison, this scaling is significantly better than the classical best-arm identification algorithm that scales as $HGX\log(1/δ)$.

stat.ML↗

arXiv2022

Tube-Based Zonotopic Data-Driven Predictive Control

We present a novel tube-based data-driven predictive control method for linear systems affected by a bounded addictive disturbance. Our method leverages recent results in the reachability analysis of unknown linear systems to formulate and solve a robust tube-based predictive control problem. More precisely, our approach consists in deriving, from the collected data, a zonotope that includes the true state error set. We show how to guarantee the stability of the resulting error zonotope, which can be exploited to increase the computational efficiency of existing zonotopic data-driven MPC formulations. Results on a double-integrator affected by strong adversarial noise demonstrate the effectiveness of the proposed control approach.

eess.SY↗

arXiv2022

Self-Tuning Tube-based Model Predictive Control

We present Self-Tuning Tube-based Model Predictive Control (STT-MPC), an adaptive robust control algorithm for uncertain linear systems with additive disturbances based on the least-squares estimator and polytopic tubes. Our algorithm leverages concentration results to bound the system uncertainty set with prescribed confidence, and guarantees robust constraint satisfaction for this set, along with recursive feasibility and input-to-state stability. Persistence of excitation is ensured without compromising the algorithm's asymptotic performance or increasing its computational complexity. We demonstrate the performance of our algorithm using numerical experiments.

eess.SY↗

arXiv2021

Minimizing Information Leakage of Abrupt Changes in Stochastic Systems

This work investigates the problem of analyzing privacy of abrupt changes for general Markov processes. These processes may be affected by changes, or exogenous signals, that need to remain private. Privacy refers to the disclosure of information of these changes through observations of the underlying Markov chain. In contrast to previous work on privacy, we study the problem for an online sequence of data. We use theoretical tools from optimal detection theory to motivate a definition of online privacy based on the average amount of information per observation of the stochastic system in consideration. Two cases are considered: the full-information case, where the eavesdropper measures all but the signals that indicate a change, and the limited-information case, where the eavesdropper only measures the state of the Markov process. For both cases, we provide ways to derive privacy upper-bounds and compute policies that attain a higher privacy level. It turns out that the problem of computing privacy-aware policies is concave, and we conclude with some examples and numerical simulations for both cases.

eess.SY↗

arXiv2021

Balancing detectability and performance of attacks on the control channel of Markov Decision Processes

We investigate the problem of designing optimal stealthy poisoning attacks on the control channel of Markov decision processes (MDPs). This research is motivated by the recent interest of the research community for adversarial and poisoning attacks applied to MDPs, and reinforcement learning (RL) methods. The policies resulting from these methods have been shown to be vulnerable to attacks perturbing the observations of the decision-maker. In such an attack, drawing inspiration from adversarial examples used in supervised learning, the amplitude of the adversarial perturbation is limited according to some norm, with the hope that this constraint will make the attack imperceptible. However, such constraints do not grant any level of undetectability and do not take into account the dynamic nature of the underlying Markov process. In this paper, we propose a new attack formulation, based on information-theoretical quantities, that considers the objective of minimizing the detectability of the attack as well as the performance of the controlled process. We analyze the trade-off between the efficiency of the attack and its detectability. We conclude with examples and numerical simulations illustrating this trade-off.

eess.SY↗

arXiv2021

Some Ethical Issues in the Review Process of Machine Learning Conferences

Recent successes in the Machine Learning community have led to a steep increase in the number of papers submitted to conferences. This increase made more prominent some of the issues that affect the current review process used by these conferences. The review process has several issues that may undermine the nature of scientific research, which is of being fully objective, apolitical, unbiased and free of misconduct (such as plagiarism, cheating, improper influence, and other improprieties). In this work, we study the problem of reviewers' recruitment, infringements of the double-blind process, fraudulent behaviors, biases in numerical ratings, and the appendix phenomenon (i.e., the fact that it is becoming more common to publish results in the appendix section of a paper). For each of these problems, we provide a short description and possible solutions. The goal of this work is to raise awareness in the Machine Learning community regarding these issues.

cs.LG↗

arXiv2021

Poisoning Attacks against Data-Driven Control Methods

This paper investigates poisoning attacks against data-driven control methods. This work is motivated by recent trends showing that, in supervised learning, slightly modifying the data in a malicious manner can drastically deteriorate the prediction ability of the trained model. We extend these analyses to the case of data-driven control methods. Specifically, we investigate how a malicious adversary can poison the data so as to minimize the performance of a controller trained using this data. We show that identifying the most impactful attack boils down to solving a bi-level non-convex optimization problem, and provide theoretical insights on the attack. We present a generic algorithm finding a local optimum of this problem and illustrate our analysis in the case of a model-reference based approach, the Virtual Reference Feedback Tuning technique, and on data-driven methods based on Willems et al. lemma. Numerical experiments reveal that minimal but well-crafted changes in the dataset are sufficient to deteriorate the performance of data-driven control methods significantly, and even make the closed-loop system unstable.

eess.SY↗

arXiv2021

Data-Driven Control and Data-Poisoning attacks in Buildings: the KTH Live-In Lab case study

This work investigates the feasibility of using input-output data-driven control techniques for building control and their susceptibility to data-poisoning techniques. The analysis is performed on a digital replica of the KTH Livein Lab, a non-linear validated model representing one of the KTH Live-in Lab building testbeds. This work is motivated by recent trends showing a surge of interest in using data-based techniques to control cyber-physical systems. We also analyze the susceptibility of these controllers to data-poisoning methods, a particular type of machine learning threat geared towards finding imperceptible attacks that can undermine the performance of the system under consideration. We consider the Virtual Reference Feedback Tuning (VRFT), a popular data-driven control technique, and show its performance on the KTH Live-In Lab digital replica. We then demonstrate how poisoning attacks can be crafted and illustrate the impact of such attacks. Numerical experiments reveal the feasibility of using data-driven control methods for finding efficient control laws. However, a subtle change in the datasets can significantly deteriorate the performance of VRFT.

eess.SY↗

arXiv2019

Optimal Attacks on Reinforcement Learning Policies

Control policies, trained using the Deep Reinforcement Learning, have been recently shown to be vulnerable to adversarial attacks introducing even very small perturbations to the policy input. The attacks proposed so far have been designed using heuristics, and build on existing adversarial example crafting techniques used to dupe classifiers in supervised learning. In contrast, this paper investigates the problem of devising optimal attacks, depending on a well-defined attacker's objective, e.g., to minimize the main agent average reward. When the policy and the system dynamics, as well as rewards, are known to the attacker, a scenario referred to as a white-box attack, designing optimal attacks amounts to solving a Markov Decision Process. For what we call black-box attacks, where neither the policy nor the system is known, optimal attacks can be trained using Reinforcement Learning techniques. Through numerical experiments, we demonstrate the efficiency of our attacks compared to existing attacks (usually based on Gradient methods). We further quantify the potential impact of attacks and establish its connection to the smoothness of the policy under attack. Smooth policies are naturally less prone to attacks (this explains why Lipschitz policies, with respect to the state, are more resilient). Finally, we show that from the main agent perspective, the system uncertainties and the attacker can be modeled as a Partially Observable Markov Decision Process. We actually demonstrate that using Reinforcement Learning techniques tailored to POMDP (e.g. using Recurrent Neural Networks) leads to more resilient policies.

cs.LG↗