arXiv subjects

Anders Ahlén

Publications and source records attributed to Anders Ahlén.

10 recordsLinked to original sources

arXiv2023

Continual Learning with Distributed Optimization: Does CoCoA Forget?

We focus on the continual learning problem where the tasks arrive sequentially and the aim is to perform well on the newly arrived task without performance degradation on the previously seen tasks. In contrast to the continual learning literature focusing on the centralized setting, we investigate the distributed estimation framework. We consider the well-established distributed learning algorithm COCOA. We derive closed form expressions for the iterations for the overparametrized case. We illustrate the convergence and the error performance of the algorithm based on the over/under-parameterization of the problem. Our results show that depending on the problem dimensions and data generation assumptions, COCOA can perform continual learning over a sequence of tasks, i.e., it can learn a new task without forgetting previously learned tasks, with access only to one task at a time.

stat.ML↗

arXiv2023

Regularization Trade-offs with Fake Features

Recent successes of massively overparameterized models have inspired a new line of work investigating the underlying conditions that enable overparameterized models to generalize well. This paper considers a framework where the possibly overparametrized model includes fake features, i.e., features that are present in the model but not in the data. We present a non-asymptotic high-probability bound on the generalization error of the ridge regression problem under the model misspecification of having fake features. Our highprobability results provide insights into the interplay between the implicit regularization provided by the fake features and the explicit regularization provided by the ridge parameter. Numerical results illustrate the trade-off between the number of fake features and how the optimal ridge parameter may heavily depend on the number of fake features.

cs.LG↗

arXiv2023

Distributed Continual Learning with CoCoA in High-dimensional Linear Regression

We consider estimation under scenarios where the signals of interest exhibit change of characteristics over time. In particular, we consider the continual learning problem where different tasks, e.g., data with different distributions, arrive sequentially and the aim is to perform well on the newly arrived task without performance degradation on the previously seen tasks. In contrast to the continual learning literature focusing on the centralized setting, we investigate the problem from a distributed estimation perspective. We consider the well-established distributed learning algorithm COCOA, which distributes the model parameters and the corresponding features over the network. We provide exact analytical characterization for the generalization error of COCOA under continual learning for linear regression in a range of scenarios, where overparameterization is of particular interest. These analytical results characterize how the generalization error depends on the network structure, the task similarity and the number of tasks, and show how these dependencies are intertwined. In particular, our results show that the generalization error can be significantly reduced by adjusting the network size, where the most favorable network size depends on task similarity and the number of tasks. We present numerical results verifying the theoretical analysis and illustrate the continual learning performance of COCOA with a digit classification task.

cs.LG↗

arXiv2023

Quickest Detection of Deception Attacks on Cyber-Physical Systems with a Parsimonious Watermarking Policy

The addition of a physical watermarking signal to the control input increases the detection probability of data deception attacks at the expense of increased control cost. In this paper, we propose a parsimonious policy to reduce the average number of watermarking events when the attack is not present, which in turn reduces the control cost. We model the system as a stochastic optimal control problem and apply the dynamic programming to minimize the average detection delay (ADD) for fixed upper bounds on false alarm rate (FAR) and increased control cost. The optimal solution results in a two threshold policy on the posterior probability of attack, which is derived from the Shiryaev statistics for sequential change detection assuming the change point is a random variable with a geometric distribution. We derive approximate expressions of ADD and FAR applying the non-linear renewal theory. The relationship between the average number of watermarking added before the attack and the increase in control cost is also derived. We design the optimal watermarking that maximizes the Kullback-Leibler divergence for a fixed increase in the control cost. Simulation studies are performed to illustrate and validate the theoretical results.

math.OC↗

arXiv2022

Estimation under Model Misspecification with Fake Features

We consider estimation under model misspecification where there is a model mismatch between the underlying system, which generates the data, and the model used during estimation. We propose a model misspecification framework which enables a joint treatment of the model misspecification types of having fake features as well as incorrect covariance assumptions on the unknowns and the noise. We present a decomposition of the output error into components that relate to different subsets of the model parameters corresponding to underlying, fake and missing features. Here, fake features are features which are included in the model but are not present in the underlying system. Under this framework, we characterize the estimation performance and reveal trade-offs between the number of samples, number of fake features, and the possibly incorrect noise level assumption. In contrast to existing work focusing on incorrect covariance assumptions or missing features, fake features is a central component of our framework. Our results show that fake features can significantly improve the estimation performance, even though they are not correlated with the features in the underlying system. In particular, we show that the estimation error can be decreased by including more fake features in the model, even to the point where the model is overparametrized, i.e., the model contains more unknowns than observations.

eess.SP↗

arXiv2022

Risk assessment and optimal allocation of security measures under stealthy false data injection attacks

This paper firstly addresses the problem of risk assessment under false data injection attacks on uncertain control systems. We consider an adversary with complete system knowledge, injecting stealthy false data into an uncertain control system. We then use the Value-at-Risk to characterize the risk associated with the attack impact caused by the adversary. The worst-case attack impact is characterized by the recently proposed output-to-output gain. We observe that the risk assessment problem corresponds to an infinite non-convex robust optimization problem. To this end, we use dissipative system theory and the scenario approach to approximate the risk-assessment problem into a convex problem and also provide probabilistic certificates on approximation. Secondly, we consider the problem of security measure allocation. We consider an operator with a constraint on the security budget. Under this constraint, we propose an algorithm to optimally allocate the security measures using the calculated risk such that the resulting Value-at-risk is minimized. Finally, we illustrate the results through a numerical example. The numerical example also illustrates that the security allocation using the Value-at-risk, and the impact on the nominal system may have different outcomes: thereby depicting the benefit of using risk metrics.

eess.SY↗

arXiv2022

Risk Assessment of Stealthy Attacks on Uncertain Control Systems

In this article, we address the problem of risk assessment of stealthy attacks on uncertain control systems. Considering data injection attacks that aim at maximizing impact while remaining undetected, we use the recently proposed output-to-output gain to characterize the risk associated with the impact of attacks under a limited system knowledge attacker. The risk is formulated using a well-established risk metric, namely the maximum expected loss. Under this setups, the risk assessment problem corresponds to an untractable infinite non-convex optimization problem. To address this limitation, we adopt the framework of scenario-based optimization to approximate the infinite non-convex optimization problem by a sampled non-convex optimization problem. Then, based on the framework of dissipative system theory and S-procedure, the sampled non-convex risk assessment problem is formulated as an equivalent convex semi-definite program. Additionally, we derive the necessary and sufficient conditions for the risk to be bounded. Finally, we illustrate the results through numerical simulation of a hydro-turbine power system.

math.OC↗

arXiv2021

Linear Regression with Distributed Learning: A Generalization Error Perspective

Distributed learning provides an attractive framework for scaling the learning task by sharing the computational load over multiple nodes in a network. Here, we investigate the performance of distributed learning for large-scale linear regression where the model parameters, i.e., the unknowns, are distributed over the network. We adopt a statistical learning approach. In contrast to works that focus on the performance on the training data, we focus on the generalization error, i.e., the performance on unseen data. We provide high-probability bounds on the generalization error for both isotropic and correlated Gaussian data as well as sub-gaussian data. These results reveal the dependence of the generalization performance on the partitioning of the model over the network. In particular, our results show that the generalization error of the distributed solution can be substantially higher than that of the centralized solution even when the error on the training data is at the same level for both the centralized and distributed approaches. Our numerical results illustrate the performance with both real-world image data as well as synthetic data.

stat.ML↗

arXiv2020

Generalization Error for Linear Regression under Distributed Learning

Distributed learning facilitates the scaling-up of data processing by distributing the computational burden over several nodes. Despite the vast interest in distributed learning, generalization performance of such approaches is not well understood. We address this gap by focusing on a linear regression setting. We consider the setting where the unknowns are distributed over a network of nodes. We present an analytical characterization of the dependence of the generalization error on the partitioning of the unknowns over nodes. In particular, for the overparameterized case, our results show that while the error on training data remains in the same range as that of the centralized solution, the generalization error of the distributed solution increases dramatically compared to that of the centralized solution when the number of unknowns estimated at any node is close to the number of observations. We further provide numerical examples to verify our analytical expressions.

stat.ML↗

arXiv2016

Disturbance scaling in bidirectional vehicle platoons with different asymmetry in position and velocity coupling

This paper considers a string of vehicles where the local control law uses the states of the vehicle's immediate predecessor and follower. The coupling towards the preceding vehicle can be chosen different to the coupling towards the following vehicle, which is often referred to as an asymmetric bidirectional string. Further, the asymmetry for the velocity coupling can be chosen differently to the asymmetry in the position coupling. It is investigated how the effect of disturbance on the control errors in the string depends on the string length. It is shown, that in case of symmetric position coupling and asymmetric velocity coupling, linear scaling can be achieved. For symmetric interaction in both states, i.e., in symmetric bidirectional strings, the errors scale quadratically in the number of vehicles. When the coupling in position is asymmetric, exponential scaling may occur or the system might even become unstable. The paper thus gives a comprehensive overview of the achievable performance in linear, asymmetric, bidirectional platoons. The results reveal that symmetry in the position coupling and asymmetry in velocity coupling qualitatively improves the performance of the string. Extensive numerical results illustrate the theoretical findings.

eess.SY↗