arXiv subjects

Ana Bušić

Publications and source records attributed to Ana Bušić.

At least 37 records · Page 2Linked to original sources

arXiv2020

Zap Q-Learning With Nonlinear Function Approximation

Zap Q-learning is a recent class of reinforcement learning algorithms, motivated primarily as a means to accelerate convergence. Stability theory has been absent outside of two restrictive classes: the tabular setting, and optimal stopping. This paper introduces a new framework for analysis of a more general class of recursive algorithms known as stochastic approximation. Based on this general theory, it is shown that Zap Q-learning is consistent under a non-degeneracy assumption, even when the function approximation architecture is nonlinear. Zap Q-learning with neural network function approximation emerges as a special case, and is tested on examples from OpenAI Gym. Based on multiple experiments with a range of neural network sizes, it is found that the new algorithms converge quickly and are robust to choice of function approximation architecture.

cs.LG↗

arXiv2020

Towards Phase Balancing using Energy Storage

Ad-hoc growth of single-phase-connected distributed energy resources, such as solar generation and electric vehicles, can lead to network unbalance with negative consequences on the quality and efficiency of electricity supply. Case-studies are presented for a substation in Madeira, Portugal and an EV charging facility in Pasadena, California. These case studies show that phase imbalance can happen due to a large amount of distributed generation (DG) and electric vehicle (EV) integration. We conducted stylized load-flow analysis on a radial distribution network using an openDSS-based simulator to understand such negative effects of phase imbalance on neutral and phase conductor losses, and in voltage drop/rise. We evaluate the integration of storage in the distribution network as a possible solution for mitigating effects caused by imbalance. We present control architectures of storage operation for phase balancing. Numerically we show that relatively small-sized storage (compared to unbalance magnitude) can significantly reduce network imbalance. We identify the end node of the feeder as the best location to install storage.

eess.SY↗

arXiv2020

Energy storage applications for low voltage consumers in Uruguay

Energy storage can be used for many applications in the Smart Grid such as energy arbitrage, peak demand shaving, power factor correction, energy backup to name a few, and can play a major role at increasing the capacity of power networks to host renewable energy sources. Often, storage control algorithms will need to be \textit{tailored} according to power networks billing structure, reliability restrictions, and other local power networks norms. In this paper we explore residential energy storage applications in Uruguay, one of the global leaders in renewable energies, where new low-voltage consumer contracts were recently introduced. Based on these billing mechanisms, we focus on energy arbitrage and reactive energy compensation with the aim of minimizing the cost of consumption of an end-user. Given that in the new contacts the buying and selling price of electricity are equal and that reactive power compensation is primarily governed by the installed converter, the storage operation is not sensitive to parameter uncertainties and, therefore, no lookahead is required for decision making. A threshold-based \textit{hierarchical} controller is proposed which decides on the optimal active energy for arbitrage and uses the remaining converter capacity for reactive power compensation, which is shown to increase end-user profit. Numerical results indicate that storage could be profitable, even considering battery degradation, under some but not all of the studied contracts. For the cases in which it is not, we propose the best-suited contract. Results presented here can be naturally applied whenever the tariff structure satisfies the hypothesis considered in this work.

eess.SY↗

arXiv2020

Explicit Mean-Square Error Bounds for Monte-Carlo and Linear Stochastic Approximation

This paper concerns error bounds for recursive equations subject to Markovian disturbances. Motivating examples abound within the fields of Markov chain Monte Carlo (MCMC) and Reinforcement Learning (RL), and many of these algorithms can be interpreted as special cases of stochastic approximation (SA). It is argued that it is not possible in general to obtain a Hoeffding bound on the error sequence, even when the underlying Markov chain is reversible and geometrically ergodic, such as the M/M/1 queue. This is motivation for the focus on mean square error bounds for parameter estimates. It is shown that mean square error achieves the optimal rate of $O(1/n)$, subject to conditions on the step-size sequence. Moreover, the exact constants in the rate are obtained, which is of great value in algorithm design.

math.PR↗

arXiv2020

Storage Optimal Control under Net Metering Policies

Electricity prices and the end user net load vary with time. Electricity consumers equipped with energy storage devices can perform energy arbitrage, i.e., buy when energy is cheap or when there is a deficit of energy, and sell it when it is expensive or in excess, taking into account future variations in price and net load. Net metering policies indicate that many of the utilities apply a {customer selling} rate lower than or equal to the retail {customer buying rate} in order to compensate excess energy generated by end users. In this paper, we formulate the optimal control problem for an end user energy storage device in presence of net metering. We propose a computationally efficient algorithm, with worst case run time complexity of quadratic in terms of number of samples in lookahead horizon, that computes the optimal energy ramping rates in a time horizon. The proposed algorithm exploits the problem's piecewise linear structure and convexity properties for the \textit{discretization} of optimal Lagrange multipliers. The solution has a \textit{threshold-based structure} in which optimal control decisions are independent of past or future price as well as of net load values beyond a certain time horizon, defined as a \textit{sub-horizon}. Numerical results show the effectiveness of the proposed model and algorithm. Furthermore, we investigate the impact of forecasting errors on the proposed technique. We consider an Auto-Regressive Moving Average (ARMA) based forecasting of net load together with the Model Predictive Control (MPC). We numerically show that adaptive forecasting and MPC significantly mitigate the effects of forecast error on energy arbitrage gains.

eess.SY↗

arXiv2019

Sizing and Profitability of Energy Storage for Prosumers in Madeira, Portugal

This paper proposes a framework to select the best-suited battery for co-optimizing for peak demand shaving, energy arbitrage and increase self-sufficiency in the context of power network in Madeira, Portugal. Feed-in-tariff for electricity network in Madeira is zero, which implies consumers with excess production should locally consume the excess generation rather than wasting it. Further, the power network {operator} applies a peak power contract for consumers which imposes an upper bound on the peak power seen by the power grid interfaced by energy meter. We investigate the value of storage in Madeira, using four different types of prosumers, categorized based on the relationship between their inelastic load and renewable generation. We observe that the marginal increase in the value of storage deteriorates with increase in size and ramping capabilities. We propose the use of profit per cycle per unit of battery capacity and expected payback period as indices for selecting the best-suited storage parameters to ensure profitability. This mechanism takes into account the consumption and generation patterns, profit, storage degradation, and cycle and calendar life of the battery. We also propose the inclusion of a friction coefficient in the original co-optimization formulation to increase the value of storage by reducing the operational cycles and eliminate low returning transactions.

eess.SY↗

arXiv2019

Demand Dispatch with Heterogeneous Intelligent Loads

A distributed control architecture is presented that is intended to make a collection of heterogeneous loads appear to the grid operator as a nearly perfect battery. Local control is based on randomized decision rules advocated in prior research, and extended in this paper to any load with a discrete number of power states. Additional linear filtering at the load ensures that the input-output dynamics of the aggregate has a nearly flat input-output response: the behavior of an ideal, multi-GW battery system.

eess.SY↗

arXiv2019

Zap Q-Learning for Optimal Stopping Time Problems

The objective in this paper is to obtain fast converging reinforcement learning algorithms to approximate solutions to the problem of discounted cost optimal stopping in an irreducible, uniformly ergodic Markov chain, evolving on a compact subset of $\mathbb{R}^n$. We build on the dynamic programming approach taken by Tsitsikilis and Van Roy, wherein they propose a Q-learning algorithm to estimate the optimal state-action value function, which then defines an optimal stopping rule. We provide insights as to why the convergence rate of this algorithm can be slow, and propose a fast-converging alternative, the "Zap-Q-learning" algorithm, designed to achieve optimal rate of convergence. For the first time, we prove the convergence of the Zap-Q-learning algorithm under the assumption of linear function approximation setting. We use ODE analysis for the proof, and the optimal asymptotic variance property of the algorithm is reflected via fast convergence in a finance example.

eess.SY↗

arXiv2019

Energy Storage in Madeira, Portugal: Co-optimizing for Arbitrage, Self-Sufficiency, Peak Shaving and Energy Backup

Energy storage applications are explored from a prosumer (consumers with generation) perspective for the island of Madeira in Portugal. These applications could also be relevant to other power networks. We formulate a convex co-optimization problem for performing arbitrage under zero feed-in tariff, increasing self-sufficiency by increasing self-consumption of locally generated renewable energy, provide peak shaving and act as a backup power source during anticipated and scheduled power outages. Using real data from Madeira we perform short and long time-scale simulations in order to select end-user contract which maximizes their gains considering storage degradation based on operational cycles. We observe energy storage ramping capability decides peak shaving potential, fast ramping batteries can significantly reduce peak demand charge. The numerical experiment indicates that storage providing backup does not significantly reduce gains performing arbitrage and peak demand shaving. Furthermore, we also use AutoRegressive Moving Average (ARMA) forecasting along with Model Predictive Control (MPC) for real-time implementation of the proposed optimization problem in the presence of uncertainty.

eess.SY↗

arXiv2019

Optimal Storage Arbitrage under Net Metering using Linear Programming

We formulate the optimal energy arbitrage problem for a piecewise linear cost function for energy storage devices using linear programming (LP). The LP formulation is based on the equivalent minimization of the epigraph. This formulation considers ramping and capacity constraints, charging and discharging efficiency losses of the storage, inelastic consumer load and local renewable generation in presence of net-metering which facilitates selling of energy to the grid and incentivizes consumers to install renewable generation and energy storage. We consider the case where the consumer loads, electricity prices, and renewable generations at different instances are uncertain. These uncertain quantities are predicted using an Auto-Regressive Moving Average (ARMA) model and used in a model predictive control (MPC) framework to obtain the arbitrage decision at each instance. In numerical results we present the sensitivity analysis of storage performing arbitrage with varying ramping batteries and different ratio of selling and buying price of electricity.

eess.SY↗

arXiv2019

Optimal Matrix Momentum Stochastic Approximation and Applications to Q-learning

Acceleration is an increasingly common theme in the stochastic optimization literature. The two most common examples are Nesterov's method, and Polyak's momentum technique. In this paper two new algorithms are introduced for root finding problems: 1) PolSA is a root finding algorithm with specially designed matrix momentum, and 2) NeSA can be regarded as a variant of Nesterov's algorithm, or a simplification of PolSA. The PolSA algorithm is new even in the context of optimization (when cast as a root finding problem). The research surveyed in this paper is motivated by applications to reinforcement learning. It is well known that most variants of TD- and Q-learning may be cast as SA (stochastic approximation) algorithms, and the tools from general SA theory can be used to investigate convergence and bounds on convergence rate. In particular, the asymptotic variance is a common metric of performance for SA algorithms, and is also one among many metrics used in assessing the performance of stochastic optimization algorithms. There are two well known SA techniques that are known to have optimal asymptotic variance: the Ruppert-Polyak averaging technique, and stochastic Newton-Raphson (SNR). The former algorithm can have extremely bad transient performance, and the latter can be computationally expensive. It is demonstrated here that parameter estimates from the new PolSA algorithm couple with those of the ideal (but more complex) SNR algorithm. The new algorithm is thus a third approach to obtain optimal asymptotic covariance. These strong results require assumptions on the model. A linearized model is considered, and the noise is assumed to be a martingale difference sequence. Numerical results are obtained in a non-linear setting that is the motivation for this work: In PolSA implementations of Q-learning it is observed that coupling occurs with SNR in this non-ideal setting.

math.OC↗

arXiv2018

Action-Constrained Markov Decision Processes With Kullback-Leibler Cost

This paper concerns computation of optimal policies in which the one-step reward function contains a cost term that models Kullback-Leibler divergence with respect to nominal dynamics. This technique was introduced by Todorov in 2007, where it was shown under general conditions that the solution to the average-reward optimality equations reduce to a simple eigenvector problem. Since then many authors have sought to apply this technique to control problems and models of bounded rationality in economics. A crucial assumption is that the input process is essentially unconstrained. For example, if the nominal dynamics include randomness from nature (e.g., the impact of wind on a moving vehicle), then the optimal control solution does not respect the exogenous nature of this disturbance. This paper introduces a technique to solve a more general class of action-constrained MDPs. The main idea is to solve an entire parameterized family of MDPs, in which the parameter is a scalar weighting the one-step reward function. The approach is new and practical even in the original unconstrained formulation.

math.OC↗

arXiv2016

Ordinary Differential Equation Methods For Markov Decision Processes and Application to Kullback-Leibler Control Cost

A new approach to computation of optimal policies for MDP (Markov decision process) models is introduced. The main idea is to solve not one, but an entire family of MDPs, parameterized by a weighting factor $ζ$ that appears in the one-step reward function. For an MDP with $d$ states, the family of value functions $\{ h^*_ζ: ζ\in\Re\}$ is the solution to an ODE, $$ \frac{d}{dζ} h^*_ζ= {\cal V}(h^*_ζ) $$ where the vector field ${\cal V}\colon\Re^d\to\Re^d$ has a simple form, based on a matrix inverse. This general methodology is applied to a family of average-cost optimal control models in which the one-step reward function is defined by Kullback-Leibler divergence. The motivation for this reward function in prior work is computation: The solution to the MDP can be expressed in terms of the Perron-Frobenius eigenvector for an associated positive matrix. The drawback with this approach is that no hard constraints on the control are permitted. It is shown here that it is possible to extend this framework to model randomness from nature that cannot be modified by the controller. Perron-Frobenius theory is no longer applicable -- the resulting dynamic programming equations appear as complex as a completely unstructured MDP model. Despite this apparent complexity, it is shown that this class of MDPs admits a solution via this new ODE technique. This approach is new and practical even for the simpler problem in which randomness from nature is absent.

math.OC↗

arXiv2016

Estimation and Control of Quality of Service in Demand Dispatch

It is now well known that flexibility of energy consumption can be harnessed for the purposes of grid-level ancillary services. In particular, through distributed control of a collection of loads, a balancing authority regulation signal can be tracked accurately, while ensuring that the quality of service (QoS) for each load is acceptable {\it on average}. In this paper it is argued that a histogram of QoS is approximately Gaussian, and consequently each load will eventually receive poor service. Statistical techniques are developed to estimate the mean and variance of QoS as a function of the power spectral density of the regulation signal. It is also shown that additional local control can eliminate risk: The histogram of QoS is {\it truncated} through this local control, so that strict bounds on service quality are guaranteed. While there is a tradeoff between the grid-level tracking performance (capacity and accuracy) and the bounds imposed on QoS, it is found that the loss of capacity is minor in typical cases.

eess.SY↗

arXiv2016

Approximate optimality with bounded regret in dynamic matching models

We consider a discrete-time bipartite matching model with random arrivals of units of supply and demand that can wait in queues located at the nodes in the network. A control policy determines which are matched at each time. The focus is on the infinite-horizon average-cost optimal control problem. A relaxation of the stochastic control problem is proposed, which is found to be a special case of an inventory model, as treated in the classical theory of Clark and Scarf. The optimal policy for the relaxation admits a closed-form expression. Based on the policy for this relaxation, a new matching policy is proposed. For a parameterized family of models in which the network load approaches capacity, this policy is shown to be approximately optimal, with bounded regret, even though the average cost grows without bound.

cs.DM↗

arXiv2016

Ergodic Theory for Controlled Markov Chains with Stationary Inputs

Consider a stochastic process $\{X(t)\}$ on a finite state space $ {\sf X}=\{1,\dots, d\}$. It is conditionally Markov, given a real-valued `input process' $\{ζ(t)\}$. This is assumed to be small, which is modeled through the scaling, \[ ζ_t = \varepsilon ζ^1_t, \qquad 0\le \varepsilon \le 1\,, \] where $\{ζ^1(t)\}$ is a bounded stationary process. The following conclusions are obtained, subject to smoothness assumptions on the controlled transition matrix and a mixing condition on $\{ζ(t)\}$: (i) A stationary version of the process is constructed, that is coupled with a stationary version of the Markov chain $\{X^\bullet$(t)\}obtained with $\{ζ(t)\}\equiv 0$. The triple $(\{X(t)\}, \{X^\bullet(t)\},\{ζ(t)\})$ is a jointly stationary process satisfying \[ {\sf P}\{X(t) \neq X^\bullet(t)\} = O(\varepsilon) \] Moreover, a second-order Taylor-series approximation is obtained: \[ {\sf P}\{X(t) =i \} ={\sf P}\{X^\bullet(t) =i \} + \varepsilon^2 \varrho(i) + o(\varepsilon^2),\quad 1\le i\le d, \] with an explicit formula for the vector $\varrho\in\mathbb{R}^d$. (ii) For any $m\ge 1$ and any function $f\colon \{1,\dots,d\}\times \mathbb{R}\to\mathbb{R}^m$, the stationary stochastic process $Y(t) = f(X(t),ζ(t))$ has a power spectral density $\text{S}_f$ that admits a second order Taylor series expansion: A function $\text{S}^{(2)}_f\colon [-π,π] \to \mathbb{C}^{ m\times m}$ is constructed such that \[ \text{S}_f(θ) = \text{S}^\bullet_f(θ) + \varepsilon^2 \text{S}_f^{(2)}(θ) + o(\varepsilon^2),\quad θ\in [-π,π] . \] An explicit formula for the function $\text{S}_f^{(2)}$ is obtained, based in part on the bounds in (i). The results are illustrated using a version of the timing channel of Anantharam and Verdu.

cs.PF↗

arXiv2016

State Estimation for the Individual and the Population in Mean Field Control with Application to Demand Dispatch

This paper concerns state estimation problems in a mean field control setting. In a finite population model, the goal is to estimate the joint distribution of the population state and the state of a typical individual. The observation equations are a noisy measurement of the population. The general results are applied to demand dispatch for regulation of the power grid, based on randomized local control algorithms. In prior work by the authors it has been shown that local control can be carefully designed so that the aggregate of loads behaves as a controllable resource with accuracy matching or exceeding traditional sources of frequency regulation. The operational cost is nearly zero in many cases. The information exchange between grid and load is minimal, but it is assumed in the overall control architecture that the aggregate power consumption of loads is available to the grid operator. It is shown that the Kalman filter can be constructed to reduce these communication requirements,

eess.SY↗

arXiv2016

Distributed Randomized Control for Demand Dispatch

The paper concerns design of control systems for Demand Dispatch to obtain ancillary services to the power grid by harnessing inherent flexibility in many loads. The role of "local intelligence" at the load has been advocated in prior work, randomized local controllers that manifest this intelligence are convenient for loads with a finite number of states. The present work introduces two new design techniques for these randomized controllers: (i) The Individual Perspective Design (IPD) is based on the solution to a one-dimensional family of Markov Decision Processes, whose objective function is formulated from the point of view of a single load. The family of dynamic programming equation appears complex, but it is shown that it is obtained through the solution of a single ordinary differential equation. (ii) The System Perspective Design (SPD) is motivated by a single objective of the grid operator: Passivity of any linearization of the aggregate input-output model. A solution is obtained that can again be computed through the solution of a single ordinary differential equation. Numerical results complement these theoretical results.

math.OC↗