arXiv subjects

Damien Ernst

Publications and source records attributed to Damien Ernst.

At least 55 records · Page 3Linked to original sources

arXiv2023

Distributional Reinforcement Learning with Unconstrained Monotonic Neural Networks

The distributional reinforcement learning (RL) approach advocates for representing the complete probability distribution of the random return instead of only modelling its expectation. A distributional RL algorithm may be characterised by two main components, namely the representation of the distribution together with its parameterisation and the probability metric defining the loss. The present research work considers the unconstrained monotonic neural network (UMNN) architecture, a universal approximator of continuous monotonic functions which is particularly well suited for modelling different representations of a distribution. This property enables the efficient decoupling of the effect of the function approximator class from that of the probability metric. The research paper firstly introduces a methodology for learning different representations of the random return distribution (PDF, CDF and QF). Secondly, a novel distributional RL algorithm named unconstrained monotonic deep Q-network (UMDQN) is presented. To the authors' knowledge, it is the first distributional RL method supporting the learning of three, valid and continuous representations of the random return distribution. Lastly, in light of this new algorithm, an empirical comparison is performed between three probability quasi-metrics, namely the Kullback-Leibler divergence, Cramer distance, and Wasserstein distance. The results highlight the main strengths and weaknesses associated with each probability metric together with an important limitation of the Wasserstein distance.

cs.LG↗

arXiv2023

Multi-objective near-optimal necessary conditions for multi-sectoral planning

This paper extends the concepts of epsilon-optimal spaces and necessary conditions for near-optimality from single-objective to multi-objective optimisation. These notions are first presented for single-objective optimisation, and the mathematical formulation is adapted to address the multi-objective framework. Afterwards, we illustrate the newly developed methodology by conducting multi-sectoral planning of the Belgian energy system with an open-source model called EnergyScope TD. The cost and energy invested in the system are used as objectives. Optimal and efficient solutions for these two objectives are computed and analysed. These results are then used to obtain necessary conditions corresponding to the minimum amount of energy from different sets of resources, including endogenous and exogenous resources. This case study highlights the high dependence of Belgium on imported energy while demonstrating that no individual resource is essential on its own.

math.OC↗

arXiv2023

Matching of Everyday Power Supply and Demand with Dynamic Pricing: Problem Formalisation and Conceptual Analysis

The energy transition is expected to significantly increase the share of renewable energy sources whose production is intermittent in the electricity mix. Apart from key benefits, this development has the major drawback of generating a mismatch between power supply and demand. The innovative dynamic pricing approach may significantly contribute to mitigating that critical problem by taking advantage of the flexibility offered by the demand side. At its core, this idea consists in providing the consumer with a price signal which is evolving over time, in order to influence its consumption. This novel approach involves a challenging decision-making problem that can be summarised as follows: how to determine a price signal maximising the synchronisation between power supply and demand under the constraints of maintaining the producer/retailer's profitability and benefiting the final consumer at the same time? As a contribution, this research work presents a detailed formalisation of this particular decision-making problem. Moreover, the paper discusses the diverse algorithmic components required to efficiently design a dynamic pricing policy: different forecasting models together with an accurate statistical modelling of the demand response to dynamic prices.

q-fin.TR↗

arXiv2022

Risk-Sensitive Policy with Distributional Reinforcement Learning

Classical reinforcement learning (RL) techniques are generally concerned with the design of decision-making policies driven by the maximisation of the expected outcome. Nevertheless, this approach does not take into consideration the potential risk associated with the actions taken, which may be critical in certain applications. To address that issue, the present research work introduces a novel methodology based on distributional RL to derive sequential decision-making policies that are sensitive to the risk, the latter being modelled by the tail of the return probability distribution. The core idea is to replace the $Q$ function generally standing at the core of learning schemes in RL by another function taking into account both the expected return and the risk. Named the risk-based utility function $U$, it can be extracted from the random return distribution $Z$ naturally learnt by any distributional RL algorithm. This enables to span the complete potential trade-off between risk minimisation and expected return maximisation, in contrast to fully risk-averse methodologies. Fundamentally, this research yields a truly practical and accessible solution for learning risk-sensitive policies with minimal modification to the distributional RL algorithm, and with an emphasis on the interpretability of the resulting decision-making process.

cs.LG↗

arXiv2022

Value-based CTDE Methods in Symmetric Two-team Markov Game: from Cooperation to Team Competition

In this paper, we identify the best learning scenario to train a team of agents to compete against multiple possible strategies of opposing teams. We evaluate cooperative value-based methods in a mixed cooperative-competitive environment. We restrict ourselves to the case of a symmetric, partially observable, two-team Markov game. We selected three training methods based on the centralised training and decentralised execution (CTDE) paradigm: QMIX, MAVEN and QVMix. For each method, we considered three learning scenarios differentiated by the variety of team policies encountered during training. For our experiments, we modified the StarCraft Multi-Agent Challenge environment to create competitive environments where both teams could learn and compete simultaneously. Our results suggest that training against multiple evolving strategies achieves the best results when, for scoring their performances, teams are faced with several strategies.

cs.LG↗

arXiv2022

Recurrent networks, hidden states and beliefs in partially observable environments

Reinforcement learning aims to learn optimal policies from interaction with environments whose dynamics are unknown. Many methods rely on the approximation of a value function to derive near-optimal policies. In partially observable environments, these functions depend on the complete sequence of observations and past actions, called the history. In this work, we show empirically that recurrent neural networks trained to approximate such value functions internally filter the posterior probability distribution of the current state given the history, called the belief. More precisely, we show that, as a recurrent neural network learns the Q-function, its hidden states become more and more correlated with the beliefs of state variables that are relevant to optimal control. This correlation is measured through their mutual information. In addition, we show that the expected return of an agent increases with the ability of its recurrent architecture to reach a high mutual information between its hidden states and the beliefs. Finally, we show that the mutual information between the hidden states and the beliefs of variables that are irrelevant for optimal control decreases through the learning process. In summary, this work shows that in its hidden states, a recurrent neural network approximating the Q-function of a partially observable environment reproduces a sufficient statistic from the history that is correlated to the relevant part of the belief for taking optimal actions.

cs.LG↗

arXiv2022

Optimal Connection Phase Selection of Residential Distributed Energy Resources and its Impact on Aggregated Demand

The recent major increase in decentralized energy resources (DERs) such as photovoltaic (PV) panels alters the loading profile of distribution systems (DS) and impacts higher voltage levels. Distribution system operators (DSOs) try to manage the deployment of new DERs to decrease the operational costs. However, DER location and size are factors beyond any DSO's reach. This paper presents a practical method to minimize the DS operational costs due to new DER deployments, through optimal selection of their connection phase. The impact of such distribution grid management efforts on aggregated demand for higher voltage levels is also evaluated and discussed in this paper. Simulation results on a real-life Belgian network show the effectiveness of optimal connection phase selection in decreasing DS operational costs, and the considerable impact of such simple DS management efforts on the aggregated demand.

cs.CE↗

arXiv2022

M4Depth: Monocular depth estimation for autonomous vehicles in unseen environments

Estimating the distance to objects is crucial for autonomous vehicles when using depth sensors is not possible. In this case, the distance has to be estimated from on-board mounted RGB cameras, which is a complex task especially in environments such as natural outdoor landscapes. In this paper, we present a new method named M4Depth for depth estimation. First, we establish a bijective relationship between depth and the visual disparity of two consecutive frames and show how to exploit it to perform motion-invariant pixel-wise depth estimation. Then, we detail M4Depth which is based on a pyramidal convolutional neural network architecture where each level refines an input disparity map estimate by using two customized cost volumes. We use these cost volumes to leverage the visual spatio-temporal constraints imposed by motion and to make the network robust for varied scenes. We benchmarked our approach both in test and generalization modes on public datasets featuring synthetic camera trajectories recorded in a wide variety of outdoor scenes. Results show that our network outperforms the state of the art on these datasets, while also performing well on a standard depth estimation benchmark. The code of our method is publicly available at https://github.com/michael-fonder/M4Depth.

cs.CV↗

arXiv2022

Computing Necessary Conditions for Near-Optimality in Capacity Expansion Planning Problems

In power systems, large-scale optimisation problems are extensively used to plan for capacity expansion at the supra-national level. However, their cost-optimal solutions are often not exploitable by decision-makers who are preferably looking for features of solutions that can accommodate their different requirements. This paper proposes a generic framework for addressing this problem. It is based on the concept of the epsilon-optimal feasible space of a given optimisation problem and the identification of necessary conditions over this space. This framework has been developed in a generic case, and an approach for solving this problem is subsequently described for a specific case where conditions are constrained sums of variables. The approach is tested on a case study about capacity expansion planning of the European electricity network to determine necessary conditions on the minimal investments in transmission, storage and generation capacity.

math.OC↗

arXiv2022

Allocation of locally generated electricity in renewable energy communities

Local electricity markets represent a way of supplementing traditional retailing contracts for end consumers -- among these markets, the renewable energy community has gained momentum over the last few years. This paper proposes a practical and readily to be adopted modelling solution for these communities, one that allows their members to share the economic benefits derived from them. The proposed solution relies on an \emph{ex-post} allocation of the electricity that is generated within energy communities (i.e., local electricity) based on the optimisation of \emph{repartition keys}. Repartition keys are therefore optimally computed to represent the proportion of total local electricity to be allocated to each community member, and aim to minimise the sum of electricity bills of all community members. Since the optimisation takes place \emph{ex-post} the repartition keys do not modify the actual electricity flows, but rather the financial flows of the community members. Then, the billing process of the community will take these keys into account to correctly send the electricity bills to each member. Building on this concept, we also introduce two additions to the basic algorithm to enhance the stability of the community, which a global bill minimisation may fail to ensure (e.g., very asymmetrical solutions between members may lead to some of them opting out).

eess.SY↗

arXiv2022

Churn prediction in online gambling

In business retention, churn prevention has always been a major concern. This work contributes to this domain by formalizing the problem of churn prediction in the context of online gambling as a binary classification task. We also propose an algorithmic answer to this problem based on recurrent neural network. This algorithm is tested with online gambling data that have the form of time series, which can be efficiently processed by recurrent neural networks. To evaluate the performances of the trained models, standard machine learning metrics were used, such as accuracy, precision and recall. For this problem in particular, the conducted experiments allowed to assess that the choice of a specific architecture depends on the metric which is given the greatest importance. Architectures using nBRC favour precision, those using LSTM give better recall, while GRU-based architectures allow a higher accuracy and balance two other metrics. Moreover, further experiments showed that using only the more recent time-series histories to train the networks decreases the quality of the results. We also study the performances of models learned at a specific instant $t$, at other times $t^{\prime} > t$. The results show that the performances of the models learned at time $t$ remain good at the following instants $t^{\prime} > t$, suggesting that there is no need to refresh the models at a high rate. However, the performances of the models were subject to noticeable variance due to one-off events impacting the data.

cs.LG↗

arXiv2022

Jointly Learning Environments and Control Policies with Projected Stochastic Gradient Ascent

We consider the joint design and control of discrete-time stochastic dynamical systems over a finite time horizon. We formulate the problem as a multi-step optimization problem under uncertainty seeking to identify a system design and a control policy that jointly maximize the expected sum of rewards collected over the time horizon considered. The transition function, the reward function and the policy are all parametrized, assumed known and differentiable with respect to their parameters. We then introduce a deep reinforcement learning algorithm combining policy gradient methods with model-based optimization techniques to solve this problem. In essence, our algorithm iteratively approximates the gradient of the expected return via Monte-Carlo sampling and automatic differentiation and takes projected gradient ascent steps in the space of environment and policy parameters. This algorithm is referred to as Direct Environment and Policy Search (DEPS). We assess the performance of our algorithm in three environments concerned with the design and control of a mass-spring-damper system, a small-scale off-grid power system and a drone, respectively. In addition, our algorithm is benchmarked against a state-of-the-art deep reinforcement learning algorithm used to tackle joint design and control problems. We show that DEPS performs at least as well or better in all three environments, consistently yielding solutions with higher returns in fewer iterations. Finally, solutions produced by our algorithm are also compared with solutions produced by an algorithm that does not jointly optimize environment and policy parameters, highlighting the fact that higher returns can be achieved when joint optimization is performed.

cs.LG↗

arXiv2021

Gym-ANM: Reinforcement Learning Environments for Active Network Management Tasks in Electricity Distribution Systems

Active network management (ANM) of electricity distribution networks include many complex stochastic sequential optimization problems. These problems need to be solved for integrating renewable energies and distributed storage into future electrical grids. In this work, we introduce Gym-ANM, a framework for designing reinforcement learning (RL) environments that model ANM tasks in electricity distribution networks. These environments provide new playgrounds for RL research in the management of electricity networks that do not require an extensive knowledge of the underlying dynamics of such systems. Along with this work, we are releasing an implementation of an introductory toy-environment, ANM6-Easy, designed to emphasize common challenges in ANM. We also show that state-of-the-art RL algorithms can already achieve good performance on ANM6-Easy when compared against a model predictive control (MPC) approach. Finally, we provide guidelines to create new Gym-ANM environments differing in terms of (a) the distribution network topology and parameters, (b) the observation space, (c) the modelling of the stochastic processes present in the system, and (d) a set of hyperparameters influencing the reward signal. Gym-ANM can be downloaded at https://github.com/robinhenry/gym-anm.

cs.LG↗

arXiv2021

Remote Renewable Hubs For Carbon-Neutral Synthetic Fuel Production

This paper studies the economics of carbon-neutral synthetic fuel production from renewable electricity in remote areas where high-quality renewable resources are abundant. To this end, a graph-based optimisation modelling framework directly applicable to the strategic planning of remote renewable energy supply chains is proposed. More precisely, a hypergraph abstraction of planning problems is introduced, wherein nodes can be viewed as optimisation subproblems with their own parameters, variables, constraints and local objective. Nodes typically represent a subsystem such as a technology, a plant or a process. Hyperedges, on the other hand, express the connectivity between subsystems. The framework is leveraged to study the economics of carbon-neutral synthetic methane production from solar and wind energy in North Africa and its delivery to Northwestern European markets. The full supply chain is modelled in an integrated fashion, which makes it possible to accurately capture the interaction between various technologies on an hourly time scale. Results suggest that the cost of synthetic methane production and delivery would be slightly under 150 EUR/MWh (higher heating value) by 2030 for a system supplying 10 TWh annually and relying on a combination of solar photovoltaic and wind power plants, assuming a uniform weighted average cost of capital of 7%. A comprehensive sensitivity analysis is also carried out in order to assess the impact of various techno-economic parameters and assumptions on synthetic methane cost, including the availability of wind power plants, the investment costs of electrolysis, methanation and direct air capture plants, their operational flexibility, the energy consumption of direct air capture plants, and financing costs.

eess.SY↗

arXiv2021

Assessing the Impact of Offshore Wind Siting Strategies on the Design of the European Power System

This paper provides a detailed account of the impact of different offshore wind siting strategies on the design of the European power system. To this end, a two-stage method is proposed. In the first stage, a highly-granular siting problem identifies a suitable set of sites where offshore wind plants could be deployed according to a pre-specified criterion. Two siting schemes are analysed and compared within a realistic case study. These schemes essentially select a pre-specified number of sites so as to maximise their aggregate power output and their spatiotemporal complementarity, respectively. In addition, two variants of these siting schemes are provided, wherein the number of sites to be selected is specified on a country-by-country basis rather than Europe-wide. In the second stage, the subset of previously identified sites is passed to a capacity expansion planning (CEP) framework that sizes the power generation, transmission and storage assets that should be deployed and operated in order to satisfy pre-specified electricity demand levels at minimum cost. Results show that the complementarity-based siting criterion leads to system designs which are up to 5% cheaper than the ones relying the power output-based criterion when offshore wind plants are deployed with no consideration for country-based deployment targets. On the contrary, the power output-based scheme leads to system designs which are consistently 2% cheaper than the ones leveraging the complementarity-based siting strategy when such constraints are enforced. The robustness of the results is supported by a sensitivity analysis on offshore wind capital expenditure and inter-annual weather variability, respectively.

eess.SY↗

arXiv2021

Gym-ANM: Open-source software to leverage reinforcement learning for power system management in research and education

Gym-ANM is a Python package that facilitates the design of reinforcement learning (RL) environments that model active network management (ANM) tasks in electricity networks. Here, we describe how to implement new environments and how to write code to interact with pre-existing ones. We also provide an overview of ANM6-Easy, an environment designed to highlight common ANM challenges. Finally, we discuss the potential impact of Gym-ANM on the scientific community, both in terms of research and education. We hope this package will facilitate collaboration between the power system and RL communities in the search for algorithms to control future energy systems.

cs.LG↗

arXiv2021

Model Reduction in Capacity Expansion Planning Problems via Renewable Generation Site Selection

The accurate representation of variable renewable generation (RES, e.g., wind, solar PV) assets in capacity expansion planning (CEP) studies is paramount to capture spatial and temporal correlations that may exist between sites and impact both power system design and operation. However, it typically has a high computational cost. This paper proposes a method to reduce the spatial dimension of CEP problems while preserving an accurate representation of renewable energy sources. A two-stage approach is proposed to this end. In the first stage, relevant sites are identified via a screening routine that discards the locations with little impact on system design. In the second stage, the subset of relevant RES sites previously identified is used in a CEP problem to determine the optimal configuration of the power system. The proposed method is tested on a realistic EU case study and its performance is benchmarked against a CEP set-up in which the entire set of candidate RES sites is available. The method shows great promise, with the screening stage consistently identifying 90% of the optimal RES sites while discarding up to 54% of the total number of candidate locations. This leads to a peak memory reduction of up to 41% and solver runtime gains between 31% and 46%, depending on the weather year considered.

eess.SY↗

arXiv2021

Sparse Training Theory for Scalable and Efficient Agents

A fundamental task for artificial intelligence is learning. Deep Neural Networks have proven to cope perfectly with all learning paradigms, i.e. supervised, unsupervised, and reinforcement learning. Nevertheless, traditional deep learning approaches make use of cloud computing facilities and do not scale well to autonomous agents with low computational resources. Even in the cloud, they suffer from computational and memory limitations, and they cannot be used to model adequately large physical worlds for agents which assume networks with billions of neurons. These issues are addressed in the last few years by the emerging topic of sparse training, which trains sparse networks from scratch. This paper discusses sparse training state-of-the-art, its challenges and limitations while introducing a couple of new theoretical research directions which has the potential of alleviating sparse training limitations to push deep learning scalability well beyond its current boundaries. Nevertheless, the theoretical advancements impact in complex multi-agents settings is discussed from a real-world perspective, using the smart grid case study.

cs.AI↗