arXiv subjects

Damien Ernst

Publications and source records attributed to Damien Ernst.

At least 37 records · Page 2Linked to original sources

arXiv2025

Behind the Myth of Exploration in Policy Gradients

In order to compute near-optimal policies with policy-gradient algorithms, it is common in practice to include intrinsic exploration terms in the learning objective. Although the effectiveness of these terms is usually justified by an intrinsic need to explore environments, we propose a novel analysis with the lens of numerical optimization. Two criteria are introduced on the learning objective and two others on its stochastic gradient estimates, and are afterwards used to discuss the quality of the policy after optimization. The analysis sheds light on two separate effects of exploration techniques. First, they make it possible to smooth the learning objective and to eliminate local optima while preserving the global maximum. Second, they modify the gradient estimates, increasing the probability that the stochastic parameter updates eventually provide an optimal policy. We empirically illustrate these effects with exploration strategies based on entropy bonuses, identifying limitations and suggesting directions for future work.

cs.LG↗

arXiv2025

Remote Renewable Energy Hubs: a Taxonomy

Serving the energy demand with renewable energy is hindered by its limited availability near load centres (i.e. places where the energy demand is high). To address this challenge, the concept of Remote Renewable Energy Hubs (RREH) emerges as a promising solution. RREHs are energy hubs located in areas with abundant renewable energy sources, such as sun in the Sahara Desert or wind in Greenland. In these hubs, renewable energy sources are used to synthetise energy molecules. To produce specific energy molecules, a tailored hub configuration must be designed, which means choosing a set of technologies that are interacting with each other as well as defining how they are integrated in their local environment. The plurality of technologies that may be employed in RREHs results in a large diversity of hubs. In order to characterize this diversity, we propose in this paper a taxonomy for accurately defining these hubs. This taxonomy allows to better describe and compare designs of hubs as well as to identify new ones. Thus, it may guide policymakers and engineers in hub design, contributing to cost efficiency and/or improving local integration.

eess.SY↗

arXiv2025

Ammonia, Methane, Hydrogen and Methanol Produced in Remote Renewable Energy Hubs: a Comparative Quantitative Analysis

Remote renewable energy hubs (RREHs) for synthetic fuel production are engineering systems harvesting renewable energy where it is particularly abundant. They produce transportable synthetic fuels for export to distant load centers. This article aims to evaluate the production costs of different energy carriers, and includes a discussion on advantages and disadvantages in terms of technical performance. To do so, we extend the study of Berger et al., (2021) which focuses on methane (CH4) as energy carrier and introduce three new carriers: ammonia (NH3), hydrogen (H2) and methanol (CH3OH). The four different RREHs are located in the Algerian Sahara desert and must serve to the load center, Belgium, a constant electro-fuel demand of 10 TWh per year. The modelling and optimisation of these systems are performed using the modelling language GBOML (Graph-Based Optimisation Modelling Language). Our findings reveal that the three new RREHs, each with its respective carrier (ammonia, hydrogen, and methanol), are all more cost-effective than the methane-based system. Ammonia demonstrates the most favourable cost-to-energy exported ratio.

eess.SY↗

arXiv2025

Investigating the Monte-Carlo Tree Search Approach for the Job Shop Scheduling Problem

The Job Shop Scheduling Problem (JSSP) is a well-known optimization problem in manufacturing, where the goal is to determine the optimal sequence of jobs across different machines to minimize a given objective. In this work, we focus on minimising the weighted sum of job completion times. We explore the potential of Monte Carlo Tree Search (MCTS), a heuristic-based reinforcement learning technique, to solve large-scale JSSPs, especially those with recirculation. We propose several Markov Decision Process (MDP) formulations to model the JSSP for the MCTS algorithm. In addition, we introduce a new synthetic benchmark derived from real manufacturing data, which captures the complexity of large, non-rectangular instances often encountered in practice. Our experimental results show that MCTS effectively produces good-quality solutions for large-scale JSSP instances, outperforming our constraint programming approach.

cs.AI↗

arXiv2024

Cost Estimation in Unit Commitment Problems Using Simulation-Based Inference

The Unit Commitment (UC) problem is a key optimization task in power systems to forecast the generation schedules of power units over a finite time period by minimizing costs while meeting demand and technical constraints. However, many parameters required by the UC problem are unknown, such as the costs. In this work, we estimate these unknown costs using simulation-based inference on an illustrative UC problem, which provides an approximated posterior distribution of the parameters given observed generation schedules and demands. Our results highlight that the learned posterior distribution effectively captures the underlying distribution of the data, providing a range of possible values for the unknown parameters given a past observation. This posterior allows for the estimation of past costs using observed past generation schedules, enabling operators to better forecast future costs and make more robust generation scheduling forecasts. We present avenues for future research to address overconfidence in posterior estimation, enhance the scalability of the methodology and apply it to more complex UC problems modeling the network constraints and renewable energy sources.

cs.LG↗

arXiv2024

An Optimization Algorithm for Customer Topological Paths Identification in Electrical Distribution Networks

A customer topological path represents the sequence of network elements connecting an MV/LV transformer to a customer. Accurate knowledge of these paths is crucial for distribution system operators (DSOs) in digitalization, analysis, and network planning. This paper introduces an innovative approach to address the challenge of customer topological path identification (TPI) using only the limited and often inaccurate data available to DSOs. Specifically, our method relies only on geographic information system (GIS) data of network elements and the customer to MV/LV transformers connection information. We introduce an integer linear programming (ILP) optimization algorithm designed to identify customer topological paths that closely approximate the real electricity paths. The effectiveness of the proposed approach is demonstrated through its application to both an academic and a real-world electrical distribution network. Results show that the method effectively addresses data inaccuracies and successfully identifies customer topological paths, providing a valuable tool for DSOs in developing accurate digital twins of their distribution networks.

eess.SY↗

arXiv2024

A Systematic Procedure for Topological Path Identification with Raw Data Transformation in Electrical Distribution Networks

This paper introduces a systematic approach to address the topological path identification (TPI) problem in power distribution networks. Our approach starts by listing the DSO's raw information coming from several sources. The raw information undergoes a transformation process using a set of transformation functions. This process converts the raw information into well-defined information exploitable by an algorithm. Then a set of hypothetical paths is generated, considering any potential connections between the elements of the power distribution system. This set of hypothetical paths is processed by the algorithm that identifies the hypothetical paths that are compatible with the well-defined information. This procedure operates iteratively, adapting the set of transformation functions based on the result obtained: if the identified paths fail to meet the DSO's expectations, new data is collected, and/or the transformation functions found to be responsible for the discrepancies are modified. The systematic procedure offers practical advantages for DSOs, including improved accuracy in path identification and high adaptability to diverse network configurations, even with incomplete or inaccurate data. Consequently, it emerges as a useful tool for the construction of digital twins of power distribution networks that aligns with DSO expectations.

eess.SY↗

arXiv2024

Fair Reinforcement Learning Algorithm for PV Active Control in LV Distribution Networks

The increasing adoption of distributed energy resources, particularly photovoltaic (PV) panels, has presented new and complex challenges for power network control. With the significant energy production from PV panels, voltage issues in the network have become a problem. Currently, PV smart inverters (SIs) are used to mitigate the voltage problems by controlling their active power generation and reactive power injection or absorption. However, reducing the active power output of PV panels can be perceived as unfair to some customers, discouraging future installations. To solve this issue, in this paper, a reinforcement learning technique is proposed to address voltage issues in a distribution network, while considering fairness in active power curtailment among customers. The feasibility of the proposed approach is explored through experiments, demonstrating its ability to effectively control voltage in a fair and efficient manner.

eess.SY↗

arXiv2024

Parallelizing Autoregressive Generation with Variational State Space Models

Attention-based models such as Transformers and recurrent models like state space models (SSMs) have emerged as successful methods for autoregressive sequence modeling. Although both enable parallel training, none enable parallel generation due to their autoregressiveness. We propose the variational SSM (VSSM), a variational autoencoder (VAE) where both the encoder and decoder are SSMs. Since sampling the latent variables and decoding them with the SSM can be parallelized, both training and generation can be conducted in parallel. Moreover, the decoder recurrence allows generation to be resumed without reprocessing the whole sequence. Finally, we propose the autoregressive VSSM that can be conditioned on a partial realization of the sequence, as is common in language generation tasks. Interestingly, the autoregressive VSSM still enables parallel generation. We highlight on toy problems (MNIST, CIFAR) the empirical gains in speed-up and show that it competes with traditional models in terms of generation quality (Transformer, Mamba SSM).

cs.LG↗

arXiv2024

Reinforcement Learning to improve delta robot throws for sorting scrap metal

This study proposes a novel approach based on reinforcement learning (RL) to enhance the sorting efficiency of scrap metal using delta robots and a Pick-and-Place (PaP) process, widely used in the industry. We use three classical model-free RL algorithms (TD3, SAC and PPO) to reduce the time to sort metal scraps. We learn the release position and speed needed to throw an object in a bin instead of moving to the exact bin location, as with the classical PaP technique. Our contribution is threefold. First, we provide a new simulation environment for learning RL-based Pick-and-Throw (PaT) strategies for parallel grippers. Second, we use RL algorithms for learning this task in this environment resulting in 89% accuracy while speeding up the throughput by 51% in simulation. Third, we evaluate the performances of RL algorithms and compare them to a PaP and a state-of-the-art PaT method both in simulation and reality, learning only from simulation with domain randomisation and without fine tuning in reality to transfer our policies. This work shows the benefits of RL-based PaT compared to PaP or classical optimization PaT techniques used in the industry.

cs.RO↗

arXiv2024

Informed POMDP: Leveraging Additional Information in Model-Based RL

In this work, we generalize the problem of learning through interaction in a POMDP by accounting for eventual additional information available at training time. First, we introduce the informed POMDP, a new learning paradigm offering a clear distinction between the information at training and the observation at execution. Next, we propose an objective that leverages this information for learning a sufficient statistic of the history for the optimal control. We then adapt this informed objective to learn a world model able to sample latent trajectories. Finally, we empirically show a learning speed improvement in several environments using this informed world model in the Dreamer algorithm. These results and the simplicity of the proposed adaptation advocate for a systematic consideration of eventual additional information when learning in a POMDP using model-based RL.

cs.LG↗

arXiv2024

Spike-based computation using classical recurrent neural networks

Spiking neural networks are a type of artificial neural networks in which communication between neurons is only made of events, also called spikes. This property allows neural networks to make asynchronous and sparse computations and therefore drastically decrease energy consumption when run on specialised hardware. However, training such networks is known to be difficult, mainly due to the non-differentiability of the spike activation, which prevents the use of classical backpropagation. This is because state-of-the-art spiking neural networks are usually derived from biologically-inspired neuron models, to which are applied machine learning methods for training. Nowadays, research about spiking neural networks focuses on the design of training algorithms whose goal is to obtain networks that compete with their non-spiking version on specific tasks. In this paper, we attempt the symmetrical approach: we modify the dynamics of a well-known, easily trainable type of recurrent neural network to make it event-based. This new RNN cell, called the Spiking Recurrent Cell, therefore communicates using events, i.e. spikes, while being completely differentiable. Vanilla backpropagation can thus be used to train any network made of such RNN cell. We show that this new network can achieve performance comparable to other types of spiking networks in the MNIST benchmark and its variants, the Fashion-MNIST and the Neuromorphic-MNIST. Moreover, we show that this new cell makes the training of deep spiking networks achievable.

cs.NE↗

arXiv2024

Optimal Control of Renewable Energy Communities subject to Network Peak Fees with Model Predictive Control and Reinforcement Learning Algorithms

We propose in this paper an optimal control framework for renewable energy communities (RECs) equipped with controllable assets. Such RECs allow its members to exchange production surplus through an internal market. The objective is to control their assets in order to minimise the sum of individual electricity bills. These bills account for the electricity exchanged through the REC and with the retailers. Typically, for large companies, another important part of the bills are the costs related to the power peaks; in our framework, they are determined from the energy exchanges with the retailers. We compare rule-based control strategies with the two following control algorithms. The first one is derived from model predictive control techniques, and the second one is built with reinforcement learning techniques. We also compare variants of these algorithms that neglect the peak power costs. Results confirm that using policies accounting for the power peaks lead to a significantly lower sum of electricity bills and thus better control strategies at the cost of higher computation time. Furthermore, policies trained with reinforcement learning approaches appear promising for real-time control of the communities, where model predictive control policies may be computationally expensive in practice. These findings encourage pursuing the efforts toward development of scalable control algorithms, operating from a centralised standpoint, for renewable energy communities equipped with controllable assets.

eess.SY↗

arXiv2024

Synthetic methane for closing the carbon loop: Comparative study of three carbon sources for remote carbon-neutral fuel synthetization

Achieving carbon neutrality is probably one of the most important challenges of the 21st century for our societies. Part of the solution to this challenge is to leverage renewable energies. However, these energy sources are often located far away from places that need the energy, and their availability is intermittent, which makes them challenging to work with. In this paper, we build upon the concept of Remote Renewable Energy Hubs (RREHs), which are hubs located at remote places with abundant renewable energy sources whose purpose is to produce carbon-neutral synthetic fuels. More precisely, we model and study the Energy Supply Chain (ESC) that would be required to provide a constant source of carbon-neutral synthetic methane, also called e-NG (electric Natural Gas) or e-methane (electric methane), in Belgium from an RREH located in Morocco. To be carbon neutral, a synthetic fuel has to be produced from existing carbon dioxide (CO2) that needs to be captured using either Direct Air Capture (DAC) or Post Combustion Carbon Capture (PCCC). In this work, we detail the impact of three different carbon sourcing configurations on the price of the e-methane delivered in Belgium. Our results show that sourcing CO2 through a combination of DAC and PCCC is more cost-effective, resulting in a cost of 146 e/MWh for e-methane delivered in Belgium, as opposed to relying solely on DAC, which leads to a cost of 158 e/MWh. Moreover, these scenarios are compared to a scenario where CO2 is captured in Morocco from a CO2 emitting asset that allows to deliver e-methane for a cost of 136 e/MWh.

math.OC↗

arXiv2023

IMP-MARL: a Suite of Environments for Large-scale Infrastructure Management Planning via MARL

We introduce IMP-MARL, an open-source suite of multi-agent reinforcement learning (MARL) environments for large-scale Infrastructure Management Planning (IMP), offering a platform for benchmarking the scalability of cooperative MARL methods in real-world engineering applications. In IMP, a multi-component engineering system is subject to a risk of failure due to its components' damage condition. Specifically, each agent plans inspections and repairs for a specific system component, aiming to minimise maintenance costs while cooperating to minimise system failure risk. With IMP-MARL, we release several environments including one related to offshore wind structural systems, in an effort to meet today's needs to improve management strategies to support sustainable and reliable energy systems. Supported by IMP practical engineering environments featuring up to 100 agents, we conduct a benchmark campaign, where the scalability and performance of state-of-the-art cooperative MARL methods are compared against expert-based heuristic policies. The results reveal that centralised training with decentralised execution methods scale better with the number of agents than fully centralised or decentralised RL approaches, while also outperforming expert-based heuristic policies in most IMP environments. Based on our findings, we additionally outline remaining cooperation and scalability challenges that future MARL methods should still address. Through IMP-MARL, we encourage the implementation of new environments and the further development of MARL methods.

cs.LG↗

arXiv2023

Policy Gradient Algorithms Implicitly Optimize by Continuation

Direct policy optimization in reinforcement learning is usually solved with policy-gradient algorithms, which optimize policy parameters via stochastic gradient ascent. This paper provides a new theoretical interpretation and justification of these algorithms. First, we formulate direct policy optimization in the optimization by continuation framework. The latter is a framework for optimizing nonconvex functions where a sequence of surrogate objective functions, called continuations, are locally optimized. Second, we show that optimizing affine Gaussian policies and performing entropy regularization can be interpreted as implicitly optimizing deterministic policies by continuation. Based on these theoretical results, we argue that exploration in policy-gradient algorithms consists in computing a continuation of the return of the policy at hand, and that the variance of policies should be history-dependent functions adapted to avoid local extrema rather than to maximize the return of the policy.

cs.LG↗

arXiv2023

Warming up recurrent neural networks to maximise reachable multistability greatly improves learning

Training recurrent neural networks is known to be difficult when time dependencies become long. In this work, we show that most standard cells only have one stable equilibrium at initialisation, and that learning on tasks with long time dependencies generally occurs once the number of network stable equilibria increases; a property known as multistability. Multistability is often not easily attained by initially monostable networks, making learning of long time dependencies between inputs and outputs difficult. This insight leads to the design of a novel way to initialise any recurrent cell connectivity through a procedure called "warmup" to improve its capability to learn arbitrarily long time dependencies. This initialisation procedure is designed to maximise network reachable multistability, i.e., the number of equilibria within the network that can be reached through relevant input trajectories, in few gradient steps. We show on several information restitution, sequence classification, and reinforcement learning benchmarks that warming up greatly improves learning speed and performance, for multiple recurrent cells, but sometimes impedes precision. We therefore introduce a double-layer architecture initialised with a partial warmup that is shown to greatly improve learning of long time dependencies while maintaining high levels of precision. This approach provides a general framework for improving learning abilities of any recurrent cell when long time dependencies are present. We also show empirically that other initialisation and pretraining procedures from the literature implicitly foster reachable multistability of recurrent cells.

cs.LG↗

arXiv2023

Towards CO2 valorization in a multi remote renewable energy hub framework

In this paper, we propose a multi-RREH (Remote Renewable Energy Hub) based optimization framework. This framework allows a valorization of CO2 using carbon capture technologies. This valorization is grounded on the idea that CO2 gathered from the atmosphere or post combustion can be combined with hydrogen to produce synthetic methane. The hydrogen is obtained from water electrolysis using renewable energy (RE). Such renewable energy is generated in RREHs, which are locations where RE is cheap and abundant (e.g., solar PV in the Sahara Desert, or wind in Greenland). We instantiate our framework on a case study focusing on Belgium and 2 RREHs, and we conduct a techno-economic analysis. This analysis highlights the key role played by the cost of the two main carbon capture technologies: Post Combustion Carbon Capture (PCCC) and Direct Air Capture (DAC). In addition, we use our framework to derive a carbon price threshold above which carbon capture technologies may start playing a pivotal role in the decarbonation process of our industries. For example, this price threshold may give relevant information for calibrating the EU Emission Trading System so as to trigger the emergence of the multi-RREH.

math.OC↗