arXiv subjects

Max Mowbray

Publications and source records attributed to Max Mowbray.

8 recordsLinked to original sources

arXiv2025

A Decomposition Approach to Solving Numerical Constraint Satisfaction Problems on Directed Acyclic Graphs

Certifying feasibility in decision-making, critical in many industries, can be framed as a constraint satisfaction problem. This paper focuses on characterising a subset of parameter values from an a priori set that satisfy constraints on a directed acyclic graph of constituent functions. The main assumption is that these functions and constraints may be evaluated for given parameter values, but they need not be known in closed form and could result from expensive or proprietary simulations. This setting lends itself to using sampling methods to gain an inner approximation of the feasible domain. To mitigate the curse of dimensionality, the paper contributes new methodology to leverage the graph structure for decomposing the problem into lower-dimensional subproblems defined on the respective nodes. The working hypothesis that the Cartesian product of the solution sets yielded by the subproblems will tighten the a priori parameter domain, before solving the full problem defined on the graph, is demonstrated through four case studies relevant to machine learning and engineering. Future research will extend this approach to cyclic graphs and account for parametric uncertainty.

eess.SY↗

arXiv2025

Survey and Tutorial of Reinforcement Learning Methods in Process Systems Engineering

Sequential decision making under uncertainty is central to many Process Systems Engineering (PSE) challenges, where traditional methods often face limitations related to controlling and optimizing complex and stochastic systems. Reinforcement Learning (RL) offers a data-driven approach to derive control policies for such challenges. This paper presents a survey and tutorial on RL methods, tailored for the PSE community. We deliver a tutorial on RL, covering fundamental concepts and key algorithmic families including value-based, policy-based and actor-critic methods. Subsequently, we survey existing applications of these RL techniques across various PSE domains, such as in fed-batch and continuous process control, process optimization, and supply chains. We conclude with PSE focused discussion of specialized techniques and emerging directions. By synthesizing the current state of RL algorithm development and implications for PSE this work identifies successes, challenges, trends, and outlines avenues for future research at the interface of these fields.

eess.SY↗

arXiv2024

PC-Gym: Benchmark Environments For Process Control Problems

PC-Gym is an open-source tool for developing and evaluating reinforcement learning (RL) algorithms in chemical process control. It features environments that simulate various chemical processes, incorporating nonlinear dynamics, disturbances, and constraints. The tool includes customizable constraint handling, disturbance generation, reward function design, and enables comparison of RL algorithms against Nonlinear Model Predictive Control (NMPC) across different scenarios. Case studies demonstrate the framework's effectiveness in evaluating RL approaches for systems like continuously stirred tank reactors, multistage extraction processes, and crystallization reactors. The results reveal performance gaps between RL algorithms and NMPC oracles, highlighting areas for improvement and enabling benchmarking. By providing a standardized platform, PC-Gym aims to accelerate research at the intersection of machine learning, control, and process systems engineering. By connecting theoretical RL advances with practical industrial process control applications, offering researchers a tool for exploring data-driven control solutions.

eess.SY↗

arXiv2023

An Analysis of Multi-Agent Reinforcement Learning for Decentralized Inventory Control Systems

Most solutions to the inventory management problem assume a centralization of information that is incompatible with organisational constraints in real supply chain networks. The inventory management problem is a well-known planning problem in operations research, concerned with finding the optimal re-order policy for nodes in a supply chain. While many centralized solutions to the problem exist, they are not applicable to real-world supply chains made up of independent entities. The problem can however be naturally decomposed into sub-problems, each associated with an independent entity, turning it into a multi-agent system. Therefore, a decentralized data-driven solution to inventory management problems using multi-agent reinforcement learning is proposed where each entity is controlled by an agent. Three multi-agent variations of the proximal policy optimization algorithm are investigated through simulations of different supply chain networks and levels of uncertainty. The centralized training decentralized execution framework is deployed, which relies on offline centralization during simulation-based policy identification, but enables decentralization when the policies are deployed online to the real system. Results show that using multi-agent proximal policy optimization with a centralized critic leads to performance very close to that of a centralized data-driven solution and outperforms a distributed model-based solution in most cases while respecting the information constraints of the system.

cs.LG↗

arXiv2022

Distributional Reinforcement Learning for Scheduling of Chemical Production Processes

Reinforcement Learning (RL) has recently received significant attention from the process systems engineering and control communities. Recent works have investigated the application of RL to identify optimal scheduling decision in the presence of uncertainty. In this work, we present a RL methodology tailored to efficiently address production scheduling problems in the presence of uncertainty. We consider commonly imposed restrictions on these problems such as precedence and disjunctive constraints which are not naturally considered by RL in other contexts. Additionally, this work naturally enables the optimization of risk-sensitive formulations such as the conditional value-at-risk (CVaR), which are essential in realistic scheduling processes. The proposed strategy is investigated thoroughly in a parallel batch production environment, and benchmarked against mixed integer linear programming (MILP) strategies. We show that the policy identified by our approach is able to account for plant uncertainties in online decision-making, with expected performance comparable to existing MILP methods. Additionally, the framework gains the benefits of optimizing for risk-sensitive measures, and identifies online decisions orders of magnitude faster than the most efficient optimization approaches. This promises to mitigate practical issues and ease in handling realizations of process uncertainty in the paradigm of online production scheduling.

eess.SY↗

arXiv2021

Integrating process design and control using reinforcement learning

To create efficient-high performing processes, one must find an optimal design with its corresponding controller that ensures optimal operation in the presence of uncertainty. When comparing different process designs, for the comparison to be meaningful, each design must involve its optimal operation. Therefore, to optimize a process' design, one must address design and control simultaneously. For this, one can formulate a bilevel optimization problem, with the design as the outer problem in the form of a mixed-integer nonlinear program (MINLP) and a stochastic optimal control as the inner problem. This is intractable by most approaches. In this paper we propose to compute the optimal control using reinforcement learning, and then embed this controller into the design problem. This allows to decouple the solution procedure, while having the same optimal result as if solving the bilevel problem. The approach is tested in two case studies and the performance of the controller is evaluated. The case studies indicate that the proposed approach outperforms current state-of-the-art simultaneous design and control strategies. This opens a new avenue to address simultaneous design and control of engineering systems.

eess.SY↗

arXiv2021

Safe Chance Constrained Reinforcement Learning for Batch Process Control

Reinforcement Learning (RL) controllers have generated excitement within the control community. The primary advantage of RL controllers relative to existing methods is their ability to optimize uncertain systems independently of explicit assumption of process uncertainty. Recent focus on engineering applications has been directed towards the development of safe RL controllers. Previous works have proposed approaches to account for constraint satisfaction through constraint tightening from the domain of stochastic model predictive control. Here, we extend these approaches to account for plant-model mismatch. Specifically, we propose a data-driven approach that utilizes Gaussian processes for the offline simulation model and use the associated posterior uncertainty prediction to account for joint chance constraints and plant-model mismatch. The method is benchmarked against nonlinear model predictive control via case studies. The results demonstrate the ability of the methodology to account for process uncertainty, enabling satisfaction of joint chance constraints even in the presence of plant-model mismatch.

cs.LG↗

arXiv2020

Constrained Model-Free Reinforcement Learning for Process Optimization

Reinforcement learning (RL) is a control approach that can handle nonlinear stochastic optimal control problems. However, despite the promise exhibited, RL has yet to see marked translation to industrial practice primarily due to its inability to satisfy state constraints. In this work we aim to address this challenge. We propose an 'oracle'-assisted constrained Q-learning algorithm that guarantees the satisfaction of joint chance constraints with a high probability, which is crucial for safety critical tasks. To achieve this, constraint tightening (backoffs) are introduced and adjusted using Broyden's method, hence making them self-tuned. This results in a general methodology that can be imbued into approximate dynamic programming-based algorithms to ensure constraint satisfaction with high probability. Finally, we present case studies that analyze the performance of the proposed approach and compare this algorithm with model predictive control (MPC). The favorable performance of this algorithm signifies a step toward the incorporation of RL into real world optimization and control of engineering systems, where constraints are essential in ensuring safety.

cs.LG↗