arXiv subjects

Georg Schäfer

Publications and source records attributed to Georg Schäfer.

16 recordsLinked to original sources

arXiv2026

Chebyshev Policies and the Mountain Car Problem: Reinforcement Learning for Low-Dimensional Control Tasks

We analytically solve the Mountain Car problem, a canonical benchmark in RL, and derive an optimal control solution, closing a gap after 36 years. This enables us to reveal two surprising insights: The optimal control is quite simple, yet modern RL agents display a large gap to optimality. Motivated by the analysis of the optimal control, we introduce Chebyshev policies as a universal (i.e. dense) class of RL policies from first principles. They can be trained as drop-in replacements of neural nets, reducing the regret by a factor of 6.18, while requiring 277 times fewer parameters, fostering sample efficiency, explainability and realtime capability. Chebyshev policies are evaluated on further RL tasks, including a real-world nonlinear motion control testbed. They consistently improve performance over neural nets with PPO, ARS and REINFORCE. Our results demonstrate how Chebyshev policies offer a compelling and lightweight alternative or addition to neural nets for low-dimensional control tasks.

cs.LG↗

arXiv2026

Safe Reinforcement Learning using Ideas from Model Predictive Control

Reinforcement learning (RL) enables the synthesis of control policies directly from data, making it highly appealing for complex cyber-physical systems (CPSs) and robotics. A persistent challenge, however, is ensuring strict, hard safety constraints during the active learning phase. In real-world physical systems, violating mechanical limits can cause irreversible damage, necessitating that exploration remains strictly within safe operational regions. We propose a generalized framework that combines the adaptive, high-performance nature of deep reinforcement learning (DRL) with the formal safety guarantees of model predictive control (MPC). Using a mathematical model of the system dynamics, offline MPC computations define a feasible state-action space, representing all safe combinations of system states and control inputs that guarantee constraint satisfaction. During training and deployment, the RL agent's instantaneous actions are projected onto this globally verified feasible set via a safety filter. We systematically evaluate our generalized approach on a non-linear 1-DoF laboratory testbed, demonstrating successful exploration and stable policy convergence on physical hardware.

cs.LG↗

arXiv2026

Integrating Physics-Informed Neural Networks for Safe Reinforcement Learning in a 1-DoF Helicopter System

Deep reinforcement learning (DRL) offers powerful control for industrial cyber-physical systems (ICPSs), but its "black-box" exploration risks violating strict hardware safety limits. Typically, these constraints are managed through complex reward shaping. In this work-in-progress paper, we embed a differentiable physics model directly into the proximal policy optimization (PPO) actor loss function. By simulating short-horizon future trajectories during training, the policy is penalized for anticipated safety violations independent of the task-reward signal. Evaluated on a simulated 1-degree-of-freedom helicopter testbed with strict pitch constraints, our physics-informed soft regularizations substantially reduce constraint violations while maintaining reliable target tracking.

cs.LG↗

arXiv2026

Anticipatory Reinforcement Learning for Trajectory Tracking

Deep reinforcement learning (DRL) in industrial control often suffers from lag and overshoot due to purely reactive control based on the current tracking error. To achieve anticipatory control without high computational overhead, we introduce a predictive formulation that augments the DRL state space with target velocities and future reference horizons. Evaluating eight configurations using proximal policy optimization (PPO) on a 1-degree-of-freedom (1-DoF) helicopter testbed, simulation results showed a 9-fold error reduction, lowering the mean absolute deviation from 2.73° to 0.31°. However, zero-shot transfer to physical hardware revealed a sim-to-real gap. Interestingly, a simpler configuration using a single, further look-ahead horizon matched the real-world top performance of the most complex model (1.11°). Overall, evaluating various combinations of prediction horizons and target velocities demonstrated that highly granular predictive data is not necessarily required for physical transfer.

cs.LG↗

arXiv2026

Sample-Efficient Pareto Front Modeling for Energy-Aware Reinforcement Learning Using Bayesian Optimization

Industrial automation increasingly demands control strategies that balance operational performance with strict energy efficiency requirements. A common approach to solving this multi-objective problem, particularly within the framework of reinforcement learning (RL), is to formulate a single, scalar reward function that linearly combines the competing objectives. However, the manual weighting of these different objectives is heavily reliant on domain intuition, incredibly time-consuming, prone to human bias, and frequently fails to uncover optimal trade-off solutions. This work addresses the critical challenge of automating the weight selection process to systematically and efficiently discover the Pareto front of optimal trade-off policies. We formulate the weight selection process as a multi-objective Bayesian optimization (MOBO) problem and evaluate its sample efficiency against a standard uniform grid search baseline. Using a physical Quanser Aero 2 testbed configured for 1-DoF pitch control, our results demonstrate that the MOBO approach, utilizing the expected hypervolume improvement (qEHVI) acquisition function, consistently outperforms uniform grid sampling. MOBO achieves superior hypervolume and maximum spread, successfully identifying high-quality, diverse trade-off policies with a reduced evaluation budget, thereby enabling highly efficient energy-aware control in complex mechatronic systems.

cs.LG↗

arXiv2025

Multi-Objective Reinforcement Learning for Energy-Efficient Industrial Control

Industrial automation increasingly demands energy-efficient control strategies to balance performance with environmental and cost constraints. In this work, we present a multi-objective reinforcement learning (MORL) framework for energy-efficient control of the Quanser Aero 2 testbed in its one-degree-of-freedom configuration. We design a composite reward function that simultaneously penalizes tracking error and electrical power consumption. Preliminary experiments explore the influence of varying the Energy penalty weight, alpha, on the trade-off between pitch tracking and energy savings. Our results reveal a marked performance shift for alpha values between 0.0 and 0.25, with non-Pareto optimal solutions emerging at lower alpha values, on both the simulation and the real system. We hypothesize that these effects may be attributed to artifacts introduced by the adaptive behavior of the Adam optimizer, which could bias the learning process and favor bang-bang control strategies. Future work will focus on automating alpha selection through Gaussian Process-based Pareto front modeling and transitioning the approach from simulation to real-world deployment.

eess.SY↗

arXiv2025

The Crucial Role of Problem Formulation in Real-World Reinforcement Learning

Reinforcement Learning (RL) offers promising solutions for control tasks in industrial cyber-physical systems (ICPSs), yet its real-world adoption remains limited. This paper demonstrates how seemingly small but well-designed modifications to the RL problem formulation can substantially improve performance, stability, and sample efficiency. We identify and investigate key elements of RL problem formulation and show that these enhance both learning speed and final policy quality. Our experiments use a one-degree-of-freedom (1-DoF) helicopter testbed, the Quanser Aero~2, which features non-linear dynamics representative of many industrial settings. In simulation, the proposed problem design principles yield more reliable and efficient training, and we further validate these results by training the agent directly on physical hardware. The encouraging real-world outcomes highlight the potential of RL for ICPS, especially when careful attention is paid to the design principles of problem formulation. Overall, our study underscores the crucial role of thoughtful problem formulation in bridging the gap between RL research and the demands of real-world industrial systems.

eess.SY↗

arXiv2024

A Generative Model Based Honeypot for Industrial OPC UA Communication

Industrial Operational Technology (OT) systems are increasingly targeted by cyber-attacks due to their integration with Information Technology (IT) systems in the Industry 4.0 era. Besides intrusion detection systems, honeypots can effectively detect these attacks. However, creating realistic honeypots for brownfield systems is particularly challenging. This paper introduces a generative model-based honeypot designed to mimic industrial OPC UA communication. Utilizing a Long ShortTerm Memory (LSTM) network, the honeypot learns the characteristics of a highly dynamic mechatronic system from recorded state space trajectories. Our contributions are twofold: first, we present a proof-of concept for a honeypot based on generative machine-learning models, and second, we publish a dataset for a cyclic industrial process. The results demonstrate that a generative model-based honeypot can feasibly replicate a cyclic industrial process via OPC UA communication. In the short-term, the generative model indicates a stable and plausible trajectory generation, while deviations occur over extended periods. The proposed honeypot implementation operates efficiently on constrained hardware, requiring low computational resources. Future work will focus on improving model accuracy, interaction capabilities, and extending the dataset for broader applications.

cs.NI↗

arXiv2024

Comparison of Model Predictive Control and Proximal Policy Optimization for a 1-DOF Helicopter System

This study conducts a comparative analysis of Model Predictive Control (MPC) and Proximal Policy Optimization (PPO), a Deep Reinforcement Learning (DRL) algorithm, applied to a 1-Degree of Freedom (DOF) Quanser Aero 2 system. Classical control techniques such as MPC and Linear Quadratic Regulator (LQR) are widely used due to their theoretical foundation and practical effectiveness. However, with advancements in computational techniques and machine learning, DRL approaches like PPO have gained traction in solving optimal control problems through environment interaction. This paper systematically evaluates the dynamic response characteristics of PPO and MPC, comparing their performance, computational resource consumption, and implementation complexity. Experimental results show that while LQR achieves the best steady-state accuracy, PPO excels in rise-time and adaptability, making it a promising approach for applications requiring rapid response and adaptability. Additionally, we have established a baseline for future RL-related research on this specific testbed. We also discuss the strengths and limitations of each control strategy, providing recommendations for selecting appropriate controllers for real-world scenarios.

eess.SY↗

arXiv2024

Python-Based Reinforcement Learning on Simulink Models

This paper proposes a framework for training Reinforcement Learning agents using Python in conjunction with Simulink models. Leveraging Python's superior customization options and popular libraries like Stable Baselines3, we aim to bridge the gap between the established Simulink environment and the flexibility of Python for training bleeding edge agents. Our approach is demonstrated on the Quanser Aero 2, a versatile dual-rotor helicopter. We show that policies trained on Simulink models can be seamlessly transferred to the real system, enabling efficient development and deployment of Reinforcement Learning agents for control tasks. Through systematic integration steps, including C-code generation from Simulink, DLL compilation, and Python interface development, we establish a robust framework for training agents on Simulink models. Experimental results demonstrate the effectiveness of our approach, surpassing previous efforts and highlighting the potential of combining Simulink with Python for Reinforcement Learning research and applications.

cs.LG↗

arXiv2024

IT/OT Integration by Design

The four Industry 4.0 design principles information transparency, technical assistance, interconnection, and decentralized decisions pose challenges in integrating information technology (IT) and operational technology (OT) solutions in industrial systems. These different solutions have conflicting requirements, making interfaces between them problematic for both systems and organizations. An Industrial Business Process Twin (IBPT) entity, acting as an intermediary between the realms of IT and OT, has been proposed in a previous work, to effectively reduce the amount of required IT/OT interfaces in an attempt of overcoming this situation. In this work, we investigate the effects of this approach during the design phase. We argue that, by eliminating potentially conflicting direct interfaces between IT and OT stakeholders within the organizational structure, this approach effectively eliminates conflicting communication channels within the system design. In order to verify our argument, we develop a model of our IBPT concept according to the Reference Architecture Model Industrie 4.0 (RAMI4.0) using an Industry 4.0 scenario addressing the four essential Industry 4.0 design principles. Results show that the IBPT approach indeed eliminates potentially conflicting IT/OT interfaces during the system design phase.

cs.SE↗

arXiv2023

A Modular Test Bed for Reinforcement Learning Incorporation into Industrial Applications

This application paper explores the potential of using reinforcement learning (RL) to address the demands of Industry 4.0, including shorter time-to-market, mass customization, and batch size one production. Specifically, we present a use case in which the task is to transport and assemble goods through a model factory following predefined rules. Each simulation run involves placing a specific number of goods of random color at the entry point. The objective is to transport the goods to the assembly station, where two rivets are installed in each product, connecting the upper part to the lower part. Following the installation of rivets, blue products must be transported to the exit, while green products are to be transported to storage. The study focuses on the application of reinforcement learning techniques to address this problem and improve the efficiency of the production process.

cs.AI↗

arXiv2023

An Architecture for Deploying Reinforcement Learning in Industrial Environments

Industry 4.0 is driven by demands like shorter time-to-market, mass customization of products, and batch size one production. Reinforcement Learning (RL), a machine learning paradigm shown to possess a great potential in improving and surpassing human level performance in numerous complex tasks, allows coping with the mentioned demands. In this paper, we present an OPC UA based Operational Technology (OT)-aware RL architecture, which extends the standard RL setting, combining it with the setting of digital twins. Moreover, we define an OPC UA information model allowing for a generalized plug-and-play like approach for exchanging the RL agent used. In conclusion, we demonstrate and evaluate the architecture, by creating a proof of concept. By means of solving a toy example, we show that this architecture can be used to determine the optimal policy using a real control system.

cs.AI↗

arXiv2023

Digital Twins of Business Processes as Enablers for IT / OT Integration

The vision of Industry 4.0 introduces new requirements to Operational Technology (OT) systems. Solutions for these requirements already exist in the Information Technology (IT) world, however, due to the different characteristics of both worlds, these solutions often cannot be directly used in the world of OT. We therefore propose an Industrial Business Process Twin (IBPT), allowing to apply methods of one world to another not directly but, instead, to a representation, that is in bidirectional exchange with the other world. The proposed IBPT entity acts as an intermediary, decoupling the worlds of IT and OT, thus allowing for an integration of IT and OT components of different manufacturers and platforms. Using this approach, we demonstrate the four essential Industry 4.0 design principles information transparency, technical assistance, interconnection and decentralized decisions based on the gamified Industry 4.0 scenario of playing the game of Nine Men's Morris. This scenario serves well for agent based Artificial Intelligence (AI)-research and education. We develop an Open Platform Communications Unified Architecture (OPC UA) information and communication model and then evaluate the IBPT component with respect to the different views of the Reference Architecture Model Industry 4.0 (RAMI4.0).

cs.SE↗

arXiv2022

Experimental estimates of the photon background in a potential light-by-light scattering study

High power short pulse lasers provide a promising route to study the strong field effects of the quantum vacuum, for example by direct photon-photon scattering in the all-optical regime. Theoretical predictions based on realistic laser parameters achievable today or in the near future predict scattering of a few photons with colliding Petawatt laser pulses, requiring single photon sensitive detection schemes and very good spatio-temporal filtering and background suppression. In this article, we present experimental investigations of this photon background by employing only a single high power laser pulse tightly focused in residual gas of a vacuum chamber. The focal region was imaged onto a single-photon sensitive, time gated camera. As no detectable quantum vacuum signature was expected in our case, the setup allowed for characterization and first mitigation of background contributions. For the setup employed, scattering off surfaces of imperfect optics dominated below residual gas pressures of $1\times 10^{-4}$ mbar. Extrapolation of the findings to intensities relevant for photon-photon scattering studies is discussed.

hep-ex↗

arXiv2021

Immutable and Democratic Data in permissionless Peer-to-Peer Systems

Conventional data storage methods like SQL and NoSQL offer a huge amount of possibilities with one major disadvantage, having to use a centralized authority. This authority may be in the form of a centralized or decentralized master server or a permissioned peer-to-peer setting. This paper looks at different technologies on how to persist data without using a central authority, mainly looking at permissionless peer-to-peer networks, primarily Distributed Ledger Technologies (DLTs) and a combination of DLTs with conventional databases. Afterwards it is shown how a system like this might be implemented in two prototypes which are then evaluated against conventional databases.

cs.DB↗