arXiv subjects

Gautam Biswas

Publications and source records attributed to Gautam Biswas.

51 records · Page 3Linked to original sources

arXiv2024

MARVEL: Multi-Agent Reinforcement-Learning for Large-Scale Variable Speed Limits

Variable Speed Limit (VSL) control acts as a promising highway traffic management strategy with worldwide deployment, which can enhance traffic safety by dynamically adjusting speed limits according to real-time traffic conditions. Most of the deployed VSL control algorithms so far are rule-based, lacking generalizability under varying and complex traffic scenarios. In this work, we propose MARVEL (Multi-Agent Reinforcement-learning for large-scale Variable spEed Limits), a novel framework for large-scale VSL control on highway corridors with real-world deployment settings. MARVEL utilizes only sensing information observable in the real world as state input and learns through a reward structure that incorporates adaptability to traffic conditions, safety, and mobility, thereby enabling multi-agent coordination. With parameter sharing among all VSL agents, the proposed framework scales to cover corridors with many agents. The policies are trained in a microscopic traffic simulation environment, focusing on a short freeway stretch with 8 VSL agents spanning 7 miles. For testing, these policies are applied to a more extensive network with 34 VSL agents spanning 17 miles of I-24 near Nashville, TN, USA. MARVEL-based method improves traffic safety by 63.4% compared to the no control scenario and enhances traffic mobility by 58.6% compared to a state-of-the-practice algorithm that has been deployed on I-24. Besides, we conduct an explainability analysis to examine the decision-making process of the agents and explore the learned policy under different traffic conditions. Finally, we test the response of the policy learned from the simulation-based experiments with real-world data collected from I-24 and illustrate its deployment capability.

cs.MA↗

arXiv2023

Collision of two drops moving in the same direction

The collision dynamics of two drops of the same liquid moving in the same direction has been studied numerically. A wide range of radius ratios of trailing drop and leading drop ($R_r$) and the velocity ratios ($U_r$) have been deployed to understand the collision outcomes. A volume of fluid (VOF) based open-source fluid flow solver, Basilisk, has been used with its adaptive mesh refinement feature to capture the nuances of the interface morphology. The simulations are analyzed for the evolving time instances. Different collision outcomes, such as coalescence and reflexive separation with and without the formation of satellite drops, have been observed for various combinations of $U_r$ and $R_r$. The study analyzes the evolution of kinetic energy and surface energy before and after the collision for plausible outcomes. The collision outcomes are depicted on a regime map with $U_r-R_r$ space, highlighting distinct regimes formed due to variations in relevant governing parameters.

physics.flu-dyn↗

arXiv2023

Modeling binary collision of evaporating drops

We investigate the interactions between two drops in a heated environment and analyze the effect of evaporation on bouncing, coalescence and reflexive separation phenomena. A reliable mass transfer model is incorporated in a coupled level-set and volume-of-fluid framework to accurately model the evaporation process and the evolution of drop interfaces during the interactions. The numerical technique is extensively validated against the benchmark problems involving the evaporation of a single drop. We analyze the contour plots of temperature and vapor mass fraction fields for each collision outcome. Our numerical simulations reveal that vapor entrapment during the separation process, with high-velocity vapor manages to escape. Increasing evaporation rates result in slower post-collision drop separation. Furthermore, the differences in kinetic energy and surface energy are analyzed for different Stefan numbers. The coalescence of drops exhibits energy oscillations until dissipation, while the bouncing and reflexive separations lack such oscillations. In the reflexive separation regime, the kinetic energy of the drops becomes zero after detachment.

physics.flu-dyn↗

arXiv2023

An investigation on the impact of two vertically aligned drops on a liquid surface

The dynamics of two vertically coalescing drops and a pool of the same liquid have been investigated using a Coupled Level Set and Volume of Fluid (CLSVOF) method. Such a configuration enables us to study the dynamic interaction of an arbitrary-shaped liquid conglomerate, formed owing to drop-drop coalescence, with a pool. Similar to drop-pool and drop-drop interactions, partial coalescence is observed when a conglomerate interacts with a pool. The presence of the pool below the father drop is found to influence the coalescence characteristic of the two drops. At the same time, the movement of the capillary waves resulting from the interaction of two drops governs the coalescence dynamics of the conglomerate with the pool. As liquid interfaces interact and generate capillary waves at multiple locations, complex trajectories of capillary waves are observed, which play a crucial role in determining the pinch-off characteristics of the satellite during conglomerate-pool interaction. We examine the effect of the ratio of the diameters of the lower/father drop to the upper/mother drop (D_r) on the coalescence dynamics while maintaining the size of the mother drop constant. The variation in the coalescence dynamics due to change in $D_r$ is quantified in terms of the residence time (tau_r), pinch-off time (tau_p) and the satellite diameter to conglomerate diameter ratio (Ds/Dc). The coalescence dynamics of the conglomerate is then compared with that of an equivalent spherical drop of the same volume and also with that of a drop initialized with the same shape as that of the conglomerate. Finally, the regions of complete and partial coalescence for the conglomerate-pool interactions are demarcated on the Weber number - diameter ratio (We-Dr) space.

physics.flu-dyn↗

arXiv2023

Dynamics of an evaporating drop migrating in a Poiseuille flow

The evaporation of a liquid drop of initial diameter (Ddrop) migrating in a tube of diameter (D0) is investigated using the coupled level set and volume of fluid (CLSVOF) method focusing on determining the heat and mass transfer coefficients for a deforming drop. A robust phase change model is developed using an embedded boundary method under a finite difference framework to handle vaporizing flows. The model is extensively validated through simulations of benchmark problems such as arbitrary evaporation of a static drop and reproduction of psychrometric data. The results show that the Sherwood number (Sh) and the Nusselt number (Nu) reach a steady value after an initial transient period for the drop subjected to Hagen-Poiseuille flow. A parametric study is conducted to investigate the effect of drop deformation on the rate of evaporation. It is observed that Stefan flow due to evaporation has a negligible impact on the drop deformation dynamics. We also observed that, for different values of Ddrop/D0, the Sh follows a linear correlation with Re^{1/2}Sc^{1/3}.

physics.flu-dyn↗

arXiv2023

A Reinforcement Learning Approach for Robust Supervisory Control of UAVs Under Disturbances

In this work, we present an approach to supervisory reinforcement learning control for unmanned aerial vehicles (UAVs). UAVs are dynamic systems where control decisions in response to disturbances in the environment have to be made in the order of milliseconds. We formulate a supervisory control architecture that interleaves with extant embedded control and demonstrates robustness to environmental disturbances in the form of adverse wind conditions. We run case studies with a Tarot T-18 Octorotor to demonstrate the effectiveness of our approach and compare it against a classic cascade control architecture used in most vehicles. While the results show the performance difference is marginal for nominal operations, substantial performance improvement is obtained with the supervisory RL approach under unseen wind conditions.

eess.SY↗

arXiv2023

Model-based adaptation for sample efficient transfer in reinforcement learning control of parameter-varying systems

In this paper, we leverage ideas from model-based control to address the sample efficiency problem of reinforcement learning (RL) algorithms. Accelerating learning is an active field of RL highly relevant in the context of time-varying systems. Traditional transfer learning methods propose to use prior knowledge of the system behavior to devise a gradual or immediate data-driven transformation of the control policy obtained through RL. Such transformation is usually computed by estimating the performance of previous control policies based on measurements recently collected from the system. However, such retrospective measures have debatable utility with no guarantees of positive transfer in most cases. Instead, we propose a model-based transformation, such that when actions from a control policy are applied to the target system, a positive transfer is achieved. The transformation can be used as an initialization for the reinforcement learning process to converge to a new optimum. We validate the performance of our approach through four benchmark examples. We demonstrate that our approach is more sample-efficient than fine-tuning with reinforcement learning alone and achieves comparable performance to linear-quadratic-regulators and model-predictive control when an accurate linear model is known in the three cases. If an accurate model is not known, we empirically show that the proposed approach still guarantees positive transfer with jump-start improvement.

eess.SY↗

arXiv2022

Analyzing Adaptive Scaffolds that Help Students Develop Self-Regulated Learning Behaviors

Providing adaptive scaffolds to help learners develop self-regulated learning (SRL) processes has been an important goal for intelligent learning environments. Adaptive scaffolding is especially important in open-ended learning environments (OELE), where novice learners often face difficulties in completing their learning tasks. This paper presents a systematic framework for adaptive scaffolding in Betty's Brain, a learning-by-teaching OELE for middle school science, where students construct a causal model to teach a virtual agent, generically named Betty. We evaluate the adaptive scaffolding framework and discuss its implications on the development of more effective scaffolds for SRL in OELEs. We detect key cognitive/metacognitive inflection points, i.e., instances where students' behaviors and performance change as they work on their learning tasks. At such inflection points, Mr. Davis (a mentor agent) or Betty (the teachable agent) provide conversational feedback, focused on strategies to help students become productive learners. We conduct a classroom study with 98 middle schoolers to analyze the impact of adaptive scaffolds on students' learning behaviors and performance. Adaptive scaffolding produced mixed results, with some scaffolds (viz., strategic hints that supported debugging and assessment of causal models) being generally more useful to students than others (viz., encouragement prompts). We also note differences in learning behaviors of High and Low performers after receiving scaffolds. Overall, our findings suggest how adaptive scaffolding in OELEs like Betty's Brain can be further improved to narrow the gap between High and Low performers.

cs.CY↗

arXiv2022

Concurrent Policy Blending and System Identification for Generalized Assistive Control

In this work, we address the problem of solving complex collaborative robotic tasks subject to multiple varying parameters. Our approach combines simultaneous policy blending with system identification to create generalized policies that are robust to changes in system parameters. We employ a blending network whose state space relies solely on parameter estimates from a system identification technique. As a result, this blending network learns how to handle parameter changes instead of trying to learn how to solve the task for a generalized parameter set simultaneously. We demonstrate our scheme's ability on a collaborative robot and human itching task in which the human has motor impairments. We then showcase our approach's efficiency with a variety of system identification techniques when compared to standard domain randomization.

cs.RO↗

arXiv2020

Performance-Weighed Policy Sampling for Meta-Reinforcement Learning

This paper discusses an Enhanced Model-Agnostic Meta-Learning (E-MAML) algorithm that generates fast convergence of the policy function from a small number of training examples when applied to new learning tasks. Built on top of Model-Agnostic Meta-Learning (MAML), E-MAML maintains a set of policy parameters learned in the environment for previous tasks. We apply E-MAML to developing reinforcement learning (RL)-based online fault tolerant control schemes for dynamic systems. The enhancement is applied when a new fault occurs, to re-initialize the parameters of a new RL policy that achieves faster adaption with a small number of samples of system behavior with the new fault. This replaces the random task sampling step in MAML. Instead, it exploits the extant previously generated experiences of the controller. The enhancement is sampled to maximally span the parameter space to facilitate adaption to the new fault. We demonstrate the performance of our approach combining E-MAML with proximal policy optimization (PPO) on the well-known cart pole example, and then on the fuel transfer system of an aircraft.

cs.LG↗

arXiv2020

Complementary Meta-Reinforcement Learning for Fault-Adaptive Control

Faults are endemic to all systems. Adaptive fault-tolerant control maintains degraded performance when faults occur as opposed to unsafe conditions or catastrophic events. In systems with abrupt faults and strict time constraints, it is imperative for control to adapt quickly to system changes to maintain system operations. We present a meta-reinforcement learning approach that quickly adapts its control policy to changing conditions. The approach builds upon model-agnostic meta learning (MAML). The controller maintains a complement of prior policies learned under system faults. This "library" is evaluated on a system after a new fault to initialize the new policy. This contrasts with MAML, where the controller derives intermediate policies anew, sampled from a distribution of similar systems, to initialize a new policy. Our approach improves sample efficiency of the reinforcement learning process. We evaluate our approach on an aircraft fuel transfer system under abrupt faults.

cs.LG↗

arXiv2020

Comparison of Model Predictive and Reinforcement Learning Methods for Fault Tolerant Control

A desirable property in fault-tolerant controllers is adaptability to system changes as they evolve during systems operations. An adaptive controller does not require optimal control policies to be enumerated for possible faults. Instead it can approximate one in real-time. We present two adaptive fault-tolerant control schemes for a discrete time system based on hierarchical reinforcement learning. We compare their performance against a model predictive controller in presence of sensor noise and persistent faults. The controllers are tested on a fuel tank model of a C-130 plane. Our experiments demonstrate that reinforcement learning-based controllers perform more robustly than model predictive controllers under faults, partially observable system models, and varying sensor noise levels.

eess.SY↗

arXiv2020

Fault-Tolerant Control of Degrading Systems with On-Policy Reinforcement Learning

We propose a novel adaptive reinforcement learning control approach for fault tolerant control of degrading systems that is not preceded by a fault detection and diagnosis step. Therefore, \textit{a priori} knowledge of faults that may occur in the system is not required. The adaptive scheme combines online and offline learning of the on-policy control method to improve exploration and sample efficiency, while guaranteeing stable learning. The offline learning phase is performed using a data-driven model of the system, which is frequently updated to track the system's operating conditions. We conduct experiments on an aircraft fuel transfer system to demonstrate the effectiveness of our approach.

eess.SY↗

arXiv2020

A Relearning Approach to Reinforcement Learning for Control of Smart Buildings

This paper demonstrates that continual relearning of control policies using incremental deep reinforcement learning (RL) can improve policy learning for non-stationary processes. We demonstrate this approach for a data-driven 'smart building environment' that we use as a test-bed for developing HVAC controllers for reducing energy consumption of large buildings on our university campus. The non-stationarity in building operations and weather patterns makes it imperative to develop control strategies that are adaptive to changing conditions. On-policy RL algorithms, such as Proximal Policy Optimization (PPO) represent an approach for addressing this non-stationarity, but exploration on the actual system is not an option for safety-critical systems. As an alternative, we develop an incremental RL technique that simultaneously reduces building energy consumption without sacrificing overall comfort. We compare the performance of our incremental RL controller to that of a static RL controller that does not implement the relearning function. The performance of the static controller diminishes significantly over time, but the relearning controller adjusts to changing conditions while ensuring comfort and optimal energy performance.

cs.LG↗

arXiv2013

Using the Dempster-Shafer Scheme in a Diagnostic Expert System Shell

This paper discusses an expert system shell that integrates rule-based reasoning and the Dempster-Shafer evidence combination scheme. Domain knowledge is stored as rules with associated belief functions. The reasoning component uses a combination of forward and backward inferencing mechanisms to allow interaction with users in a mixed-initiative format.

cs.AI↗