arXiv subjects

Aaron Kandel

Publications and source records attributed to Aaron Kandel.

6 recordsLinked to original sources

arXiv2025

Reinforcement Learning-based Approach for Vehicle-to-Building Charging with Heterogeneous Agents and Long Term Rewards

Strategic aggregation of electric vehicle batteries as energy reservoirs can optimize power grid demand, benefiting smart and connected communities, especially large office buildings that offer workplace charging. This involves optimizing charging and discharging to reduce peak energy costs and net peak demand, monitored over extended periods (e.g., a month), which involves making sequential decisions under uncertainty and delayed and sparse rewards, a continuous action space, and the complexity of ensuring generalization across diverse conditions. Existing algorithmic approaches, e.g., heuristic-based strategies, fall short in addressing real-time decision-making under dynamic conditions, and traditional reinforcement learning (RL) models struggle with large state-action spaces, multi-agent settings, and the need for long-term reward optimization. To address these challenges, we introduce a novel RL framework that combines the Deep Deterministic Policy Gradient approach (DDPG) with action masking and efficient MILP-driven policy guidance. Our approach balances the exploration of continuous action spaces to meet user charging demands. Using real-world data from a major electric vehicle manufacturer, we show that our approach comprehensively outperforms many well-established baselines and several scalable heuristic approaches, achieving significant cost savings while meeting all charging requirements. Our results show that the proposed approach is one of the first scalable and general approaches to solving the V2B energy management challenge.

cs.LG↗

arXiv2021

Distributionally Robust Surrogate Optimal Control for High-Dimensional Systems

This paper presents a novel methodology for tractably solving optimal control and offline reinforcement learning problems for high-dimensional systems. This work is motivated by the ongoing challenges of safety, computation, and optimality in high-dimensional optimal control. We address these key questions with the following approach. First, we identify a sequence-modeling surrogate methodology which takes as input the initial state and a time series of control inputs, and outputs an approximation of the objective function and trajectories of constraint functions. Importantly this approach entirely absorbs the individual state transition dynamics. The sole dependence on the initial state means we can apply dimensionality reduction to compress the model input while retaining most of its information. Uncertainty in the surrogate objective will affect the result optimality. Critically, however, uncertainty in the surrogate constraint functions will lead to infeasibility, i.e. unsafe actions. When considering offline reinforcement learning, the most significant modeling error will be encountered on out-of-distribution data. Therefore, we apply Wasserstein ambiguity sets to ``robustify'' our surrogate modeling approach subject to worst-case out-of-sample modeling error based on the distribution of test data residuals. We demonstrate the efficacy of this combined approach through a case study of safe optimal fast charging of a high-dimensional lithium-ion battery model at low temperatures.

math.OC↗

arXiv2020

Fuel and Time Optimal USV Trajectory Planning under Flexible Refueling Constraints

This paper addresses the problem of trajectory optimization for an unmanned surface vehicle while considering direction-dependent ocean currents and flexible refueling constraints. This work is motivated by the rising interest in developing autonomous navigation technology for commercial, scientific, and military applications for ocean-bound unmanned surface vehicles. Relevant literature on such vehicles has addressed energy-efficient trajectory optimization and time-efficient trajectory optimization. However, the application of trajectory optimization techniques which include refueling stops and multi-objective optimization remains relatively unexplored. We address this open challenge by formulating the trajectory design problem as a nonconvex mixed-integer optimization program with a multi-objective cost function. Then, we apply dynamic programming to solve this optimization program for fuel and time optimal trajectories. We synthesize these results into a series of Pareto fronts which demonstrates the tradeoff between fuel consumption and trip time for a prototypical route with direction-dependent ocean currents. Furthermore, the optimal trajectories for which we solve illustrate the changes in the vehicle's behavior as the fuel level becomes low, and as the vehicle encounters counterproductive ocean currents. Our results indicate several meaningful insights about the overall trajectory optimization process.

eess.SY↗

arXiv2020

Fisher Identifiability Analysis of Longitudinal Vehicle Dynamics

This paper investigates the theoretical Cramer-Rao bounds on estimation accuracy of longitudinal vehicle dynamics parameters. This analysis is motivated by the value of parameter estimation in various applications, including chassis model validation and active safety. Relevant literature addresses this demand through algorithms capable of estimating chassis parameters for diverse conditions. While the implementation of such algorithms has been studied, the question of fundamental limits on their accuracy remains largely unexplored. We address this question by presenting two contributions. First, this paper presents theoretical findings which reveal the prevailing effects underpinning vehicle chassis parameter identifiability. We then validate these findings with data from on-road experiments. Our results demonstrate, among a variety of effects, the strong relevance of road grade variability in determining parameter identifiability from a drive cycle. These findings can motivate improved experimental designs in the future.

eess.SY↗

arXiv2020

Safe Learning MPC with Limited Model Knowledge and Data

This paper presents an end-to-end framework for safe learning-based control (LbC) using nonlinear stochastic MPC and distributionally robust optimization (DRO). This work is motivated by several open challenges in LbC literature. In particular, many control-theoretic LbC methods require subject matter expertise in order to translate their own safety guarantees, often manifested as preexisting data of safe trajectories or structural model knowledge. In this paper, we focus on LbC where the controller is applied directly to a system of which it has no or extremely limited direct experience, towards safety during \textit{tabula-rasa} or ``\textit{blank slate''} model-based learning and control as a challenging case for validation. This explores the boundary of the status-quo in control theory relating to requirements for subject matter expertise. We show under basic and limited assumptions on the underlying problem, we can translate probabilistic guarantees on feasibility to nonlinear systems using results in stochastic MPC and DRO literature whose relevance we formally extend in a mathematical analysis. We also present a coupled and intuitive formulation for persistence of excitation (PoE), and illustrate the connection between PoE and applicability of the proposed method. Our case studies of vehicle obstacle avoidance and safe extreme fast charging of lithium-ion batteries reveal powerful empirical results supporting the underlying DRO theory. Our method is widely applicable within the LbC domain to, for example, airborne wind energy systems, vehicle obstacle avoidance, and energy storage systems management. It is also applicable to quantifying uncertainty beyond the LbC case.

eess.SY↗

arXiv2020

Safe Wasserstein Constrained Deep Q-Learning

This paper presents a distributionally robust Q-Learning algorithm (DrQ) which leverages Wasserstein ambiguity sets to provide idealistic probabilistic out-of-sample safety guarantees during online learning. First, we follow past work by separating the constraint functions from the principal objective to create a hierarchy of machines which estimate the feasible state-action space within the constrained Markov decision process (CMDP). DrQ works within this framework by augmenting constraint costs with tightening offset variables obtained through Wasserstein distributionally robust optimization (DRO). These offset variables correspond to worst-case distributions of modeling error characterized by the TD-errors of the constraint Q-functions. This procedure allows us to safely approach the nominal constraint boundaries. Using a case study of lithium-ion battery fast charging, we explore how idealistic safety guarantees translate to generally improved safety relative to conventional methods.

cs.LG↗