arXiv subjects
Publications and source records attributed to Rushikesh Kamalapurkar.
This paper develops a data-driven inverse reinforcement learning technique for a class of linear systems to estimate the cost function of an agent online, using input-output measurements. A simultaneous state and parameter estimator is utilized to facilitate output-feedback inverse reinforcement learning, and cost function estimation is achieved up to multiplication by a constant.
Online approximation of the optimal station keeping strategy for a fully actuated six degrees-of-freedom marine craft subject to an irrotational ocean current is considered. An approximate solution to the optimal control problem is obtained using an adaptive dynamic programming technique. The hydrodynamic drift dynamics of the dynamic model are assumed to be unknown; therefore, a concurrent learning-based system identifier is developed to identify the unknown model parameters. The identified model is used to implement an adaptive model-based reinforcement learning technique to estimate the unknown value function. The developed policy guarantees uniformly ultimately bounded convergence of the vehicle to the desired station and uniformly ultimately bounded convergence of the approximated policies to the optimal polices without the requirement of persistence of excitation. The developed strategy is validated using an autonomous underwater vehicle, where the three degrees-of-freedom in the horizontal plane are regulated. The experiments are conducted in a second-magnitude spring located in central Florida.
This paper generalizes the Lasalle-Yoshizawa Theorem to switched nonsmooth systems. Filippov and Krasovskii regularizations of a switched system are shown to be contained within the convex hull of the Filippov and Krasovskii regularizations of the subsystems, respectively. A candidate common Lyapunov function that has a negative semidefinite derivative along the trajectories of the subsystems is shown to be sufficient to establish LaSalle-Yoshizawa results for the switched system. Results for regular and non-regular candidate Lyapunov functions are presented using an appropriate generalization of the time derivative. The developed generalization is motivated by adaptive control of switched systems where the derivative of the candidate Lyapunov function is typically negative semidefinite.
This paper seeks to combine differential game theory with the actor-critic-identifier architecture to determine forward-in-time, approximate optimal controllers for formation tracking in multi-agent systems, where the agents have uncertain heterogeneous nonlinear dynamics. A continuous control strategy is proposed, using communication feedback from extended neighbors on a communication topology that has a spanning tree. A model-based reinforcement learning technique is developed to cooperatively control a group of agents to track a trajectory in a desired formation. Simulation results are presented to demonstrate the performance of the developed technique.
In this paper, a concurrent learning based adaptive observer is developed for a class of second-order linear time-invariant systems with uncertain system matrices. The developed technique yields an exponentially convergent state estimator and an exponentially convergent parameter estimator. As opposed to persistent excitation required for parameter convergence in traditional adaptive methods, excitation over a finite time-interval is sufficient for the developed technique to achieve exponential convergence. Simulation results in both noise-free and noisy environments are presented to validate the design.
Concurrent learning is a recently developed adaptive update scheme that can be used to guarantee parameter convergence without requiring persistent excitation. However, this technique requires knowledge of state derivatives, which are usually not directly sensed and therefore must be estimated. A novel integral concurrent learning method is developed in this paper that removes the need to estimate state derivatives while maintaining parameter convergence properties. A Monte Carlo simulation illustrates improved robustness to noise compared to the traditional derivative formulation.
A function approximation method is developed that aims to approximate a function in a small neighborhood of a state that travels within a compact set. The development is based on the theory of universal reproducing kernel Hilbert spaces over the $n$-dimensional Euclidean space. Several theorems are introduced that support the development of this State Following (StaF) method. In particular, it is shown that there is a bound on the number of kernel functions required for the maintenance of an accurate function approximation as a state moves through a compact set. Additionally, a weight update law, based on gradient descent, is introduced where arbitrarily close accuracy can be achieved provided the weight update law is iterated at a sufficient frequency, as detailed in Theorem 6.1. To illustrate the advantage, the impact of the StaF method is that for some applications the number of basis functions can be reduced. The StaF method is applied to an adaptive dynamic programming (ADP) application to demonstrate that stability is maintained with a reduced number of basis functions. Simulation results demonstrate the utility of the StaF methodology for the maintenance of accurate function approximation as well as solving an infinite horizon optimal regulation problem through ADP. The results of the simulation indicate that fewer basis functions are required to guarantee stability and approximate optimality than are required when a global approximation approach is used.
A concurrent learning (CL)-based parameter estimator is developed to identify the unknown parameters in a linearly parameterized uncertain control-affine nonlinear system. Unlike state-of-the-art CL techniques that assume knowledge of the state-derivative or rely on numerical smoothing, CL is implemented using a dynamic state-derivative estimator. A novel purging algorithm is introduced to discard possibly erroneous data recorded during the transient phase for concurrent learning. Since purging results in a discontinuous parameter adaptation law, the closed-loop error system is modeled as a switched system. Asymptotic convergence of the error states to the origin is established under a persistent excitation condition, and the error states are shown to be ultimately bounded under a finite excitation condition. Simulation results are provided to demonstrate the effectiveness of the developed parameter estimator.
This paper provides an approximate online adaptive solution to the infinite-horizon optimal tracking problem for control-affine continuous-time nonlinear systems with unknown drift dynamics. Model-based reinforcement learning is used to relax the persistence of excitation condition. Model-based reinforcement learning is implemented using a concurrent learning-based system identifier to simulate experience by evaluating the Bellman error over unexplored areas of the state space. Tracking of the desired trajectory and convergence of the developed policy to a neighborhood of the optimal policy are established via Lyapunov-based stability analysis. Simulation results demonstrate the effectiveness of the developed technique.
In this paper the infinite horizon optimal regulation problem is solved online for a deterministic control-affine nonlinear dynamical system using the state following (StaF) kernel method to approximate the value function. Unlike traditional methods that aim to approximate a function over a large compact set, the StaF kernel method aims to approximate a function in a small neighborhood of a state that travels within a compact set. Simulation results demonstrate that stability and approximate optimality of the control system can be achieved with significantly fewer basis functions than may be required for global approximation methods.
A robust controller is developed for uncertain, second-order nonlinear systems subject to simultaneous unknown, time-varying state delays and known, time-varying input delays in addition to additive, sufficiently smooth disturbances. An integral term composed of previous control values facilitates a delay-free open-loop error system and the development of the feedback control structure. A stability analysis based on Lyapunov-Krasovskii (LK) functionals guarantees uniformly ultimately bounded tracking under the assumption that the delays are bounded and slowly varying.
This paper presents a concurrent learning-based actor-critic-identifier architecture to obtain an approximate feedback-Nash equilibrium solution to an infinite horizon N-player nonzero-sum differential game online, without requiring persistence of excitation (PE), for a nonlinear control-affine system. Under a condition milder than PE, uniformly ultimately bounded convergence of the developed control policies to the feedback-Nash equilibrium policies is established.
Approximate dynamic programming has been investigated and used as a method to approximately solve optimal regulation problems. However, the extension of this technique to optimal tracking problems for continuous time nonlinear systems has remained a non-trivial open problem. The control development in this paper guarantees ultimately bounded tracking of a desired trajectory, while also ensuring that the controller converges to an approximate optimal policy.
Online approximation of an infinite horizon optimal path-following strategy for a kinematic unicycle is considered. The solution to the optimal control problem is approximated using an approximate dynamic programming technique that uses concurrent-learning-based adaptive update laws to estimate the unknown value function. The developed controller overcomes challenges with the approximation of the infinite horizon value function using an auxiliary function that describes the motion of a virtual target on the desired path. The developed controller guarantees uniformly ultimately bounded (UUB) convergence of the vehicle to a desired path while maintaining a desired speed profile and UUB convergence of the approximate policy to the optimal policy. Simulation results are included to demonstrate the controller's performance.
In deterministic systems, reinforcement learning-based online approximate optimal control methods typically require a restrictive persistence of excitation (PE) condition for convergence. This paper presents a concurrent learning-based solution to the online approximate optimal regulation problem that eliminates the need for PE. The development is based on the observation that given a model of the system, the Bellman error, which quantifies the deviation of the system Hamiltonian from the optimal Hamiltonian, can be evaluated at any point in the state space. Further, a concurrent learning-based parameter identifier is developed to compensate for parametric uncertainty in the plant dynamics. Uniformly ultimately bounded (UUB) convergence of the system states to the origin, and UUB convergence of the developed policy to the optimal policy are established using a Lyapunov-based analysis, and simulations are performed to demonstrate the performance of the developed controller.
Efforts in this paper seek to combine graph theory with adaptive dynamic programming (ADP) as a reinforcement learning (RL) framework to determine forward-in-time, real-time, approximate optimal controllers for distributed multi-agent systems with uncertain nonlinear dynamics. A decentralized continuous time-varying control strategy is proposed, using only local communication feedback from two-hop neighbors on a communication topology that has a spanning tree. An actor-critic-identifier architecture is proposed that employs a nonlinear state derivative estimator to estimate the unknown dynamics online and uses the estimate thus obtained for value function approximation.