Searcharxiv⌕ Search

arXiv subjects

Ajinkya Bhole

Publications and source records attributed to Ajinkya Bhole.

3 recordsLinked to original sources

KL-Regularized Policy Iteration for Adaptive LQR: Gain Updates and Probing from One Policy Improvement

Policy-gradient methods for adaptive linear quadratic control improve the feedback gain of an unknown plant online by gradient-type steps computed from closed-loop data. A recent framework of such methods, policy-gradient adaptive control (PGAC), takes one gradient-type step per sample, which its indirect form computes from an estimated model and its direct form directly from the data. Because data generated by a fixed linear feedback cannot identify the plant, PGAC adds a probing perturbation to the input and leaves its covariance as a design parameter. We study the adaptive LQR problem through a KL-regularized formulation, in which the probing is the randomness of a Gaussian policy kept close to a Gaussian reference policy, so that the cost of the probing is part of the objective. One policy improvement of this formulation gives a gain update together with a probing covariance. We show that, for suitable references centred at the current gain, this gain update is exactly PGAC's Gauss-Newton or natural-gradient update with any step size below a bound. We repeat this policy improvement, each time against a tempered copy of the current policy, and refer to this scheme as KL-regularized policy iteration (KLPI). On a known plant and with tempering, the cost of KLPI's gains converges linearly to the optimal cost, and the probing covariance keeps a floor proportional to the regularization weight. On an estimated plant, under conditions like PGAC's (small updates and sufficiently informative data), the closed loop stays stable, the cost of the gains converges linearly up to a bias, and the probing covariance again keeps this floor. In simulations at equal probing cost, the shape of KLPI's probing covariance, which PGAC can adopt, lowers the extra cost caused by suboptimal gains on nearly all tested plants, whereas KLPI's use of past iterations in its gain update changes this cost little.

math.OC↗

Unifying Entropy Regularization in Optimal Control: From and Back to Classical Objectives via Iterated Soft Policies and Path Integral Solutions

This paper develops a unified perspective on several optimal control formulations through the lens of Kullback-Leibler (KL) regularization. We propose a central problem that separates the KL penalties on policies and transitions with independent weights, thus generalizing the standard trajectory-level KL-regularization used in probabilistic optimal control. This umbrella formulation recovers various control problems: the classical Stochastic Optimal Control (SOC), Risk-Sensitive Stochastic Optimal Control (RSOC), and their policy-based KL-regularized counterparts, termed soft-policy SOC and RSOC, which yield tractable surrogates. Beyond being regularized variants, these soft-policy formulations majorize the original SOC and RSOC, thus, iterating their solutions recovers the original objectives. We further identify a synchronized case of soft-policy RSOC where the policy and transition KL weights coincide, yielding a linear Bellman operator, path-integral solution, and compositionality -- extending these computationally favourable properties to a broad class of control problems.

math.OC↗

Design of a Robust Stair Climbing Compliant Modular Robot to Tackle Overhang on Stairs

This paper discusses the concept and parameter design of a Robust Stair Climbing Compliant Modular Robot, capable of tackling stairs with overhangs. Modifying the geometry of the periphery of the wheels of our robot helps in tackling overhangs. Along with establishing a concept design, robust design parameters are set to minimize performance variation. The Grey-based Taguchi Method is adopted for providing an optimal setting for the design parameters of the robot. The robot prototype is shown to have successfully scaled stairs of varying dimensions, with overhang, thus corroborating the analysis performed.

cs.RO↗