arXiv subjects

Rahul Jain

Publications and source records attributed to Rahul Jain.

At least 91 records · Page 5Linked to original sources

arXiv2022

Online Bayesian Optimization for Beam Alignment in the SECAR Recoil Mass Separator

The SEparator for CApture Reactions (SECAR) is a next-generation recoil separator system at the Facility for Rare Isotope Beams (FRIB) designed for the direct measurement of capture reactions on unstable nuclei in inverse kinematics. To maximize the performance of the device, careful beam alignment to the central ion optical axis needs to be achieved. This can be difficult to attain through manual tuning by human operators without potentially leaving the system in a sub-optimal and irreproducible state. In this work, we present the first development of online Bayesian optimization with a Gaussian process model to tune an ion beam through a nuclear astrophysics recoil separator. We show that the method achieves small incoming angular deviations (0-1 mrad) in an efficient and reproducible manner that is at least 3 times faster than standard hand-tuning. This method is now routinely used for all separator tuning.

physics.ins-det↗

arXiv2021

Improved No-Regret Algorithms for Stochastic Shortest Path with Linear MDP

We introduce two new no-regret algorithms for the stochastic shortest path (SSP) problem with a linear MDP that significantly improve over the only existing results of (Vial et al., 2021). Our first algorithm is computationally efficient and achieves a regret bound $\widetilde{O}\left(\sqrt{d^3B_{\star}^2T_{\star} K}\right)$, where $d$ is the dimension of the feature space, $B_{\star}$ and $T_{\star}$ are upper bounds of the expected costs and hitting time of the optimal policy respectively, and $K$ is the number of episodes. The same algorithm with a slight modification also achieves logarithmic regret of order $O\left(\frac{d^3B_{\star}^4}{c_{\min}^2\text{gap}_{\min}}\ln^5\frac{dB_{\star} K}{c_{\min}} \right)$, where $\text{gap}_{\min}$ is the minimum sub-optimality gap and $c_{\min}$ is the minimum cost over all state-action pairs. Our result is obtained by developing a simpler and improved analysis for the finite-horizon approximation of (Cohen et al., 2021) with a smaller approximation error, which might be of independent interest. On the other hand, using variance-aware confidence sets in a global optimization problem, our second algorithm is computationally inefficient but achieves the first "horizon-free" regret bound $\widetilde{O}(d^{3.5}B_{\star}\sqrt{K})$ with no polynomial dependency on $T_{\star}$ or $1/c_{\min}$, almost matching the $Ω(dB_{\star}\sqrt{K})$ lower bound from (Min et al., 2021).

cs.LG↗

arXiv2021

Scheduling Flexible Non-Preemptive Loads in Smart-Grid Networks

A market consisting of a generator with thermal and renewable generation capability, a set of non-preemptive loads (i.e., loads which cannot be interrupted once started), and an independent system operator (ISO) is considered. Loads are characterized by durations, power demand rates and utility for receiving service, as well as disutility functions giving preferences for time slots in which service is preferred. Given this information, along with the generator's thermal generation cost function and forecast renewable generation, the social planner solves a mixed integer program to determine a load activation schedule which maximizes social welfare. Assuming price taking behavior, we develop a competitive equilibrium concept based on a relaxed version of the social planner's problem which includes prices for consumption and incentives for flexibility, and allows for probabilistic allocation of power to loads. Considering each load as representative of a population of identical loads with scaled characteristics, we demonstrate that the relaxed social planner's problem gives an exact solution to the original mixed integer problem in the large population limit, and give a market mechanism for implementing the competitive equilibrium. Finally, we evaluate via case study the benefit of incorporating load flexibility information into power consumption and generation scheduling in terms of proportion of loads served and overall social welfare.

eess.SY↗

arXiv2021

Implicit Finite-Horizon Approximation and Efficient Optimal Algorithms for Stochastic Shortest Path

We introduce a generic template for developing regret minimization algorithms in the Stochastic Shortest Path (SSP) model, which achieves minimax optimal regret as long as certain properties are ensured. The key of our analysis is a new technique called implicit finite-horizon approximation, which approximates the SSP model by a finite-horizon counterpart only in the analysis without explicit implementation. Using this template, we develop two new algorithms: the first one is model-free (the first in the literature to our knowledge) and minimax optimal under strictly positive costs; the second one is model-based and minimax optimal even with zero-cost state-action pairs, matching the best existing result from [Tarbouriech et al., 2021b]. Importantly, both algorithms admit highly sparse updates, making them computationally more efficient than all existing algorithms. Moreover, both can be made completely parameter-free.

cs.LG↗

arXiv2021

Model-Free Reinforcement Learning for Optimal Control of MarkovDecision Processes Under Signal Temporal Logic Specifications

We present a model-free reinforcement learning algorithm to find an optimal policy for a finite-horizon Markov decision process while guaranteeing a desired lower bound on the probability of satisfying a signal temporal logic (STL) specification. We propose a method to effectively augment the MDP state space to capture the required state history and express the STL objective as a reachability objective. The planning problem can then be formulated as a finite-horizon constrained Markov decision process (CMDP). For a general finite horizon CMDP problem with unknown transition probability, we develop a reinforcement learning scheme that can leverage any model-free RL algorithm to provide an approximately optimal policy out of the general space of non-stationary randomized policies. We illustrate the effectiveness of our approach in the context of robotic motion planning for complex missions under uncertainty and performance objectives.

eess.SY↗

arXiv2021

Dynamic Meta-theorems for Distance and Matching

Reachability, distance, and matching are some of the most fundamental graph problems that have been of particular interest in dynamic complexity theory in recent years [DKMSZ18, DMVZ18, DKMTVZ20]. Reachability can be maintained with first-order update formulas, or equivalently in DynFO in general graphs with n nodes [DKMSZ18], even under O(log n/loglog n) changes per step [DMVZ18]. In the context of how large the number of changes can be handled, it has recently been shown [DKMTVZ20] that under a polylogarithmic number of changes, reachability is in DynFOpar in planar, bounded treewidth, and related graph classes -- in fact in any graph where small non-zero circulation weights can be computed in NC. We continue this line of investigation and extend the meta-theorem for reachability to distance and bipartite maximum matching with the same bounds. These are amongst the most general classes of graphs known where we can maintain these problems deterministically without using a majority quantifier and even maintain witnesses. For the bipartite matching result, modifying the approach from [FGT], we convert the static non-zero circulation weights to dynamic matching-isolating weights. While reachability is in DynFOar under O(log n/loglog n) changes, no such bound is known for either distance or matching in any non-trivial class of graphs under non-constant changes. We show that, in the same classes of graphs as before, bipartite maximum matching is in DynFOar under O(log n/loglog n) changes per step. En route to showing this we prove that the rank of a matrix can be maintained in DynFOar, also under O(log n/loglog n) entry changes, improving upon the previous O(1) bound [DKMSZ18]. This implies similar extension for the non-uniform DynFO bound for maximum matching in general graphs and an alternate algorithm for maintaining reachability under O(log n/loglog n) changes [DMVZ18].

cs.LO↗

arXiv2021

Online Learning for Cooperative Multi-Player Multi-Armed Bandits

We introduce a framework for decentralized online learning for multi-armed bandits (MAB) with multiple cooperative players. The reward obtained by the players in each round depends on the actions taken by all the players. It's a team setting, and the objective is common. Information asymmetry is what makes the problem interesting and challenging. We consider three types of information asymmetry: action information asymmetry when the actions of the players can't be observed but the rewards received are common; reward information asymmetry when the actions of the other players are observable but rewards received are IID from the same distribution; and when we have both action and reward information asymmetry. For the first setting, we propose a UCB-inspired algorithm that achieves $O(\log T)$ regret whether the rewards are IID or Markovian. For the second section, we offer an environment such that the algorithm given for the first setting gives linear regret. For the third setting, we show that a variation of the `explore then commit' algorithm achieves almost log regret.

cs.LG↗

arXiv2021

Reachability and Matching in Single Crossing Minor Free Graphs

We show that for each single crossing graph $H$, a polynomially bounded weight function for all $H$-minor free graphs $G$ can be constructed in Logspace such that it gives nonzero weights to all the cycles in $G$. This class of graphs subsumes almost all classes of graphs for which such a weight function is known to be constructed in Logspace. As a consequence, we obtain that for the class of $H$-minor free graphs where $H$ is a single crossing graph, reachability can be solved in UL, and bipartite maximum matching can be solved in SPL, which are small subclasses of the parallel complexity class NC. In the restrictive case of bipartite graphs, our maximum matching result improves upon the recent result of Eppstein and Vazirani, where they show an NC bound for constructing perfect matching in general single crossing minor free graphs.

cs.CC↗

arXiv2021

Space-Efficient Algorithms for Reachability in Geometric Graphs

The problem of graph Reachability is to decide whether there is a path from one vertex to another in a given graph. In this paper, we study the Reachability problem on three distinct graph families - intersection graphs of Jordan regions, unit contact disk graphs (penny graphs), and chordal graphs. For each of these graph families, we present space-efficient algorithms for the Reachability problem. For intersection graphs of Jordan regions, we show how to obtain a "good" vertex separator in a space-efficient manner and use it to solve the Reachability in polynomial time and $O(m^{1/2}\log n)$ space, where $n$ is the number of Jordan regions, and $m$ is the total number of crossings among the regions. We use a similar approach for chordal graphs and obtain a polynomial-time and $O(m^{1/2}\log n)$ space algorithm, where $n$ and $m$ are the number of vertices and edges, respectively. However, we use a more involved technique for unit contact disk graphs (penny graphs) and obtain a better algorithm. We show that for every $ε> 0$, there exists a polynomial-time algorithm that can solve Reachability in an $n$ vertex directed penny graph, using $O(n^{1/4+ε})$ space. We note that the method used to solve penny graphs does not extend naturally to the class of geometric intersection graphs that include arbitrary size cliques.

cs.CC↗

arXiv2021

Designing Interpretable Approximations to Deep Reinforcement Learning

In an ever expanding set of research and application areas, deep neural networks (DNNs) set the bar for algorithm performance. However, depending upon additional constraints such as processing power and execution time limits, or requirements such as verifiable safety guarantees, it may not be feasible to actually use such high-performing DNNs in practice. Many techniques have been developed in recent years to compress or distill complex DNNs into smaller, faster or more understandable models and controllers. This work seeks to identify reduced models that not only preserve a desired performance level, but also, for example, succinctly explain the latent knowledge represented by a DNN. We illustrate the effectiveness of the proposed approach on the evaluation of decision tree variants and kernel machines in the context of benchmark reinforcement learning tasks.

cs.LG↗

arXiv2021

Two-Stage Electricity Markets with Renewable Energy Integration: Market Mechanisms and Equilibrium Analysis

We consider a two-stage market mechanism for trading electricity including renewable generation as an alternative to the widely used multi-settlement market structure. The two-stage market structure allows for recourse decisions by the market operator, which are not possible in today's markets. We allow for different conventional generation cost curves in the forward and the real-time stages. We have considered costs of demand response programs and black outs, and adopt a DC power flow model to account for network constraints. Our first result is to show existence (by construction) of a sequential competitive equilibrium (SCEq) in such a two-stage market. We argue social welfare properties of such an SCEq, and then design a market mechanism that achieves social welfare maximization when the market participants are non-strategic. We also show that under either a congestion-free or a monopoly-free condition, an efficient Nash equilibrium exists.

cs.GT↗

arXiv2021

Online Learning for Unknown Partially Observable MDPs

Solving Partially Observable Markov Decision Processes (POMDPs) is hard. Learning optimal controllers for POMDPs when the model is unknown is harder. Online learning of optimal controllers for unknown POMDPs, which requires efficient learning using regret-minimizing algorithms that effectively tradeoff exploration and exploitation, is even harder, and no solution exists currently. In this paper, we consider infinite-horizon average-cost POMDPs with unknown transition model, though a known observation model. We propose a natural posterior sampling-based reinforcement learning algorithm (PSRL-POMDP) and show that it achieves a regret bound of $O(\log T)$, where $T$ is the time horizon, when the parameter set is finite. In the general case (continuous parameter set), we show that the algorithm achieves $O (T^{2/3})$ regret under two technical assumptions. To the best of our knowledge, this is the first online RL algorithm for POMDPs and has sub-linear regret.

cs.LG↗

arXiv2021

Online Learning for Stochastic Shortest Path Model via Posterior Sampling

We consider the problem of online reinforcement learning for the Stochastic Shortest Path (SSP) problem modeled as an unknown MDP with an absorbing state. We propose PSRL-SSP, a simple posterior sampling-based reinforcement learning algorithm for the SSP problem. The algorithm operates in epochs. At the beginning of each epoch, a sample is drawn from the posterior distribution on the unknown model dynamics, and the optimal policy with respect to the drawn sample is followed during that epoch. An epoch completes if either the number of visits to the goal state in the current epoch exceeds that of the previous epoch, or the number of visits to any of the state-action pairs is doubled. We establish a Bayesian regret bound of $O(B_\star S\sqrt{AK})$, where $B_\star$ is an upper bound on the expected cost of the optimal policy, $S$ is the size of the state space, $A$ is the size of the action space, and $K$ is the number of episodes. The algorithm only requires the knowledge of the prior distribution, and has no hyper-parameters to tune. It is the first such posterior sampling algorithm and outperforms numerically previously proposed optimism-based algorithms.

cs.LG↗

arXiv2021

Noisy quantum state redistribution with promise and the Alpha-bit

We consider a variation of the well-studied quantum state redistribution task, in which the starting state is known only to the receiver Bob and not to the sender Alice. We refer to this as quantum state redistribution with a one-sided promise. In addition, we consider communication from Alice to Bob over a noisy channel $\mathcal{N}$, instead of the noiseless channel, as is usually considered in state redistribution. We take a natural approach towards the solution of this problem where we "embed" the promise as part of the state and then invoke known protocols for quantum state redistribution composed with known protocols for transfer of quantum information over noisy channels. Using our approach, we are able to reproduce the Alpha-bit capacities with or without entanglement assistance in Ref. [ArXiv:1706.09434], using known protocols for quantum state redistribution and quantum communication over noisy channels. Furthermore, we generalize the entanglement assisted classical Alpha-bit capacity, showing that any quantum state redistribution protocol can be used as a black box to simulate classical communication.

quant-ph↗

arXiv2021

Learning Infinite-horizon Average-reward MDPs with Linear Function Approximation

We develop several new algorithms for learning Markov Decision Processes in an infinite-horizon average-reward setting with linear function approximation. Using the optimism principle and assuming that the MDP has a linear structure, we first propose a computationally inefficient algorithm with optimal $\widetilde{O}(\sqrt{T})$ regret and another computationally efficient variant with $\widetilde{O}(T^{3/4})$ regret, where $T$ is the number of interactions. Next, taking inspiration from adversarial linear bandits, we develop yet another efficient algorithm with $\widetilde{O}(\sqrt{T})$ regret under a different set of assumptions, improving the best existing result by Hao et al. (2020) with $\widetilde{O}(T^{2/3})$ regret. Moreover, we draw a connection between this algorithm and the Natural Policy Gradient algorithm proposed by Kakade (2002), and show that our analysis improves the sample complexity bound recently given by Agarwal et al. (2020).

cs.LG↗

arXiv2021

Optimal communication and control strategies in a multi-agent MDP problem

The problem of controlling multi-agent systems under different models of information sharing among agents has received significant attention in the recent literature. In this paper, we consider a setup where rather than committing to a fixed information sharing protocol (e.g. periodic sharing or no sharing etc), agents can dynamically decide at each time step whether to share information with each other and incur the resulting communication cost. This setup requires a joint design of agents' communication and control strategies in order to optimize the trade-off between communication costs and control objective. We first show that agents can ignore a big part of their private information without compromising the system performance. We then provide a common information approach based solution for the strategy optimization problem. This approach relies on constructing a fictitious POMDP whose solution (obtained via a dynamic program) characterizes the optimal strategies for the agents. We also show that our solution can be easily modified to incorporate constraints on when and how frequently agents can communicate.

math.OC↗

arXiv2021

Synthesis of Discounted-Reward Optimal Policies for Markov Decision Processes Under Linear Temporal Logic Specifications

We present a method to find an optimal policy with respect to a reward function for a discounted Markov decision process under general linear temporal logic (LTL) specifications. Previous work has either focused on maximizing a cumulative reward objective under finite-duration tasks, specified by syntactically co-safe LTL, or maximizing an average reward for persistent (e.g., surveillance) tasks. This paper extends and generalizes these results by introducing a pair of occupancy measures to express the LTL satisfaction objective and the expected discounted reward objective, respectively. These occupancy measures are then connected to a single policy via a novel reduction resulting in a mixed integer linear program whose solution provides an optimal policy. Our formulation can also be extended to include additional constraints with respect to secondary reward functions. We illustrate the effectiveness of our approach in the context of robotic motion planning for complex missions under uncertainty and performance objectives.

eess.SY↗

arXiv2020

A Model-free Learning Algorithm for Infinite-horizon Average-reward MDPs with Near-optimal Regret

Recently, model-free reinforcement learning has attracted research attention due to its simplicity, memory and computation efficiency, and the flexibility to combine with function approximation. In this paper, we propose Exploration Enhanced Q-learning (EE-QL), a model-free algorithm for infinite-horizon average-reward Markov Decision Processes (MDPs) that achieves regret bound of $O(\sqrt{T})$ for the general class of weakly communicating MDPs, where $T$ is the number of interactions. EE-QL assumes that an online concentrating approximation of the optimal average reward is available. This is the first model-free learning algorithm that achieves $O(\sqrt T)$ regret without the ergodic assumption, and matches the lower bound in terms of $T$ except for logarithmic factors. Experiments show that the proposed algorithm performs as well as the best known model-based algorithms.

cs.LG↗