arXiv subjects

Jonathan Lee

Publications and source records attributed to Jonathan Lee.

36 records · Page 2Linked to original sources

arXiv2022

Online Learning with Continuous Variations: Dynamic Regret and Reductions

Online learning is a powerful tool for analyzing iterative algorithms. However, the classic adversarial setup sometimes fails to capture certain regularity in online problems in practice. Motivated by this, we establish a new setup, called Continuous Online Learning (COL), where the gradient of online loss function changes continuously across rounds with respect to the learner's decisions. We show that COL covers and more appropriately describes many interesting applications, from general equilibrium problems (EPs) to optimization in episodic MDPs. In particular, we show monotone EPs admits a reduction to achieving sublinear static regret in COL. Using this new setup, we revisit the difficulty of sublinear dynamic regret. We prove a fundamental equivalence between achieving sublinear dynamic regret in COL and solving certain EPs. With this insight, we offer conditions for efficient algorithms that achieve sublinear dynamic regret, even when the losses are chosen adaptively without any a priori variation budget. Furthermore, we show for COL a reduction from dynamic regret to both static regret and convergence in the associated EP, allowing us to analyze the dynamic regret of many existing algorithms.

cs.LG↗

arXiv2022

Electric dipole moments and the search for new physics

Static electric dipole moments of nondegenerate systems probe mass scales for physics beyond the Standard Model well beyond those reached directly at high energy colliders. Discrimination between different physics models, however, requires complementary searches in atomic-molecular-and-optical, nuclear and particle physics. In this report, we discuss the current status and prospects in the near future for a compelling suite of such experiments, along with developments needed in the encompassing theoretical framework.

hep-ph↗

arXiv2021

Pricing and Energy Trading in Peer-to-peer Zero Marginal-cost Microgrids

Efforts to utilize 100% renewable energy in community microgrids require new approaches to energy markets and transactions to efficiently address periods of scarce energy supply. In this paper we contribute to the promising approach of peer-to-peer (P2P) energy trading in two main ways: analysis of a centralized, welfare-maximizing economic dispatch that characterizes optimal price and allocations, and a novel P2P system for negotiating energy trades that yields physically feasible and at least weakly Pareto-optimal outcomes. Our main results are 1) that optimal pricing is insufficient to induce agents with batteries to take optimal actions, 2) a novel P2P algorithm to address this while keeping private information, 3) a formal proof that this algorithm converges to the centralized solution in the case of two agents negotiating for a single period, and 4)numerical simulations of the P2P algorithm performance with up to 10 agents and 24 periods that show it converges on average to total welfare within 0.1% of the social optimum in on the order of 10s to 100s of iterations, increasing with the number of agents, time periods, and total storage capacity.

math.OC↗

arXiv2021

Design of Experiments for Stochastic Contextual Linear Bandits

In the stochastic linear contextual bandit setting there exist several minimax procedures for exploration with policies that are reactive to the data being acquired. In practice, there can be a significant engineering overhead to deploy these algorithms, especially when the dataset is collected in a distributed fashion or when a human in the loop is needed to implement a different policy. Exploring with a single non-reactive policy is beneficial in such cases. Assuming some batch contexts are available, we design a single stochastic policy to collect a good dataset from which a near-optimal policy can be extracted. We present a theoretical analysis as well as numerical experiments on both synthetic and real-world datasets.

cs.LG↗

arXiv2021

Near Optimal Policy Optimization via REPS

Since its introduction a decade ago, \emph{relative entropy policy search} (REPS) has demonstrated successful policy learning on a number of simulated and real-world robotic domains, not to mention providing algorithmic components used by many recently proposed reinforcement learning (RL) algorithms. While REPS is commonly known in the community, there exist no guarantees on its performance when using stochastic and gradient-based solvers. In this paper we aim to fill this gap by providing guarantees and convergence rates for the sub-optimality of a policy learned using first-order optimization methods applied to the REPS objective. We first consider the setting in which we are given access to exact gradients and demonstrate how near-optimality of the objective translates to near-optimality of the policy. We then consider the practical setting of stochastic gradients, and introduce a technique that uses \emph{generative} access to the underlying Markov decision process to compute parameter updates that maintain favorable convergence to the optimal regularized policy.

cs.LG↗

arXiv2020

Is Q-Learning Provably Efficient? An Extended Analysis

This work extends the analysis of the theoretical results presented within the paper Is Q-Learning Provably Efficient? by Jin et al. We include a survey of related research to contextualize the need for strengthening the theoretical guarantees related to perhaps the most important threads of model-free reinforcement learning. We also expound upon the reasoning used in the proofs to highlight the critical steps leading to the main result showing that Q-learning with UCB exploration achieves a sample efficiency that matches the optimal regret that can be achieved by any model-based approach.

cs.LG↗

arXiv2020

On-Policy Robot Imitation Learning from a Converging Supervisor

Existing on-policy imitation learning algorithms, such as DAgger, assume access to a fixed supervisor. However, there are many settings where the supervisor may evolve during policy learning, such as a human performing a novel task or an improving algorithmic controller. We formalize imitation learning from a "converging supervisor" and provide sublinear static and dynamic regret guarantees against the best policy in hindsight with labels from the converged supervisor, even when labels during learning are only from intermediate supervisors. We then show that this framework is closely connected to a class of reinforcement learning (RL) algorithms known as dual policy iteration (DPI), which alternate between training a reactive learner with imitation learning and a model-based supervisor with data from the learner. Experiments suggest that when this framework is applied with the state-of-the-art deep model-based RL algorithm PETS as an improving supervisor, it outperforms deep RL baselines on continuous control tasks and provides up to an 80-fold speedup in policy evaluation.

cs.LG↗

arXiv2019

Continuous Online Learning and New Insights to Online Imitation Learning

Online learning is a powerful tool for analyzing iterative algorithms. However, the classic adversarial setup sometimes fails to capture certain regularity in online problems in practice. Motivated by this, we establish a new setup, called Continuous Online Learning (COL), where the gradient of online loss function changes continuously across rounds with respect to the learner's decisions. We show that COL covers and more appropriately describes many interesting applications, from general equilibrium problems (EPs) to optimization in episodic MDPs. Using this new setup, we revisit the difficulty of achieving sublinear dynamic regret. We prove that there is a fundamental equivalence between achieving sublinear dynamic regret in COL and solving certain EPs, and we present a reduction from dynamic regret to both static regret and convergence rate of the associated EP. At the end, we specialize these new insights into online imitation learning and show improved understanding of its learning stability.

cs.LG↗

arXiv2019

Phonon and Thermal Properties of Quasi-Two-Dimensional FePS3 and MnPS3 Antiferromagnetic Semiconductor Materials

We report results of investigation of the phonon and thermal properties of the exfoliated films of layered single crystals of antiferromagnetic FePS3 and MnPS3 semiconductors. The Raman spectroscopy was conducted using three different excitation lasers with the wavelengths of 325 nm (UV), 488 nm (blue), and 633 nm (red). The resonant UV-Raman spectroscopy reveals new spectral features, which are not detectable via visible Raman light scattering. The thermal conductivity of FePS3 and MnPS3 thin films was measured by two different techniques: the steady-state Raman optothermal and transient time-resolved magneto-optical Kerr effect. The Raman optothermal measurements provided the orientation-average thermal conductivity of FePS3 to be 1.35 W/mK at room temperature. The transient measurements revealed that the through-plane and in-plane thermal conductivity of FePS3 is 0.85 W/mK and 2.7 W/mK, respectively. The films of MnPS3 have higher thermal conductivity of 1.1 W/mK through-plane and 6.3 W/mK in-plane. The data obtained by both techniques reveal strong thermal anisotropy of the films and the dominant contribution of phonons to heat conduction. Our results are important for the proposed applications of the antiferromagnetic semiconductor thin films in spintronic devices.

cond-mat.mtrl-sci↗

arXiv2018

Generalizing Robot Imitation Learning with Invariant Hidden Semi-Markov Models

Generalizing manipulation skills to new situations requires extracting invariant patterns from demonstrations. For example, the robot needs to understand the demonstrations at a higher level while being invariant to the appearance of the objects, geometric aspects of objects such as its position, size, orientation and viewpoint of the observer in the demonstrations. In this paper, we propose an algorithm that learns a joint probability density function of the demonstrations with invariant formulations of hidden semi-Markov models to extract invariant segments (also termed as sub-goals or options), and smoothly follow the generated sequence of states with a linear quadratic tracking controller. The algorithm takes as input the demonstrations with respect to different coordinate systems describing virtual landmarks or objects of interest with a task-parameterized formulation, and adapt the segments according to the environmental changes in a systematic manner. We present variants of this algorithm in latent space with low-rank covariance decompositions, semi-tied covariances, and non-parametric online estimation of model parameters under small variance asymptotics; yielding considerably low sample and model complexity for acquiring new manipulation skills. The algorithm allows a Baxter robot to learn a pick-and-place task while avoiding a movable obstacle based on only 4 kinesthetic demonstrations.

cs.RO↗

arXiv2018

Constraint Estimation and Derivative-Free Recovery for Robot Learning from Demonstrations

Learning from human demonstrations can facilitate automation but is risky because the execution of the learned policy might lead to collisions and other failures. Adding explicit constraints to avoid unsafe states is generally not possible when the state representations are complex. Furthermore, enforcing these constraints during execution of the learned policy can be challenging in environments where dynamics are difficult to model such as push mechanics in grasping. In this paper, we propose Derivative-Free Recovery (DFR), a two-phase method for generating robust policies from demonstrations in robotic manipulation tasks where the system comes to rest at each time step. In the first phase, we use support estimation of supervisor demonstrations and treat the support as implicit constraints on states. We also propose a time-varying modification for sequential tasks. In the second phase, we use this support estimate to derive a switching policy that employs the learned policy in the interior of the support and switches to a recovery policy to steer the robot away from the boundary of the support if it drifts too close. We present additional conditions, which linearly bound the difference in state at each time step by the magnitude of control, allowing us to prove that the robot will not violate the constraints using the recovery policy. A simulated pushing task in MuJoCo suggests that DFR can reduce collisions by 83\%. On a physical line tracking task using a da Vinci Surgical Robot and a moving Stewart platform, DFR reduced collisions by 84\%.

cs.RO↗

arXiv2018

Rigorous Analysis of a Randomised Number Field Sieve

Factorisation of integers $n$ is of number theoretic and cryptographic significance. The Number Field Sieve (NFS) introduced circa 1990, is still the state of the art algorithm, but no rigorous proof that it halts or generates relationships is known. We propose and analyse an explicitly randomised variant. For each $n$, we show that these randomised variants of the NFS and Coppersmith's multiple polynomial sieve find congruences of squares in expected times matching the best-known heuristic estimates.

math.NT↗

arXiv2017

DART: Noise Injection for Robust Imitation Learning

One approach to Imitation Learning is Behavior Cloning, in which a robot observes a supervisor and infers a control policy. A known problem with this "off-policy" approach is that the robot's errors compound when drifting away from the supervisor's demonstrations. On-policy, techniques alleviate this by iteratively collecting corrective actions for the current robot policy. However, these techniques can be tedious for human supervisors, add significant computation burden, and may visit dangerous states during training. We propose an off-policy approach that injects noise into the supervisor's policy while demonstrating. This forces the supervisor to demonstrate how to recover from errors. We propose a new algorithm, DART (Disturbances for Augmenting Robot Trajectories), that collects demonstrations with injected noise, and optimizes the noise level to approximate the error of the robot's trained policy during data collection. We compare DART with DAgger and Behavior Cloning in two domains: in simulation with an algorithmic supervisor on the MuJoCo tasks (Walker, Humanoid, Hopper, Half-Cheetah) and in physical experiments with human supervisors training a Toyota HSR robot to perform grasping in clutter. For high dimensional tasks like Humanoid, DART can be up to $3x$ faster in computation time and only decreases the supervisor's cumulative reward by $5\%$ during training, whereas DAgger executes policies that have $80\%$ less cumulative reward than the supervisor. On the grasping in clutter task, DART obtains on average a $62\%$ performance increase over Behavior Cloning.

cs.LG↗

arXiv2017

Coalescence on the real line

We study a geometrically constrained coalescence model derived from spin systems. Given two probability distributions $\mathbb{P}_R$ and $\mathbb{P}_B$ on the positive reals with finite means, colour the real line alternately with red and blue intervals so that the lengths of the red intervals have distribution $\mathbb{P}_R$, the lengths of the blue intervals have distribution $\mathbb{P}_B$, and distinct intervals have independent lengths. Now, iteratively update this colouring of the line by coalescing intervals: change the colour of any interval that is surrounded by longer intervals so that these three consecutive intervals subsequently form a single monochromatic interval. We say that a colour (either red or blue) wins if every point of the line is eventually of that colour. Holroyd, in 2011, asked the following question: under what natural conditions on the initial distributions is one of the colours almost surely guaranteed to win? It turns out that the answer to this question can be quite counter-intuitive due to the non-monotone dynamics of the model. In this paper, we investigate various notions of advantage one of the colours might initially possess, and in the course of doing so, we determine which of the two colours emerges victorious for various nontrivial pairs of distributions.

math.PR↗

arXiv2017

Line percolation

We study a new geometric bootstrap percolation model, line percolation, on the $d$-dimensional integer grid $[n]^d$. In line percolation with infection parameter $r$, infection spreads from a subset $A\subset [n]^d$ of initially infected lattice points as follows: if there exists an axis-parallel line $L$ with $r$ or more infected lattice points on it, then every lattice point of $[n]^d$ on $L$ gets infected, and we repeat this until the infection can no longer spread. The elements of the set $A$ are usually chosen independently, with some density $p$, and the main question is to determine $p_c(n,r,d)$, the density at which percolation (infection of the entire grid) becomes likely. In this paper, we determine $p_c(n,r,2)$ up to a multiplicative factor of $1+o(1)$ and $p_c(n,r,3)$ up to a multiplicative constant as $n\rightarrow \infty$ for every fixed $r\in \mathbb{N}$. We also determine the size of the minimal percolating sets in all dimensions and for all values of the infection parameter.

math.PR↗

arXiv2017

Comparing Human-Centric and Robot-Centric Sampling for Robot Deep Learning from Demonstrations

Motivated by recent advances in Deep Learning for robot control, this paper considers two learning algorithms in terms of how they acquire demonstrations. "Human-Centric" (HC) sampling is the standard supervised learning algorithm, where a human supervisor demonstrates the task by teleoperating the robot to provide trajectories consisting of state-control pairs. "Robot-Centric" (RC) sampling is an increasingly popular alternative used in algorithms such as DAgger, where a human supervisor observes the robot executing a learned policy and provides corrective control labels for each state visited. RC sampling can be challenging for human supervisors and prone to mislabeling. RC sampling can also induce error in policy performance because it repeatedly visits areas of the state space that are harder to learn. Although policies learned with RC sampling can be superior to HC sampling for standard learning models such as linear SVMs, policies learned with HC sampling may be comparable with highly-expressive learning models such as deep learning and hyper-parametric decision trees, which have little model error. We compare HC and RC using a grid world and a physical robot singulation task, where in the latter the input is a binary image of a connected set of objects on a planar worksurface and the policy generates a motion of the gripper to separate one object from the rest. We observe in simulation that for linear SVMs, policies learned with RC outperformed those learned with HC but that with deep models this advantage disappears. We also find that with RC, the corrective control labels provided by humans can be highly inconsistent. We prove there exists a class of examples where in the limit, HC is guaranteed to converge to an optimal policy while RC may fail to converge.

cs.RO↗

arXiv2016

Eigenvalues of subgraphs of the cube

We consider the problem of maximising the largest eigenvalue of subgraphs of the hypercube $Q_d$ of a given order. We believe that in most cases, Hamming balls are maximisers, and our results support this belief. We show that the Hamming balls of radius $o(d)$ have largest eigenvalue that is within $1 + o(1)$ of the maximum value. We also prove that Hamming balls with fixed radius maximise the largest eigenvalue exactly, rather than asymptotically, when $d$ is sufficiently large. Our proofs rely on the method of compressions.

math.CO↗

arXiv2012

Enumerating regular expressions and their languages

In this chapter we discuss the problem of enumerating distinct regular expressions by size and the regular languages they represent. We discuss various notions of the size of a regular expression that appear in the literature and their advantages and disadvantages. We consider a formal definition of regular expressions using a context-free grammar. We then show how to enumerate strings generated by an unambiguous context-free grammar using the Chomsky-Schützenberger theorem. This theorem allows one to construct an algebraic equation whose power series expansion provides the enumeration. Classical tools from complex analysis, such as singularity analysis, can then be used to determine the asymptotic behavior of the enumeration. We use these algebraic and analytic methods to obtain asymptotic estimates on the number of regular expressions of size n. A single regular language can often be described by several regular expressions, and we estimate the number of distinct languages denoted by regular expressions of size n. We also give asymptotic estimates for these quantities. For the first few values, we provide exact enumeration results.

cs.FL↗