arXiv subjects

Jeffrey Tsang

Publications and source records attributed to Jeffrey Tsang.

4 recordsLinked to original sources

arXiv2017

Hybrid Reward Architecture for Reinforcement Learning

One of the main challenges in reinforcement learning (RL) is generalisation. In typical deep RL methods this is achieved by approximating the optimal value function with a low-dimensional representation using a deep network. While this approach works well in many domains, in domains where the optimal value function cannot easily be reduced to a low-dimensional representation, learning can be very slow and unstable. This paper contributes towards tackling such challenging domains, by proposing a new method, called Hybrid Reward Architecture (HRA). HRA takes as input a decomposed reward function and learns a separate value function for each component reward function. Because each component typically only depends on a subset of all features, the corresponding value function can be approximated more easily by a low-dimensional representation, enabling more effective learning. We demonstrate HRA on a toy-problem and the Atari game Ms. Pac-Man, where HRA achieves above-human performance.

cs.LG↗

arXiv2014

Taking all positive eigenvectors is suboptimal in classical multidimensional scaling

It is hard to overstate the importance of multidimensional scaling as an analysis technique in the broad sciences. Classical, or Torgerson multidimensional scaling is one of the main variants, with the advantage that it has a closed-form analytic solution. However, this solution is exact if and only if the distances are Euclidean. Conversely, there has been comparatively little discussion on what to do in the presence of negative eigenvalues: the intuitive solution, prima facie justifiable in least-squares terms, is to take every positive eigenvector as a dimension. We show that this, minimizing least-squares to the centred distances instead of the true distances, is suboptimal - throwing away positive eigenvectors can decrease the error even as we project to fewer dimensions. We provide provably better methods for handling this common case.

math.ST↗

arXiv2014

The parametrized probabilistic finite-state transducer probe game player fingerprint model

Fingerprinting operators generate functional signatures of game players and are useful for their automated analysis independent of representation or encoding. The theory for a fingerprinting operator which returns the length-weighted probability of a given move pair occurring from playing the investigated agent against a general parametrized probabilistic finite-state transducer (PFT) is developed, applicable to arbitrary iterated games. Results for the distinguishing power of the 1-state opponent model, uniform approximability of fingerprints of arbitrary players, analyticity and Lipschitz continuity of fingerprints for logically possible players, and equicontinuity of the fingerprints of bounded-state probabilistic transducers are derived. Algorithms for the efficient computation of special instances are given; the shortcomings of a previous model, strictly generalized here from a simple projection of the new model, are explained in terms of regularity condition violations, and the extra power and functional niceness of the new fingerprints demonstrated. The 2-state deterministic finite-state transducers (DFTs) are fingerprinted and pairwise distances computed; using this the structure of DFTs in strategy space is elucidated.

cs.GT↗

arXiv2014

N derivatives are necessary for order N+1 convergence in quadrature: a converse result

Results on the error bounds of quadrature methods are well known - most state that if the method has degree N, and the integrand has N derivatives, then the error is order N+1. We prove here a converse: that if the integrand fails to have N derivatives, even only at a finite number of points, no method, regardless of its degree, can guarantee convergence more than order N. Even if the integrand fails to have N derivatives at just 3 (for even N, 2) points, no method can produce order more than N+1 convergence. This is done by an adversarial proof: we explicitly construct the functions that exhibit such error; simple splines turn out to suffice.

math.NA↗