arXiv subjects

Simon Schmitt

Publications and source records attributed to Simon Schmitt.

At least 19 recordsLinked to original sources

arXiv2025

General Uncertainty Estimation with Delta Variances

Decision makers may suffer from uncertainty induced by limited data. This may be mitigated by accounting for epistemic uncertainty, which is however challenging to estimate efficiently for large neural networks. To this extent we investigate Delta Variances, a family of algorithms for epistemic uncertainty quantification, that is computationally efficient and convenient to implement. It can be applied to neural networks and more general functions composed of neural networks. As an example we consider a weather simulator with a neural-network-based step function inside -- here Delta Variances empirically obtain competitive results at the cost of a single gradient computation. The approach is convenient as it requires no changes to the neural network architecture or training procedure. We discuss multiple ways to derive Delta Variances theoretically noting that special cases recover popular techniques and present a unified perspective on multiple related methods. Finally we observe that this general perspective gives rise to a natural extension and empirically show its benefit.

cs.LG↗

arXiv2023

Exploration via Epistemic Value Estimation

How to efficiently explore in reinforcement learning is an open problem. Many exploration algorithms employ the epistemic uncertainty of their own value predictions -- for instance to compute an exploration bonus or upper confidence bound. Unfortunately the required uncertainty is difficult to estimate in general with function approximation. We propose epistemic value estimation (EVE): a recipe that is compatible with sequential decision making and with neural network function approximators. It equips agents with a tractable posterior over all their parameters from which epistemic value uncertainty can be computed efficiently. We use the recipe to derive an epistemic Q-Learning agent and observe competitive performance on a series of benchmarks. Experiments confirm that the EVE recipe facilitates efficient exploration in hard exploration tasks.

cs.LG↗

arXiv2022

Phase sensitive quantum spectroscopy with high frequency resolution

Classical sensors for spectrum analysis are widely used but lack micro- or nanoscale spatial resolution. On the other hand, quantum sensors, capable of working with nanoscale precision, do not provide precise frequency resolution over a wide range of frequencies. Using a single spin in diamond, we present a measurement protocol for quantum probes which enables full signal reconstruction on a nanoscale spatial resolution up to potentially 100\,GHz. We achieve $58\,\mathrm{nT/\sqrt{Hz}}$ amplitude and $0.095\,\mathrm{rad/\sqrt{Hz}}$ phase sensitivity and a relative frequency uncertainty of $10^{-12}$ for a $1.51\,\mathrm{GHz}$ signal within $10\,\mathrm{s}$ of integration. This technique opens the way to quantum spectrum analysis methods with potential applications in electron spin detection and nanocircuitry in quantum technologies.

quant-ph↗

arXiv2022

Muesli: Combining Improvements in Policy Optimization

We propose a novel policy update that combines regularized policy optimization with model learning as an auxiliary loss. The update (henceforth Muesli) matches MuZero's state-of-the-art performance on Atari. Notably, Muesli does so without using deep search: it acts directly with a policy network and has computation speed comparable to model-free baselines. The Atari results are complemented by extensive ablations, and by additional results on continuous control and 9x9 Go.

cs.LG↗

arXiv2022

Chaining Value Functions for Off-Policy Learning

To accumulate knowledge and improve its policy of behaviour, a reinforcement learning agent can learn `off-policy' about policies that differ from the policy used to generate its experience. This is important to learn counterfactuals, or because the experience was generated out of its own control. However, off-policy learning is non-trivial, and standard reinforcement-learning algorithms can be unstable and divergent. In this paper we discuss a novel family of off-policy prediction algorithms which are convergent by construction. The idea is to first learn on-policy about the data-generating behaviour, and then bootstrap an off-policy value estimate on this on-policy estimate, thereby constructing a value estimate that is partially off-policy. This process can be repeated to build a chain of value functions, each time bootstrapping a new estimate on the previous estimate in the chain. Each step in the chain is stable and hence the complete algorithm is guaranteed to be stable. Under mild conditions this comes arbitrarily close to the off-policy TD solution when we increase the length of the chain. Hence it can compute the solution even in cases where off-policy TD diverges. We prove that the proposed scheme is convergent and corresponds to an iterative decomposition of the inverse key matrix. Furthermore it can be interpreted as estimating a novel objective -- that we call a `k-step expedition' -- of following the target policy for finitely many steps before continuing indefinitely with the behaviour policy. Empirically we evaluate the idea on challenging MDPs such as Baird's counter example and observe favourable results.

cs.LG↗

arXiv2021

A Network Control Theory Approach to Longitudinal Symptom Dynamics in Major Depressive Disorder

Background: The evolution of symptoms over time is at the heart of understanding and treating mental disorders. However, a principled, quantitative framework explaining symptom dynamics remains elusive. Here, we propose a Network Control Theory of Psychopathology allowing us to formally derive a theoretical control energy which we hypothesize quantifies resistance to future symptom improvement in Major Depressive Disorder (MDD). We test this hypothesis and investigate the relation to genetic and environmental risk as well as resilience. Methods: We modelled longitudinal symptom-network dynamics derived from N=2,059 Beck Depression Inventory measurements acquired over a median of 134 days in a sample of N=109 patients suffering from MDD. We quantified the theoretical energy required for each patient and time-point to reach a symptom-free state given individual symptom-network topology (E 0 ) and 1) tested if E 0 predicts future symptom improvement and 2) whether this relationship is moderated by Polygenic Risk Scores (PRS) of mental disorders, childhood maltreatment experience, and self-reported resilience. Outcomes: We show that E 0 indeed predicts symptom reduction at the next measurement and reveal that this coupling between E 0 and future symptom change increases with higher genetic risk and childhood maltreatment while it decreases with resilience. Interpretation: Our study provides a mechanistic framework capable of predicting future symptom improvement based on individual symptom-network topology and clarifies the role of genetic and environmental risk as well as resilience. Our control-theoretic framework makes testable, quantitative predictions for individual therapeutic response and provides a starting-point for the theory-driven design of personalized interventions. Funding: German Research Foundation and Interdisciplinary Centre for Clinical Research, Münster

eess.SY↗

arXiv2021

Learning and Planning in Complex Action Spaces

Many important real-world problems have action spaces that are high-dimensional, continuous or both, making full enumeration of all possible actions infeasible. Instead, only small subsets of actions can be sampled for the purpose of policy evaluation and improvement. In this paper, we propose a general framework to reason in a principled way about policy evaluation and improvement over such sampled action subsets. This sample-based policy iteration framework can in principle be applied to any reinforcement learning algorithm based upon policy iteration. Concretely, we propose Sampled MuZero, an extension of the MuZero algorithm that is able to learn in domains with arbitrarily complex action spaces by planning over sampled actions. We demonstrate this approach on the classical board game of Go and on two continuous control benchmark domains: DeepMind Control Suite and Real-World RL Suite.

cs.LG↗

arXiv2020

AlgebraNets

Neural networks have historically been built layerwise from the set of functions in ${f: \mathbb{R}^n \to \mathbb{R}^m }$, i.e. with activations and weights/parameters represented by real numbers, $\mathbb{R}$. Our work considers a richer set of objects for activations and weights, and undertakes a comprehensive study of alternative algebras as number representations by studying their performance on two challenging problems: large-scale image classification using the ImageNet dataset and language modeling using the enwiki8 and WikiText-103 datasets. We denote this broader class of models as AlgebraNets. Our findings indicate that the conclusions of prior work, which explored neural networks constructed from $\mathbb{C}$ (complex numbers) and $\mathbb{H}$ (quaternions) on smaller datasets, do not always transfer to these challenging settings. However, our results demonstrate that there are alternative algebras which deliver better parameter and computational efficiency compared with $\mathbb{R}$. We consider $\mathbb{C}$, $\mathbb{H}$, $M_{2}(\mathbb{R})$ (the set of $2\times2$ real-valued matrices), $M_{2}(\mathbb{C})$, $M_{3}(\mathbb{R})$ and $M_{4}(\mathbb{R})$. Additionally, we note that multiplication in these algebras has higher compute density than real multiplication, a useful property in situations with inherently limited parameter reuse such as auto-regressive inference and sparse neural networks. We therefore investigate how to induce sparsity within AlgebraNets. We hope that our strong results on large-scale, practical benchmarks will spur further exploration of these unconventional architectures which challenge the default choice of using real numbers for neural network weights and activations.

cs.LG↗

arXiv2020

Gated Linear Networks

This paper presents a new family of backpropagation-free neural architectures, Gated Linear Networks (GLNs). What distinguishes GLNs from contemporary neural networks is the distributed and local nature of their credit assignment mechanism; each neuron directly predicts the target, forgoing the ability to learn feature representations in favor of rapid online learning. Individual neurons can model nonlinear functions via the use of data-dependent gating in conjunction with online convex optimization. We show that this architecture gives rise to universal learning capabilities in the limit, with effective model capacity increasing as a function of network size in a manner comparable with deep ReLU networks. Furthermore, we demonstrate that the GLN learning mechanism possesses extraordinary resilience to catastrophic forgetting, performing comparably to a MLP with dropout and Elastic Weight Consolidation on standard benchmarks. These desirable theoretical and empirical properties position GLNs as a complementary technique to contemporary offline deep learning methods.

cs.LG↗

arXiv2020

Finding polynomial roots by dynamical systems -- a case study

We investigate two well known dynamical systems that are designed to find roots of univariate polynomials by iteration: the methods known by Newton and by Ehrlich-Aberth. Both are known to have found all roots of high degree polynomials with good complexity. Our goal is to determine in which cases which of the two algorithms is more efficient. We come to the conclusion that Newton is faster when the polynomials are given by recursion so they can be evaluated in logarithmic time with respect to the degree, or when all the roots are all near the boundary of their convex hull. Conversely, Ehrlich-Aberth has the advantage when no fast evaluation of the polynomials is available, and when roots are in the interior of the convex hull of other roots.

math.NA↗

arXiv2020

Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model

Constructing agents with planning capabilities has long been one of the main challenges in the pursuit of artificial intelligence. Tree-based planning methods have enjoyed huge success in challenging domains, such as chess and Go, where a perfect simulator is available. However, in real-world problems the dynamics governing the environment are often complex and unknown. In this work we present the MuZero algorithm which, by combining a tree-based search with a learned model, achieves superhuman performance in a range of challenging and visually complex domains, without any knowledge of their underlying dynamics. MuZero learns a model that, when applied iteratively, predicts the quantities most directly relevant to planning: the reward, the action-selection policy, and the value function. When evaluated on 57 different Atari games - the canonical video game environment for testing AI techniques, in which model-based planning approaches have historically struggled - our new algorithm achieved a new state of the art. When evaluated on Go, chess and shogi, without any knowledge of the game rules, MuZero matched the superhuman performance of the AlphaZero algorithm that was supplied with the game rules.

cs.LG↗

arXiv2019

Off-Policy Actor-Critic with Shared Experience Replay

We investigate the combination of actor-critic reinforcement learning algorithms with uniform large-scale experience replay and propose solutions for two challenges: (a) efficient actor-critic learning with experience replay (b) stability of off-policy learning where agents learn from other agents behaviour. We employ those insights to accelerate hyper-parameter sweeps in which all participating agents run concurrently and share their experience via a common replay module. To this end we analyze the bias-variance tradeoffs in V-trace, a form of importance sampling for actor-critic methods. Based on our analysis, we then argue for mixing experience sampled from replay with on-policy experience, and propose a new trust region scheme that scales effectively to data distributions where V-trace becomes unstable. We provide extensive empirical validation of the proposed solution. We further show the benefits of this setup by demonstrating state-of-the-art data efficiency on Atari among agents trained up until 200M environment frames.

cs.LG↗

arXiv2019

Spin-Layer- and Spin-Valley-Locking in CVD-Grown AA'- and AB-Stacked Tungsten-Disulfide Bilayers

Valley-selective optical selection rules and a spin-valley locking in transition-metal dichalcogenide (TMDC) monolayers are at the heart of "valleytronic physics", which exploits the valley degree of freedom and has been a major research topic in recent years. In contrast, valleytronic properties of TMDC bilayers have not been in the focus so much by now. Here, we report on the valleytronic properties and optical characterization of bilayers of WS2 as a representative TMDC material. In particular, we study the influence of the relative layer alignment in TMDC homo-bilayer samples on their polarization-dependent optical properties. Therefore, CVD-grown WS2 bilayer samples have been prepared that favor either the inversion symmetric AA' stacking or AB stacking without inversion symmetry during synthesis. Subsequently, a detailed analysis of reflection contrast and photoluminescence spectra under different polarization conditions has been performed. We observe circular and linear dichroism of the photoluminescence that is more pronounced for the AB stacking configuration. Our experimental findings are supported by theoretical calculations showing that the observed dichroism can be linked to optical selection rules, that maintain the spin-valley locking in the AB-stacked WS2 bilayer, whereas a spin-layer-locking is present the inversion symmetric AA' bilayer instead. Furthermore, our theoretical calculations predict a small relative shift of the excitonic resonances in both stacking configurations, which is also experimentally observed.

cond-mat.mes-hall↗

arXiv2019

Randomisation of Pulse Phases for Unambiguous and Robust Quantum Sensing

We develop theoretically and demonstrate experimentally a universal dynamical decoupling method for robust quantum sensing with unambiguous signal identification. Our method uses randomisation of control pulses to suppress simultaneously two types of errors in the measured spectra that would otherwise lead to false signal identification. These are spurious responses due to finite-width $π$ pulses, as well as signal distortion caused by $π$ pulse imperfections. For the cases of nanoscale nuclear spin sensing and AC magnetometry, we benchmark the performance of the protocol with a single nitrogen vacancy centre in diamond against widely used non-randomised pulse sequences. Our method is general and can be combined with existing multipulse quantum sensing sequences to enhance their performance.

quant-ph↗

arXiv2019

Limits on Spectral Resolution Measurements by Quantum Probes

The limits of frequency resolution in nano-NMR experiments have been discussed extensively in recent years. It is believed that there is a crucial difference between the ability to resolve a few frequencies and the precision of estimating a single one. Whereas the efficiency of single frequency estimation gradually increases with the square root of the number of measurements, the ability to resolve two frequencies is limited by the specific timescale of the signal and cannot be compensated for by extra measurements. Here we show theoretically and demonstrate experimentally that the relationship between these quantities is more subtle and both are only limited by the Cramér-Rao bound of a single frequency estimation.

quant-ph↗

arXiv2018

Multi-task Deep Reinforcement Learning with PopArt

The reinforcement learning community has made great strides in designing algorithms capable of exceeding human performance on specific tasks. These algorithms are mostly trained one task at the time, each new task requiring to train a brand new agent instance. This means the learning algorithm is general, but each solution is not; each agent can only solve the one task it was trained on. In this work, we study the problem of learning to master not one but multiple sequential-decision tasks at once. A general issue in multi-task learning is that a balance must be found between the needs of multiple tasks competing for the limited resources of a single learning system. Many learning algorithms can get distracted by certain tasks in the set of tasks to solve. Such tasks appear more salient to the learning process, for instance because of the density or magnitude of the in-task rewards. This causes the algorithm to focus on those salient tasks at the expense of generality. We propose to automatically adapt the contribution of each task to the agent's updates, so that all tasks have a similar impact on the learning dynamics. This resulted in state of the art performance on learning to play all games in a set of 57 diverse Atari games. Excitingly, our method learned a single trained policy - with a single set of weights - that exceeds median human performance. To our knowledge, this was the first time a single agent surpassed human-level performance on this multi-task domain. The same approach also demonstrated state of the art performance on a set of 30 tasks in the 3D reinforcement learning platform DeepMind Lab.

cs.LG↗

arXiv2018

Magnetic-field-learning using a single electronic spin in diamond with one-photon-readout at room temperature

Nitrogen-vacancy (NV) centres in diamond are appealing nano-scale quantum sensors for temperature, strain, electric fields and, most notably, for magnetic fields. However, the cryogenic temperatures required for low-noise single-shot readout that have enabled the most sensitive NV-magnetometry reported to date, are impractical for key applications, e.g. biological sensing. Overcoming the noisy readout at room-temperature has until now demanded repeated collection of fluorescent photons, which increases the time-cost of the procedure thus reducing its sensitivity. Here we show how machine learning can process the noisy readout of a single NV centre at room-temperature, requiring on average only one photon per algorithm step, to sense magnetic field strength with a precision comparable to those reported for cryogenic experiments. Analysing large data sets from NV centres in bulk diamond, we report absolute sensitivities of $60$ nT s$^{1/2}$ including initialisation, readout, and computational overheads. We show that dephasing times can be simultaneously estimated, and that time-dependent fields can be dynamically tracked at room temperature. Our results dramatically increase the practicality of early-term single spin sensors.

quant-ph↗

arXiv2018

Kickstarting Deep Reinforcement Learning

We present a method for using previously-trained 'teacher' agents to kickstart the training of a new 'student' agent. To this end, we leverage ideas from policy distillation and population based training. Our method places no constraints on the architecture of the teacher or student agents, and it regulates itself to allow the students to surpass their teachers in performance. We show that, on a challenging and computationally-intensive multi-task benchmark (DMLab-30), kickstarted training improves the data efficiency of new agents, making it significantly easier to iterate on their design. We also show that the same kickstarting pipeline can allow a single student agent to leverage multiple 'expert' teachers which specialize on individual tasks. In this setting kickstarting yields surprisingly large gains, with the kickstarted agent matching the performance of an agent trained from scratch in almost 10x fewer steps, and surpassing its final performance by 42 percent. Kickstarting is conceptually simple and can easily be incorporated into reinforcement learning experiments.

cs.LG↗