arXiv subjects

David Lindner

Publications and source records attributed to David Lindner.

41 records · Page 3Linked to original sources

arXiv2021

Learning What To Do by Simulating the Past

Since reward functions are hard to specify, recent work has focused on learning policies from human feedback. However, such approaches are impeded by the expense of acquiring such feedback. Recent work proposed that agents have access to a source of information that is effectively free: in any environment that humans have acted in, the state will already be optimized for human preferences, and thus an agent can extract information about what humans want from the state. Such learning is possible in principle, but requires simulating all possible past trajectories that could have led to the observed state. This is feasible in gridworlds, but how do we scale it to complex tasks? In this work, we show that by combining a learned feature encoder with learned inverse models, we can enable agents to simulate human actions backwards in time to infer what they must have done. The resulting algorithm is able to reproduce a specific skill in MuJoCo environments given a single state sampled from the optimal policy for that skill.

cs.LG↗

arXiv2021

Challenges for Using Impact Regularizers to Avoid Negative Side Effects

Designing reward functions for reinforcement learning is difficult: besides specifying which behavior is rewarded for a task, the reward also has to discourage undesired outcomes. Misspecified reward functions can lead to unintended negative side effects, and overall unsafe behavior. To overcome this problem, recent work proposed to augment the specified reward function with an impact regularizer that discourages behavior that has a big impact on the environment. Although initial results with impact regularizers seem promising in mitigating some types of side effects, important challenges remain. In this paper, we examine the main current challenges of impact regularizers and relate them to fundamental design decisions. We discuss in detail which challenges recent approaches address and which remain unsolved. Finally, we explore promising directions to overcome the unsolved challenges in preventing negative side effects with impact regularizers.

cs.LG↗

arXiv2019

Detecting Spiky Corruption in Markov Decision Processes

Current reinforcement learning methods fail if the reward function is imperfect, i.e. if the agent observes reward different from what it actually receives. We study this problem within the formalism of Corrupt Reward Markov Decision Processes (CRMDPs). We show that if the reward corruption in a CRMDP is sufficiently "spiky", the environment is solvable. We fully characterize the regret bound of a Spiky CRMDP, and introduce an algorithm that is able to detect its corrupt states. We show that this algorithm can be used to learn the optimal policy with any common reinforcement learning algorithm. Finally, we investigate our algorithm in a pair of simple gridworld environments, finding that our algorithm can detect the corrupt states and learn the optimal policy despite the corruption.

cs.LG↗

arXiv2019

Sensing Social Media Signals for Cryptocurrency News

The ability to track and monitor relevant and important news in real-time is of crucial interest in multiple industrial sectors. In this work, we focus on the set of cryptocurrency news, which recently became of emerging interest to the general and financial audience. In order to track relevant news in real-time, we (i) match news from the web with tweets from social media, (ii) track their intraday tweet activity and (iii) explore different machine learning models for predicting the number of the article mentions on Twitter within the first 24 hours after its publication. We compare several machine learning models, such as linear extrapolation, linear and random forest autoregressive models, and a sequence-to-sequence neural network. We find that the random forest autoregressive model behaves comparably to more complex models in the majority of tasks.

cs.SI↗

arXiv2018

Topological semimetals and insulators in three-dimensional honeycomb materials

Semimetals, in which conduction and valence bands touch but do not form Fermi surfaces, have attracted considerable interest for their anomalous properties starting with the discovery of Dirac matter in graphene and other two-dimensional honeycomb materials. Here we introduce a family of three-dimensional honeycomb systems whose electronic band structures exhibit a variety of topological semimetals with Dirac nodal lines. We show that these nodal lines appear in varying numbers and mutual geometries, depending on the underlying lattice structure. They are stabilized, in most cases, by a combination of time-reversal and inversion symmetries and are accompanied by topologically protected "drumhead" surface states. In the bulk, these nodal line systems exhibit Landau level quantization and flat bands upon applying a magnetic field. In the presence of spin-orbit coupling, these topological semimetals are found to generically form (strong) topological insulators. This comprehensive classification of the electronic band structures of three-dimensional honeycomb systems might serve as guidance for future material synthesis.

cond-mat.str-el↗