arXiv subjects

Ronald Ortner

Publications and source records attributed to Ronald Ortner.

17 recordsLinked to original sources

arXiv2025

On cubic vertex-transitive graphs of given girth

A set of vertices of a graph is distinguishing if the only automorphism that preserves it is the identity. The minimal size of such sets, if they exist, is the distinguishing cost. The distinguishing costs of vertex transitive cubic graphs are well known if they are 1-arc-transitive, or if they have two edge orbits and either have girth 3 or vertex-stabilizers of order 1 or 2. There are many results about vertex-transitive cubic graphs of girth 4 with two edge orbits, but for larger girth almost nothing is known about %the existence or the distinguishing costs of such graphs. We prove that cubic vertex-transitive graphs of girth 5 with two edge orbits have distinguishing cost 2, and prove the non-existence of infinite 3-arc-transitive cubic graphs of girth 6.

math.CO↗

arXiv2024

A Note on the Bias and Kemeny's Constant in Markov Reward Processes with an Application to Markov Chain Perturbation

Given a unichain Markov reward process (MRP), we provide an explicit expression for the bias values in terms of mean first passage times. This result implies a generalization of known Markov chain perturbation bounds for the stationary distribution to the case where the perturbed chain is not irreducible. It further yields an improved perturbation bound in 1-norm. As a special case, Kemeny's constant can be interpreted as the translated bias in an MRP with constant reward 1, which offers an intuitive explanation why it is a constant.

math.PR↗

arXiv2022

Transfer in Reinforcement Learning via Regret Bounds for Learning Agents

We present an approach for the quantification of the usefulness of transfer in reinforcement learning via regret bounds for a multi-agent setting. Considering a number of $\aleph$ agents operating in the same Markov decision process, however possibly with different reward functions, we consider the regret each agent suffers with respect to an optimal policy maximizing her average reward. We show that when the agents share their observations the total regret of all agents is smaller by a factor of $\sqrt{\aleph}$ compared to the case when each agent has to rely on the information collected by herself. This result demonstrates how considering the regret in multi-agent settings can provide theoretical bounds on the benefit of sharing observations in transfer learning.

cs.LG↗

arXiv2019

Autonomous exploration for navigating in non-stationary CMPs

We consider a setting in which the objective is to learn to navigate in a controlled Markov process (CMP) where transition probabilities may abruptly change. For this setting, we propose a performance measure called exploration steps which counts the time steps at which the learner lacks sufficient knowledge to navigate its environment efficiently. We devise a learning meta-algorithm, MNM and prove an upper bound on the exploration steps in terms of the number of changes.

cs.LG↗

arXiv2019

Variational Regret Bounds for Reinforcement Learning

We consider undiscounted reinforcement learning in Markov decision processes (MDPs) where both the reward functions and the state-transition probabilities may vary (gradually or abruptly) over time. For this problem setting, we propose an algorithm and provide performance guarantees for the regret evaluated against the optimal non-stationary policy. The upper bound on the regret is given in terms of the total variation in the MDP. This is the first variational regret bound for the general reinforcement learning setting.

cs.LG↗

arXiv2019

Regret Bounds for Reinforcement Learning via Markov Chain Concentration

We give a simple optimistic algorithm for which it is easy to derive regret bounds of $\tilde{O}(\sqrt{t_{\rm mix} SAT})$ after $T$ steps in uniformly ergodic Markov decision processes with $S$ states, $A$ actions, and mixing time parameter $t_{\rm mix}$. These bounds are the first regret bounds in the general, non-episodic setting with an optimal dependence on all given parameters. They could only be improved by using an alternative mixing time parameter.

cs.LG↗

arXiv2018

Efficient Bias-Span-Constrained Exploration-Exploitation in Reinforcement Learning

We introduce SCAL, an algorithm designed to perform efficient exploration-exploitation in any unknown weakly-communicating Markov decision process (MDP) for which an upper bound $c$ on the span of the optimal bias function is known. For an MDP with $S$ states, $A$ actions and $Γ\leq S$ possible next states, we prove a regret bound of $\widetilde{O}(c\sqrt{ΓSAT})$, which significantly improves over existing algorithms (e.g., UCRL and PSRL), whose regret scales linearly with the MDP diameter $D$. In fact, the optimal bias span is finite and often much smaller than $D$ (e.g., $D=\infty$ in non-communicating MDPs). A similar result was originally derived by Bartlett and Tewari (2009) for REGAL.C, for which no tractable algorithm is available. In this paper, we relax the optimization problem at the core of REGAL.C, we carefully analyze its properties, and we provide the first computationally efficient algorithm to solve it. Finally, we report numerical simulations supporting our theoretical findings and showing how SCAL significantly outperforms UCRL in MDPs with large diameter and small span.

cs.LG↗

arXiv2018

A Sliding-Window Algorithm for Markov Decision Processes with Arbitrarily Changing Rewards and Transitions

We consider reinforcement learning in changing Markov Decision Processes where both the state-transition probabilities and the reward functions may vary over time. For this problem setting, we propose an algorithm using a sliding window approach and provide performance guarantees for the regret evaluated against the optimal non-stationary policy. We also characterize the optimal window size suitable for our algorithm. These results are complemented by a sample complexity bound on the number of sub-optimal steps taken by the algorithm. Finally, we present some experimental results to support our theoretical analysis.

cs.LG↗

arXiv2014

Selecting Near-Optimal Approximate State Representations in Reinforcement Learning

We consider a reinforcement learning setting introduced in (Maillard et al., NIPS 2011) where the learner does not have explicit access to the states of the underlying Markov decision process (MDP). Instead, she has access to several models that map histories of past interactions to states. Here we improve over known regret bounds in this setting, and more importantly generalize to the case where the models given to the learner do not contain a true model resulting in an MDP representation but only approximations of it. We also give improved error bounds for state aggregation.

cs.LG↗

arXiv2013

Optimal Regret Bounds for Selecting the State Representation in Reinforcement Learning

We consider an agent interacting with an environment in a single stream of actions, observations, and rewards, with no reset. This process is not assumed to be a Markov Decision Process (MDP). Rather, the agent has several representations (mapping histories of past interactions to a discrete state space) of the environment with unknown dynamics, only some of which result in an MDP. The goal is to minimize the average regret criterion against an agent who knows an MDP representation giving the highest optimal reward, and acts optimally in it. Recent regret bounds for this setting are of order $O(T^{2/3})$ with an additive term constant yet exponential in some characteristics of the optimal MDP. We propose an algorithm whose regret after $T$ time steps is $O(\sqrt{T})$, with all constants reasonably small. This is optimal in $T$ since $O(\sqrt{T})$ is the optimal regret in the setting of learning in a (single discrete) MDP.

cs.LG↗

arXiv2013

Online Regret Bounds for Undiscounted Continuous Reinforcement Learning

We derive sublinear regret bounds for undiscounted reinforcement learning in continuous state space. The proposed algorithm combines state aggregation with the use of upper confidence bounds for implementing optimism in the face of uncertainty. Beside the existence of an optimal policy which satisfies the Poisson equation, the only assumptions made are Holder continuity of rewards and transition probabilities.

cs.LG↗

arXiv2012

Regret Bounds for Restless Markov Bandits

We consider the restless Markov bandit problem, in which the state of each arm evolves according to a Markov process independently of the learner's actions. We suggest an algorithm that after $T$ steps achieves $\tilde{O}(\sqrt{T})$ regret with respect to the best policy that knows the distributions of all arms. No assumptions on the Markov chains are made except that they are irreducible. In addition, we show that index-based policies are necessarily suboptimal for the considered problem.

cs.LG↗

arXiv2008

Upper Bounds on the Number of Vertices of Weight <=k in Particular Arrangements of Pseudocircles

In arrangements of pseudocircles (Jordan curves) the weight of a vertex (intersection point) is the number of pseudocircles that contain the vertex in its interior. We give improved upper bounds on the number of vertices of weight <=k in certain arrangements of pseudocircles in the plane. In particular, forbidding certain subarrangements we improve the known bound of 6n-12 (cf. Kedem et al., 1986) for vertices of weight 0 in arrangements of n pseudocircles to 4n-6. In complete arrangements (i.e. arrangements with each two pseudocircles intersecting) we identify two subarrangements of three and four pseudocircles, respectively, whose absence gives improved bounds for vertices of weight 0 and more generally for vertices of weight <=k.

math.CO↗

arXiv2006

A Short Note on Stationary Distributions of Unichain Markov Decision Processes

Dealing with unichain MDPs, we consider stationary distributions of policies that coincide in all but $n$ states. In these states each policy chooses one of two possible actions. We show that the stationary distributions of n+1 such policies uniquely determine the stationary distributions of all other such policies. An explicit formula for calculation is given.

math.PR↗

arXiv2005

Combinations and Mixtures of Optimal Policies in Unichain Markov Decision Processes are Optimal

We show that combinations of optimal (stationary) policies in unichain Markov decision processes are optimal. That is, let M be a unichain Markov decision process with state space S, action space A and policies π_j^*: S -> A (1\leq j\leq n) with optimal average infinite horizon reward. Then any combination πof these policies, where for each state i in S there is a j such that π(i)=π_j^*(i), is optimal as well. Furthermore, we prove that any mixture of optimal policies, where at each visit in a state i an arbitrary action π_j^*(i) of an optimal policy is chosen, yields optimal average reward, too.

math.CO↗

arXiv2005

Embeddability of Arrangements of Pseudocircles into the Sphere

An arrangement of pseudocircles is a finite set of oriented closed Jordan curves each two of which cross each other in exactly two points. To describe the combinatorial structure of arrangements on closed orientable surfaces, in (Linhart, Ortner 2004) so-called *intersection schemes* were introduced. Building up on results about the latter, we first clarify the notion of embedding of an arrangement. Once this is done it is shown how the embeddability of an arrangement depends on the embeddability of its subarrangements. The main result presented is that an arrangement of pseudocircles can be embedded into the sphere if and only if all of its subarrangements of four pseudocircles are embeddable into the sphere as well.

math.CO↗

arXiv2004

Non-backtracking random walks and cogrowth of graphs

Let X be a locally finite, connected graph without vertices of degree 1. Non-backtracking random walk moves at each step with equal probability to one of the "forward" neighbours of the actual state, i.e., it does not go back along the preceding edge to the preceding state. This is not a Markov chain, but can be turned into a Markov chain whose state space is the set of oriented edges of X. Thus we obtain for infinite X that the n-step non-backtracking transition probabilities tend to zero, and we can also compute their limit when X is finite. This provides a short proof of old results concerning cogrowth of groups, and makes the extension of that result to arbitrary regular graphs rigorous. Even when X is non-regular, but "small cycles are dense" in X, we show that the graph X is non-amenable if and only if the non-backtracking n-step transition probabilities decay exponentially fast. This is a partial generalization of the cogrowth criterion for regular graphs which comprises the original cogrowth criterion for finitely generated groups of Grigorchuk and Cohen.

math.PR↗