Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2206.04547

Value Iteration is Optic Composition

Jules Hedges·Riu Rodríguez Sakamoto
View original source ↗Read source document ↗Export metadata ↓

Abstract

Dynamic programming is a class of algorithms used to compute optimal control policies for Markov decision processes. Dynamic programming is ubiquitous in control theory, and is also the foundation of reinforcement learning. In this paper, we show that value improvement, one of the main steps of dynamic programming, can be naturally seen as composition in a category of optics, and intuitively, the optimal value function is the limit of a chain of optic compositions. We illustrate this with three classic examples: the gridworld, the inverted pendulum and the savings problem. This is a first step towards a complete account of reinforcement learning in terms of parametrised optics.

Explore related subjects

math.CTmath.OC

Keep this discovery

BibTeXRIS

Jules Hedges, Riu Rodríguez Sakamoto. 2022-06-09. Value Iteration is Optic Composition. https://doi.org/10.4204/eptcs.380.24

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

A model structure for cartesian 2-fibrations

Cartesian 2-fibrations provide a way to understand indexed categories, but their classical ``straightening'' construction requires several layers of weak coherence data. This paper develops a homotopical framework that replaces much of this bookkeeping with a fully strict model. By using marked 2-categories to record the cartesian morphisms and 2-cells, we construct a model structure whose fibrant objects are precisely the cartesian 2-fibrations over a fixed 2-category $\mathcal{C}$. We then show that the marked Grothendieck construction identifies these 2-fibrations, up to weak equivalence, with strict 2-functors from $\mathcal{C}$ into $2\mathrm{Cat}$. As an additional contribution, we construct localizations of 2-categories that simultaneously invert selected morphisms and 2-cells.

math.CT↗
arXiv2026

A Natural Fuzzy Order on Fuzzy Numbers

This paper introduces a natural fuzzy order on fuzzy numbers that extends the natural orders on real numbers and interval numbers. We investigate its completeness properties and show that the space of uniformly bounded fuzzy numbers is conically complete and conically cocomplete, and that it is complete if and only if the underlying continuous t-norm is the G\"odel t-norm. Moreover, it is proved that this space constitutes a \([0,1]\)-enriched domain if and only if the underlying continuous t-norm satisfies the (S) condition. These results provide a foundation for ordering fuzzy numbers.

math.CT↗
arXiv2026

Noetherian forms of free non-symmetric operads

In this paper, we study certain categories of labeled finite rooted ordered trees over a fixed set of labels where each label is equipped with an arity: a fixed number of children that the vertex with the given label must have. Equivalently, these are expression trees for operations in a free non-symmetric operad. A morphism between these trees matches a pruning of one tree (a prefix) with an entire subtree of another (a suffix). We characterize such categories, up to isomorphism, in terms of suitable exactness properties. It turns out that these categories exhibit strong algebraic behavior, in the sense that every such category, when appended with a strict initial object, has a particularly nice noetherian form.

math.CT↗