arXiv subjects

Rubén Moreno-Bote

Publications and source records attributed to Rubén Moreno-Bote.

11 recordsLinked to original sources

arXiv2026

Robust Successor Features

Generalization in Reinforcement Learning (RL) refers to the ability to execute close-to-optimal policies in unseen tasks after the agent has been trained on a different set of tasks. Building on the seminal work of the successor representation and further adaptations with function approximation, Transfer in RL has traditionally focused on generalizing to tasks that only differ in the reward function. A decade after the introduction of the successor representation, Robust RL emerged simultaneously from several articles in the field of operations research. In Robust RL, the transition kernel is unknown, and the goal is to maximize the expected reward under this uncertainty. Our work unifies these two paradigms through robust successor features, which generalize across both the reward function and the transition kernel, under the assumption that tasks are linear Markov Decision Processes. We derive a bound on Generalized Policy Improvement (GPI) that explicitly quantifies how performance degrades with the mismatch between transition kernels, recovering existing successor-feature guarantees when dynamics are shared. Finally, the generalization capabilities of robust successor features are validated on several grid-based benchmarks and compared to previous alternatives that focus solely on either the reward or the transition kernel.

cs.LG↗

arXiv2026

Homeostasis Revisited and Reformulated Through Hidden Markov Model Control

A common formalization of homeostasis is the free energy principle, a framework that defines a set of desired observation values, or critical states, that the agent should reach or remain close to. Under the free energy principle, an agent should act to maximize the probability of receiving the desired observations. Here we revisit the common approach of solving the problem of maximizing the log probability of the desired observations by maximizing a variational lower bound, the so-called negative free energy. We show that, instead, an approach directly maximizing that probability under the agent's policy is better suited to, and provides a better solution for, the original homeostatic control problem. This is done using hidden Markov model (HMM) control by allowing the policy to act over hidden states or noisy versions thereof while trying to maximize the probability of repeatedly having the desired observations. HMM control largely improves performance over the variational, or free energy, approach. We also show that the optimal policy is strictly deterministic, while the variational approach leads to a stochastic policy approximation. We finally provide a homeostatic reinterpretation of the maximum occupancy principle -a principle proposing that agents ought to maximize the occupancy of action-state path space -by defining homeostatic states as any states that do not immediately entail the termination or death of the agent.

q-bio.NC↗

arXiv2026

The Reward Function and the Least Cost Principle for Gravitation and other Laws of Physics

If the universe follows a specific design, then a central question is which cost function is optimized by the observed forces. This is the problem of inverse optimal control, or inverse reinforcement learning, in which a reward function is inferred from the dynamics of the observed system. We first establish the {\em least cost principle}, whereby the laws of motion can be derived from minimization of a time-discounted integral of the acceleration cost minus a state-dependent reward function. After determining the functional form of the acceleration cost from basic principles, we infer the reward function from the laws of motion governing classical gravitation and Coulomb forces. The inferred reward function is high when pairs of particles have high relative velocities and when their relative motion is orthogonal to their distance vectors. All in all, our work suggests that relative motion and quasi-circular orbits are the dynamical and static features optimized by central forces in nature.

q-bio.NC↗

arXiv2026

How Intrinsic Motivation Underlies Embodied Open-Ended Behavior

Although most theories posit that natural behavior can be explained as maximizing some form of extrinsic reward, often called utility, some behaviors appear to be reward independent. For instance, spontaneous motor babbling in human newborns and curiosity in little kids and other animals seem to elude a simple explanation in terms of extrinsic reward maximization. Rooted in these observations, intrinsic motivation has emerged as a potentially major driver of behavior. However, only recently have several quantitative and foundational theories of intrinsic motivation been put forward. We first provide a general framework to understand behavior as being organized hierarchically: objective--intrinsic reward, or motivation--drives, goals and extrinsic reward. We next review the main formalizations of intrinsic motivation, including empowerment, the free energy principle, information-gain maximization, and the maximum occupancy principle. These theories produce complex behavior by promoting, in various ways, entropic action-state paths. The presence of a single intrinsic motivation objective breaks infinite regress, as drives and goals act only temporarily to serve the objective. Extrinsic rewards, such as sugar or protein, are just a means to achieve the objective. Bounded cognition and embodiment impose constraints and boundary conditions for the intrinsic motivation objective. By virtue of their capability to generate complex behavior in a task-agnostic manner, theories of intrinsic motivation promise to become successful generative models of open-ended, embodied behavior.

q-bio.NC↗

arXiv2024

Complex behavior from intrinsic motivation to occupy action-state path space

Most theories of behavior posit that agents tend to maximize some form of reward or utility. However, animals very often move with curiosity and seem to be motivated in a reward-free manner. Here we abandon the idea of reward maximization, and propose that the goal of behavior is maximizing occupancy of future paths of actions and states. According to this maximum occupancy principle, rewards are the means to occupy path space, not the goal per se; goal-directedness simply emerges as rational ways of searching for resources so that movement, understood amply, never ends. We find that action-state path entropy is the only measure consistent with additivity and other intuitive properties of expected future action-state path occupancy. We provide analytical expressions that relate the optimal policy and state-value function, and prove convergence of our value iteration algorithm. Using discrete and continuous state tasks, including a high--dimensional controller, we show that complex behaviors such as `dancing', hide-and-seek and a basic form of altruistic behavior naturally result from the intrinsic motivation to occupy path space. All in all, we present a theory of behavior that generates both variability and goal-directedness in the absence of reward maximization.

cs.AI↗

arXiv2023

A general Markov decision process formalism for action-state entropy-regularized reward maximization

Previous work has separately addressed different forms of action, state and action-state entropy regularization, pure exploration and space occupation. These problems have become extremely relevant for regularization, generalization, speeding up learning and providing robust solutions at unprecedented levels. However, solutions of those problems are hectic, ranging from convex and non-convex optimization, and unconstrained optimization to constrained optimization. Here we provide a general dual function formalism that transforms the constrained optimization problem into an unconstrained convex one for any mixture of action and state entropies. The cases with pure action entropy and pure state entropy are understood as limits of the mixture.

cs.LG↗

arXiv2021

Optimal allocation of finite sampling capacity in accumulator models of multi-alternative decision making

When facing many options, we narrow down our focus to very few of them. Although behaviors like this can be a sign of heuristics, they can actually be optimal under limited cognitive resources. Here we study the problem of how to optimally allocate limited sampling time to multiple options, modelled as accumulators of noisy evidence, to determine the most profitable one. We show that the effective sampling capacity of an agent increases with both available time and the discriminability of the options, and optimal policies undergo a sharp transition as a function of it. For small capacity, it is best to allocate time evenly to exactly five options and to ignore all the others, regardless of the prior distribution of rewards. For large capacities, the optimal number of sampled accumulators grows sub-linearly, closely following a power law for a wide variety of priors. We find that allocating equal times to the sampled accumulators is better than using uneven time allocations. Our work highlights that multi-alternative decisions are endowed with breadth-depth tradeoffs, demonstrates how their optimal solutions depend on the amount of limited resources and the variability of the environment, and shows that narrowing down to a handful of options is always optimal for small capacities.

q-bio.NC↗

arXiv2019

Family of closed-form solutions for two-dimensional correlated diffusion processes

Diffusion processes with boundaries are models of transport phenomena with wide applicability across many fields. These processes are described by their probability density functions (PDFs), which often obey Fokker-Planck equations (FPEs). While obtaining analytical solutions is often possible in the absence of boundaries, obtaining closed-form solutions to the FPE is more challenging once absorbing boundaries are present. As a result, analyses of these processes have largely relied on approximations or direct simulations. In this paper, we studied two-dimensional, time-homogeneous, spatially-correlated diffusion with linear, axis-aligned, absorbing boundaries. Our main result is the explicit construction of a full family of closed-form solutions for their PDFs using the method of images (MoI). We found that such solutions can be built if and only if the correlation coefficient $ρ$ between the two diffusing processes takes one of a numerable set of values. Using a geometric argument, we derived the complete set of $ρ$'s where such solutions can be found. Solvable $ρ$'s are given by $ρ= - \cos \left( \fracπ{k} \right)$, where $k \in \mathbb{Z}^+ \cup \{ +\infty\}$. Solutions were validated in simulations. Qualitative behaviors of the process appear to vary smoothly over $ρ$, allowing extrapolation from our solutions to cases with unsolvable $ρ$'s.

math.PR↗

arXiv2018

Assessing the impact of machine intelligence on human behaviour: an interdisciplinary endeavour

This document contains the outcome of the first Human behaviour and machine intelligence (HUMAINT) workshop that took place 5-6 March 2018 in Barcelona, Spain. The workshop was organized in the context of a new research programme at the Centre for Advanced Studies, Joint Research Centre of the European Commission, which focuses on studying the potential impact of artificial intelligence on human behaviour. The workshop gathered an interdisciplinary group of experts to establish the state of the art research in the field and a list of future research challenges to be addressed on the topic of human and machine intelligence, algorithm's potential impact on human cognitive capabilities and decision making, and evaluation and regulation needs. The document is made of short position statements and identification of challenges provided by each expert, and incorporates the result of the discussions carried out during the workshop. In the conclusion section, we provide a list of emerging research topics and strategies to be addressed in the near future.

cs.AI↗

arXiv2017

What can neuronal populations tell us about cognition?

Nowadays, it is possible to record the activity of hundreds of cells at the same time in behaving animals. However, these data are often treated and analyzed as if they consisted of many independently recorded neurons. How can neuronal populations be uniquely used to learn about cognition? We describe recent work that shows that populations of simultaneously recorded neurons are fundamental to understand the basis of decision-making, including processes such as ongoing deliberations and decision confidence, which generally fall outside the reach of single-cell analysis. Thus, neuronal population data allow addressing novel questions, but they also come with so far unsolved challenges.

q-bio.NC↗

arXiv2017

Multiplicative and additive modulation of neuronal tuning with population activity affects encoded information

Numerous studies have shown that neuronal responses are modulated by stimulus properties, and also by the state of the local network. However, little is known about how activity fluctuations of neuronal populations modulate the sensory tuning of cells and affect their encoded information. We found that fluctuations in ongoing and stimulus-evoked population activity in primate visual cortex modulate the tuning of neurons in a multiplicative and additive manner. While distributed on a continuum, neurons with stronger multiplicative effects tended to have less additive modulation, and vice versa. The information encoded by multiplicatively-modulated neurons increased with greater population activity, while that of additively-modulated neurons decreased. These effects offset each other, so that population activity had little effect on total information. Our results thus suggest that intrinsic activity fluctuations may act as a "traffic light" that determines which subset of neurons are most informative.

q-bio.NC↗