arXiv subjects

Yuji Kanagawa

Publications and source records attributed to Yuji Kanagawa.

4 recordsLinked to original sources

arXiv2025

Evolution of Fear and Social Rewards in Prey-Predator Relationship

Fear is a critical brain function that enables us to learn to avoid danger via reinforcement learning (RL). While many researchers have argued that fear has evolved to escape predators, how varying predatory pressures have shaped fear and other rewards, including positive social rewards for collective grouping, remains an open question. In this study, we investigate the relationship between predatory pressure and fear using an evolutionary simulation of RL agents with evolving rewards. In our simulation, prey and predator RL agents co-evolve their reward functions, including visual rewards for observing prey and predators. While fear-like negative visual rewards for predators often evolved in prey, we also observed cases in which positive rewards for both predators and prey evolved, the latter serving as a social reward for collective grouping. A comparison between different environmental conditions revealed that stronger predator hunting capability promoted stronger fear reward, while less food supply promoted more negative social reward. Moreover, fear did not evolve in response to static pitfalls with non-lethal damage, suggesting that actively hunting predators played an important role in its evolution. These results highlight the special role of predators in the diverse evolution of fear and social rewards.

q-bio.PE↗

arXiv2024

Evolution of Rewards for Food and Motor Action by Simulating Birth and Death

The reward system is one of the fundamental drivers of animal behaviors and is critical for survival and reproduction. Despite its importance, the problem of how the reward system has evolved is underexplored. In this paper, we try to replicate the evolution of biologically plausible reward functions and investigate how environmental conditions affect evolved rewards' shape. For this purpose, we developed a population-based decentralized evolutionary simulation framework, where agents maintain their energy level to live longer and produce more children. Each agent inherits its reward function from its parent subject to mutation and learns to get rewards via reinforcement learning throughout its lifetime. Our results show that biologically reasonable positive rewards for food acquisition and negative rewards for motor action can evolve from randomly initialized ones. However, we also find that the rewards for motor action diverge into two modes: largely positive and slightly negative. The emergence of positive motor action rewards is surprising because it can make agents too active and inefficient in foraging. In environments with poor and poisonous foods, the evolution of rewards for less important foods tends to be unstable, while rewards for normal foods are still stable. These results demonstrate the usefulness of our simulation environment and energy-dependent birth and death model for further studies of the origin of reward systems.

cs.NE↗

arXiv2023

Learning Diverse Options via InfoMax Termination Critic

We consider the problem of autonomously learning reusable temporally extended actions, or options, in reinforcement learning. While options can speed up transfer learning by serving as reusable building blocks, learning reusable options for unknown task distribution remains challenging. Motivated by the recent success of mutual information (MI) based skill learning, we hypothesize that more diverse options are more reusable. To this end, we propose a method for learning termination conditions of options by maximizing MI between options and corresponding state transitions. We derive a scalable approximation of this MI maximization via gradient ascent, yielding the InfoMax Termination Critic (IMTC) algorithm. Our experiments demonstrate that IMTC significantly improves the diversity of learned options without extrinsic rewards combined with an intrinsic option learning method. Moreover, we test the reusability of learned options by transferring options into various tasks, confirming that IMTC helps quick adaptation, especially in complex domains where an agent needs to manipulate objects.

cs.LG↗

arXiv2019

Rogue-Gym: A New Challenge for Generalization in Reinforcement Learning

In this paper, we propose Rogue-Gym, a simple and classic style roguelike game built for evaluating generalization in reinforcement learning (RL). Combined with the recent progress of deep neural networks, RL has successfully trained human-level agents without human knowledge in many games such as those for Atari 2600. However, it has been pointed out that agents trained with RL methods often overfit the training environment, and they work poorly in slightly different environments. To investigate this problem, some research environments with procedural content generation have been proposed. Following these studies, we propose the use of roguelikes as a benchmark for evaluating the generalization ability of RL agents. In our Rogue-Gym, agents need to explore dungeons that are structured differently each time they start a new game. Thanks to the very diverse structures of the dungeons, we believe that the generalization benchmark of Rogue-Gym is sufficiently fair. In our experiments, we evaluate a standard reinforcement learning method, PPO, with and without enhancements for generalization. The results show that some enhancements believed to be effective fail to mitigate the overfitting in Rogue-Gym, although others slightly improve the generalization ability.

cs.LG↗