Competitive exploration in structured environments

Abstract

Organisms must constantly weigh exploiting the resources they know against exploring for better ones. When foragers compete, the best behavior for each forager also depends on what its competitors do. How does competition change the strategic balance between exploration and exploitation? Here we address this question in a minimal model: two $ε$-greedy temporal-difference learners on a one-dimensional lattice, who start at a small reward at one end and are unaware of a larger reward at the other. For a lone forager, the optimal exploration rate increases monotonically with the value of the distant reward, and with its distance up to a critical distance. When two foragers share the lattice and split rewards, the choice of exploration rate becomes a game whose Nash equilibria have both foragers exploring, both staying, or one exploring while the other stays (anti-coordination). Under local best-response dynamics, pairs of foragers often converge on both exploring or both staying, even when anti-coordination would be socially optimal. This social dilemma arises for a wide range of parameters, and it persists even in a simplified model without explicit spatial structure. A minimal, naturalistic signal, in which one forager informs the other that it has found the distant reward, can remove many of the dilemmas in which both foragers explore, and can produce a Nash equilibrium whose collective payoff exceeds that of every outcome available without signaling.