Benchmarking In-context Experiential Learning Through Repeated Product Recommendations

Abstract

To navigate ever-shifting real-world environments, agents must grapple with incomplete knowledge and adapt their strategies through experience. However, current evaluations of LLM-based agents largely overlook this capability. Crucially, we stress not just the ability to contend with uncertainty within a task (episode), as episode-specific information is progressively revealed across turns, but also the refinement of such adaptive ability across similar episodes, as agents accumulate experiences and infer their shared latent structure. Repeated product recommendation offers a natural setting to isolate this need for experiential learning: within each interaction, an intelligent recommender must elicit unknown customer preferences through questions; across multiple interactions, it should tailor its questioning strategy to the observed customer and product distributions. We instantiate the Benchmark for Experiential Learning and Active exploration (BELA) by combining (1) a rich catalog of real-world products from Amazon, (2) a diverse collection of synthetic customer personas aimed to capture heterogeneous latent preferences, and (3) an LLM-based customer simulator framework that emulate preference-revealing interactions. Rather than aiming to faithfully replicate real consumer behavior, BELA provides a controlled and scalable testbed for whether agents can exploit consistent latent preferences across episodes. Benchmarking current models reveals that they can learn across turns, but struggle to improve across episodes. This underscore the need for frontier models to advance in experiential learning capabilities.