When and Whom to Collaborate with in a Changing Environment: A Collaborative Dynamic Bandit Solution

Abstract

Collaborative bandit learning, i.e., bandit algorithms that utilize collaborative filtering techniques to improve sample efficiency in online interactive recommendation, has attracted much research attention as it enjoys the best of both worlds. However, all existing collaborative bandit learning solutions impose a stationary assumption about the environment, i.e., both user preferences and the dependency among users are assumed static over time. Unfortunately, this assumption hardly holds in practice due to users' ever-changing interests and dependence relations, which inevitably costs a recommender system sub-optimal performance in practice. In this work, we develop a collaborative dynamic bandit solution to handle a changing environment for recommendation. We explicitly model the underlying changes in both user preferences and their dependency relation as a stochastic process. Individual user's preference is modeled by a mixture of globally shared contextual bandit models with a Dirichlet Process prior. Collaboration among users is thus achieved via Bayesian inference over the global bandit models. Model selection and arm selection for each user are done via Thompson sampling to balance exploitation and exploration. Our solution is proved to maintain a standard $\tilde O(\sqrt{T})$ sublinear regret even in such a challenging environment. And extensive empirical evaluations on both synthetic and real-world datasets further confirmed the necessity of modeling a changing environment and our algorithm's practical advantages against several state-of-the-art online learning solutions.