Training for Alignment

Abstract

We propose a simple model of training an agent with an endogenous bias. A principal chooses in which states to train the agent and what actions to prescribe in these states. In response, the agent updates their bias and then acts optimally given their new bias in untrained states, while following their training in trained states. The agent chooses their new bias to minimize a weighted average of their loss under their initial and final biases. Training can thus generalize to untrained states via the agent's bias updating. We characterize optimal training, which involves \emph{over-correction} in trained states and \emph{generalization-aware} (non-greedy) selection of training states. If only a small share of states can be trained, greedy state selection is approximately optimal, but over-correction remains essential. If any bias function is possible, or if the principal evaluates training robustly over bias functions that admit the possibility of full alignment, the value of training generalization vanishes, and greedy training becomes optimal. With a more pessimistic objective, \emph{under-correction}, or even no training at all, may be optimal. The principal benefits from the agent's preferences being more flexible (\emph{corrigibility}) when the set of possible biases is known and contains full alignment, but otherwise corrigibility can be harmful. Applications include socialization in organizations and families, and AI post-training and recursive self-improvement.