Preference-Conditioned Group-Relative Policy Optimization for Terminally Constrained Multi-Objective Control

Abstract

Multi-objective optimal control is commonly addressed by reinforcement learning with fixed scalarization weights, requiring a separate training run and constraint enforcement for each trade-off. This letter extends critic-free group-relative policy optimization to terminally constrained multi-objective control: a single preference-conditioned policy is trained over random preferences and returns a feasible approximation of the Pareto front in one run. Three structural results are established. Group normalization after scalarization preserves the prescribed preference, whereas per-objective normalization induces policy-dependent distortion; with pairwise distinct normalization scales, only single-objective preferences remain unchanged. A positive floor prevents the terminal-constraint multiplier from vanishing and renders the penalty exact when the floor exceeds a geometric multiplier. Deterministic terminal violation is bounded by expected stochastic violation plus a term depending on terminal-state curvature and action variance, so stochastic feasibility alone does not certify deterministic execution. On a heavy-duty range-extender powertrain with six stage costs and a terminal state-of-charge band, a representative training run places all 32 preference-grid solutions in band on six cycles inside and outside the training pool. Averaged over ten runs, the method yields more in-band solutions under deterministic execution than a critic-based conditioned baseline with the same terminal-constraint handling.