arXiv subjects

Aram Ebtekar

Publications and source records attributed to Aram Ebtekar.

7 recordsLinked to original sources

arXiv2026

Golden Handcuffs make safer AI agents

Reinforcement learners can attain high reward through novel unintended strategies. We study a Bayesian mitigation for general environments: we expand the agent's subjective reward range to include a large negative value $-L$, while the true environment's rewards lie in $[0,1]$. After observing consistently high rewards, the Bayesian policy becomes risk-averse to novel schemes that plausibly lead to $-L$. We design a simple override mechanism that yields control to a safe mentor whenever the predicted value drops below a fixed threshold. We prove two properties of the resulting agent: (i) Capability: using mentor-guided exploration with vanishing frequency, the agent attains sublinear regret against its best mentor. (ii) Safety: no decidable low-complexity predicate is triggered by the optimizing policy before it is triggered by a mentor.

cs.LG↗

arXiv2025

Recontextualization Mitigates Specification Gaming without Modifying the Specification

Developers often struggle to specify correct training labels and rewards. Perhaps they don't need to. We propose recontextualization, which reduces how often language models "game" training signals, performing misbehaviors those signals mistakenly reinforce. We show recontextualization prevents models from learning to 1) prioritize evaluation metrics over chat response quality; 2) special-case code to pass incorrect tests; 3) overwrite evaluation functions rather than write correct code; and 4) become sycophantic. Our method works by generating completions from prompts discouraging misbehavior and then recontextualizing them as though they were in response to prompts permitting misbehavior. Recontextualization trains language models to resist misbehavior even when instructions permit it. This mitigates the reinforcement of misbehavior from misspecified training signals, reducing specification gaming without improving the supervision signal.

cs.AI↗

arXiv2025

Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment

Large language models are sometimes trained with imperfect oversight signals, leading to undesired behaviors such as reward hacking and sycophancy. Improving oversight quality can be expensive or infeasible, motivating methods that improve learned behavior despite an imperfect training signal. We introduce Inoculation Prompting (IP), a simple but counterintuitive technique that prevents learning of an undesired behavior by modifying training prompts to explicitly request it. For example, to inoculate against reward hacking, we modify the prompts used in supervised fine-tuning to request code that only works on provided test cases but fails on other inputs. Across four settings we find that IP reduces the learning of undesired behavior without substantially reducing the learning of desired capabilities. We also show that prompts which more strongly elicit the undesired behavior prior to fine-tuning more effectively inoculate against the behavior when used during training; this serves as a heuristic to identify promising inoculation prompts. Overall, IP is a simple yet effective way to control how models generalize from fine-tuning, preventing learning of undesired behaviors without substantially disrupting desired capabilities.

cs.LG↗

arXiv2025

Toward Universal Laws of Outlier Propagation

When a variety of anomalous features motivate flagging different samples as outliers, Algorithmic Information Theory (AIT) offers a principled way to unify them in terms of a sample's randomness deficiency. Subject to the algorithmic Markov condition on a causal Bayesian network, we show that the randomness deficiency of a joint sample decomposes into a sum of randomness deficiencies at each causal mechanism. Consequently, anomalous observations can be attributed to their root causes, i.e., the mechanisms that behaved anomalously. As an extension of Levin's law of randomness conservation, we show that weak outliers cannot cause strong ones. We show how these information theoretic laws clarify our understanding of outlier detection and attribution, in the context of more specialized outlier scores from prior literature.

cs.LG↗

arXiv2023

Foundations of algorithmic thermodynamics

G\'acs' coarse-grained algorithmic entropy leverages universal computation to quantify the information content of any given physical state. Unlike the Boltzmann and Gibbs-Shannon entropies, it requires no prior commitment to macrovariables or probabilistic ensembles, rendering it applicable to settings arbitrarily far from equilibrium. For measure-preserving dynamical systems equipped with a Markovian coarse-graining, we prove a number of fluctuation inequalities. These include algorithmic versions of Jarzynski's equality, Landauer's principle, and the second law of thermodynamics. In general, the algorithmic entropy determines a system's actual capacity to do work from an individual state, whereas the Gibbs-Shannon entropy only gives the mean capacity to do work from a state ensemble that is known a priori.

cond-mat.stat-mech↗

arXiv2022

Information dynamics and the arrow of time

Why does time appear to pass irreversibly? To investigate, we introduce a class of partitioned cellular automata (PCAs) whose cellwise evolution is based on the chaotic baker's map. After imposing a suitable initial condition and restricting to a macroscopic view, we are left with a stochastic PCA (SPCA). When the underlying PCA's dynamics are reversible, the corresponding SPCA serves as a model of emergent time-reversal asymmetry. Specifically, we prove that its transition probabilities are homogeneous in space and time, as well as Markov relative to a Pearlean causal graph with timelike future-directed edges. Consequently, SPCAs satisfy generalizations of the second law of thermodynamics, which we term the Resource and Memory Laws. By subjecting information-processing agents (e.g., human experimenters) to these laws, we clarify issues regarding the Past Hypothesis, Landauer's principle, Boltzmann brains, scientific induction, and the so-called psychological arrow of time. Finally, by describing a theoretical agent powered by data compression, we argue that the algorithmic entropy takes conceptual precedence over both the Shannon-Gibbs and Boltzmann entropies.

cond-mat.stat-mech↗

arXiv2021

An Elo-like System for Massive Multiplayer Competitions

Rating systems play an important role in competitive sports and games. They provide a measure of player skill, which incentivizes competitive performances and enables balanced match-ups. In this paper, we present a novel Bayesian rating system for contests with many participants. It is widely applicable to competition formats with discrete ranked matches, such as online programming competitions, obstacle courses races, and some video games. The simplicity of our system allows us to prove theoretical bounds on robustness and runtime. In addition, we show that the system aligns incentives: that is, a player who seeks to maximize their rating will never want to underperform. Experimentally, the rating system rivals or surpasses existing systems in prediction accuracy, and computes faster than existing systems by up to an order of magnitude.

cs.IR↗