arXiv subjects

Benjamin Edelman

Publications and source records attributed to Benjamin Edelman.

3 recordsLinked to original sources

arXiv2026

Understanding and Mitigating Role Spoofing in LLMs

Despite growing capabilities, LLMs remain susceptible to prompt injections: malicious inputs that can manipulate or hijack models. In particular, it has recently been observed that LLMs can exhibit role confusion (Ye et al., 2026), where models attribute the source of text based on its style instead of its labeled role. Malicious text in tool outputs (for instance, a malicious webpage accessed by an agent) can leverage role confusion to impersonate the user, an attack we call role spoofing}. In this work, we seek to understand and mitigate role spoofing. We study how models track "who said what" in a conversation, showing that they use cues from both content (e.g., semantics and writing style) and structure (e.g., control tokens and formatting). We show that LLMs are highly susceptible to role spoofing attacks that combine both fake structure and content. Additionally, in a toy setting, we show that adversarial training which teaches the model to ignore content-based injections does mitigate content-based attacks but leaves the model susceptible to structure-based attacks. This can explain why frontier models, with increased resistance to many injection attacks, can still be susceptible to role spoofing. Finally, we propose a new method for detecting role spoofing using linear probes trained to predict roles in a conversation, and show that it is effective across different settings.

cs.CR↗

arXiv2022

Causal Strategic Linear Regression

In many predictive decision-making scenarios, such as credit scoring and academic testing, a decision-maker must construct a model that accounts for agents' propensity to "game" the decision rule by changing their features so as to receive better decisions. Whereas the strategic classification literature has previously assumed that agents' outcomes are not causally affected by their features (and thus that strategic agents' goal is deceiving the decision-maker), we join concurrent work in modeling agents' outcomes as a function of their changeable attributes. As our main contribution, we provide efficient algorithms for learning decision rules that optimize three distinct decision-maker objectives in a realizable linear setting: accurately predicting agents' post-gaming outcomes (prediction risk minimization), incentivizing agents to improve these outcomes (agent outcome maximization), and estimating the coefficients of the true underlying model (parameter estimation). Our algorithms circumvent a hardness result of Miller et al. (2020) by allowing the decision maker to test a sequence of decision rules and observe agents' responses, in effect performing causal interventions through the decision rules.

cs.LG↗

arXiv2017

Matrix rigidity and the Croot-Lev-Pach lemma

Matrix rigidity is a notion put forth by Valiant as a means for proving arithmetic circuit lower bounds. A matrix is rigid if it is far, in Hamming distance, from any low rank matrix. Despite decades of efforts, no explicit matrix rigid enough to carry out Valiant's plan has been found. Recently, Alman and Williams showed, contrary to common belief, that the $2^n \times 2^n$ Hadamard matrix could not be used for Valiant's program as it is not sufficiently rigid. In this note we observe a similar `non rigidity' phenomena for any $q^n \times q^n$ matrix $M$ of the form $M(x,y) = f(x+y)$, where $f:F_q^n \to F_q$ is any function and $F_q$ is a fixed finite field of $q$ elements ($n$ goes to infinity). The theorem follows almost immediately from a recent lemma of Croot, Lev and Pach which is also the main ingredient in the recent solution of the cap-set problem.

cs.CC↗