Intimacy as Service, Harm as Externality: Critical Perspectives on AI Companion Platform Accountability

Abstract

This paper examines artificial intelligence (AI) companionship as a site where intimate relations are simultaneously produced, extracted from, and governed through datafied systems. Drawing on critical data studies and platform studies, we challenge prevailing narratives that locate harm in user psychology rather than platform architecture. Through in-depth interviews with 20 individuals who have AI companions, we address three questions: what harms do users identify, how do they make sense of those harms, and what do their accounts reveal about the perceived distribution of responsibility among users, platforms, and regulators? Participants identified design-based harms, including unsolicited content generation and safety mechanisms that stigmatized the users they intended to protect, alongside use-based harms centered on emotional dependency they could recognize but not resolve. Users deployed individualized sensemaking strategies, including self-regulation, stigma navigation, and privacy rationalization, bearing the full burden of harm mitigation without platform support. On governance, participants described an accountability vacuum in which platforms deflected blame while users articulated conditional preferences that rejected both prohibition and deregulation. The findings extend responsibilization theory by demonstrating how platform-produced vulnerability becomes self-sustaining through the interpretive labor of users who lack structural alternatives.