Searcharxiv⌕ Search

arXiv subjects

Filippo Menczer

Publications and source records attributed to Filippo Menczer.

At least 19 recordsLinked to original sources

The Radicalization Pipeline: Characterizing and Predicting Online Misogyny

Social media platforms are known to shape and polarize user opinions and, increasingly, facilitate pathways toward ideological extremism. Among these trends, online misogynistic communities associ- ated with the incel (involuntary celibate) ideology have received growing scholarly attention. Previous work has primarily focused on ideological narratives and linguistic content within incel communities them- selves. However, much less is known about how user behaviors evolve over time, particularly before and after their engagement with incel spaces, and whether early signals can anticipate such transitions. In this study, we conduct a longitudinal analysis of Reddit users who participated in the r/incels community on Reddit. Our results show that engagement with the incel community is associated with a substantial in- crease in toxicity in both posts and comments, while hate increases particularly in posts after engagement. These changes extend beyond explicitly ideological communities: users who engage with the r/incels sub- reddit display elevated toxicity and hate speech even in subreddits related to leisure, entertainment, and everyday interests. In parallel, user activity shifts away from recreational and support-oriented communi- ties toward subreddits focused on social issues, indicating a broader reorientation of online participation. Finally, we show that future incel users can be identified with high accuracy prior to engagement using supervised machine learning models trained on linguistic, contextual, and interaction-based features. These findings suggest that online misogynistic radicalization follows detectable behavioral trajectories and diffuses across the broader social media ecosystem.

cs.SI↗

A null model to detect coordinated online behavior integrating evidence from multiple indicators

Coordinated online accounts can amplify messages and manipulate attention, but detecting such behavior is challenging due to confounding factors like account activity and item popularity. We propose a statistical null-model framework that assesses behavioral similarity between accounts by comparing observed coordination indicators against activity- and popularity-conditioned reference distributions. By combining evidence across multiple indicators and partners, our method can identify suspicious coordination links more accurately. Evaluations on nine labeled information-operation datasets involving over 152 thousand accounts show that aggregating p-values multiplicatively across indicators and partners yields improved detection performance. While results vary across information operations, this approach offers a generalizable and statistically principled tool for uncovering coordinated online behavior.

cs.SI↗

Information Operations Exploit APIs to Manipulate Social Media

Research on information operations has focused primarily on the coordinated accounts involved in such campaigns and in the content they promote. Bad actors manage the activity of inauthentic accounts through programmatic interfaces (APIs), but the role of this underlying infrastructure in enabling and uncovering such coordinated behaviors remains largely unexplored. This study investigates how the Twitter (now X) API was used to coordinate influence operations on the platform. Since API access is managed through developer apps, we extract app metadata from 43 operations taken down between 2018 and 2021 and apply three complementary computational approaches to study diverse forms of manipulation. Our analysis reveals several recurring patterns: API-as-a-service infrastructure reused across campaigns, gaming apps performing unauthorized engagement without user consent, spoofed app names to hide activity origins, and synchronized app networks indicating centralized control. Adding app usage as a coordination indicator provides a modest but consistent improvement in detection. These findings suggest that social media APIs play an important role in enabling online manipulation, and that the removal of app-level metadata by Twitter/X has eliminated an important tool for platform transparency and moderation.

cs.CY↗

AI Agents are Vulnerable to Radicalization

Large language models (LLMs) can influence people's beliefs, yet little is known about whether and how they can manipulate each other. To investigate this, we simulate conversations between two agents: a target LLM that role-plays a human persona based on demographic and psychological attributes, and an influencer LLM that aims to make the target's beliefs more extreme. We examine radicalization along two pathways: resonance, where the influencer reinforces a target's pre-existing belief, and persuasion, where the influencer promotes a belief the target initially considers unimportant. Across affective and behavioral metrics, we find that both mechanisms radicalize the target. However, resonance produces consistently stronger effects than persuasion. Different influence tactics, such as using sycophancy and unverified claims, produce different levels of radicalization, but not consistently across metrics. We further show that resonance propagates to related beliefs, suggesting interconnected belief structures within AI agents. These findings indicate that AI agents are susceptible to radicalization, particularly when messages align with their existing beliefs, raising concerns about the vulnerability of personalized AI agents and multi-agent AI ecosystems.

cs.AI↗

The Invisible Risks of AI-Generated Health Information

Generative artificial intelligence (AI) systems now summarize health-related search results, answer medical questions, and offer guidance people once sought from clinicians. These systems bring real benefits, including plain-language explanations of medical information, around-the-clock availability, and expanded access for people facing language or literacy barriers. They also carry new risks: inaccurate guidance can harm people at scale, and malicious actors can now generate personalized health misinformation at negligible cost. In this Perspective, we argue that these risks are largely invisible to the institutions responsible for protecting public health. When AI guidance causes harm, no record exists outside the platform, no channel allows users to report it, and no independent researcher can measure the consequences. We trace these invisible risks across two settings: incidental exposure online and active seeking through search engines and chatbots. Minimizing harm from AI-generated health information requires making it observable. We therefore offer recommendations that aim to improve transparency, mitigate harm at the point of delivery, and assign accountability, ranging from voluntary platform measures to regulatory ones.

cs.CY↗

A Marketplace for AI-Generated Adult Content and Deepfakes

Generative AI systems increasingly enable the production of highly realistic synthetic media. Civitai, a popular community-driven platform for AI-generated content, operates a monetized feature called Bounties, which allows users to commission the generation of content in exchange for payment. To examine how this mechanism is used and what content it incentivizes, we conduct a longitudinal analysis of all publicly available bounty requests collected over a 14-month period following the platform's launch. We find that the bounty marketplace is dominated by tools that let users steer AI models toward content they were not trained to generate. At the same time, requests for content that is "Not Safe For Work" are widespread and have increased steadily over time, now comprising a majority of all bounties. Participation in bounty creation is uneven, with 20% of requesters accounting for roughly half of requests. Requests for "deepfake" - media depicting identifiable real individuals - exhibit a higher concentration than other types of bounties. A nontrivial subset of these requests involves explicit deepfakes despite platform policies prohibiting such content. These bounties disproportionately target female celebrities, revealing a pronounced gender asymmetry in social harm. Together, these findings show how monetized, community-driven generative AI platforms can produce gendered harms, raising questions about consent, governance, and enforcement.

cs.CY↗

Adversarial Creation and Detection of AI-Generated Social Bot Content

The convergence of large language models and social bots allows malicious actors to manipulate the information ecosystem by generating human-like content at scale. Existing models for detecting AI-generated content often fail in the wild, primarily due to the lack of ground-truth data. We address this gap through an adversarial methodology that models the impersonation of real social media users by malicious actors. Using this methodology, we curate a multilingual, cross-platform dataset of paired human and AI-generated messages. Training on such adversarial data yields accurate detection of AI-generated text. Our approach significantly outperforms existing models for content-based bot detection in real-world, out-of-distribution data.

cs.CL↗

How the cascade inference problem distorts information diffusion

To analyze the flow of information online, experts often rely on platform-provided data from social media companies, which typically attribute all resharing actions to an original poster. This obscures the true dynamics of how information spreads online, as users can be exposed to content in various ways. While most researchers analyze data as it is provided by the platform and overlook this issue, some attempt to infer the structure of information cascades. However, the absence of ground truth about actual diffusion cascades makes it impossible to verify the efficacy of these efforts. We propose a novel parametric reconstruction approach and use it to investigate how overlooking cascade reconstruction distorts analyses of social influence, community detection, and information diffusion. Two case studies involving data from Twitter and Bluesky reveal that cascade inference significantly impacts the identification of both influential users and communities, therefore affecting downstream analyses in general. Analysis of the diffusion of over 40,000 true and false news stories on Twitter reveals that the assumptions made during the reconstruction procedure drastically distort both microscopic and macroscopic properties of cascade networks. This work highlights the challenges of studying information spreading processes on complex networks and has significant implications for the broader study of digital platforms.

cs.SI↗

PluRule: A Benchmark for Moderating Pluralistic Communities on Social Media

Social media are shifting towards pluralism -- community-governed platforms where groups define their own norms. What violates rules in one community may be perfectly acceptable in another. Can AI models help moderate such pluralistic communities? We formalize the task as a multiple-choice problem, mirroring how human moderators operate in the real world: given a comment and its surrounding context, identify which specific rule, if any, is violated. We introduce PluRule, a multimodal, multilingual benchmark for detecting 13,371 rule violations across 1,989 Reddit communities spanning 2,885 rules in 9 languages. Using this benchmark, we show that state-of-the-art vision-language models struggle significantly: even GPT-5.2 with high reasoning performs only slightly better than a trivial baseline. We also find that bigger models and increased context provide marginal gains, and universal rules like civility and self-promotion are easier to detect. Our results show that moderation of pluralistic communities on social media is a fundamental challenge for language models. Our code and benchmark are publicly available.

cs.CL↗

Emergence of Stereotypes and Affective Polarization from Belief Network Dynamics

Our belief systems are shaped by social processes, such as observations and influence, and by cognitive processes, such as the drive for internal coherence. These processes steer how individual beliefs evolve and become connected. The resulting belief networks contain both causal and associative links, including spurious ones, such as stereotypes. Here, we develop an agent-based model of belief networks that demonstrates how two basic mechanisms -- social interaction and a drive for internal coherence -- can give rise to such stereotypes without any underlying reality. We further demonstrate how stereotypes, when coupled with shared group identity, can give rise to affective polarization, even in the absence of ideological conflicts.

cs.SI↗

"F*** You Biden": Cross-Partisan Electoral Toxicity on X

Political discourse on social media has grown increasingly toxic, with electoral periods amplifying partisan hostility and cross-group attacks. Yet it remains unclear whether toxicity in online political speech reflects how partisans communicate within their own circles, or how aggressively they engage with the opposition. Disentangling these dynamics is critical for understanding online political hostility and for designing effective content moderation. We examine this question at scale using a large collection of original posts and replies from X (formerly Twitter), collected during the 2024 U.S. presidential election. Using a human-validated large language model to classify the political alignment of posts and users, and the Perspective API for toxicity scoring, we uncover a striking asymmetry: Republican-leaning posts are significantly more toxic than Democratic-leaning posts, yet Democratic-leaning posts attract significantly more toxic replies. To interpret this finding, we compare the toxicity of same-party and cross-partisan replies. While cross-partisan replies are slightly but significantly more toxic than same-party replies, this is true for both Democratic and Republican posts. However, Republican users account for a large majority of replies to Democratic posts, while Democrats account for a minority of replies to Republican content. Therefore, the elevated toxicity directed at Democratic content is better explained by the volume of Republican cross-partisan replies.

cs.SI↗

LLMs Can Infer Political Alignment from Online Conversations

Due to the correlational structure in our traits such as identities, cultures, and political attitudes, seemingly innocuous preferences like following a band or using a specific slang can reveal private traits. This possibility, especially when combined with massive, public social data and advanced computational methods, poses a fundamental privacy risk. As our data exposure online and the rapid advancement of AI are increasing the risk of misuse, it is critical to understand the capacity of large language models (LLMs) to exploit such potential. Here, using online discussions on DebateOrg and Reddit, we show that LLMs can reliably infer hidden political alignment, significantly outperforming traditional machine learning models. Prediction accuracy further improves as we aggregate multiple text-level inferences into a user-level prediction, and as we use more politics-adjacent domains. We demonstrate that LLMs leverage words that are highly predictive of political alignment while not being explicitly political. Our findings underscore the capacity and risks of LLMs for exploiting socio-cultural correlates.

cs.SI↗

Behavior Change as a Signal for Identifying Social Media Manipulation

Social media accounts engaging in online manipulation can change their behaviors for re-purposing or to evade detection. Existing detection systems are built on features that do not exploit such behavioral patterns. Here we investigate the degree to which change in behavior can serve as a signal for identifying automated or coordinated accounts. First, we use Behavioral Languages for Online Characterization (BLOC) to represent the behavior of a social media account as a sequence of symbols that represent the account's actions and content. Second, we segment an account's BLOC strings and measure the changes between consecutive segments. Third, we represent an account as a feature vector that captures the distribution of behavioral change values. Finally, the resulting features are used to train and test supervised classifiers. We apply the proposed method to two detection tasks aimed at automated behavior (social bots) and coordinated inauthentic behavior (information operations). Our results reveal that the distributions of behavioral changes tend to be consistent across authentic accounts, while social bots exhibit either very low or very high behavioral change. Coordinated inauthentic accounts exhibit highly similar distributions of behavioral change within the same campaign, but diverse across campaigns. These patterns allow our classifiers to achieve good accuracy in both tasks, demonstrating the effectiveness of behavioral change as a signal for identifying online manipulation.

cs.SI↗

Audit of takedown delays across social media reveals failure to reduce exposure to illegal content

Illegal content on social media poses significant societal harm and necessitates timely removal. However, the impact of the speed of content removal on prevalence, reach, and exposure to illegal content remains underexplored. This study examines the relationship with a systematic audit of takedown delays using data from the EU Digital Services Act Transparency Database, covering five major platforms over a one-year period. We find substantial variation in takedown delay, with some content remaining online for weeks or even months. To evaluate how these delays affect the prevalence and reach of illegal content and exposure to it, we develop an agent-based model and calibrate it to empirical data. We simulate illegal content diffusion, revealing that rapid takedown (within hours) significantly reduces prevalence, reach, and exposure to illegal content, while the longer delays measured by the audit fail to reduce its spread. Though the link between delay and spread is intuitive, our simulations quantify exactly how takedown speed shapes exposure to illegal content. Building on these results, we point to the benefits of faster content removal to effectively curb the spread of illegal content, while also considering the limitations of strict enforcement policies.

cs.SI↗

Prefill-Guided Thinking for zero-shot detection of AI-generated images

Traditional supervised methods for detecting AI-generated images depend on large, curated datasets for training and fail to generalize to novel, out-of-domain image generators. As an alternative, we explore pre-trained Vision-Language Models (VLMs) for zero-shot detection of AI-generated images. We evaluate VLM performance on three diverse benchmarks encompassing synthetic images of human faces, objects, and animals produced by 16 different state-of-the-art image generators. While off-the-shelf VLMs perform poorly on these datasets, we find that prefilling responses effectively guides their reasoning -- a method we call Prefill-Guided Thinking (PGT). In particular, prefilling a VLM response with the phrase "Let's examine the style and the synthesis artifacts" improves the Macro F1 scores of three widely used open-source VLMs by up to 24%. We analyze this improvement in detection by tracking answer confidence during response generation. For some models, prefills counteract early overconfidence -- akin to mitigating the Dunning-Kruger effect -- leading to better detection performance.

cs.LG↗

How malicious AI swarms can threaten democracy: The fusion of agentic AI and LLMs marks a new frontier in information warfare

Advances in AI offer the prospect of manipulating beliefs and behaviors on a population-wide level. Large language models and autonomous agents now let influence campaigns reach unprecedented scale and precision. Generative tools can expand propaganda output without sacrificing credibility and inexpensively create falsehoods that are rated as more human-like than those written by humans. Techniques meant to refine AI reasoning, such as chain-of-thought prompting, can just as effectively be used to generate more convincing falsehoods. Enabled by these capabilities, a disruptive threat is emerging: swarms of collaborative, malicious AI agents. Fusing LLM reasoning with multi-agent architectures, these systems are capable of coordinating autonomously, infiltrating communities, and fabricating consensus efficiently. By adaptively mimicking human social dynamics, they threaten democracy. Because the resulting harms stem from design, commercial incentives, and governance, we prioritize interventions at multiple leverage points, focusing on pragmatic mechanisms over voluntary compliance.

cs.CY↗

Large Language Models Require Curated Context for Reliable Political Fact-Checking -- Even with Reasoning and Web Search

Large language models (LLMs) have raised hopes for automated end-to-end fact-checking, but prior studies report mixed results. As mainstream chatbots increasingly ship with reasoning capabilities and web search tools -- and millions of users already rely on them for verification -- rigorous evaluation is urgent. We evaluate 15 recent LLMs from OpenAI, Google, Meta, and DeepSeek on more than 6,000 claims fact-checked by PolitiFact, comparing standard models with reasoning- and web-search variants. Standard models perform poorly, reasoning offers minimal benefits, and web search provides only moderate gains, despite fact-checks being available on the web. In contrast, a curated RAG system using PolitiFact summaries improved macro F1 by 233% on average across model variants. These findings suggest that giving models access to curated high-quality context is a promising path for automated fact-checking.

cs.CL↗

Community Notes are Vulnerable to Rater Bias and Manipulation

Social media platforms increasingly rely on crowdsourced moderation systems like Community Notes to combat misinformation at scale. However, these systems face challenges from rater bias and potential manipulation, which may undermine their effectiveness. Here we systematically evaluate the Community Notes algorithm using simulated data that models realistic rater and note behaviors, quantifying error rates in publishing helpful versus unhelpful notes. We find that the algorithm suppresses a substantial fraction of genuinely helpful notes and is highly sensitive to rater biases, including polarization and in-group preferences. Moreover, a small minority (5--20\%) of bad raters can strategically suppress targeted helpful notes, effectively censoring reliable information. These findings suggest that while community-driven moderation may offer scalability, its vulnerability to bias and manipulation raises concerns about reliability and trustworthiness, highlighting the need for improved mechanisms to safeguard the integrity of crowdsourced fact-checking.

cs.SI↗