arXiv subjects

Filippo Menczer

Publications and source records attributed to Filippo Menczer.

At least 19 recordsLinked to original sources

arXiv2026

Adversarial Creation and Detection of AI-Generated Social Bot Content

The convergence of large language models and social bots allows malicious actors to manipulate the information ecosystem by generating human-like content at scale. Existing models for detecting AI-generated content often fail in the wild, primarily due to the lack of ground-truth data. We address this gap through an adversarial methodology that models the impersonation of real social media users by malicious actors. Using this methodology, we curate a multilingual, cross-platform dataset of paired human and AI-generated messages. Training on such adversarial data yields accurate detection of AI-generated text. Our approach significantly outperforms existing models for content-based bot detection in real-world, out-of-distribution data.

cs.CL↗

arXiv2026

The Invisible Risks of AI-Generated Health Information

Generative artificial intelligence (AI) systems now summarize health-related search results, answer medical questions, and offer guidance people once sought from clinicians. These systems bring real benefits, including plain-language explanations of medical information, around-the-clock availability, and expanded access for people facing language or literacy barriers. They also carry new risks: inaccurate guidance can harm people at scale, and malicious actors can now generate personalized health misinformation at negligible cost. In this Perspective, we argue that these risks are largely invisible to the institutions responsible for protecting public health. When AI guidance causes harm, no record exists outside the platform, no channel allows users to report it, and no independent researcher can measure the consequences. We trace these invisible risks across two settings: incidental exposure online and active seeking through search engines and chatbots. Minimizing harm from AI-generated health information requires making it observable. We therefore offer recommendations that aim to improve transparency, mitigate harm at the point of delivery, and assign accountability, ranging from voluntary platform measures to regulatory ones.

cs.CY↗

arXiv2026

PluRule: A Benchmark for Moderating Pluralistic Communities on Social Media

Social media are shifting towards pluralism -- community-governed platforms where groups define their own norms. What violates rules in one community may be perfectly acceptable in another. Can AI models help moderate such pluralistic communities? We formalize the task as a multiple-choice problem, mirroring how human moderators operate in the real world: given a comment and its surrounding context, identify which specific rule, if any, is violated. We introduce PluRule, a multimodal, multilingual benchmark for detecting 13,371 rule violations across 1,989 Reddit communities spanning 2,885 rules in 9 languages. Using this benchmark, we show that state-of-the-art vision-language models struggle significantly: even GPT-5.2 with high reasoning performs only slightly better than a trivial baseline. We also find that bigger models and increased context provide marginal gains, and universal rules like civility and self-promotion are easier to detect. Our results show that moderation of pluralistic communities on social media is a fundamental challenge for language models. Our code and benchmark are publicly available.

cs.CL↗

arXiv2026

Emergence of Stereotypes and Affective Polarization from Belief Network Dynamics

Our belief systems are shaped by social processes, such as observations and influence, and by cognitive processes, such as the drive for internal coherence. These processes steer how individual beliefs evolve and become connected. The resulting belief networks contain both causal and associative links, including spurious ones, such as stereotypes. Here, we develop an agent-based model of belief networks that demonstrates how two basic mechanisms -- social interaction and a drive for internal coherence -- can give rise to such stereotypes without any underlying reality. We further demonstrate how stereotypes, when coupled with shared group identity, can give rise to affective polarization, even in the absence of ideological conflicts.

cs.SI↗

arXiv2026

"F*** You Biden": Cross-Partisan Electoral Toxicity on X

Political discourse on social media has grown increasingly toxic, with electoral periods amplifying partisan hostility and cross-group attacks. Yet it remains unclear whether toxicity in online political speech reflects how partisans communicate within their own circles, or how aggressively they engage with the opposition. Disentangling these dynamics is critical for understanding online political hostility and for designing effective content moderation. We examine this question at scale using a large collection of original posts and replies from X (formerly Twitter), collected during the 2024 U.S. presidential election. Using a human-validated large language model to classify the political alignment of posts and users, and the Perspective API for toxicity scoring, we uncover a striking asymmetry: Republican-leaning posts are significantly more toxic than Democratic-leaning posts, yet Democratic-leaning posts attract significantly more toxic replies. To interpret this finding, we compare the toxicity of same-party and cross-partisan replies. While cross-partisan replies are slightly but significantly more toxic than same-party replies, this is true for both Democratic and Republican posts. However, Republican users account for a large majority of replies to Democratic posts, while Democrats account for a minority of replies to Republican content. Therefore, the elevated toxicity directed at Democratic content is better explained by the volume of Republican cross-partisan replies.

cs.SI↗

arXiv2026

LLMs Can Infer Political Alignment from Online Conversations

Due to the correlational structure in our traits such as identities, cultures, and political attitudes, seemingly innocuous preferences like following a band or using a specific slang can reveal private traits. This possibility, especially when combined with massive, public social data and advanced computational methods, poses a fundamental privacy risk. As our data exposure online and the rapid advancement of AI are increasing the risk of misuse, it is critical to understand the capacity of large language models (LLMs) to exploit such potential. Here, using online discussions on DebateOrg and Reddit, we show that LLMs can reliably infer hidden political alignment, significantly outperforming traditional machine learning models. Prediction accuracy further improves as we aggregate multiple text-level inferences into a user-level prediction, and as we use more politics-adjacent domains. We demonstrate that LLMs leverage words that are highly predictive of political alignment while not being explicitly political. Our findings underscore the capacity and risks of LLMs for exploiting socio-cultural correlates.

cs.SI↗

arXiv2026

Behavior Change as a Signal for Identifying Social Media Manipulation

Social media accounts engaging in online manipulation can change their behaviors for re-purposing or to evade detection. Existing detection systems are built on features that do not exploit such behavioral patterns. Here we investigate the degree to which change in behavior can serve as a signal for identifying automated or coordinated accounts. First, we use Behavioral Languages for Online Characterization (BLOC) to represent the behavior of a social media account as a sequence of symbols that represent the account's actions and content. Second, we segment an account's BLOC strings and measure the changes between consecutive segments. Third, we represent an account as a feature vector that captures the distribution of behavioral change values. Finally, the resulting features are used to train and test supervised classifiers. We apply the proposed method to two detection tasks aimed at automated behavior (social bots) and coordinated inauthentic behavior (information operations). Our results reveal that the distributions of behavioral changes tend to be consistent across authentic accounts, while social bots exhibit either very low or very high behavioral change. Coordinated inauthentic accounts exhibit highly similar distributions of behavioral change within the same campaign, but diverse across campaigns. These patterns allow our classifiers to achieve good accuracy in both tasks, demonstrating the effectiveness of behavioral change as a signal for identifying online manipulation.

cs.SI↗

arXiv2026

A Marketplace for AI-Generated Adult Content and Deepfakes

Generative AI systems increasingly enable the production of highly realistic synthetic media. Civitai, a popular community-driven platform for AI-generated content, operates a monetized feature called Bounties, which allows users to commission the generation of content in exchange for payment. To examine how this mechanism is used and what content it incentivizes, we conduct a longitudinal analysis of all publicly available bounty requests collected over a 14-month period following the platform's launch. We find that the bounty marketplace is dominated by tools that let users steer AI models toward content they were not trained to generate. At the same time, requests for content that is "Not Safe For Work" are widespread and have increased steadily over time, now comprising a majority of all bounties. Participation in bounty creation is uneven, with 20% of requesters accounting for roughly half of requests. Requests for "deepfake" - media depicting identifiable real individuals - exhibit a higher concentration than other types of bounties. A nontrivial subset of these requests involves explicit deepfakes despite platform policies prohibiting such content. These bounties disproportionately target female celebrities, revealing a pronounced gender asymmetry in social harm. Together, these findings show how monetized, community-driven generative AI platforms can produce gendered harms, raising questions about consent, governance, and enforcement.

cs.CY↗

arXiv2025

Large Language Models Require Curated Context for Reliable Political Fact-Checking -- Even with Reasoning and Web Search

Large language models (LLMs) have raised hopes for automated end-to-end fact-checking, but prior studies report mixed results. As mainstream chatbots increasingly ship with reasoning capabilities and web search tools -- and millions of users already rely on them for verification -- rigorous evaluation is urgent. We evaluate 15 recent LLMs from OpenAI, Google, Meta, and DeepSeek on more than 6,000 claims fact-checked by PolitiFact, comparing standard models with reasoning- and web-search variants. Standard models perform poorly, reasoning offers minimal benefits, and web search provides only moderate gains, despite fact-checks being available on the web. In contrast, a curated RAG system using PolitiFact summaries improved macro F1 by 233% on average across model variants. These findings suggest that giving models access to curated high-quality context is a promising path for automated fact-checking.

cs.CL↗

arXiv2025

Community Notes are Vulnerable to Rater Bias and Manipulation

Social media platforms increasingly rely on crowdsourced moderation systems like Community Notes to combat misinformation at scale. However, these systems face challenges from rater bias and potential manipulation, which may undermine their effectiveness. Here we systematically evaluate the Community Notes algorithm using simulated data that models realistic rater and note behaviors, quantifying error rates in publishing helpful versus unhelpful notes. We find that the algorithm suppresses a substantial fraction of genuinely helpful notes and is highly sensitive to rater biases, including polarization and in-group preferences. Moreover, a small minority (5--20\%) of bad raters can strategically suppress targeted helpful notes, effectively censoring reliable information. These findings suggest that while community-driven moderation may offer scalability, its vulnerability to bias and manipulation raises concerns about reliability and trustworthiness, highlighting the need for improved mechanisms to safeguard the integrity of crowdsourced fact-checking.

cs.SI↗

arXiv2025

Identifying Constructive Conflict in Online Discussions through Controversial yet Toxicity Resilient Posts

Bridging content that brings together individuals with opposing viewpoints on social media remains elusive, overshadowed by echo chambers and toxic exchanges. We propose that algorithmic curation could surface such content by considering constructive conflicts as a foundational criterion. We operationalize this criterion through controversiality to identify challenging dialogues and toxicity resilience to capture respectful conversations. We develop high-accuracy models to capture these dimensions. Analyses based on these models demonstrate that assessing resilience to toxic responses is not the same as identifying low-toxicity posts. We also find that political posts are often controversial and tend to attract more toxic responses. However, some posts, even the political ones, are resilient to toxicity despite being highly controversial, potentially sparking civil engagement. Toxicity resilient posts tend to use politeness cues, such as showing gratitude and hedging. These findings suggest the potential for framing the tone of posts to encourage constructive political discussions.

cs.SI↗

arXiv2025

Community Moderation and the New Epistemology of Fact Checking on Social Media

Social media platforms have traditionally relied on internal moderation teams and partnerships with independent fact-checking organizations to identify and flag misleading content. Recently, however, platforms including X (formerly Twitter) and Meta have shifted towards community-driven content moderation by launching their own versions of crowd-sourced fact-checking -- Community Notes. If effectively scaled and governed, such crowd-checking initiatives have the potential to combat misinformation with increased scale and speed as successfully as community-driven efforts once did with spam. Nevertheless, general content moderation, especially for misinformation, is inherently more complex. Public perceptions of truth are often shaped by personal biases, political leanings, and cultural contexts, complicating consensus on what constitutes misleading content. This suggests that community efforts, while valuable, cannot replace the indispensable role of professional fact-checkers. Here we systemically examine the current approaches to misinformation detection across major platforms, explore the emerging role of community-driven moderation, and critically evaluate both the promises and challenges of crowd-checking at scale.

cs.SI↗

arXiv2025

Prefill-Guided Thinking for zero-shot detection of AI-generated images

Traditional supervised methods for detecting AI-generated images depend on large, curated datasets for training and fail to generalize to novel, out-of-domain image generators. As an alternative, we explore pre-trained Vision-Language Models (VLMs) for zero-shot detection of AI-generated images. We evaluate VLM performance on three diverse benchmarks encompassing synthetic images of human faces, objects, and animals produced by 16 different state-of-the-art image generators. While off-the-shelf VLMs perform poorly on these datasets, we find that prefilling responses effectively guides their reasoning -- a method we call Prefill-Guided Thinking (PGT). In particular, prefilling a VLM response with the phrase "Let's examine the style and the synthesis artifacts" improves the Macro F1 scores of three widely used open-source VLMs by up to 24%. We analyze this improvement in detection by tracking answer confidence during response generation. For some models, prefills counteract early overconfidence -- akin to mitigating the Dunning-Kruger effect -- leading to better detection performance.

cs.LG↗

arXiv2025

How malicious AI swarms can threaten democracy: The fusion of agentic AI and LLMs marks a new frontier in information warfare

Advances in AI offer the prospect of manipulating beliefs and behaviors on a population-wide level. Large language models and autonomous agents now let influence campaigns reach unprecedented scale and precision. Generative tools can expand propaganda output without sacrificing credibility and inexpensively create falsehoods that are rated as more human-like than those written by humans. Techniques meant to refine AI reasoning, such as chain-of-thought prompting, can just as effectively be used to generate more convincing falsehoods. Enabled by these capabilities, a disruptive threat is emerging: swarms of collaborative, malicious AI agents. Fusing LLM reasoning with multi-agent architectures, these systems are capable of coordinating autonomously, infiltrating communities, and fabricating consensus efficiently. By adaptively mimicking human social dynamics, they threaten democracy. Because the resulting harms stem from design, commercial incentives, and governance, we prioritize interventions at multiple leverage points, focusing on pragmatic mechanisms over voluntary compliance.

cs.CY↗

arXiv2025

A longitudinal analysis of misinformation, polarization and toxicity on Bluesky after its public launch

Bluesky is a decentralized, Twitter-like social media platform that has rapidly gained popularity. Following an invite-only phase, it officially opened to the public on February 6th, 2024, leading to a significant expansion of its user base. In this paper, we present a longitudinal analysis of user activity in the two months surrounding its public launch, examining how the platform evolved due to this rapid growth. Our analysis reveals that Bluesky exhibits an activity distribution comparable to more established social platforms, yet it features a higher volume of original content relative to reshared posts and maintains low toxicity levels. We further investigate the political leanings of its user base, misinformation dynamics, and engagement in harmful conversations. Our findings indicate that Bluesky users predominantly lean left politically and tend to share high-credibility sources. After the platform's public launch, an influx of new users, particularly those posting in English and Japanese, contributed to a surge in activity. Among them, several accounts displayed suspicious behaviors, such as mass-following users and sharing content from low-credibility news sources. Some of these accounts have already been flagged as spam or suspended, suggesting that Bluesky's moderation efforts have been effective.

cs.SI↗

arXiv2025

The Rise of Bluesky

This study investigates the rapid growth and evolving network structure of Bluesky from August 2023 to February 2025. Through multiple waves of user migrations, the platform has reached a stable, persistently active user base. The growth process has given rise to a dense follower network with clustering and hub features that favor viral information diffusion. These developments highlight engagement and structural similarities between Bluesky and established platforms.

cs.SI↗

arXiv2025

Beyond Interaction Patterns: Assessing Claims of Coordinated Inter-State Information Operations on Twitter/X

Social media platforms have become key tools for coordinated influence operations, enabling state actors to manipulate public opinion through strategic, collective actions. While previous research has suggested collaboration between states, such research failed to leverage state-of-the-art coordination indicators or control datasets. In this study, we investigate inter-state coordination by analyzing multiple online behavioral traces and using sophisticated coordination detection models. By incorporating a control dataset to differentiate organic user activity from coordinated efforts, our findings reveal no evidence of inter-state coordination. These results challenge earlier claims and underscore the importance of robust methodologies and control datasets in accurately detecting online coordination.

cs.SI↗

arXiv2025

Audit of takedown delays across social media reveals failure to reduce exposure to illegal content

Illegal content on social media poses significant societal harm and necessitates timely removal. However, the impact of the speed of content removal on prevalence, reach, and exposure to illegal content remains underexplored. This study examines the relationship with a systematic audit of takedown delays using data from the EU Digital Services Act Transparency Database, covering five major platforms over a one-year period. We find substantial variation in takedown delay, with some content remaining online for weeks or even months. To evaluate how these delays affect the prevalence and reach of illegal content and exposure to it, we develop an agent-based model and calibrate it to empirical data. We simulate illegal content diffusion, revealing that rapid takedown (within hours) significantly reduces prevalence, reach, and exposure to illegal content, while the longer delays measured by the audit fail to reduce its spread. Though the link between delay and spread is intuitive, our simulations quantify exactly how takedown speed shapes exposure to illegal content. Building on these results, we point to the benefits of faster content removal to effectively curb the spread of illegal content, while also considering the limitations of strict enforcement policies.

cs.SI↗