arXiv subjects

Ali Naseh

Publications and source records attributed to Ali Naseh.

At least 19 recordsLinked to original sources

arXiv2026

Identifying Models Behind Text-to-Image Leaderboards

Text-to-image (T2I) models are increasingly popular, producing a large share of AI-generated images online. To compare model quality, voting-based leaderboards have become the standard, relying on anonymized model outputs for fairness. In this work, we show that such anonymity can be easily broken. We find that generations from each T2I model form distinctive clusters in the image embedding space, enabling accurate deanonymization without prompt control or training data. Using 22 models and 280 prompts (150K images), our centroid-based method achieves high accuracy and reveals systematic model-specific signatures. We further introduce a prompt-level distinguishability metric and conduct large-scale analyses showing how certain prompts can lead to near-perfect distinguishability. Our findings expose fundamental security flaws in T2I leaderboards and motivate stronger anonymization defenses.

cs.CV↗

arXiv2025

Text-to-Image Models Leave Identifiable Signatures: Implications for Leaderboard Security

Generative AI leaderboards are central to evaluating model capabilities, but remain vulnerable to manipulation. Among key adversarial objectives is rank manipulation, where an attacker must first deanonymize the models behind displayed outputs -- a threat previously demonstrated and explored for large language models (LLMs). We show that this problem can be even more severe for text-to-image leaderboards, where deanonymization is markedly easier. Using over 150,000 generated images from 280 prompts and 19 diverse models spanning multiple organizations, architectures, and sizes, we demonstrate that simple real-time classification in CLIP embedding space identifies the generating model with high accuracy, even without prompt control or historical data. We further introduce a prompt-level separability metric and identify prompts that enable near-perfect deanonymization. Our results indicate that rank manipulation in text-to-image leaderboards is easier than previously recognized, underscoring the need for stronger defenses.

cs.LG↗

arXiv2025

Throttling Web Agents Using Reasoning Gates

AI web agents use Internet resources at far greater speed, scale, and complexity -- changing how users and services interact. Deployed maliciously or erroneously, these agents could overload content providers. At the same time, web agents can bypass CAPTCHAs and other defenses by mimicking user behavior or flood authentication systems with fake accounts. Yet providers must protect their services and content from denial-of-service attacks and scraping by web agents. In this paper, we design a framework that imposes tunable costs on agents before providing access to resources; we call this Web Agent Throttling. We start by formalizing Throttling Gates as challenges issued to an agent that are asymmetric, scalable, robust, and compatible with any agent. Focusing on a common component -- the language model -- we require the agent to solve reasoning puzzles, thereby incurring excessive token-generation costs. However, we find that using existing puzzles, e.g., coding or math, as throttling gates fails to satisfy our properties. To address this, we introduce rebus-based Reasoning Gates, synthetic text puzzles that require multi-hop reasoning over world knowledge (thereby throttling an agent's model). We design a scalable generation and verification protocol for such reasoning gates. Our framework achieves computational asymmetry, i.e., the response-generation cost is 9.2x higher than the generation cost for SOTA models. We further deploy reasoning gates on a custom website and Model Context Protocol (MCP) servers and evaluate with real-world web agents. Finally, we discuss the limitations and environmental impact of real-world deployment of our framework.

cs.AI↗

arXiv2025

Exploiting Leaderboards for Large-Scale Distribution of Malicious Models

While poisoning attacks on machine learning models have been extensively studied, the mechanisms by which adversaries can distribute poisoned models at scale remain largely unexplored. In this paper, we shed light on how model leaderboards -- ranked platforms for model discovery and evaluation -- can serve as a powerful channel for adversaries for stealthy large-scale distribution of poisoned models. We present TrojanClimb, a general framework that enables injection of malicious behaviors while maintaining competitive leaderboard performance. We demonstrate its effectiveness across four diverse modalities: text-embedding, text-generation, text-to-speech and text-to-image, showing that adversaries can successfully achieve high leaderboard rankings while embedding arbitrary harmful functionalities, from backdoors to bias injection. Our findings reveal a significant vulnerability in the machine learning ecosystem, highlighting the urgent need to redesign leaderboard evaluation mechanisms to detect and filter malicious (e.g., poisoned) models, while exposing broader security implications for the machine learning community regarding the risks of adopting models from unverified sources.

cs.LG↗

arXiv2025

R1dacted: Investigating Local Censorship in DeepSeek's R1 Language Model

DeepSeek recently released R1, a high-performing large language model (LLM) optimized for reasoning tasks. Despite its efficient training pipeline, R1 achieves competitive performance, even surpassing leading reasoning models like OpenAI's o1 on several benchmarks. However, emerging reports suggest that R1 refuses to answer certain prompts related to politically sensitive topics in China. While existing LLMs often implement safeguards to avoid generating harmful or offensive outputs, R1 represents a notable shift - exhibiting censorship-like behavior on politically charged queries. In this paper, we investigate this phenomenon by first introducing a large-scale set of heavily curated prompts that get censored by R1, covering a range of politically sensitive topics, but are not censored by other models. We then conduct a comprehensive analysis of R1's censorship patterns, examining their consistency, triggers, and variations across topics, prompt phrasing, and context. Beyond English-language queries, we explore censorship behavior in other languages. We also investigate the transferability of censorship to models distilled from the R1 language model. Finally, we propose techniques for bypassing or removing this censorship. Our findings reveal possible additional censorship integration likely shaped by design choices during training or alignment, raising concerns about transparency, bias, and governance in language model deployment.

cs.CL↗

arXiv2025

LLM Misalignment via Adversarial RLHF Platforms

Reinforcement learning has shown remarkable performance in aligning language models with human preferences, leading to the rise of attention towards developing RLHF platforms. These platforms enable users to fine-tune models without requiring any expertise in developing complex machine learning algorithms. While these platforms offer useful features such as reward modeling and RLHF fine-tuning, their security and reliability remain largely unexplored. Given the growing adoption of RLHF and open-source RLHF frameworks, we investigate the trustworthiness of these systems and their potential impact on behavior of LLMs. In this paper, we present an attack targeting publicly available RLHF tools. In our proposed attack, an adversarial RLHF platform corrupts the LLM alignment process by selectively manipulating data samples in the preference dataset. In this scenario, when a user's task aligns with the attacker's objective, the platform manipulates a subset of the preference dataset that contains samples related to the attacker's target. This manipulation results in a corrupted reward model, which ultimately leads to the misalignment of the language model. Our results demonstrate that such an attack can effectively steer LLMs toward undesirable behaviors within the targeted domains. Our work highlights the critical need to explore the vulnerabilities of RLHF platforms and their potential to cause misalignment in LLMs during the RLHF fine-tuning process.

cs.LG↗

arXiv2025

OverThink: Slowdown Attacks on Reasoning LLMs

Most flagship language models generate explicit reasoning chains, enabling inference-time scaling. However, producing these reasoning chains increases token usage (i.e., reasoning tokens), which in turn increases latency and costs. Our OverThink attack increases overhead for applications that rely on reasoning language models (RLMs) and external context by forcing them to spend substantially more reasoning tokens while still producing contextually correct answers. An adversary mounts an attack by injecting decoy reasoning problems into public content that is consumed by RLM at inference time. Because our decoys (e.g., Markov decision processes, Sudokus, etc.) are benign, they evade safety filters. We evaluate OverThink on both closed-source and open-source reasoning models across the FreshQA, SQuAD, and MuSR datasets. We also explore the attack in multi-modal settings by creating images that cause excessive reasoning. We show that the resulting slowdown transfers across models. Finally, we explore both LLM-based and systems-level defenses, and discuss the societal, financial, and energy implications of the OverThink attacks.

cs.LG↗

arXiv2025

Riddle Me This! Stealthy Membership Inference for Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) enables Large Language Models (LLMs) to generate grounded responses by leveraging external knowledge databases without altering model parameters. Although the absence of weight tuning prevents leakage via model parameters, it introduces the risk of inference adversaries exploiting retrieved documents in the model's context. Existing methods for membership inference and data extraction often rely on jailbreaking or carefully crafted unnatural queries, which can be easily detected or thwarted with query rewriting techniques common in RAG systems. In this work, we present Interrogation Attack (IA), a membership inference technique targeting documents in the RAG datastore. By crafting natural-text queries that are answerable only with the target document's presence, our approach demonstrates successful inference with just 30 queries while remaining stealthy; straightforward detectors identify adversarial prompts from existing methods up to ~76x more frequently than those generated by our attack. We observe a 2x improvement in TPR@1%FPR over prior inference attacks across diverse RAG configurations, all while costing less than $0.02 per document inference.

cs.CR↗

arXiv2025

Synthetic Data Can Mislead Evaluations: Membership Inference as Machine Text Detection

Recent work shows membership inference attacks (MIAs) on large language models (LLMs) produce inconclusive results, partly due to difficulties in creating non-member datasets without temporal shifts. While researchers have turned to synthetic data as an alternative, we show this approach can be fundamentally misleading. Our experiments indicate that MIAs function as machine-generated text detectors, incorrectly identifying synthetic data as training samples regardless of the data source. This behavior persists across different model architectures and sizes, from open-source models to commercial ones such as GPT-3.5. Even synthetic text generated by different, potentially larger models is classified as training data by the target model. Our findings highlight a serious concern: using synthetic data in membership evaluations may lead to false conclusions about model memorization and data leakage. We caution that this issue could affect other evaluations using model signals such as loss where synthetic or machine-generated translated data substitutes for real-world samples.

cs.CL↗

arXiv2024

Renormalized Volume, Polyakov Anomaly and Orbifold Riemann Surfaces

In arXiv:2310.17536, two of the authors studied the function $\mathscr{S}_{\boldsymbol{m}} = S_{\boldsymbol{m}} - \pi \sum_{i=1}^n (m_i - \tfrac{1}{m_i}) \log \mathsf{h}_{i}$ for orbifold Riemann surfaces of signature $(g;m_1,...,m_{n_e};n_p)$ on the generalized Schottky space $\mathfrak{S}_{g,n}(\boldsymbol{m})$. In this paper, we prove the holographic duality between $\mathscr{S}_{\boldsymbol{m}}$ and the renormalized hyperbolic volume $V_{\text{ren}}$ of the corresponding Schottky 3-orbifolds with lines of conical singularity that reach the conformal boundary. In case of the classical Liouville action on $\mathfrak{S}_{g}$ and $\mathfrak{S}_{g,n}(\boldsymbol{\infty})$, the holography principle was proved in arXiv:hep-th/0005106v2 and arXiv:1508.02102, respectively. Our result implies that $V_{\text{ren}}$ acts as K\"ahler potential for a particular combination of the Weil-Petersson and Takhtajan-Zograf metrics that appears in the local index theorem for orbifold Riemann surfaces arXiv:1701.00771. Moreover, we demonstrate that under the conformal transformations, the change of function $\mathscr{S}_{\boldsymbol{m}}$ is equivalent to the Polyakov anomaly, which indicates that the function $\mathscr{S}_{\boldsymbol{m}}$ is a consistent height function with a unique hyperbolic solution. Consequently, the associated renormalized hyperbolic volume $V_{\text{ren}}$ also admits a Polyakov anomaly formula. The method we used to establish this equivalence may provide an alternative approach to derive the renormalized Polyakov anomaly for Riemann surfaces with punctures (cusps), as described in arXiv:0909.0807.

hep-th↗

arXiv2024

Backdooring Bias ($B^2$) into Stable Diffusion Models

Recent advances in large text-conditional diffusion models have revolutionized image generation by enabling users to create realistic, high-quality images from textual prompts, significantly enhancing artistic creation and visual communication. However, these advancements also introduce an underexplored attack opportunity: the possibility of inducing biases by an adversary into the generated images for malicious intentions, e.g., to influence public opinion and spread propaganda. In this paper, we study an attack vector that allows an adversary to inject arbitrary bias into a target model. The attack leverages low-cost backdooring techniques using a targeted set of natural textual triggers embedded within a small number of malicious data samples produced with public generative models. An adversary could pick common sequences of words that can then be inadvertently activated by benign users during inference. We investigate the feasibility and challenges of such attacks, demonstrating how modern generative models have made this adversarial process both easier and more adaptable. On the other hand, we explore various aspects of the detectability of such attacks and demonstrate that the model's utility remains intact in the absence of the triggers. Our extensive experiments using over 200,000 generated images and against hundreds of fine-tuned models demonstrate the feasibility of the presented backdoor attack. We illustrate how these biases maintain strong text-image alignment, highlighting the challenges in detecting biased images without knowing that bias in advance. Our cost analysis confirms the low financial barrier (\$10-\$15) to executing such attacks, underscoring the need for robust defensive strategies against such vulnerabilities in diffusion models.

cs.LG↗

arXiv2024

Iteratively Prompting Multimodal LLMs to Reproduce Natural and AI-Generated Images

With the digital imagery landscape rapidly evolving, image stocks and AI-generated image marketplaces have become central to visual media. Traditional stock images now exist alongside innovative platforms that trade in prompts for AI-generated visuals, driven by sophisticated APIs like DALL-E 3 and Midjourney. This paper studies the possibility of employing multi-modal models with enhanced visual understanding to mimic the outputs of these platforms, introducing an original attack strategy. Our method leverages fine-tuned CLIP models, a multi-label classifier, and the descriptive capabilities of GPT-4V to create prompts that generate images similar to those available in marketplaces and from premium stock image providers, yet at a markedly lower expense. In presenting this strategy, we aim to spotlight a new class of economic and security considerations within the realm of digital imagery. Our findings, supported by both automated metrics and human assessment, reveal that comparable visual content can be produced for a fraction of the prevailing market prices ($0.23 - $0.27 per image), emphasizing the need for awareness and strategic discussions about the integrity of digital media in an increasingly AI-integrated landscape. Our work also contributes to the field by assembling a dataset consisting of approximately 19 million prompt-image pairs generated by the popular Midjourney platform, which we plan to release publicly.

cs.CR↗

arXiv2023

Diffence: Fencing Membership Privacy With Diffusion Models

Deep learning models, while achieving remarkable performances, are vulnerable to membership inference attacks (MIAs). Although various defenses have been proposed, there is still substantial room for improvement in the privacy-utility trade-off. In this work, we introduce a novel defense framework against MIAs by leveraging generative models. The key intuition of our defense is to remove the differences between member and non-member inputs, which is exploited by MIAs, by re-generating input samples before feeding them to the target model. Therefore, our defense, called DIFFENCE, works pre inference, which is unlike prior defenses that are either training-time or post-inference time. A unique feature of DIFFENCE is that it works on input samples only, without modifying the training or inference phase of the target model. Therefore, it can be cascaded with other defense mechanisms as we demonstrate through experiments. DIFFENCE is designed to preserve the model's prediction labels for each sample, thereby not affecting accuracy. Furthermore, we have empirically demonstrated it does not reduce the usefulness of confidence vectors. Through extensive experimentation, we show that DIFFENCE can serve as a robust plug-n-play defense mechanism, enhancing membership privacy without compromising model utility. For instance, DIFFENCE reduces MIA accuracy against an undefended model by 15.8\% and attack AUC by 14.0\% on average across three datasets, all without impacting model utility. By integrating DIFFENCE with prior defenses, we can achieve new state-of-the-art performances in the privacy-utility trade-off. For example, when combined with the state-of-the-art SELENA defense it reduces attack accuracy by 9.3\%, and attack AUC by 10.0\%. DIFFENCE achieves this by imposing a negligible computation overhead, adding only 57ms to the inference time per sample processed on average.

cs.CR↗

arXiv2023

Memory Triggers: Unveiling Memorization in Text-To-Image Generative Models through Word-Level Duplication

Diffusion-based models, such as the Stable Diffusion model, have revolutionized text-to-image synthesis with their ability to produce high-quality, high-resolution images. These advancements have prompted significant progress in image generation and editing tasks. However, these models also raise concerns due to their tendency to memorize and potentially replicate exact training samples, posing privacy risks and enabling adversarial attacks. Duplication in training datasets is recognized as a major factor contributing to memorization, and various forms of memorization have been studied so far. This paper focuses on two distinct and underexplored types of duplication that lead to replication during inference in diffusion-based models, particularly in the Stable Diffusion model. We delve into these lesser-studied duplication phenomena and their implications through two case studies, aiming to contribute to the safer and more responsible use of generative models in various applications.

cs.CR↗

arXiv2023

Understanding (Un)Intended Memorization in Text-to-Image Generative Models

Multimodal machine learning, especially text-to-image models like Stable Diffusion and DALL-E 3, has gained significance for transforming text into detailed images. Despite their growing use and remarkable generative capabilities, there is a pressing need for a detailed examination of these models' behavior, particularly with respect to memorization. Historically, memorization in machine learning has been context-dependent, with diverse definitions emerging from classification tasks to complex models like Large Language Models (LLMs) and Diffusion models. Yet, a definitive concept of memorization that aligns with the intricacies of text-to-image synthesis remains elusive. This understanding is vital as memorization poses privacy risks yet is essential for meeting user expectations, especially when generating representations of underrepresented entities. In this paper, we introduce a specialized definition of memorization tailored to text-to-image models, categorizing it into three distinct types according to user expectations. We closely examine the subtle distinctions between intended and unintended memorization, emphasizing the importance of balancing user privacy with the generative quality of the model outputs. Using the Stable Diffusion model, we offer examples to validate our memorization definitions and clarify their application.

cs.CV↗

arXiv2023

Classical Liouville Action and Uniformization of Orbifold Riemann Surfaces

We study the classical Liouville field theory on Riemann surfaces of genus $g>1$ in the presence of vertex operators associated with branch points of orders $m_i>1$. In order to do so, we consider the generalized Schottky space $\mathfrak{S}_{g,n}(\boldsymbol{m})$ obtained as a holomorphic fibration over the Schottky space $\mathfrak{S}_g$ of the (compactified) underlying Riemann surface. Those fibers correspond to configuration spaces of $n$ orbifold points of orders $\boldsymbol{m}=(m_1,\dots,m_n)$. Drawing on the previous work of Park, Teo, and Takhtajan \cite{park2015potentials} as well as Takhtajan and Zograf \cite{ZT_2018}, we define Hermitian metrics $\mathsf{h}_i$ for tautological line bundles $\mathscr{L}_i$ over $\mathfrak{S}_{g,n}(\boldsymbol{m})$. These metrics are expressed in terms of the first coefficient of the expansion of covering map $J$ of the Schottky domain. Additionally, we define the regularized classical Liouville action $S_{\boldsymbol{m}}$ using Schottky global coordinates on Riemann orbisurfaces with genus $g>1$. We demonstrate that $\exp{S_{\boldsymbol{m}}/\pi}$ serves as a Hermitian metric on the $\mathbb{Q}$-line bundle $\mathscr{L}=\bigotimes_{i=1}^{n}\mathscr{L}_i^{\otimes (1-1/m_i^2)}$ over $\mathfrak{S}_{g,n}(\boldsymbol{m})$. Furthermore, we explicitly compute the first and second variations of the smooth real-valued function $\mathscr{S}_{\boldsymbol{m}}=S_{\boldsymbol{m}}-\pi\sum_{i=1}^n(m_i-\tfrac{1}{m_i})\log\mathsf{h}_{i}$ on the Schottky deformation space $\mathfrak{S}_{g,n}(\boldsymbol{m})$. We establish two key results: (i) $\mathscr{S}_{\boldsymbol{m}}$ generates a combination of accessory and auxiliary parameters, and (ii) $-\mathscr{S}_{\boldsymbol{m}}$ acts as a K\"{a}hler potential for a specific combination of Weil-Petersson and Takhtajan-Zograf metrics that appear in the local index theorem for orbifold Riemann surfaces \cite{ZT_2018}.

hep-th↗

arXiv2023

Stealing the Decoding Algorithms of Language Models

A key component of generating text from modern language models (LM) is the selection and tuning of decoding algorithms. These algorithms determine how to generate text from the internal probability distribution generated by the LM. The process of choosing a decoding algorithm and tuning its hyperparameters takes significant time, manual effort, and computation, and it also requires extensive human evaluation. Therefore, the identity and hyperparameters of such decoding algorithms are considered to be extremely valuable to their owners. In this work, we show, for the first time, that an adversary with typical API access to an LM can steal the type and hyperparameters of its decoding algorithms at very low monetary costs. Our attack is effective against popular LMs used in text generation APIs, including GPT-2, GPT-3 and GPT-Neo. We demonstrate the feasibility of stealing such information with only a few dollars, e.g., $\$0.8$, $\$1$, $\$4$, and $\$40$ for the four versions of GPT-3.

cs.LG↗

arXiv2022

Islands in Flat-Space Cosmology

Flat-space cosmologies (FSC) are solutions to three dimensional theories of gravity without cosmological constant that have cosmological horizons. A detector located near the time-like singularity of the spacetime can absorb Hawking modes that are created near the horizon. Continuation of this process will eventually cause the entropy of the radiation to be larger than the entropy of the FSC, which leads to the information paradox. In this paper, we resolve this paradox for the FSC using the island proposal. To do this, we couple an auxiliary flat bath system to this spacetime in timelike singularity so that Hawking modes are allowed to enter the bath and the entropy of radiation can be measured in its asymptotic region where gravity is also weak. We show that adding island regions that receive the partners of Hawking modes cause the entropy of radiation to follow a Page curve which leads to resolving the information paradox. Moreover, we design a quantum teleportation protocol by which one can extract the information residing in islands.

hep-th↗