Michael Todasco | Searcharxiv

arXiv2026

Know Your Author: Does the AI Penalty Hold in Short Fiction?

Public concern about an "AI penalty" suggests that labeling content as AI-generated may negatively influence how it is evaluated. We tested this claim in a preregistered experiment (N = 254, per protocol) using a pure attribution design: participants read one of two ~200-word vignettes and were randomly assigned to see it labeled as Human-written, AI-written, or presented with no author line. Authorship labels did not produce reliable main effects on creativity, enjoyment, recommendation, or originality; observed effect sizes were uniformly small. However, labels strongly influenced inferred effort: participants estimated that Human-labeled stories took far longer to create than AI-labeled stories (back-transformed geometric means from ln[minutes + 1]: 148 vs. 6 minutes). Across conditions, higher inferred effort predicted greater enjoyment, and this relationship was also present within the AI-labeled condition. Additionally, participants' prior attitudes toward AI moderated recommendation judgments: more positive attitudes were associated with higher recommendation ratings for AI-labeled stories, but not for Human-labeled stories. These findings suggest that while AI authorship labels do not systematically alter average evaluations of short fiction, they meaningfully shape perceptions of effort and interact with prior beliefs to influence downstream judgments.

cs.HC↗

arXiv2025

Going All-In on LLM Accuracy: Fake Prediction Markets, Real Confidence Signals

Large language models are increasingly used to evaluate other models, yet these judgments typically lack any representation of confidence. This pilot study tests whether framing an evaluation task as a betting game (a fictional prediction market with its own LLM currency) improves forecasting accuracy and surfaces calibrated confidence signals. We generated 100 math and logic questions with verifiable answers. Six Baseline models (three current-generation, three prior-generation) answered all items. Three Predictor models then forecasted, for each question-baseline pair, if the baseline would answer correctly. Each predictor completed matched runs in two conditions: Control (simple correct/incorrect predictions) and Incentive (predictions plus wagers of 1-100,000 LLMCoin under even odds, starting from a 1,000,000 LLMCoin bankroll). Across 5,400 predictions per condition, Incentive runs showed modestly higher accuracy (81.5% vs. 79.1%, p = .089, d = 0.86) and significantly faster learning across rounds (12.0 vs. 2.9 percentage-point improvement from Round 1 to Round 4, p = .011). Most notably, stake size tracked confidence. "Whale" bets of 40,000+ coins were correct ~99% of the time, while small bets (<1,000 coins) showed only ~74% accuracy. The key finding is not that fictional money makes models smarter; accuracy gains were modest and did not reach statistical significance (p = .089) in this pilot. Rather, the betting mechanic created a legible confidence signal absent from binary yes/no outputs. This suggests that simple financial framing may help transform LLMs into risk-aware forecasters, making their internal beliefs visible and usable. The protocol offers a foundation for future work for meta-evaluation systems and what may become LLM-to-LLM prediction markets.

cs.AI↗