Searcharxiv⌕ Search

arXiv · 2611.15010

Calibrate, Don't Retrain: Decomposing the Value of Annotator Disagreement in Cyberbullying Severity

Abstract

Machine learning models are often trained on labeled data produced by multiple human annotators. When those annotators disagree, their labels are commonly collapsed into a single majority vote, which discards information about the underlying disagreement. We examine the cost of this practice for cyberbullying severity, a subjective judgment on which annotators disagree heavily. In the corpus we study, all 5 annotators assigned the same severity label to only 4.1% of comments. Turning 5 judgments into one label hides this disagreement and obscures how much the annotators differed in their assessments. In this paper, we ask whether a model can preserve this information by predicting the distribution of annotator judgments instead of a single majority label. We find 3 main results. First, accuracy against the majority label cannot tell competent models apart: 2 models with similar accuracy can differ greatly in how well they match the distribution of annotator judgments. Second, predicting the full distribution requires little extra effort. Calibrating an existing model, which adjusts its output probabilities after training, captures most of the improvement without retraining, across 3 datasets covering severity, abuse, and irony, while training on label distributions adds a small further gain. Third, we examine a dataset that includes annotator demographics, where a single shared model has one advantage and one limit. The advantage is that a model predicting the distribution of judgments matches each demographic group's judgments significantly better than a model trained on majority labels, so both groups gain. The limit is that it does not narrow the gap in performance between the groups, because one shared model cannot encode which group is right on a given comment. Preserving annotator disagreement is therefore a low-cost step toward fairer moderation, though not a complete one.

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Arslan Bisharat, Katelyn Skees, Manuel Sandoval Madrigal, Mujtaba Nazari, Ayaan Khan, Deborah L. Hall, Mohammed Abuhamad, Yasin N. Silva. 2026-11-11. Calibrate, Don't Retrain: Decomposing the Value of Annotator Disagreement in Cyberbullying Severity. https://searcharxiv.com/abs/2611.15010

Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.

KEEP EXPLORING

Related papers

Network sampling: a comparative review

Network sampling is essential for the study of real world systems, since access to the topological structure and dynamical characteristics of a complex system is usually infeasible, and can also prove computationally intractable for large and dense networks. However, the field still lacks systematic methodologies that allow a procedural analysis of sampling plans for connected data. Hence, this study aims to compare network sampling methods through the usage of multivariate statistical analysis and machine learning techniques that are anchored in network metrics' estimation of synthetic and real-world networks. It was demonstrated that no single sampling strategy dominates both cases, although Forest fire and Frontier sampling have presented the most consistent performance across all topologies investigated. The impact of each structural property of these networks for principal component analysis and classification results were also discussed, with consistent pattern groups seen for both data types. Transfer learning aimed at real-world network identification has also demonstrated that realistic structural connections can be correlated to well-known network models through the usage of sampled subgraphs' properties for model training, providing an avenue to real-world network characterization through sampling methodologies.

cs.SI↗

Manufactured Authenticity: How Behavioral Signals Are Produced and Misread in Online Markets

Online platforms increasingly ask users to infer credibility from behavioral signals whose production conditions are opaque. We introduce manufactured authenticity: the platform-mediated production of observable activity that interfaces invite users to read as organic participation. Using 996,834 cryptocurrency tokens and 50,895 creator wallets from Pump.fun, a Solana-based token launchpad, we examine how legitimacy signals are produced in an automation-enabled online market. We identify four production strategies (high-velocity launching, passive launching, template-content creation, and organic-style creation) that generate distinct but interface-similar cues of activity, narrative plausibility, and participation. High-velocity and template-content strategies emerge as alternatives rather than cumulative layers: the former produces activity at scale, while the latter produces descriptions associated with higher reply engagement. Yet visible engagement does not imply broader diffusion: template-like content attracts replies without greater ownership dispersion, and tokens associated with non-organic production strategies exhibit more concentrated investor holdings than organic-style tokens. We argue that platform governance should move beyond detecting manipulation toward making signal provenance legible. We propose authenticity disclosure as a design approach for surfacing production conditions without requiring definitive claims about automation, AI authorship, or actor intent.

cs.SI↗

Spatial Price Formation in Transport Networks: A Variational Approach

This paper formulates a model of spatial price formation in a closed regional economy, with finitely many production nodes and finitely many demand nodes embedded in a continuous geographic domain. Transportation costs depend on the geometry of the region and on an underlying network of feasible connections. Prices are modeled as a spatial field constrained by arbitrage and transport feasibility. The equilibrium price system is characterized by a variational problem. A second minimization selects an equilibrium-compatible price field with minimum spatial energy and permits the use of Euler--Lagrange methods. The framework clarifies what observed prices and quantities reveal about demand, and uses known transport costs to bound unobserved prices and exclude directed arcs from equilibrium shipments. A five-node example illustrates equilibrium, alternative shipment patterns, and the information supplied by partial price observations. We give a numerical procedure for planar domains and outline an extension to spherical domains.

cs.SI↗