arXiv subjects

Alessandra Urbinati

Publications and source records attributed to Alessandra Urbinati.

6 recordsLinked to original sources

arXiv2026

The Ghost Annotator: a Framework to Explore Human Label Variation in Content Moderation through Conformal Prediction

Current research primarily focuses on model performance, while comparatively less attention has been devoted to uncertainty estimation, particularly in settings where LLMs are increasingly used to generate annotated data. We introduce a framework combining conformal prediction with Collaborative Filtering-style annotators' representation to model LLM behavior in relation to human annotators and to analyze patterns of agreement and disagreement. Using Non-Conformity Scores, we introduce the Ghost Prediction metric and the Ghost Annotator representation to quantify cases in which model predictions diverge from all available human annotations. We compute cosine similarity measures to explore differences in model behavior across sociodemographic axes. We evaluated four LLMs of different size and families across four content moderation datasets. Our finding shows that while we find that all models uncertainty increases with annotator disagreement, larger models tend to be more confident in the classification of texts that are not aligned with any human annotation. Finally, the Ghost Annotator framework reveals a consistent and robust pattern of demographic misalignment, suggesting a structural bias likely rooted in pretraining corpora.

cs.CL↗

arXiv2026

The Retraction Epidemic in Science Across Publishers, Fields, and Countries

Retractions serve as an indicator of failures in research integrity, yet most analyses focus on absolute counts rather than risk per paper. We use one of the largest open bibliographic databases to develop incidence metrics normalized by population: retractions per publication and per active author annually. Applying an epidemiological framework that models counts with exposure, we find evidence of exponential growth in retraction incidence, with approximately a 5-year doubling time at both the paper and author levels. These patterns vary significantly across fields, publishers, and countries. While scientific output is becoming more democratized globally, retractions are concentrated in fewer countries, creating a "concentration" paradox that calls for targeted monitoring. Despite exponential growth, the absolute incidence remains low (0.12% in 2021), allowing for corrective intervention. Incidence-based monitoring provides a framework for evaluating policies that safeguard research integrity at scale.

physics.soc-ph↗

arXiv2025

Are you sure? Measuring models bias in content moderation through uncertainty

Automatic content moderation is crucial to ensuring safety in social media. Language Model-based classifiers are being increasingly adopted for this task, but it has been shown that they perpetuate racial and social biases. Even if several resources and benchmark corpora have been developed to challenge this issue, measuring the fairness of models in content moderation remains an open issue. In this work, we present an unsupervised approach that benchmarks models on the basis of their uncertainty in classifying messages annotated by people belonging to vulnerable groups. We use uncertainty, computed by means of the conformal prediction technique, as a proxy to analyze the bias of 11 models against women and non-white annotators and observe to what extent it diverges from metrics based on performance, such as the $F_1$ score. The results show that some pre-trained models predict with high accuracy the labels coming from minority groups, even if the confidence in their prediction is low. Therefore, by measuring the confidence of models, we are able to see which groups of annotators are better represented in pre-trained models and lead the debiasing process of these models before their effective use.

cs.CL↗

arXiv2021

Measuring Scientific Brain Drain with Hubs and Authorities: a Dual Perspective

In this work we study international migrations of researchers, scientists, and academics from a complex-network perspective to identify the central countries involved in the migration phenomenon. We define the scientific migration network (SMN) as temporal directed weighted network where nodes are world countries, links account for the number of scientists moving from one country to another, and timestamps represent years (from 2000 to 2016). 2.8 millions ORCID public profiles are utilized as data source. We then characterize hubs and authorities of the SMN employing the well-know weighted hyperlink-induced topic search (HITS) algorithm to catch the interplay between providing and attracting researchers from a global perspective, and relate these results to other local and global methodologies. We also investigate the local characteristics of successors of hubs and predecessors of authorities to dive deeper into the motivations that establish hubs and authorities as such. Our findings highlight the presence of a set of countries acting both as hubs and authorities, occupying a privileged position in the scientific migration network, and having similar local characteristics. Finally, we showcase how to employ network visualization to evince temporal evolutions of ego-networks of selected hubs and authorities.

cs.SI↗

arXiv2020

Young Adult Unemployment Through the Lens of Social Media: Italy as a case study

Youth unemployment rates are still in alerting levels for many countries, among which Italy. Direct consequences include poverty, social exclusion, and criminal behaviours, while negative impact on the future employability and wage cannot be obscured. In this study, we employ survey data together with social media data, and in particular likes on Facebook Pages, to analyse personality, moral values, but also cultural elements of the young unemployed population in Italy. Our findings show that there are small but significant differences in personality and moral values, with the unemployed males to be less agreeable while females more open to new experiences. At the same time, unemployed have a more collectivist point of view, valuing more in-group loyalty, authority, and purity foundations. Interestingly, topic modelling analysis did not reveal major differences in interests and cultural elements of the unemployed. Utilisation patterns emerged though; the employed seem to use Facebook to connect with local activities, while the unemployed use it mostly as for entertainment purposes and as a source of news, making them susceptible to mis/disinformation. We believe these findings can help policymakers get a deeper understanding of this population and initiatives that improve both the hard and the soft skills of this fragile population.

cs.CY↗

arXiv2019

Human Values and Attitudes towards Vaccination in Social Media

Psychological, political, cultural, and even societal factors are entangled in the reasoning and decision-making process towards vaccination, rendering vaccine hesitancy a complex issue. Here, administering a series of surveys via a Facebook-hosted application, we study the worldviews of people that "Liked" supportive or vaccine resilient Facebook Pages. In particular, we assess differences in political viewpoints, moral values, personality traits, and general interests, finding that those sceptical about vaccination, appear to trust less the government, are less agreeable, while they are emphasising more on anti-authoritarian values. Exploring the differences in moral narratives as expressed in the linguistic descriptions of the Facebook Pages, we see that pages that defend vaccines prioritise the value of the family while the vaccine hesitancy pages are focusing on the value of freedom. Finally, creating embeddings based on the health-related likes on Facebook Pages, we explore common, latent interests of vaccine-hesitant people, showing a strong preference for natural cures. This exploratory analysis aims at exploring the potentials of a social media platform to act as a sensing tool, providing researchers and policymakers with insights drawn from the digital traces, that can help design communication campaigns that build confidence, based on the values that also appeal to the socio-moral criteria of people.

cs.CY↗