Breanna E. Green | Searcharxiv

arXiv2025

A Comparison of Human and ChatGPT Classification Performance on Complex Social Media Data

Generative artificial intelligence tools, like ChatGPT, are an increasingly utilized resource among computational social scientists. Nevertheless, there remains space for improved understanding of the performance of ChatGPT in complex tasks such as classifying and annotating datasets containing nuanced language. Method. In this paper, we measure the performance of GPT-4 on one such task and compare results to human annotators. We investigate ChatGPT versions 3.5, 4, and 4o to examine performance given rapid changes in technological advancement of large language models. We craft four prompt styles as input and evaluate precision, recall, and F1 scores. Both quantitative and qualitative evaluations of results demonstrate that while including label definitions in prompts may help performance, overall GPT-4 has difficulty classifying nuanced language. Qualitative analysis reveals four specific findings. Our results suggest the use of ChatGPT in classification tasks involving nuanced language should be conducted with prudence.

cs.CL↗

arXiv2024

How Polarized are Online Conversations about Childhood?

2020 through 2023 were unusually tumultuous years for children in the United States, and children's welfare was prominent in political debate. Theories in moral psychology suggest that political parties would treat concerns for children using different moral frames, and that moral conflict might drive substantial polarization in discussions about children. However, such partisan frames may still differ very little if there is limited underlying disagreement about moral issues and everyday concerns in childhood when not explicitly referencing politics. We evaluate claims of universality and division in moral language using tweets from 2019-2023 linked to U.S. voter records, focusing on expressed morality. Our results show that mentions of children by Republicans and Democrats are usually similar, differing no more than mentions by women and men, and tend to contain no large differences in accompanying moral words. To the extent that mentions of children did differ across parties, these differences were constrained to topics polarized well before the pandemic -- and slightly heightened when co-mentioned with `kids' or `children'. These topics reflected a small fraction of conversations about children. Overall, polarization of online discussion around childhood appears to reflect escalated polarization on lines of existing partisan conflicts rather than concerns originating from new concerns about the welfare of children during and after the pandemic.

cs.SI↗