arXiv subjects

Yong-Yeol Ahn

Publications and source records attributed to Yong-Yeol Ahn.

At least 37 records · Page 2Linked to original sources

arXiv2024

Can we trust the evaluation on ChatGPT?

ChatGPT, the first large language model (LLM) with mass adoption, has demonstrated remarkable performance in numerous natural language tasks. Despite its evident usefulness, evaluating ChatGPT's performance in diverse problem domains remains challenging due to the closed nature of the model and its continuous updates via Reinforcement Learning from Human Feedback (RLHF). We highlight the issue of data contamination in ChatGPT evaluations, with a case study of the task of stance detection. We discuss the challenge of preventing data contamination and ensuring fair model evaluation in the age of closed and continuously trained models.

cs.CL↗

arXiv2024

Implicit degree bias in the link prediction task

Link prediction -- a task of distinguishing actual hidden edges from random unconnected node pairs -- is one of the quintessential tasks in graph machine learning. Despite being widely accepted as a universal benchmark and a downstream task for representation learning, the validity of the link prediction benchmark itself has been rarely questioned. Here, we show that the common edge sampling procedure in the link prediction task has an implicit bias toward high-degree nodes and produces a highly skewed evaluation that favors methods overly dependent on node degree, to the extent that a ``null'' link prediction method based solely on node degree can yield nearly optimal performance. We propose a degree-corrected link prediction task that offers a more reasonable assessment that aligns better with the performance in the recommendation task. Finally, we demonstrate that the degree-corrected benchmark can more effectively train graph machine-learning models by reducing overfitting to node degrees and facilitating the learning of relevant structures in graphs.

cs.SI↗

arXiv2024

Emergence of simple and complex contagion dynamics from weighted belief networks

Social contagion is a ubiquitous and fundamental process that drives individual and social changes. Although social contagion arises as a result of cognitive processes and biases, the integration of cognitive mechanisms with the theory of social contagion remains an open challenge. In particular, studies on social phenomena usually assume contagion dynamics to be either simple or complex, rather than allowing it to emerge from cognitive mechanisms, despite empirical evidence indicating that a social system can exhibit a spectrum of contagion dynamics -- from simple to complex -- simultaneously. Here, we propose a model of interacting beliefs, from which both simple and complex contagion dynamics can organically arise. Our model also elucidates how a fundamental mechanism of complex contagion -- resistance -- can come about from cognitive mechanisms.

cs.SI↗

arXiv2024

Benchmarking zero-shot stance detection with FlanT5-XXL: Insights from training data, prompting, and decoding strategies into its near-SoTA performance

We investigate the performance of LLM-based zero-shot stance detection on tweets. Using FlanT5-XXL, an instruction-tuned open-source LLM, with the SemEval 2016 Tasks 6A, 6B, and P-Stance datasets, we study the performance and its variations under different prompts and decoding strategies, as well as the potential biases of the model. We show that the zero-shot approach can match or outperform state-of-the-art benchmarks, including fine-tuned models. We provide various insights into its performance including the sensitivity to instructions and prompts, the decoding strategies, the perplexity of the prompts, and to negations and oppositions present in prompts. Finally, we ensure that the LLM has not been trained on test datasets, and identify a positivity bias which may partially explain the performance differences across decoding strategie

cs.CL↗

arXiv2024

Labor Space: A Unifying Representation of the Labor Market via Large Language Models

The labor market is a complex ecosystem comprising diverse, interconnected entities, such as industries, occupations, skills, and firms. Due to the lack of a systematic method to map these heterogeneous entities together, each entity has been analyzed in isolation or only through pairwise relationships, inhibiting comprehensive understanding of the whole ecosystem. Here, we introduce $\textit{Labor Space}$, a vector-space embedding of heterogeneous labor market entities, derived through applying a large language model with fine-tuning. Labor Space exposes the complex relational fabric of various labor market constituents, facilitating coherent integrative analysis of industries, occupations, skills, and firms, while retaining type-specific clustering. We demonstrate its unprecedented analytical capacities, including positioning heterogeneous entities on an economic axes, such as `Manufacturing--Healthcare'. Furthermore, by allowing vector arithmetic of these entities, Labor Space enables the exploration of complex inter-unit relations, and subsequently the estimation of the ramifications of economic shocks on individual units and their ripple effect across the labor market. We posit that Labor Space provides policymakers and business leaders with a comprehensive unifying framework for labor market analysis and simulation, fostering more nuanced and effective strategic decision-making.

physics.soc-ph↗

arXiv2024

Cooperation and interdependence in global science funding

Investments in research and development are key to scientific and economic growth and to the well-being of society. Scientific research demands significant resources making national scientific investment a crucial driver of scientific production. As scientific production becomes increasingly multinational, it is critical to study how nations' scientific activities are funded both domestically and internationally. By tracing research grants acknowledged in scholarly publications, our study reveals a shifting duopoly of China and the United States in the global funding landscape, with a contrasting funding pattern; while China has surpassed the United States in publications with acknowledged domestic and international funding, the United States largely maintains its role as the most important global research partner. Our results also highlight the precarity of low- and middle-income countries to global funding disruptions. By revealing the complex interdependence and collaboration between countries in the global scientific enterprise, this work informs future studies investigating the national and global scientific enterprise and how funding leads to both productive cooperation and vulnerable dependencies.

econ.GN↗

arXiv2023

Unsupervised embedding of trajectories captures the latent structure of scientific migration

Human migration and mobility drives major societal phenomena including epidemics, economies, innovation, and the diffusion of ideas. Although human mobility and migration have been heavily constrained by geographic distance throughout the history, advances and globalization are making other factors such as language and culture increasingly more important. Advances in neural embedding models, originally designed for natural language, provide an opportunity to tame this complexity and open new avenues for the study of migration. Here, we demonstrate the ability of the model word2vec to encode nuanced relationships between discrete locations from migration trajectories, producing an accurate, dense, continuous, and meaningful vector-space representation. The resulting representation provides a functional distance between locations, as well as a digital double that can be distributed, re-used, and itself interrogated to understand the many dimensions of migration. We show that the unique power of word2vec to encode migration patterns stems from its mathematical equivalence with the gravity model of mobility. Focusing on the case of scientific migration, we apply word2vec to a database of three million migration trajectories of scientists derived from the affiliations listed on their publication records. Using techniques that leverage its semantic structure, we demonstrate that embeddings can learn the rich structure that underpins scientific migration, such as cultural, linguistic, and prestige relationships at multiple levels of granularity. Our results provide a theoretical foundation and methodological framework for using neural embeddings to represent and understand migration both within and beyond science.

cs.LG↗

arXiv2023

Sameness Entices, but Novelty Enchants in Fanfiction Online

Cultural evolution is driven by how we choose what to consume and share with others. A common belief is that the cultural artifacts that succeed are ones that balance novelty and conventionality. This balance theory suggests that people prefer works that are familiar, but not so familiar as to be boring; novel, but not so novel as to violate the expectations of their genre. We test this idea using a large dataset of fanfiction. We apply a multiple regression model and a generalized additive model to examine how the recognition a work receives varies with its novelty, estimated through a Latent Dirichlet Allocation topic model, in the context of existing works. We find the opposite pattern of what the balance theory predicts$\unicode{x2014}$overall success decline almost monotonically with novelty and exhibits a U-shaped, instead of an inverse U-shaped, curve. This puzzle is resolved by teasing out two competing forces: sameness attracts the mass whereas novelty provides enjoyment. Taken together, even though the balance theory holds in terms of expressed enjoyment, the overall success can show the opposite pattern due to the dominant role of sameness to attract the audience. Under these two forces, cultural evolution may have to work against inertia$\unicode{x2014}$the appetite for consuming the familiar$\unicode{x2014}$and may resemble a punctuated equilibrium, marked by occasional leaps.

cs.CL↗

arXiv2023

AI exposure predicts unemployment risk

Is artificial intelligence (AI) disrupting jobs and creating unemployment? Despite many attempts to quantify occupations' exposure to AI, inconsistent validation obfuscates the relative benefits of each approach. A lack of disaggregated labor outcome data, including unemployment data, further exacerbates the issue. Here, we assess which models of AI exposure predict job separations and unemployment risk using new occupation-level unemployment data by occupation from each US state's unemployment insurance office spanning 2010 through 2020. Although these AI exposure scores have been used by governments and industry, we find that individual AI exposure models are not predictive of unemployment rates, unemployment risk, or job separation rates. However, an ensemble of those models exhibits substantial predictive power suggesting that competing models may capture different aspects of AI exposure that collectively account for AI's variable impact across occupations, regions, and time. Our results also call for dynamic, context-aware, and validated methods for assessing AI exposure. Interactive visualizations for this study are available at https://sites.pitt.edu/~mrfrank/uiRiskDemo/.

cs.CY↗

arXiv2023

Discovering collective narratives shifts in online discussions

Narrative is a foundation of human cognition and decision making. Because narratives play a crucial role in societal discourses and spread of misinformation and because of the pervasive use of social media, the narrative dynamics on social media can have profound societal impact. Yet, systematic and computational understanding of online narratives faces critical challenge of the scale and dynamics; how can we reliably and automatically extract narratives from massive amount of texts? How do narratives emerge, spread, and die? Here, we propose a systematic narrative discovery framework that fill this gap by combining change point detection, semantic role labeling (SRL), and automatic aggregation of narrative fragments into narrative networks. We evaluate our model with synthetic and empirical data two-Twitter corpora about COVID-19 and 2017 French Election. Results demonstrate that our approach can recover major narrative shifts that correspond to the major events.

cs.CL↗

arXiv2023

Evolution of the Informational Complexity of Contemporary Western Music

We measure the complexity of songs in the Million Song Dataset (MSD) in terms of pitch, timbre, loudness, and rhythm to investigate their evolution from 1960 to 2010. By comparing the Billboard Hot 100 with random samples, we find that the complexity of popular songs tends to be more narrowly distributed around the mean, supporting the idea of an inverted U-shaped relationship between complexity and hedonistic value. We then examine the temporal evolution of complexity, reporting consistent changes across decades, such as a decrease in average loudness complexity since the 1960s, and an increase in timbre complexity overall but not for popular songs. We also show, in contrast to claims that popular songs sound more alike over time, that they are not more similar than they were 50 years ago in terms of pitch or rhythm, although similarity in timbre shows distinctive patterns across eras and similarity in loudness has been increasing. Finally, we show that musical genres can be differentiated by their distinctive complexity profiles.

cs.SD↗

arXiv2023

Unique in what sense? Heterogeneous relationships between multiple types of uniqueness and popularity in music

How does our society appreciate the uniqueness of cultural products? This fundamental puzzle has intrigued scholars in many fields, including psychology, sociology, anthropology, and marketing. It has been theorized that cultural products that balance familiarity and novelty are more likely to become popular. However, a cultural product's novelty is typically multifaceted. This paper uses songs as a case study to study the multiple facets of uniqueness and their relationship with success. We first unpack the multiple facets of a song's novelty or uniqueness and, next, measure its impact on a song's popularity. We employ a series of statistical models to study the relationship between a song's popularity and novelty associated with its lyrics, chord progressions, or audio properties. Our analyses performed on a dataset of over fifty thousand songs find a consistently negative association between all types of song novelty and popularity. Overall we found a song's lyrics uniqueness to have the most significant association with its popularity. However, audio uniqueness was the strongest predictor of a song's popularity, conditional on the song's genre. We further found the theme and repetitiveness of a song's lyrics to mediate the relationship between the song's popularity and novelty. Broadly, our results contradict the "optimal distinctiveness theory" (balance between novelty and familiarity) and call for an investigation into the multiple dimensions along which a cultural product's uniqueness could manifest.

cs.CY↗

arXiv2022

Network Cards: concise, readable summaries of network data

The deluge of network datasets demands a standard way to effectively and succinctly summarize network datasets. Building on similar efforts to standardize the documentation of models and datasets in machine learning, here we propose network cards, short summaries of network datasets that can capture not only the basic statistics of the network but also information about the data construction process, provenance, ethical considerations, and other metadata. In this paper, we lay out (1) the rationales and objectives for network cards, (2) key elements that should be included in network cards, and (3) example network cards to underscore their benefits across a variety of research domains. We also provide a schema, templates, and a software package for generating network cards.

cs.SI↗

arXiv2022

The latent structure of global scientific development

Science is essential to innovation and economic prosperity. Although studies have shown that national scientific development is affected by geographic, historic, and economic factors, it remains unclear whether there are universal structures and trajectories of national scientific development that can inform forecasting and policymaking. Here, by examining countries' scientific 'exports'-publications that are indexed in international databases-we reveal a three-cluster structure in the relatedness network of disciplines that underpin national scientific development and the organization of global science. Tracing the evolution of national research portfolios reveals that while nations are proceeding to more diverse research profiles individually, scientific production is increasingly specialized in global science over the past decades. By uncovering the underlying structure of scientific development and connecting it with economic development, our results may offer a new perspective on the evolution of global science.

cs.SI↗

arXiv2021

Residual2Vec: Debiasing graph embedding with random graphs

Graph embedding maps a graph into a convenient vector-space representation for graph analysis and machine learning applications. Many graph embedding methods hinge on a sampling of context nodes based on random walks. However, random walks can be a biased sampler due to the structural properties of graphs. Most notably, random walks are biased by the degree of each node, where a node is sampled proportionally to its degree. The implication of such biases has not been clear, particularly in the context of graph representation learning. Here, we investigate the impact of the random walks' bias on graph embedding and propose residual2vec, a general graph embedding method that can debias various structural biases in graphs by using random graphs. We demonstrate that this debiasing not only improves link prediction and clustering performance but also allows us to explicitly model salient structural properties in graph embedding.

cs.LG↗

arXiv2021

Characterizing Partisan Political Narrative Frameworks about COVID-19 on Twitter

The COVID-19 pandemic is a global crisis that has been testing every society and exposing the critical role of local politics in crisis response. In the United States, there has been a strong partisan divide between the Democratic and Republican party's narratives about the pandemic which resulted in polarization of individual behaviors and divergent policy adoption across regions. As shown in this case, as well as in most major social issues, strongly polarized narrative frameworks facilitate such narratives. To understand polarization and other social chasms, it is critical to dissect these diverging narratives. Here, taking the Democratic and Republican political social media posts about the pandemic as a case study, we demonstrate that a combination of computational methods can provide useful insights into the different contexts, framing, and characters and relationships that construct their narrative frameworks which individual posts source from. Leveraging a dataset of tweets from elite politicians in the U.S., we found that the Democrats' narrative tends to be more concerned with the pandemic as well as financial and social support, while the Republicans discuss more about other political entities such as China. We then perform an automatic framing analysis to characterize the ways in which they frame their narratives, where we found that the Democrats emphasize the government's role in responding to the pandemic, and the Republicans emphasize the roles of individuals and support for small businesses. Finally, we present a semantic role analysis that uncovers the important characters and relationships in their narratives as well as how they facilitate a membership categorization process. Our findings concretely expose the gaps in the "elusive consensus" between the two parties. Our methodologies may be applied to computationally study narratives in various domains.

cs.CL↗

arXiv2021

Predicting Anti-Asian Hateful Users on Twitter during COVID-19

We investigate predictors of anti-Asian hate among Twitter users throughout COVID-19. With the rise of xenophobia and polarization that has accompanied widespread social media usage in many nations, online hate has become a major social issue, attracting many researchers. Here, we apply natural language processing techniques to characterize social media users who began to post anti-Asian hate messages during COVID-19. We compare two user groups -- those who posted anti-Asian slurs and those who did not -- with respect to a rich set of features measured with data prior to COVID-19 and show that it is possible to predict who later publicly posted anti-Asian slurs. Our analysis of predictive features underlines the potential impact of news media and information sources that report on online hate and calls for further investigation into the role of polarized communication networks and news media.

cs.CY↗

arXiv2021

FrameAxis: Characterizing Microframe Bias and Intensity with Word Embedding

Framing is a process of emphasizing a certain aspect of an issue over the others, nudging readers or listeners towards different positions on the issue even without making a biased argument. {Here, we propose FrameAxis, a method for characterizing documents by identifying the most relevant semantic axes ("microframes") that are overrepresented in the text using word embedding. Our unsupervised approach can be readily applied to large datasets because it does not require manual annotations. It can also provide nuanced insights by considering a rich set of semantic axes. FrameAxis is designed to quantitatively tease out two important dimensions of how microframes are used in the text. \textit{Microframe bias} captures how biased the text is on a certain microframe, and \textit{microframe intensity} shows how actively a certain microframe is used. Together, they offer a detailed characterization of the text. We demonstrate that microframes with the highest bias and intensity well align with sentiment, topic, and partisan spectrum by applying FrameAxis to multiple datasets from restaurant reviews to political news.} The existing domain knowledge can be incorporated into FrameAxis {by using custom microframes and by using FrameAxis as an iterative exploratory analysis instrument.} Additionally, we propose methods for explaining the results of FrameAxis at the level of individual words and documents. Our method may accelerate scalable and sophisticated computational analyses of framing across disciplines.

cs.CL↗