Topic Modeling for Free-Response Text Data from a Complex Survey

Abstract

Topic Modeling is a popular statistical tool commonly used on textual data to identify the hidden thematic structure in a document collection based on the distribution of words. Additionally, it can be used to cluster the documents, with clusters representing distinct topics. The Mixture of Unigrams (MoU) is a standard topic model for clustering document-term data and can be particularly useful for analyzing open-ended survey responses to extract meaningful information from the underlying topics. However, with complex survey designs, where data is often collected on individual (document) characteristics, it is essential to account for the sample design in order to avoid biased estimates. To address this issue, we propose the MoU model under informative sampling using a pseudolikelihood to account for the sample design in the model by incorporating survey weights. We evaluate the effectiveness of this approach through a simulation study and illustrate its application using two datasets from the American National Election Studies (ANES). We compare our pseudolikelihood-based MoU model to the traditional MoU and assess its effectiveness in extracting meaningful topics from survey data. Additionally, we introduce a hierarchical Mixture of Unigrams (hMoU) accounting for informative sampling where topic proportions are defined as functions of document-level fixed and random effects. We demonstrate the effectiveness of the proposed model through an application to ANES data comparing topic proportions across respondent-level factors such as gender, race, age group, and state.