Empirical Prompt Engineering for Construct Identification with Large Language Models

Abstract

Due to their architecture and vast pre-training data, large language models (LLMs) demonstrate strong performance on text classification tasks. However, LLM classifications are highly responsive to prompt wording, particularly, as we show, in domains like psychology, where constructs are often latent, complex, and theory driven. Here, we present and evaluate a systematic framework for improving psychological construct identification through prompt engineering. We combinatorially generate prompts by appending random selections of multiple variants of construct definitions, task instructions, coding guidance, and examples. Empirically selecting the highest performing of these combinations in a training dataset substantially improves alignment between LLM and human classifications. In contrast, prompting techniques such as personas, chain-of-thought reasoning, and explanations provide smaller and less consistent improvements. This finding holds across multiple models and constructs. Overall, the approach we describe offers a practical, systematic, and theory-aware method for increasing the alignment between human and LLM classifications in settings where validity is critical.