Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2205.02829

Foundations for NLP-assisted formative assessment feedback for short-answer tasks in large-enrollment classes

Susan Lloyd·Matthew Beckman·Dennis Pearl·Rebecca Passonneau·Zhaohui Li·Zekun Wang
View original source ↗Read source document ↗Export metadata ↓

Abstract

Research suggests "write-to-learn" tasks improve learning outcomes, yet constructed-response methods of formative assessment become unwieldy with large class sizes. This study evaluates natural language processing algorithms to assist this aim. Six short-answer tasks completed by 1,935 students were scored by several human raters, using a detailed rubric, and an algorithm. Results indicate substantial inter-rater agreement using quadratic weighted kappa for rater pairs (each QWK > 0.74) and group consensus (Fleiss Kappa = 0.68). Additionally, intra-rater agreement was estimated for one rater who had scored 178 responses seven years prior (QWK = 0.89). With compelling rater agreement, the study then pilots cluster analysis of response text toward enabling instructors to ascribe meaning to clusters as a means for scalable formative assessment.

Explore related subjects

stat.OT

Keep this discovery

BibTeXRIS

Susan Lloyd, Matthew Beckman, Dennis Pearl, Rebecca Passonneau, Zhaohui Li, Zekun Wang. 2022-05-05. Foundations for NLP-assisted formative assessment feedback for short-answer tasks in large-enrollment classes. https://doi.org/10.52041/iase.icots11.t3c3

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

The Design and Implementation of a Virtual Statistical Computing Lab to Teach R Coding to Introductory Statistics Students

Motivated by national calls for computationally enriched, data-centric instruction across the statistics curriculum, this study investigates the design, implementation, and impact of a Virtual Statistical Computing Lab (VSCL) integrated into an introductory statistics course at a medium-sized minority-serving university in the USA. The redesigned course embedded R-based coding through two virtual lab formats: Design I (a static Posit Cloud environment) and Design II (an interactive learnr-based interface). Using a quasi-experimental design across three instructional formats, traditional (no lab), Design I, and Design II, we evaluated students' conceptual learning gains, levels of data science (DS) readiness, and DS aspirations. The results indicated significant learning gains across all groups, with the highest gains observed in Design II. Students in both VSCL formats achieved greater gains in DS readiness than the traditional group, with Design II again yielding the largest gains across the demographic subgroups. Conversely, DS aspirations remained low or declined, suggesting a gap between skill acquisition and long-term interest. These findings highlight the value of structured, interactive computing environments in supporting statistical reasoning and building confidence in modern data tools. They also point to the need for intentional curricular bridges and career mentoring to help students translate early computing exposure into sustained academic and professional pathways in statistics and data science.

stat.OT↗
arXiv2026

Statistical Theory in the Age of Machine-Assisted Mathematics: Rethinking How Theory Is Made and Taught

The computational revolution is advancing at an unprecedented pace. The combination of proof-assistant technologies and generative AI tools has recently enabled the solution of complex problems in pure mathematics at a scale that seemed unattainable only a few years ago. However, these technologies have not yet become standard tools in the development of statistical theory. In this paper, we do not present new theoretical results. Instead, we discuss five case studies involving classical problems in statistics and describe how they can be analyzed using a machine proof-checking. Our goal is not to propose a definitive workflow, but to stimulate reflection on how these technologies may transform theoretical research and advanced statistical education. We focus on two main aspects. First, statistical theory often compresses substantial mathematical content into expressions such as "under the usual regularity conditions". Formalization in a machine-verifiable language forces each assumption to be explicit, reveal hidden dependencies, and provide a deeper understanding of the formalized objects. Second, we argue that the statistical community could benefit from a collaborative effort to build repositories of formalized axioms, definitions, and theorems, supporting more precise and reliable theoretical developments. Finally, we discuss the role of these tools in graduate education. Just as high-level programming languages revolutionized empirical research by enabling rapid experimentation and prototyping, machine-assisted formalization may introduce a new paradigm for the development, verification, and communication of statistical theory.

stat.OT↗
arXiv2026

Statistical Leadership of What? Statistics After AI

Statisticians have spent over a century arguing that we are more than calculators, usually by pointing to what else we know. AI is making that defense harder, since the list of what only statisticians can do grows shorter with each model release. AI makes claims cheap to generate and may eventually make the statistics behind them cheap too. However, a model cannot be answerable in the way that statistical practice requires. Statistical leadership then becomes a question of which claims we are there to answer for, including the ones we answer for in advance by building judgment into systems.

stat.OT↗