Mike Hardy | Searcharxiv

arXiv2025

Interpretability from the Ground Up: Stakeholder-Centric Design of Automated Scoring in Educational Assessments

AI-driven automated scoring systems offer scalable and efficient means of evaluating complex student-generated responses. Yet, despite increasing demand for transparency and interpretability, the field has yet to develop a widely accepted solution for interpretable automated scoring to be used in large-scale real-world assessments. This work takes a principled approach to address this challenge. We analyze the needs and potential benefits of interpretable automated scoring for various assessment stakeholder groups and develop four principles of interpretability -- (F)aithfulness, (G)roundedness, (T)raceability, and (I)nterchangeability (FGTI) -- targeted at those needs. To illustrate the feasibility of implementing these principles, we develop the AnalyticScore framework as a reference framework. When applied to the domain of text-based constructed-response scoring, AnalyticScore outperforms many uninterpretable scoring methods in terms of scoring accuracy and is, on average, within 0.06 QWK of the uninterpretable SOTA across 10 items from the ASAP-SAS dataset. By comparing against human annotators conducting the same featurization task, we further demonstrate that the featurization behavior of AnalyticScore aligns well with that of humans.

cs.CL↗

arXiv2021

Toward Educator-focused Automated Scoring Systems for Reading and Writing

This paper presents methods for improving automated essay scoring with techniques that address the computational trade-offs of self-attention and document length. To make Automated Essay Scoring (AES) more useful to practitioners, researchers must overcome the challenges of data and label availability, authentic and extended writing, domain scoring, prompt and source variety, and transfer learning. This paper addresses these challenges using neural network models by employing techniques that preserve essay length as an important feature without increasing model training costs. It introduces techniques for minimizing classification loss on ordinal labels using multi-objective learning, capturing semantic information across the entire essay using sentence embeddings to use transformer architecture across arbitrarily long documents, the use of such models for transfer learning, automated hyperparameter generation based on prompt-corpus metadata, and, most importantly, the use of semantic information to provide meaningful insights into student reading through analysis of passage-dependent writing resulting in state-of-the-art results for various essay tasks.

cs.CL↗