arXiv subjects

Matthias von Davier

Publications and source records attributed to Matthias von Davier.

11 recordsLinked to original sources

arXiv2026

Bridging Network Psychometrics and Artificial Intelligence: An Ising-Potts Model with LLM-Derived Weights

The Potts model extends the Ising model to multinomial data. We introduce a Rater Ising-Potts model that uses agreement indicators between pairs of ratings and category labels, with weights derived from LLM embeddings. The model does not presuppose ordered category thresholds or equidistant scoring; instead, it focuses on pairwise agreement among ratings and assigns category-specific positive weights, making it suited for multi-category scoring reliability. We evaluate the model on three constructed-response datasets spanning a corpus of K=14,466 short answers on a three-level rubric and two AERA essay prompts of roughly 1,200-1,400 responses on four-point rubrics. We compare three strategies for sharpening the similarity signal: top-K pruning, min-max normalization with a power transformation, and ColBERT late-interaction similarities. Top-K pruning, which replaces the dense similarity graph with a sparse local network of strongest semantic neighbors, consistently yields the highest accuracy and Cohen's kappa, and the selected neighborhoods are always a small fraction of the corpus. Power tuning consistently ranks second, while ColBERT is competitive on longer essay prompts and adds little on short answers. Across all settings, most misclassifications occur between adjacent score levels, confirming that the model preserves the ordinal structure of scoring rubrics without imposing rigid assumptions. These findings suggest that LLM-derived similarities, combined with a parsimonious Potts formulation and a sparse local graph, offer a robust and interpretable framework for reliability auditing in educational assessment. We discuss extensions to multiple raters and hierarchical rating designs.

stat.AP↗

arXiv2026

Integrating Network Psychometrics and LLMs: The Ising-Embeddings-Model applied to Reliability Auditing

Scoring consistency for constructed-response items in large-scale assessments is typically estimated through double-scoring, which uses small samples and assumes independence among responses. We present an integrated framework combining network psychometrics with the Linguistic-Integrated Reliability Audit (LiRA) via a modified Ising model. The model defines a joint distribution over binary correctness labels with pairwise interactions set to the cosine similarity of sentence embeddings and a global bias parameter for item difficulty. LiRA's weighted majority voting over semantic neighborhoods is shown to approximate the conditional logistic distributions of this Ising model. The parametric framework supports benchmark score generation, uncertainty quantification, and missing label imputation; parameters are estimated by maximum pseudo-likelihood. The approach uses the full dataset without requiring extensive double-scoring, accounts for semantic dependencies, and provides diagnostics for rater inconsistencies. The integration of LiRA's scalable methodology with a probabilistic graphical model offers a comprehensive tool for reliability assessment in international assessments such as PIRLS, PISA, and TIMSS.

stat.AP↗

arXiv2026

Multilingual Sentence Embeddings for Linguistic-Integrated Reliability Audit

Multilingual assessment systems commonly rely on translation for scoring and quality-control processes. We evaluate whether multilingual sentence embeddings can replace translated English input for Linguistic-Integrated Reliability Auditing (LiRA) across 11 PIRLS constructed-response items and three embedding models. Native-language embeddings reproduced translation-based reliability estimates closely while recovering responses excluded after translation failure, with no meaningful change in reliability.

cs.CL↗

arXiv2026

Effective Degrees of Freedom for Balanced Repeated Replication and Paired Jackknife Variance Estimates: A Unified Approach via Stratum Contrasts

Balanced repeated replication (BRR) and the jackknife are two widely used methods for estimating variances in stratified samples with two primary sampling units per stratum. While both methods produce variance estimators that can be expressed as sums of squared stratum-level contrasts, they differ fundamentally in their construction and in the dependence structure of their replicate estimates. This article examines the independence properties of the components contributing to these variance estimators. For BRR, we show that although the replicate estimates themselves are correlated, the balancing property of Hadamard matrices collapses the variance estimator into a sum of independent stratum-specific components. For the jackknife, the independence of components follows directly from the construction. Using these independence results, we derive the variance of each variance estimator and establish a direct connection to the Welch-Satterthwaite degrees of freedom approximation. This yields a practical formula for estimating degrees of freedom when constructing confidence intervals for population totals. The derivation highlights the unified treatment of both replication methods and provides insights into their relative efficiency and applicability.

stat.ME↗

arXiv2026

A Corrected Welch Satterthwaite Equation. And: What You Always Wanted to Know About Kish's Effective Sample but Were Afraid to Ask

This article presents a corrected version of the Satterthwaite (1941, 1946) approximation for the degrees of freedom of a weighted sum of independent variance components. The original formula is known to yield biased estimates when component degrees of freedom are small. The correction, derived from exact moment matching, adjusts for the bias by incorporating a factor that accounts for the estimation of fourth moments. We show that Kish's (1965) effective sample size formula emerges as a special case when all variance components are equal, and component degrees of freedom are ignored. Simulation studies demonstrate that the corrected estimator closely matches the expected degrees of freedom even for small component sizes, while the original Satterthwaite estimator exhibits substantial downward bias. Additional applications are discussed, including jackknife variance estimation, multiple imputation total variance, and the Welch test for unequal variances.

stat.AP↗

arXiv2025

An Improved Satterthwaite Effective Degrees of Freedom Correction for Weighted Syntheses of Variance

This article presents an improved approximation for the effective degrees of freedom in the Satterthwaite (1941, 1946) method which estimates the distribution of a weighted combination of variance components The standard Satterthwaite approximation assumes a scaled chisquare distribution for the composite variance estimator but is known to be biased downward when component degrees of freedom are small. Building on recent work by von Davier (2025), we propose an adjusted estimator that corrects this bias by modifying both the numerator and denominator of the traditional formula. The new approximation incorporates a weighted average of component degrees of freedom and a scaling factor that ensures consistency as the number of components or their degrees of freedom increases. We demonstrate the utility of this adjustment in practical settings, including Rubin's (1987) total variance estimation in multiple imputations, where weighted variance combinations are common. The proposed estimator generalizes and further improves von Davier's (2025) unweighted case and more accurately approximates synthetic variance estimators with arbitrary weights.

stat.ME↗

arXiv2024

A Modified Satterthwaite (1941,1946) Effective Degrees of Freedom Approximation

This study introduces a correction to the approximation of effective degrees of freedom as proposed by Satterthwaite (1941, 1946), specifically addressing scenarios where component degrees of freedom are small. The correction is grounded in analytical results concerning the moments of standard normal random variables. This modification is applicable to complex variance estimates that involve both small and large degrees of freedom, offering an enhanced approximation of the higher moments required by Satterthwaite's framework. Additionally, this correction extends and partially validates the empirically derived adjustment by Johnson & Rust (1992), as it is based on theoretical foundations rather than simulations used to derive empirical transformation constants.

stat.OT↗

arXiv2023

Variable Selection in Latent Regression IRT Models via Knockoffs: An Application to International Large-scale Assessment in Education

International large-scale assessments (ILSAs) play an important role in educational research and policy making. They collect valuable data on education quality and performance development across many education systems, giving countries the opportunity to share techniques, organizational structures, and policies that have proven efficient and successful. To gain insights from ILSA data, we identify non-cognitive variables associated with students' academic performance. This problem has three analytical challenges: 1) academic performance is measured by cognitive items under a matrix sampling design; 2) there are many missing values in the non-cognitive variables; and 3) multiple comparisons due to a large number of non-cognitive variables. We consider an application to the Programme for International Student Assessment (PISA), aiming to identify non-cognitive variables associated with students' performance in science. We formulate it as a variable selection problem under a general latent variable model framework and further propose a knockoff method that conducts variable selection with a controlled error rate for false selections.

stat.ME↗

arXiv2023

Automated Reading Passage Generation with OpenAI's Large Language Model

The widespread usage of computer-based assessments and individualized learning platforms has resulted in an increased demand for the rapid production of high-quality items. Automated item generation (AIG), the process of using item models to generate new items with the help of computer technology, was proposed to reduce reliance on human subject experts at each step of the process. AIG has been used in test development for some time. Still, the use of machine learning algorithms has introduced the potential to improve the efficiency and effectiveness of the process greatly. The approach presented in this paper utilizes OpenAI's latest transformer-based language model, GPT-3, to generate reading passages. Existing reading passages were used in carefully engineered prompts to ensure the AI-generated text has similar content and structure to a fourth-grade reading passage. For each prompt, we generated multiple passages, the final passage was selected according to the Lexile score agreement with the original passage. In the final round, the selected passage went through a simple revision by a human editor to ensure the text was free of any grammatical and factual errors. All AI-generated passages, along with original passages were evaluated by human judges according to their coherence, appropriateness to fourth graders, and readability.

cs.CL↗

arXiv2022

Automated Scoring of Graphical Open-Ended Responses Using Artificial Neural Networks

Automated scoring of free drawings or images as responses has yet to be utilized in large-scale assessments of student achievement. In this study, we propose artificial neural networks to classify these types of graphical responses from a computer based international mathematics and science assessment. We are comparing classification accuracy of convolutional and feedforward approaches. Our results show that convolutional neural networks (CNNs) outperform feedforward neural networks in both loss and accuracy. The CNN models classified up to 97.71% of the image responses into the appropriate scoring category, which is comparable to, if not more accurate, than typical human raters. These findings were further strengthened by the observation that the most accurate CNN models correctly classified some image responses that had been incorrectly scored by the human raters. As an additional innovation, we outline a method to select human rated responses for the training sample based on an application of the expected response function derived from item response theory. This paper argues that CNN-based automated scoring of image responses is a highly accurate procedure that could potentially replace the workload and cost of second human raters for large scale assessments, while improving the validity and comparability of scoring complex constructed-response items.

cs.CV↗

arXiv2019

Training Optimus Prime, M.D.: Generating Medical Certification Items by Fine-Tuning OpenAI's gpt2 Transformer Model

This article describes new results of an application using transformer-based language models to automated item generation (AIG), an area of ongoing interest in the domain of certification testing as well as in educational measurement and psychological testing. OpenAI's gpt2 pre-trained 345M parameter language model was retrained using the public domain text mining set of PubMed articles and subsequently used to generate item stems (case vignettes) as well as distractor proposals for multiple-choice items. This case study shows promise and produces draft text that can be used by human item writers as input for authoring. Future experiments with more recent transformer models (such as Grover, TransformerXL) using existing item pools are expected to improve results further and to facilitate the development of assessment materials.

cs.CL↗