Learning Item Embeddings and Hyperparameters for IRT Calibration via Monte Carlo EM

Abstract

High-stakes computerized adaptive tests (CATs) must continually calibrate new items in their item bank. When an item is new, few responses are available, so item parameter estimates -- and thus test scores -- are poor. Item features and explanatory item response theory (IRT) models mitigate this by folding item content into calibration. Neural IRT models, whose item parameters are neural-net outputs, are powerful, but tuning hyperparameters and architectures in real time while a CAT is scoring is impractical and threatens validity. We propose a pre-launch step that fits a neural net to produce low-dimensional item embeddings, so the production system can use a simple linear explanatory IRT model on top of them. We use a neural parameterization of the 3-parameter logistic (3PL) model in which a feature network maps each item's content features to a representation $h_j = z(x_j) \in \mathbb{R}^d$, from which the discrimination and difficulty $(a,b)$ follow generalized linear forms; the guessing parameter $c$ is fixed to a global constant to avoid identifiability issues. The feature network and latent abilities $\theta$ are fit jointly via Monte Carlo Expectation-Maximization (MCEM), with no separate ability-estimation or pre-calibration stage. Using an item-split protocol that holds out entire items to simulate feature-only evaluation, we apply this to two Duolingo English Test practice task types -- yes/no vocabulary and vocabulary-in-context -- searching over feature sets, architectures, and dimensions $d$. A shallow two-layer ReLU network with $d=6$ and hand-engineered scalar features matches or beats larger architectures on held-out items for both. This is a first step toward a compact, content-derived item embedding for the Scalable Parametric Item Calibration Engine (SPICE), the fully Bayesian engine at the core of the S2A3 adaptive-testing system.