Partial Identification of Mean Achievement in ILSA Studies with Multi-Stage Stratified Sample Design and Student Non-Participation

Abstract

International large-scale assessment (ILSA) studies collect information across education systems with the objective of learning about the population-wide distribution of student achievement in the assessment. In this article, we study one of the most fundamental threats that these studies face when justifying the conclusions reached about these distributions: the identification problem that arises from student non-participation during data collection. Recognizing that ILSA studies have traditionally employed a narrow range of strategies to address non-participation, we examine this problem using tools developed within the framework of partial identification of probability distributions. We tailor this framework to the problem of non-participation when data are collected using a multi-stage stratified random sample design, as in most ILSA studies. We demonstrate this approach with application to the International Computer and Information Literacy Study in 2018. We show how to use the framework to assess mean achievement under reasonable and credible sets of assumptions about the non-participating population. We also provide examples of how these results may be reported by agencies that administer ILSA studies. By doing so, we bring to the field of ILSA an alternative strategy for identification, estimation, and reporting of population parameters of interest.