TEXT SIZE

search for




CrossRef (0)
The alignment between contextual and model generalization: An application with PISA 2015
Communications for Statistical Applications and Methods 2024;31:467-485
Published online September 30, 2024
© 2024 Korean Statistical Society.

Wan Ren1,a, Wendy Chana

aGraduate School of Education, University of Pennsylvania, USA
Correspondence to: 1 Graduate School of Education, University of Pennsylvania, 3700 Walnut St, Philadelphia, PA 19104. Email: renwanmichelle@gmail.com
Received July 23, 2023; Revised January 10, 2024; Accepted April 23, 2024.
 Abstract
Policymakers and educational researchers have grown increasingly interested in the extent to which study results generalize across different groups of students. Current generalization research in education has largely focused on the compositional similarity among students based on a set of observable characteristics. However, generalization is defined differently across various disciplines. While the concept of compositional similarity is prominent in causal research, generalization among the statistical learning community refers to the extent to which a model produces accurate predictions across samples and populations. The purpose of this study is to assess the extent to which concepts related to contextual generalization (based on compositional similarity) are associated with the ideas related to model generalization (based on accuracy of prediction). We use observational data from the Programme for International Student Assessment (PISA) 2015 wave as a case study to examine the conditions under which contextual and model generalization are aligned. We assess the correlations between statistical measures that quantify compositional similarity and prediction accuracy and discuss the implications for generalization research.
Keywords : internal validity, external validity, generalizability, model predictions, covariates, PISA