Conditional Standard Error of Measurement: Classical Test Theory, Generalizability Theory and Many-Facet Rasch Measurement with Applications to Writing Assessment

Abstract

Writing assessments often consist of students responding to multiple prompts, which are judged by more than one rater. To establish the reliability of these assessments, there exist different methods to disentangle variation due to prompts and raters, including classical test theory, Many Facet Rasch Measurement (MFRM), and Generalizability Theory (G-Theory). Each of these methods defines a standard error of measurement (SEM), which is a quantity that summarizes the overall variability of student scores. However, less attention has been given to conditional SEMs (CSEM), which expresses the variability for scores of individual students. This tutorial summarizes how to obtain CSEMs for each of the three methods, illustrates the concepts on real writing assessment data, and provides computational resources for CSEMs including an example of a specification file for the FACETS program for MFRM and R code to compute CSEMs for G-theory.