Teachers as raters: Investigation of a long term writing assessment program

Abstract

In 2010, the Norwegian Writing Centre (NWC) was commissioned by the Norwegian Directorate for Education and Training to develop the National Sample-Based Writing Test (NSBWT), which was to be administered annually to a national representative sample of students in primary and lower secondary school (NSBWT-5 for school year 5 and NSBWT-8 for school year 8). The NWC was also commissioned to set up a national panel of raters (NPR), consisting of teachers, with the purpose of 1) establishing a strong interpretive community and 2) having in place a panel that would reliably rate the NSBWT. The first reliability estimates from the autumn of 2010 indicated large variation. However, it was the belief of the NWC that an interpretive community would slowly evolve through rater training over a long period of time. The present study utilized multiple data sources to explore this assumption by investigating potential variation among a sub-sample of NPR members. The data consisted of one quantitative dataset of ratings and one qualitative dataset based on semi-structured interviews and live ratings. The quantitative investigation showed large variation among the raters, as did the investigation using qualitative data. The results are discussed in depth.