On the prediction of exam aegrotat marks: Three models compared

Abstract

The Academic Quality Agency for New Zealand Universities (AQA) has released the first Cycle 5 academic audit (AQA, 2014). One item identified was the quality assurance of aegrotat decisions. Recommendation 6 included “undertaking an analysis of aegrotat outcomes”. The Committee on University Academic Programmes’ (CUAP) Handbook gives the criteria for quality assurance of assessment. Aegrotat grade decisions must be fair, valid and consistent, and require that “work undertaken during the paper reached an adequate standard” (CUAP, 2013, 4.3).

Very little has been published on the prediction of aegrotat marks. There are three models of aegrotat prediction based on quite different prediction methods. The nearest-neighbour method of McKeown and Maclean (2010) used repeated iterations to identify the most comparable students by coursework items and used their average exam mark to predict the aegrotat exam mark. Obben (2011) ran multiple regressions of the exam marks on the three coursework marks, student characteristics (internal/external and degree enrolled) and year dummies to predict the final grade in the absence of an exam mark. Hickson and Agnew (Agnew & Hickson, 2012; Hickson & Agnew, 2013) simulated putting the weight of the missing exam across the coursework items and showed how different weightings affected the final grade distribution.

All three ran simulations over multiple semesters to test for accuracy, though the results can’t be directly compared because each used a different measure of errors. There is an irreducible level of error in predicting student outcomes. We compare each of the models against the CUAP criteria for aegrotat decisions and pragmatically from an examiner’s perspective. This is a first step to a more comprehensive analysis of aegrotat decisions and to benchmarking which can inform moderation of aegrotat decisions.