The Achievement Matters calibration process was designed to facilitate consensus of graduate accounting learning standards; using anonymous reviews of assessment requirements and student work peer review workshops to help calibrate reviewers’ judgements.
The review process is outlined below.
Stage 1 Each expert reviewer assessed three samples of student work submitted as evidence of a specific learning standard/s and submitted independent reviews online via SPARKPLUS – screenshots are available in Appendix E. Reviews included an opinion and justification on whether the standard had been met. To ensure confidence, the review process was anonymous. Reviewers had the opportunity to view reviews of their peers within their group on SPARKPLUS prior to the workshop.
Stage 2 Reviewers participated in a workshop and considered the reviews from other participants. In groups of four or five, discussion focused on judgement and justifications until a consensus was reached within the group on the first sample. This also provided an opportunity for each reviewer to reflect on previously submitted individual reviews.
Stage 3 Reviewers discussed key differences until a consensus was reached across all groups. This three-step calibration process was important in developing a shared understanding of the standards that must be achieved by accounting graduates.
Stage 4 Reviewers repeated the above steps with the second and third sample.
Stage 5 Reviewers repeated the entire process with new samples at the workshop until there was confirmation that the calibration process had been effective, and there was agreement on the standard for an accounting graduate.
Substantial efforts were undertaken at each calibration workshop to assist and ensure peer reviewers provided adequate written feedback in their reviews to accompany their ratings and judgements of whether the:
- assessment tasks were valid and could allow students to demonstrate achieved learning outcomes met or exceeded agreed learning standards;
- learning outcomes achieved in the samples of five pieces of student work met or exceeded agreed learning standards.