Can we trust our exam results?

Alison Lewis, AQA’s Director of Quality and Customer Standards, reflects on what we have learned from last year’s post-results analysis and looks at how we can increase confidence in AQA’s assessments.

Students opening their exam resultsOfqual’s recent report on A-level and GCSE exams in summer 2014, published December 2014, highlights the increase in the number of Enquiries about Results. We perform our own analysis of the use and outcomes of our post-results services each year. This informs our continuous improvement activity and helps us understand the impact of any broader changes in the education environment.

Summer 2014: the landscape

While changes to the education system seem to be constant now, these exams were sat against a backdrop of some of the most significant structural reforms in recent years.

Pending the introduction of new GCSE and AS/A-level qualifications from September 2015, the Government moved existing qualifications back to a system of linear assessment. Changes to re-sit rules and school accountability measures were also in force this series. At the same time, expectations of year-on-year improvements in cohort performance continued to put pressure on an increasingly high-stakes education system.

An understanding of these issues and how they may influence users of post-results services is an important contextual frame for our analysis.

What does our analysis show?

In terms of marking quality, it is reasonable to expect there to be some legitimate differences of opinion between two equally competent examiners, particularly in discursive subjects. However, these should only ever equate to a small number of marks.

From a marking accuracy perspective, we have seen the trend continue towards a reduction in both the average mark change following a re-mark enquiry and in the proportion of enquiries resulting in ‘out of tolerance’ mark changes. That is not to say exam marking in summer 2014 was without error: we know we have a problem in some subject areas and we intend to do more to improve our performance in the future.

The move to linear assessment meant more exams were sat in the summer. This increase in entry volume doesn’t account entirely for the increase in post-results enquiries, but it did have a significant upward effect on the numbers. 

Beyond entry volume increases and marking quality issues, we can see that it is likely that the contextual factors described above have also played a part in increasing the use of post-results services. For example, changes to the re-sit rule (limiting students’ ability to improve their results) and school accountability measures significantly raised the stakes of this series’ end of course exams, increasing the likelihood of challenges to results to improve grades.

So, what can we conclude?

There is clear evidence that the overall trend continues towards increased marking accuracy, underpinned by improvements to assessment design and operational delivery/quality control procedures.

It is also clear that, in some subjects, we need to do more to improve both marking quality and underlying confidence in our assessments.

Each year, as part of our continuous improvement, we review the performance of our assessments, subject-by-subject, utilising a range of qualitative and quantitative data (including assessment validity performance, item data, post-results outcomes, customer feedback etc).

These subject-level analyses will continue to form a key part of our approach to quality of marking – informing the design of assessments to ensure that they are valid and can be marked reliably; appointing and rigorously training appropriately experienced and qualified examiners; and controlling quality throughout the marking period, detecting and correcting errant marking early.

Finally, our analyses would suggest that other, externally driven factors – specifically changes to government policy and the consequential growth in tactical use of the post-results system – have had at least as significant an impact on the increases in volumes seen in summer 2014 than concerns over marking quality.

Ofqual has announced its intention to ‘fundamentally reform’ the post-results landscape in time for summer 2015. The details are not yet clear and we’re expecting a consultation shortly on any proposed changes. In the meantime, I’ll continue focusing our efforts where we can most directly effect change – through further, evidence-based improvements to our qualification design and to our assessment delivery and quality control procedures.

Alison Lewis is Director of Quality and Customer Standards at AQA.

Share this page