Alphabet Soup – a post about A level marking.

Cambridge Assessment tell us that ‘more than half of teachers’ grade predictions are wrong’ at A level. There is an implication behind this headline that many teachers don’t know their students and perhaps have to be ‘put right’ by the cold hard reality of exam results.

What Cambridge Assessment neglect to say is that greater accuracy in predicting A level module results would actually need prophetic powers. No teacher, however good, can predict module results with any reliability because many results are VERY unpredictable. I teach history and politics, humanities subjects and so likely to have the less consistent results than subjects like sciences. Our history results have been relatively predictable of late but our A level politics results are another matter.

To illustrate we will now play a game. Your job is to look at a selection of real data from this year and predict each student’s results using their performance in previous modules as a guide. You might think that perhaps your lack of knowledge of each student’s ability will be a hindrance to your ability to make accurate predictions. Hmm well, we’ll see! (If you are a party pooper and don’t want to play you can scroll forward to the last table and see all the answers…)

It may help you to know that in politics the two AS and the two A2 modules follow similar formats and are of similar difficulty (perhaps Unit 2 is a little bit harder than Unit 1) and so a teacher will probably predict the same grade for each module as there is no reason why grades will vary markedly between modules that a teacher can anticipate. The A2 modules are a bit harder than AS modules and so a teacher will bear this in mind when predicting A2 results with the AS grades in front of them. (That said students mature and frequently do better at A2 than AS so this isn’t an entirely safe trend to anticipate.)

So using the Unit 1 grades on the first table below, what might you predict for these students’ Unit 2 module? (Remember the teacher will necessarily have given the exam board the same predicted grade for Unit 1 and 2.)

 

Candidate Unit 1 [AS] Unit 2 [AS] Unit 3 [A2] Unit 4 [A2]
1 B
2 B
3 B
4 A
5 B
6 A
7 A
8 B
9 A
10 A
11 A

(The overall results will soon be in the public domain and I have anonymised the students by not including the whole cohort (which was 19) and not listing numerical results (which interestingly would actually make prediction even harder if included). I have not listed results at retake as these would not be available when predictions were made. We use Edexcel.)

Check the table below to see if you were right.

Candidate Unit 1 [AS] Unit 2 [AS] Unit 3 [A2] Unit 4 [A2]
1 B B
2 B C
3 B E
4 A C
5 B E
6 A E
7 A D
8 B B
9 A B
10 A A
11 A A

 

Were you close? Did you get as much as 50%? If you simply predicted the same grade for Unit 2 as scored in Unit 1 (as teachers generally would) you could have only got 7/19 of the full cohort correct.

Now don’t look at the table below until you have tried to predict the first A2 grade! Go on! Just jot down your predictions on the back of an envelope. (Isn’t this fun?)

Candidate Unit 1 [AS] Unit 2 [AS] Unit 3 [A2] Unit 4 [A2]
1 B B D
2 B C E
3 B E B
4 A C E
5 B E D
6 A E D
7 A D E
8 B B C
9 A B D
10 A A A
11 A A C

Rather an unpredictably steep drop there! It was all the more puzzling for us given that our A2 history results (same teachers and many of the same students) were great.

If you’ve got this far why not predict the final module?

 

Candidate Unit 1 [AS] Unit 2 [AS] Unit 3 [A2] Unit 4 [A2]
1 B B D C
2 B C E B
3 B E B E
4 A C E D
5 B E D B
6 A E D D
7 A D E D
8 B B C B
9 A B D D
10 A A A A
11 A A C C

 

We have no idea why there is a steep drop at A2 this year.  Perhaps our department didn’t quite ‘get’ how to prepare our students for these papers in particular. I doubt it – but if so what does this say about our exams if seasoned and successful teachers can fail to see how to prepare students for a particular exam despite much anxious poring over past papers and mark schemes? Our politics AS results were superb this year for both modules.  Our department’s History results were fantastic at AS and A2. These results were entirely UNPREDICTABLE. Or to put it another way, they were only predictable in the sense that we anticipated in advance that they would look like alphabet soup – because they often do.

The first point that should be clear is that no teacher could possibly predict these module results. To even make the statement ‘more than half of teachers’ grade predictions are wrong’ is to wilfully mislead the reader as to what the real issue is here.

In fairness, it might feel like the grades have been created by a random letter generator but the results aren’t quite random. Some very able students, such as candidate 10, get As on all papers and would only ever have had A predictions. The final grades of our students were generally within one grade of expectations so the average of the four results has some validity. This said, surely there is a more worrying headline that the TES should have run with?

Just how good are these exams at reliably discriminating between the quality of different candidates? It is argued that marking is reliable but what does this then tell us about the discriminatory strength of the exams themselves or their mark schemes? It can’t have helped that there were only 23 marks (out of 90) between an A and an E on Unit 3 or 24 marks between those grades on Unit 4. I have discussed other reasons I think may be causing the unpredictability here and here.

Not all exams are as unpredictable as our Government and Politics A level but if we want good exams that reliably and fairly discriminate between students we need to feel confident we know why some exams have such unpredictable outcomes currently. Ofqual has been moving in that direction with their interesting studies of maths GCSE and MFL at A level. As well as being unfair in its implication the Cambridge Assessment headline is simply unhelpful as it obscures the real problems.

 

 

Advertisements

5 thoughts on “Alphabet Soup – a post about A level marking.

  1. Do you use Edexcel for History too? I find Edexcel History module outcomes to bear worryingly little similarity to my sense of the rank order of a cohort.

  2. Add into this mix that predictions are sometimes taken off the last interim report by exams officers/data managers and there might well have been a very different agenda at work rather than purely predicting.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s