Early years assessment is not reliable or valid and thus not helpful

The academic year my daughter was three she attended two different nursery settings. She took away two quite different EYFS assessments, one from each setting, at the end of the year. The disagreement between these was not a one off mistake or due to incompetence but inevitable because EYFS assessment does not meet the basic requirements of effective assessment – that it should be reliable and valid*.

We have a very well researched principles to guide educational assessment and these principles can and should be applied to the ‘Early Years Foundation Stage Profile’. This is the statutory assessment used nationally to assess the learning of children up to the age of 5. The purpose of the EYFS assessment profile is summative:

‘To provide an accurate national data set relating to levels of child development at the end of EYFS’

It is also used to ‘accurately inform parents about their child’s development’. The EYFS profile is not fit for these purposes and its weaknesses are exposed when it is judged using standard principles of assessment design.

EYFS profiles are created by teachers when children are 5 to report on their progress against 17 early learning goals and describe the ‘characteristics of their learning’. The assessment is through teacher observation. The profile guidance stresses that,

‘…to accurately assess these characteristics, practitioners need to observe learning which children have initiated rather than focusing on what children do when prompted.’

Illustration is taken from EYFS assessment exemplification materials for reading

Thus the EYFS Profile exemplification materials for literacy and maths only give examples of assessment through teacher observations when children are engaged in activities they have chosen to play (child initiated activities). This is a very different approach to subsequent assessment of children throughout their later schooling which is based on tests created by adults. The EYFS profile writers no doubt wanted to avoid what Wiliam and Black (Wiliam & Black, 1996) call the ‘distortions and undesirable consequences’ created by formal testing.

Reaching valid conclusions in formal testing requires:

  1.    Standard conditions – means there is reassurance that all children receive the same level of help
  2.    A range of difficulty in items used for testing – carefully chosen test items will discriminate between the proficiency of different children
  3.    Careful selection of content – from the domain to be covered to ensure they are representative enough to allow for an inference about the domain. (Koretz pp23-28)

The EYFS profile is specifically designed to avoid the distortions created by such restrictions that lead to an artificial test environment very different from the real life situations in which learning will need to be ultimately used. However, as I explain below, in so doing the profile loses necessary reliability to the extent that teacher observations cannot support valid inferences.

This is because when assessing summatively the priority is to create a shared meaning about how pupils will perform beyond school and in comparison with their peers nationally (Koretz 2008). As Wiliam and Black (1996) explain, ‘the considerable distortions and undesirable consequences [of formal testing] are often justified by the need to create consistency of interpretation.’ This is why GCSE exams are not currently sat in authentic contexts with teachers with clipboards (as in EYFS) observing children in attempted simulations of real life contexts. Using teacher observation can be very useful for an individual teacher when assessing formatively (deciding what a child needs to learn next) but the challenges of obtaining a reliable shared meaning nationally that stop observational forms of assessment being used for GCSEs do not just disappear because the children involved are very young.

Problems of reliability

Reliability: Little inconsistency between one measurement and the next (Koretz, 2008)

Assessing child initiated activities and the problem of reliability:

The variation in my daughter’s two assessments was unsurprising given that…

  • Valid summative conclusions require ‘standardised conditions of assessment’ between settings and this is not possible when observing child initiated play.
  • Nor is it possible to even create comparative tasks ranging in difficulty that all the children in one setting will attempt.
  • The teacher cannot be sure their observations effectively identify progress in each separate area as they have to make do with whatever children choose to do.
  • These limitations make it hard to standardise between children even within one setting and unsurprising that the two nurseries had built different profiles of my daughter.

The EYFS Profile Guide does instruct that practitioners ‘make sure the child has the opportunity to demonstrate what they know, understand and can do’ and does not preclude all adult initiated activities from assessment. However, the exemplification materials only reference child initiated activity and, of course, the guide instructs practitioners that

‘…to accurately assess these characteristics, practitioners need to observe learning which children have initiated rather than focusing on what children do when prompted.’

Illustration from EYFS assessment exemplification materials for writing. Note these do not have examples of assessment from written tasks a teacher has asked children to undertake – ONLY writing voluntarily undertaken by the child during play.

Assessing adult initiated activities and the problem of reliability

Even when some children are engaged in an activity initiated or prompted by an adult

  • The setting cannot ensure the conditions of the activity have been standardised, for example it isn’t possible to predict how a child will choose to approach a number game set up for them to play.
  • It’s not practically possible to ensure the same task has been given to all children in the same conditions to discriminate meaningfully between them.

Assessment using ‘a range of perspectives’ and the problem of reliability

The EYFS profile handbook suggests that:

‘Accurate assessment will depend on contributions from a range of perspectives…Practitioners should involve children fully in their own assessment by encouraging them to communicate about and review their own learning…. Assessments which don’t include the parents’ contribution give an incomplete picture of the child’s learning and development.’

A parent’s contribution taken from EYFS assessment exemplification materials for number

Given the difficulty one teacher will have observing all aspects of 30 children’s development it is unsurprising that the profile guide stresses the importance of contributions from others to increase the validity of inferences. However, it is incorrect to claim the input of the child or of parents will make the assessment more accurate for summative purposes. With this feedback the conditions, difficulty and specifics of the content will not have been considered creating unavoidable inconsistency.

Using child-led activities to assess literacy and numeracy and the problem of reliability

The reading assessment for one of my daughters seemed oddly low. The reception teacher explained that while she knew my daughter could read at a higher level the local authority guidance on the EYFS profile said her judgement must be based on ‘naturalistic’ behaviour. She had to observe my daughter (one of 30) voluntarily going to the book corner, choosing to reading out loud to herself at the requisite level and volunteering sensible comments on her reading.

 

Illustration is taken from EYFS assessment exemplification materials for reading Note these do not have examples of assessment from reading a teacher has asked children to undertake – ONLY reading voluntarily undertaken by the child during play.

The determination to preference assessment of naturalistic behaviour is understandable when assessing how well a child can interact with their peers. However, the reliability sacrificed in the process can’t be justified when assessing literacy or maths. The success of explicit testing of these areas suggests they do not need the same naturalistic criteria to ensure a valid inference can be made from the assessment.

Are teachers meant to interpret the profile guidance in this way? The profile is unclear but while the exemplification materials only include examples of naturalistic observational assessment we are unlikely to acquire accurate assessments of reading, writing and mathematical ability from EYFS profiles.

Five year olds should not sit test papers in formal exam conditions but this does not mean only observation in naturalistic settings (whether adult or child initiated) is reasonable or the most reliable option.  The inherent unreliability of observational assessment means results can’t support the inferences required for such summative assessment to be a meaningful exercise. It cannot, as intended ‘provide an accurate national data set relating to levels of child development at the end of EYFS’ or ‘accurately inform parents about their child’s development’.

In my next post I explore the problems with the validity of our national early years assessment.

 

*n.b. I have deliberately limited my discussion to a critique using assessment theory rather than arguments that would need to based on experience or practice.

References

Koretz, D. (2008). Measuring UP. Cambridge, Massachusetts: Harvard University Press.

Standards and Testing Agency. (2016). Early Years Foundation Stage Profile. Retrieved from https://www.gov.uk/government/publications/early-years-foundation-stage-profile-handbook

Standards and Testing Agency. (2016). Early Years Foundation Stage Profile: exemplification materials. Retrieved from https://www.gov.uk/government/publications/eyfs-profile-exemplication-materials

Wiliam, D., & Black, P. (1996). Meanings and consequences: a basis for distinguishing formative and summative functions of assessment? BERJ, 537-548.

Compartments

As I scrolled my twitter timeline the other day I was drawn in by a tweet written by a teacher at Michaela School in Brent. It contained a photo of a beaming Year 9 student. The girl was in the school playground very proudly holding up a copy of a book titled ‘The Malay Archipelago’.

The tweet read:

 
“This pupil arrived in UK 4 years ago without Eng. Now she’s reading ‘Malay Archipelago’ by A.R. Wallace. Rushed over to show me her fave page!”

 
It was the lovely smile that did it. Oh sweet! I thought. Sure, there was nothing so very unusual in a child enjoying a quality book but there was something so heartwarming in the thought of this girl skipping across the playground to share her book with her teacher that I could understand why the teacher had shared the event.
I wondered what she had been reading so I googled it:

 

malayThe Malay Archipelago
From Wikipedia, the free encyclopedia

The Malay Archipelago is a book by the British naturalist Alfred Russel Wallace that chronicles his scientific exploration, during the eight-year period 1854 to 1862.

 

 

 

Wow! Now I was REALLY impressed. As one teenager, myself as was, (hooked on 19thC literature) to another (intrigued by 19thC science writing) I saluted her taste! More than that – I acknowledged all that Michaela School, an inner city school with a tough intake, had achieved to create a culture in which this sort of event was likely. I’ve visited Michela. In the morning on the tube, on my way to the school, I saw Michaela kids reading their books as they travelled in. When I arrived I found out about all the policies in place to raise reading ages way beyond normal expectations and the carefully chosen knowledge curriculum that gave children genuine access to more difficult texts. My own daughter is the same age and goes to a lovely school but, goodness, it doesn’t quite have the reading culture Michaela has created. Perhaps Michaela isn’t unique but I was impressed and tweeted this:

 
“Brilliant work. @MCSBrent throws down the gauntlet to the rest of us. If they can make this happen so can we all.”

 
I, rather naively, hadn’t anticipated the barrage of negativity my little tweet of admiration would create. Apparently, according to some on twitter, there was nothing unusual about this little event. Many, many schools across the land achieve similar every single day. Yeah, sure thing I thought. Who hasn’t come across Year 9 kids reading the original works of nineteenth century explorers? Normal, normal, normal.

 
Then after reading a few more rather unfriendly twitter notifications the penny finally dropped… The readers of my original tweet hadn’t even noticed the title of the book this girl was reading. They had totally misunderstood the point I was making because they had seen one acronym:

 
EAL

 
…and got no further. The sight of this term had compartmentalised any interpretation of my tweet as a comment on…

‘Work With EAL’.

That this child had been reading for pleasure showed the success of…

‘Literacy Policies For EAL’

and presumably this could be measured on some EAL progress spreadsheet somewhere.

How tragic that this is the mentality our education system creates. Why weren’t these angry tweeters actually interested in WHAT a child (whatever her background) had been reading?

Surely the most notable aspect of the little tweet story I’d shared was NOT that an EAL student had read a book? Wasn’t it obviously more noteworthy to teachers that a 13 year old had got beyond reading pre-teen romances and made a foray into the syntactically dense but beautifully turned phrases of Victorian writing? Wasn’t it of most educational interest that she had delved into some original scientific observation?

 

Surely the quality of the literature a child is reading is of more educational interest than her background and the particular label she has been given in a progress spreadsheet?

 
Terms like EAL or even ‘literacy’ only have meaning in so far as they help us discuss how we can help children learn the actual, specific content we want to teach them. My twitter notifications over the last few days starkly illustrate how labels designed to help teachers do their job better can so often lead to the compartmentalising of children and a focus on means (EAL provision) over ends (a child learning something great).

Reading fluency and the ‘Tragedy of the Commons’

Is reading fluency important for academic success?

I’d imagine everyone reading this would agree it was very important – crucial in fact.

This description from Quirky Teacher of many children’s reading in year 6 must sound quite familiar to secondary teachers:

I am worried that children in KS2, despite being officially ‘able to read’, are still not really fluent, even when they get to year 6. When you ask them to read to you, they stumble slowly through a text, sometimes randomly substituting trickier and new words, never able to add intonation and not really getting the bigger picture.

As I outlined in my post yesterday that does not mean that at secondary level we ensure children get enough daily reading practice to ensure our students DO read fluently. One reason for this is that the need for bulk practice is not appreciated. There is another reason why, even when the importance of sustained reading is acknowledged, children still don’t end up with enough practice to become fluent readers. This reason can be explained using the economic principle of ‘The Tragedy of the Commons’.

What is this principle?

I like using this cartoon to explain the principle when teaching my A level politics students (about ecologism).

tragedy-of-the-commons

The tragedy of the commons is an economic theory of a situation within a shared-resource system where individual users acting independently according to their own self-interest behave contrary to the common good of all users by depleting that resource through their collective action.

At secondary level the ultimate goal of subject teachers is to ensure their students learn their subject, biology, geography, RE etc, probably measured through their success in public examinations. Aside from the ultimate benefit to the pupil, if children are fluent readers it helps all teachers achieve the academic goals they have for their students and therefore no teacher would quibble that reading fluency is important.  The reading fluency is a ‘common good’- a shared resource that benefits all teachers as they teach their subject to the student. However, that does NOT mean it benefits the individual teacher to focus on reading fluency. If you look at the wiki definition above, rather than gaining an individual advantage from ‘depleting’ a resource, subject teachers gain an advantage by not adding to a shared resource. It takes valuable lesson time away from subject learning to focus on building the commonly shared resource – fluent reading. I’ll explain using an example from my own teaching:

I want my year 9 students to learn about the 1916 Battle of the Somme. After a series of lessons the class will write an essay on whether the Somme was entirely a disaster. As a subject teacher I will get the most reward from getting the best possible essays as quickly and painlessly as possible. SO should I cover the key detail on the Somme by:

  1. Using a great article I found on the internet. I can abridge it to ensure that while it challenges the students it is accessible. I know this article covers the ground well but it will take a long time to read and the students won’t exactly be enthralled that they are doing lots of reading.
  2. Use a documentary from Youtube that covers all the issues in full technicolour and takes half the time the article would. Hey – we can practice note taking!
  3. Play some sort of fact finding games. Not as effective to get the content across as the article but it will look great if you are observed as the class will love it. They’ll walk out of the lesson saying they prefer history to geography. This means they’ll probably work harder on their history essay and opt for the subject at GCSE.

Let’s be honest. How often do we opt for A because we know it serves the greater good? The common goal of reading fluency is inevitably sacrificed. I don’t think individual teachers should be blamed for this. The incentives are all wrong. How many teachers can get by on a warm glow of self-righteousness when other teachers and other subjects get better results, no one opts for your subject at GCSE and the kids complain that (comparatively) your lessons are boring. We are far more likely to kid ourselves that what amounts to a tokenistic amount of reading is ‘doing our bit’ because we’re not into career suicide.

So what should be done? As my A level politics students know – this is an argument for centralised control. The incentives need to be changed by those with the power to do so – SLT. If sustained reading is the norm in ALL lessons then students just get used to it and will start to read the amount necessary each day to become fluent.

Perhaps this shift towards more sustained reading can be achieved as at Michaela School by having a quite centralised control over the materials used in class. Perhaps SLT can agree with subject departments a notional number of words a week that need to be read in each subject, as appropriate (i.e. not appropriate for maths!). Maybe lesson observation can be used or book scrutinies can look for evidence that there has been very regular sustained reading at the right level of challenge. Ultimately, a sensible, intelligently implemented,  way needs to be found to end what amounts to a ‘tragedy of the commons’.

Why can’t our students read fluently?

Lots of children learn to read well despite very poor or minimal teaching. I think I was probably one of those children. I don’t think my mum ever listened to me read but somehow I now read fluently. I remember seeing my big sister chuckling over an Enid Blyton Image result for child reading blytonand feeling jealous because I couldn’t read and then I remember being able to read Blyton myself. I think I must have read constantly from about the age of six to twelve only surfacing for school lessons. With my sister I spent every Saturday in the children’s section of the local library. I had a long walk home from school and I would hitch my bag straps over my forehead to allow me to read my latest volume as I dawdled along the pavement.
At first sight my experience does seem to suggest that motivation provides the key to growing a strong reader and that is certainly the assumption of our schools. I disagree.

There are, in fact, many factors that make it more likely that children will read well, the area is very well researched, but perhaps the most crucial factor tends to be overlooked – bulk practice. Surely, you may say, we all know children need to read lots. Schools bend over backwards to encourage children to read lots. I’m not so sure we really DO appreciate the importance of bulk practice for reading fluency OR that we do the right things to ensure that this practice happens.
At the heart of the problem is the faulty supposition that the only lever teachers have is exhortation. The assumption seems to be that we must persuade our children that reading is fun and if they won’t listen we just throw our hands up and bewail the situation or embark on ever more elaborate campaigns to entice reluctant readers to open a book. Motivation is important. We do want to ensure our children have good role models, attractive environments and great reading options. Schools are right to ensure these elements are in place. I think though that we forget that it is not motivation that grows a reader – it is reading that grows a reader – bulk reading. The world renowned reading researcher Keith Stanovich calls this the ‘Matthew Effect’.
It is true that I was motivated to read but that is hardly surprising when you consider that my unusual childhood involved no computer screens, no television, no after school activities; nothing to provide higher gratification than I could gain, aged six, from puzzling out the letters that let me into the world of St Clare’s and the hilarious tricks the girls played on their French mistress. I had nothing ‘better’ to do and so I read in bulk. Can we simply cross our fingers that children today, faced with a plethora of instant forms of gratification, will be persuaded to persist with a book?

Image result for child on a screen

One way or another children need to read enough words in a day, a week, a month, a year, to attain fluency. As Quirky Teacher pointed out in a recent blog, at primary level the typical group reading sessions involve very little sustained reading (see here also). At secondary levels teachers try and cut the amount of reading in lessons, using other mediums to make the subject learning more accessible. Thus in the average school day a child does not encounter anything like the number of words they need to read to become fluent readers. To be blunt, currently the education provided through school alone is not adequate to create fluent readers.
The fascinating work of the late Jeanne Chall suggests this drive for accessibility has been very counterproductive. She found that in America schools had been gradually reducing the reading age of subject textbooks to make learning more accessible which correlated with a decline in reading ages. Chall suggested that, as children were less challenged, the reading ages of children gradually dropped, creating a negative spiral in which textbook publishers continued to lower the challenge of text to keep up with the falling reading age of the students.
Children need to read lots because the vast majority of the vocabulary children encounter in text is not used in everyday speech. Written communication follows different conventions to the spoken word. However, even those children who do enjoy some David Walliams aren’t therefore being exposed to more academic forms of writing which build the stamina for engaging with more abstract texts that children will need if they are to succeed academically (as I explain here.) For this they need to engage with the more academic forms of writing they should repeatedly encounter at school.

Whatever children may choose to read should be a bonus. Something as crucial as reading fluency should not be dependent on whether we can cajole children to read for pleasure.  In their schooling they should encounter enough text, fiction and subject based, at the right level of challenge, to ensure they progress towards becoming fluent readers. This may be partly through initiatives like ‘Drop Everything And Read’ although for weaker readers this won’t help –they need more time reading with an adult. Not much of this bulk reading should be in maths or PE lessons BUT sustained reading should be a normal part of most lessons – including science, geography, business and RE etc. At one school (which many on the blogosphere may have heard of it) children read ten thousand words a day. I’ll repeat that with emphasis TEN THOUSAND WORDS A DAY! I think we can rest assured that Michaela school is growing fluent readers! I’d suggest that in most schools children don’t come anywhere near this total. When children present as struggling readers it would be interesting to investigate just how much bulk practice they have had in the past. I’d bet quite a lot that it was not very much!

For a very clear and more detailed outline of the problems of primary see this.

Schools shouldn’t be relying on parents to teach reading.

Most schools rely on parents to teach children to read.
No! I can already anticipate the angry response as teachers explain that they run weekly or bi-weekly group reading sessions and daily discrete phonics sessions…
It will make me horribly unpopular to say it but still I stand by my original claim.

I have spent too many hours on the TES early years and primary forums where teachers discuss their practice to be entirely ignorant. I’ve also read the interminable Mumsnet discussions in which mums compare notes on how much their children read at school. Mums with kids at all sorts of schools join in. There are voices from urban and rural schools. Some of their kids go to schools with mainly middle class intake others underprivileged. Ofsted outstanding and special measures are both frequently discussed. The standard system for reading instruction reflects my own knowledge of schools local to me. In most schools (there are exceptions) all sustained reading practice is done at home.

The middle of the road average model is a Reception or Yr1 child doing two group reading sessions a week and then reading alone with the teacher maybe once a month. In group reading sessions the children take turns to read a few sentences and follow on as others in the group read. Lots of the time is taken discussing the meaning of the text (which is great) but it means a child may only read a handful of sentences a week in this way.

What about the discrete phonics sessions? I am all for teaching reading through synthetic phonics but the vast majority of schools use ‘mixed methods’ which marginalises the use of phonic decoding when reading. This means that phonics sessions are tacked on top of contradictory ‘mixed methods’ instruction that is used when actually reading books. Phonics decoding requires scanning words left to right without guessing before you reach the end of the word. However, when given reading books the child is taught to let their eyes dart about looking for cues from context, picture, word shape or initial letter. This means they don’t tend get sustained practice applying their phonic knowledge to sound out words especially as the most popular reading books are written to encourage guessing rather than being written to allow practice of incrementally more difficult phonics knowledge. Add to this that if schools follow the good practice videos for phonics instruction issued by Ofsted, most phonics sessions will involve delightful games that may be engaging but contain little sustained practice.

I have asked myself what a child can most afford to miss, a handful of sentences and words a week at school or the sustained practice a majority get at home? The answer is clear but uncomfortable. In fact it is line with what the teachers are telling us parents themselves. At my son’s yrR curriculum meeting the teacher showed us a chart of the progress of last year’s class in reading. This was plotted against how frequently their parents had listened to them reading at home in the evening. As she pointed out to us all the conclusion is inescapable. If you want to get your child reading you’ve got to do the reading books at home.

The system is wrong and makes me very cross. I am told in every school newsletter about the delightful, engaging activities my kid is getting up to at school. There is almost a profligacy in the use of time. A numeracy session in which each child gets to throw a dice twice so manages only two calculations in half an hour. An hour of forest school a week. Phonics through parachute games. There is a cost to this indulgence but it is played out behind closed doors. In the family home the experience is a tad less joyful as the parent returns, often dog tired, from a day at work or is wrung out from a day with a fractious toddler. They prepare the meal for the children and then with determination that can only be summoned because they know their child’s future depends on it (they’ve been shown the chart) they coax, bribe and sometimes (to their own shame) threaten the tired child to read. Sure, often it is fun, sometimes delightful but often it is a struggle and that is normal. You are a lucky parent if necessary daily routines always match the child’s inclination.

Before you presume this is a gripe by a lazy mum, I taught my son to read before he even went to school and I would read with him whatever happened at school. I am actually more cross because there is a greater cost to relying on parents to teach reading. There will always be parents that don’t read with their child. It is wrong that schools farm out their core purpose to parents and then wring their hands when children don’t learn to read, blaming their home environment or the child themselves. Learning to read needs lots and lots of practice as my son’s yrR teacher knew full well. It is the job of schools to teach reading. If there is not enough time then teaching approaches and curriculum priorities have to change to make more time. I absolutely don’t blame individual teachers, they are trained in and required to follow standard practices. It is those practices I question. My children aren’t going to suffer. In fact they benefit because of the advantage I can give them and that is wrong – plain wrong. Children should be able to learn to read at school. It is a minimum entitlement and a top priority.