The Forgotten Third (or so) - Geoff Barton

That mutant algorithm

Algorithms have had a bad press since a ‘mutant algorithm’ was blamed for the furore over A-Level results in 2020. This is somewhat unfair on algorithms, which are simply sets of instructions or rules designed to provide a method for solving a problem. The one used to calculate the 2020 A-Level results actually worked as it was intended to, which was to bias results towards small settings (most obviously independent schools) and candidates from schools in more affluent areas. But without the cover provided by candidates having sat a series of exams to provide a suitable amount of uncertainty and doubt about how they had all performed ‘on the day’, the outcome that the algorithm was designed to achieve was naked and exposed for all to see.

The problem lay not with the algorithm itself, but with the outcomes that its designers intended to achieve.

You don’t have to dig very deep to find that the entire assessment system in England is, as it turns out, predicated on algorithms, sets of instructions designed to deliver planned statistical outcomes.

In 2007 The Children’s Plan [1] set out the target that by 2020, 90% of pupils would achieve the ‘headline measure’ of 5 GCSE passes at grade C or higher, including English and Maths (by the age of 19). Behind this target lies one of the most egregious of the deceptions practised on children, families and the teaching profession, because the target itself was completely illusory. It implied that the education system was entirely meritocratic and criterion-referenced, so that every child who reached the criterion could achieve a top grade.

But it was not and is not possible for anything like 90% of pupils in any one cohort to achieve this measure under England’s assessment system, even if a particular cohort happened to contain an unfeasibly large proportion of very high achievers. The structure of the system ensures that it cannot happen. Much worse, the structure ensures that more than a third of each cohort of students cannot attain the government’s headline measure, either.

This issue has been flagged most vigorously by Geoff Barton, and the union of which he is general secretary, the Association of School and College Leaders.

In an article for the Times Educational Supplement in October 2019 entitled ‘What are we trying to achieve with our education system?’ [2] he wrote:

“The percentage of pupils who did not achieve at least a grade 4 “standard pass” in GCSE English and maths was 35.6 per cent. Pause for a second at that figure, and consider its devastating significance. More than a third of the half a million state-school pupils finishing key stage 4 this year fell short of achieving qualifications that are widely seen as an important passport for onward progression. That percentage equates to more than 190,000 pupils.

And last year this percentage was strikingly similar. It was 35.8 per cent. The fact that these figures are so close gives you a clue about what is going on. Because the proportion of students who fall below this bar is no accident. Our system is deliberately designed in this way by the use of the mechanism of comparable outcomes, which decides the distribution of grades largely on the basis of how similar cohorts have performed in the past.

So, this rate of attrition – this “forgotten third”, as we have called them – is baked into the system, year in and year out.”

In September 2019, the Association of School and College Leaders published its final report of the Commission of Inquiry into the phenomenon of the ‘forgotten third’. [3]

‘When we talk about improving social justice, it is these young people who most need our attention.’

‘Grimly surreal as it may seem to the uninitiated, this level of collateral damage is an accepted part of the process for determining the distribution of grades. In other words, we judge the success of our education system by the number of young people who don’t gain that national ‘pass’. Few other high-performing jurisdictions would think that sensible or morally acceptable.’

Types of assessment

There are three ‘headline’ types of standardised assessments: criterion-referenced, norm-referenced and ipsative. Ofqual claims that GCSEs (and A-Levels) are none of the above, but instead are ‘cohort-referenced’. However, outside of Ofqual, it is generally acknowledged that cohort-referencing is essentially a type (or a subset) of norm-referencing.

“A sub-set of norm referencing is cohort referencing. This is where a normative-type distribution of scores is applied only to the cohort in question and thus the actual pass grade might vary from cohort to cohort. An example would be the former 11+ exam taken by pupils in the UK to sort those going from grammar schools from those going to technical or secondary modern schools.” [4]

The example of the 11+ exam is a good one, because it illustrates how our exam system is set up to ‘select’ a specific proportion of pupils and to ‘reject’ the remainder, rather than to recognise every child’s attainment and achievement.

Ofqual allows exam boards to have variations from ‘the norm’ of up to 3% – and this ‘flexibility’ is the somewhat sophistic basis on which Ofqual seeks to distance GCSEs from strict norm-referencing, without having to acknowledge that cohort-referencing produces almost exactly the same key outcome: that a planned proportion of candidates – give or take a percentage point here and there – will pass the exam at the ‘headline measure’, and a complementary proportion will not achieve the headline measure – every year.

This model of government-guaranteed failure through cohort-referencing applies to GCSEs and A-Levels. Debra Kidd gives an excellent summary of the corollaries that arise from this:

“In fact the proportions of [grade] 1s, 4s and 7s have been agreed in advance, regardless of what children do on the day. It means that although the government say that they have made exams harder and more rigorous, it hardly matters because the number of children passing will remain the same. So much for “raising standards.”

[….]

“This might not matter, if it were not for the fact that it creates a complete disconnect between reality and the expectations of government and Ofsted. If results are fixed in advance, then how can schools improve? If they are to be judged on data, how can it be fair to be expected not to “coast” when in fact the system is set up to ensure coasting? The fact is that every school that improves their data is doing so at the expense of another’s results.

We are pitched against each other – child against child, teacher against teacher, school against school – in a fight to protect our position and to try to improve it, knowing full well that our Outstanding comes at a cost for someone else. No wonder so many schools are starting to select by excluding pupils who may skew their data. No wonder they are looking for ways to secure advantage over others. It actually makes the idea of sharing best practice an act of folly. Why should we collaborate when doing so could hurt our students’ chances of success?’ [5]

From ‘Coasting schools’ to Progress 8

The formal concept of ‘coasting’ schools dates back to the Children’s Plan 2007, when it initially referred to schools that were ‘at risk of failure’. In 2011, David Cameron used a speech [6] to define coasting schools as ‘the ones whose results have either flat-lined, or where they haven’t improved as much as they could have done.’

Perhaps recognising that identifying which schools this would apply to would require at least two, and preferably more, data points in order to be able to discern a ‘flat line’ in respect of improvement, Ofsted seems to have implemented a ‘fast-track’ mechanism to increase the probability that schools would be at risk of failure by replacing the judgment category of ‘satisfactory’ with the judgment category ‘requires improvement’ in 2012 (all under the banner of ‘improving standards’).

By 2016, primary legislation and Regulations had established mathematical formulae for identifying schools that would be deemed legally to be ‘coasting’ and those which would be deemed not to have reached the ‘floor standard’ set by government. But by 2019, both had been dropped – because Progress 8, the government’s headline measure of a school’s progress, makes it a simple mathematical certainty that a fixed proportion of secondary schools have results that can be disapproved of (simply because they are ‘below average’) and that another fixed proportion cannot meet the government’s minimum expectation (because they are statistically ‘well below average’).

Progress 8 does for schools what the cohort-referenced assessment system does for children, viz. it ensures that an annually-specified proportion of them will not meet the government’s expectation.

An education system which secures that there will always be schools, and children, that cannot reach ‘the bar’ is not seeking to bring about any improvement in standards – in fact it holds a vested interest in not doing so. England’s is an education system that treats a significant portion of each year group as expendable ‘ballast’ whose primary function is to illustrate the success of their more highly ranked contemporaries, without regard to the reduction in life chances for those in the ballast group.

What are the aims of an education system that applies this principle both to schools and to pupils? Geoff Barton asks, ‘What are we trying to achieve in our exam system?’ Whatever the answers, it is clearly not for the benefit of the entire pupil body, or indeed every school. The impact of this system on the life chances of every young person who falls within the ‘below expectation’ cohort may be profound. Each is wholly entitled to ask at the end of Year 11, ‘What was the point of that?’

[1]https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/325111/2007-childrens-plan.pdf

[2] https://www.tes.com/news/what-are-we-trying-achieve-our-exam-system

[3] https://www.ascl.org.uk/ASCL/media/ASCL/Our%20view/Campaigns/The-Forgotten-Third_full-report.pdf

[4] Key Concepts in Educational Assessment, Isaacs, T., Zara, C. and Herbert, G. with Coombs, S. and Smith, C. (2013) London: Sage

[5] A broken system. Progress 8, GCSEs and SATs. 2017