Looking beneath the crisis discourse: A contribution for the AEC Panel Discussion

Part 1: What’s the real crisis?

As my contribution to the discussion, I want to examine the claims of crisis that underpin current policy initiatives. The policy direction is to move towards a more detailed curriculum, characterised by a careful attention to the foundations of reading, in a thorough, stepwise and cumulative approach. I will argue here, that these initiatives are based on largely unexamined claims of a crisis, created by cherry-picking of evidence from large-scale international tests.  Moreover I argue that setting up a crisis discourse and then proposing a solution is not a scientific approach to analysis. To be more scientific in our reasoning, we need to examine hypotheses, and we need to look below overall averages to understand the explanatory mechanisms that drive the outcomes we see.

The current public discourse seems to narrate that scores in international tests of reading have fallen over time, and the cause of the crisis is the teaching methods.  This argument results in a public, angry and polarized polemic, with blaming of groups or even individuals for causing the crisis. I propose, that rather than wholesale acceptance of the crisis discourse, we would be better off investing time in critical analysis, in order to examine what’s going on underneath the scores that might explain the aggregated evidence being presented.

Potentially, there is a crisis. But equally, claims of crisis might be a construction, a tactic for the creation of a discourse of panic, and creating the conditions for a change agenda, with solutions driven by haste. Or, it might be that the situation is more complicated than ‘crisis or not’. There are a multiplicity of factors and interactions that characterise the pathway of implementation between policy directives and the educational experience of our children. This multitude of factors, and the interactions between the factors should also be part of our thinking.

The claims:

The PISA test of 15 year-olds is an international comparative test of literacy. The full report can be found here. Rather than a test of A-B-Cs, the construct of high performance in Reading Literacy includes the purposes of reading for personal fulfilment, academic learning and social participation,

“Reading literacy is understanding, using, evaluating, reflecting on and engaging with texts in order to achieve one’s goals, to develop one’s knowledge and potential, and to participate in society”

To achieve high proficiency in PISA, students need to solve intertextual conflicts, that is, they need to read across more than one text, making inferences about the sources of information, and engage in high level critical thinking, about multiple lengthy texts. Somewhat counter-intuitively, literacy is not framed here as a ‘thing in itself’ or set of basic skills, but as a vehicle to engage critically with texts, for personally, socially or academically meaningful purposes. Thus, if there is a crisis, it could be in these high level, meaningful uses of literacy for enjoyment, for disciplinary learning and for critical participation in society.

Aotearoa’s average performance (501) on PISA Reading (15 year-olds) is similar to US, Hong Kong, Australia (See Figure 2.1).  Singapore tops the list, with Ireland second.  Little part of the crisis discourse references what is happening in Singapore and Ireland.  Instead, we hear a lot about the superiority of policies in countries whose rankings are very similar to ours. My suggestion here would be to be very careful about wholesale policy borrowing from economically powerful countries, with powerful publishing marketplaces, whose outcomes in large scale tests do not seem to warrant it.

Looking at scores over time, Aotearoa’s performance on PISA Reading has indeed declined. In Figure 2.2, it is clear that the outcomes have declined by 28 points since 2000.  The years of greatest decline were between 2009-2012. Since that time, there has been a steady non-significant decline in average levels. Explanations for that decline since 2012 would need to take into account the almost parallel decline in the OECD average score.

So, it seems that there is indeed something to look into, possibly at an international level. After a marked dip between 2009 – 2012, scores have continued to decline slowly.  One commonly cited explanation for these dips is that the teaching of basic decoding skills to our young children has been flawed. This is a testable hypothesis. To test this hypothesis, we can look at international tests of younger children.

The PIRLS test is another international test of reading literacy, this time for Year 5s. The full report, where I have sourced all these figures, can be found  here. Figure 3.2 shows Aotearoa New Zealand’s outcomes on this test over time.

In 2001 the country average was 529. Over the next nine years, we maintained these averages such that by 2010 we sat at 531.  Between 2010 and 2015 average the score slipped 8 points to 523.  In 2020, scores were largely similar at 521.  As a country, it seems that our Year 5 scores have not fallen away at similar rates as the Year 10s.  Therefore, it seems unlikely that the cause of the dip in 15 year old reading scores is wholly, or even mostly, the teaching of decoding skills they received in Years 1 – 4.

Part 2: A crisis for whom, under what conditions?

While an overall average country outcome of 521 doesn’t seem to constitute a crisis for Year 5, there is a steady, small decline in average scores. Beneath this average, when the data are disaggregated, other hypotheses do emerge. When the distribution of outcomes is broken down by Decile (Figure 7.3) a picture of inequity in outcome emerges. The average score in PIRLs is standardised to 500. Averages for deciles 5 – 10 are all above that 500 mark. Deciles 1- 4 sit below.

Indeed, the difference in achievement between affluent and disadvantaged children in these assessments is 70 points (See Figure 7.1).  For some reason, our country’s educational system creates a pattern of outcome where children are more vulnerable to the effects of poverty than almost all other systems presented.

Impacts of moving schools. Figure 6.3 divides the outcomes by numbers of schools that students attended. Only when children have attended more than three primary schools does the outcome measure fall beneath the international test average. The picture created seems to indicate the fragility of education for children whose housing is precarious.

Similar outcome patterns can be seen in attendance at school. Students who are regular attenders at school have average scores well above our country average. Conversely absence of once a week is associated a large dip in outcomes (see Figure 6.4). This figure suggests at least two interpretations: either children who are absent achieve less highly, or children who are achieving less highly are more often absent. Or both. Likely, the causality is two-way. Not coming to school impacts opportunities to learn. Not learning makes it less engaging to be at school.

The correlations are also stark for bullying.  Again, outcomes dip below the 500 mark for those who report being bullied ‘about weekly’, a large difference from children who reported being bullied less often.

The data on students’ feelings about reading and about school are also potentially explanatory. Factors such as their feelings about reading (Figure 5.1) and their sense of belonging or safety at school (Figure 7.6) are probable contributors to their outcomes.

What to focus on?

For me, the international test data point us toward a focus on equity, not just of educational outcomes, but also on equity of the educational opportunities and educational experiences.

The outcomes also point toward a focus on high level literacies (critical, disciplinary) such as those required beyond Year 5. The high level outcomes are those required by content-rich (for learning curriculum knowledge) and context-rich (socially relevant and personally interesting) literacy learning.

I would also argue for a focus on good use of time. We do still need to be parsimonious with children’s time.  This would require teaching important content. It would also require careful formative use of assessment, so that nobody is ‘learning’ things they already know. This attention to pace and significance requires, in my view, a stronger system focus on teacher expertise and quality decision making.

High expectations of success coupled with equitable opportunities to learn requires a closer understanding of classroom processes of learning and teaching. Given our inequity of outcomes, it seems important that we understand the processes of inequity as they play out.  Bullying, attendance, transience are just some of the indicators measured by the PIRLS test. There are likely to be further processes of inequity at play in the system.

What to let go of?

I’d like to suggest that we should let go of hurried responding to weaponised crisis discourses, designed to undermine teaching and teachers. I’d also argue against binary thinking.  While algorithms create binaries by assigning people and positions to 0/1 categories, logically this is a fallacy. In most things, there is a wide distribution of position, and many ways to achieve more than one outcome at once.

Finally, I’d like to suggest we should think critically about the allure of simplistic solutions, and a ready-made package as cure-all.  We should avoid, in my view, framing minimum standards as our system goals. We risk aiming for very low goals, for a very long time in children’s schooling.  We should keep sight of pace and purpose, and we should avoid low expectations framed as basics. If we are going to aim for higher rankings in international tests, then we need to aim for those high-level literacy outcomes that are focused on learning and participating through literacy. It seems to me that focusing on basic skills, framed as literacy, at the expense of curriculum learning through literacy, would work against both the curriculum and the literacy outcomes we are wanting to achieve.

Rebecca Jesson (as part of the AEC panel discussion).