Online learning: How have children fared by subject?
17th February 2022 by Timo Hannay [link]
Update 17th February 2022: See also this coverage from TES.
Since April 2020, Oak National Academy has been providing free online resources to support distance learning for children between the ages of 4 and 16, particularly those affected by school closures and other restrictions during the COVID-19 pandemic. We have previously looked at patterns of Oak usage to determine overall levels of activity, characterise the kinds of schools making use of Oak and determine the types of devices being used by their pupils.
This post looks at how those children performed in different subjects and topics. Among other uses, this may help to identify areas in which pupils need extra support, both during the current post-lockdown catch-up period and, perhaps, in the longer term too. We would like to thank Oak for making their data available, supporting its analysis and allowing these results to be shared.
We conclude that:
- In general, comprehension scores declined with age: primary pupils performed better than secondary pupils in post-lesson quizzes and young primary pupils (Key Stage 1) performed best of all.
- There was also considerable variation by subject. For example, maths and science showed relatively large declines in comprehension between primary and secondary, while geography and religious education were more consistent across age groups. We have also explored variations between different units within each subject.
- Pupils tended to score higher in post-lesson 'exit' quizzes than in the same questions posed at the start of the next lesson. This suggests a degree of (natural) forgetting between lessons and provides a putative indicator of knowledge retention.
- In contrast to comprehension, maths showed very high retention rates across all Key Stages, while history and religious education performed less well; science and geography were generally somewhere in between.
- As well as delivering valuable learning resources for pupils, online platforms such as Oak can also provide detailed, real-time insights into children's collective challenges and accomplishments as they progress through the curriculum.
Extra curricular activities
Oak learning resources are composed primarily of video lessons delivered by expert teachers, each followed by a multi-choice 'exit quiz' to check pupils' comprehension of the content and provide them with feedback. Each subsequent lesson in a series also usually contains an 'intro quiz' containing the same questions. This acts as a reminder of the previous lesson's material and provides a measure of knowledge retention.
Figure 1 shows the total numbers of exit and intro quiz questions answered each day between 1st February and 31st December 2021, which is the period we will cover in this analysis. (Data were also colllected in January 2021, but were not readily available in a consistent form, so have been omitted here.)
Daily numbers show extremely heavy usage of up to 3 million or more responses each day during February and early March, when most schools in England were closed to in-person teaching for the majority of pupils. Activity fell to roughly 10% of that level by mid-March following school re-openings, though this still represents many hundreds of thousands of quiz responses a day.
Cumulative data show that this amounts to almost 100 million responses in total, about 60% of which came from the period of particularly intense use in February and early March.
(Use the menu below to switch between daily and cumulative views. Hover over the line to see corresponding values.)
Figure 1: Oak quiz questions answered
Examined nation
Our main focus in this analysis will be the proportions of questions that pupils answered correctly, and how these vary by age and subject. Figure 2 shows the proportions of exit quiz questions answered correctly, grouped by Key Stage (across all subjects). In general, around 75% of exit-quiz questions were answered correctly, though average scores were higher than this for younger pupils and lower for older ones.
Of course, the material covered by older pupils is more complex and therefore may be more challenging, even after allowing for their greater maturity. It is also important to note that the structure of questions are often more complex for older pupils, for example involving greater numbers of options or the need to choose more than one option from a list. The tendency for Key Stage 4 pupils to slightly buck this trend might be due in part to the fact that about half of them were to be assessed for GCSE grades, which may have increased motivation.
Average scores also varied by subject (across all Key Stages), with languages and maths generally doing quite well, while hands-on subjects like music, computing and art produced somewhat lower average scores.
(Use the menu below to switch between Key Stage and subject views. Hover over the columns to see corresponding values and sample sizes.)
Figure 2: Exit quiz performance by Key Stage or subject
Note, however, that the subject mix varies by Key Stage, so it is more informative to look at individual subject performance within each Key Stage, as shown in Figure 3.
At Key Stage 1, maths and science showed the highest scores, but by Key Stage 2 they declined relative to the growing list of other subjects. At Key Stage 3 and Key Stage 4 they were more or less bottom of the table. Other subjects such as geography and religious education showed more consistent performance across different age groups.
(Use the menu below to switch between Key Stages: 1, 2, 3 and 4. Hover over the columns to see corresponding values and sample sizes.)
Figure 3: Exit quiz performance by subject for each Key Stage
These trends can be seen even more clearly in Figure 4, which shows exit quiz performance for the six subjects that were included in all four Key Stages.
Maths and science showed relatively large drops, especially from Key Stage 1 to Key Stage 3, while geography and religious education showed much less change. History and RSHE/PSHE were somewhere in between. (Click here to show all subjects again.)
Whilst we can only speculate, it is possible that the greater quantity of relatively abstract content in secondary maths and science is one driver behind their lower performance among older pupils. In addition, these subjects arguably rely more than most on using previously learned concepts as a basis for acquiring new knowledge, so the effects of any insecure understanding might be expected to magnify over time.
(Hover over the points to see corresponding values and sample sizes; click on the figure legend to show or hide individual subjects. Please note that in order to make the changes between Key Stages easier to see, the Y-axis is not zero-based.)
Figure 4: Exit quiz performance by Key Stage for selected subjects
Unit tests
So much for quiz performance by subject, what about the individual topics, or 'units', within each subject? These are shown in Figure 5.
There is a lot of information here and we invite you to explore it using the menu below. But to highlight briefly the example of maths, at Key Stage 1, most units were in the 80%-90% range, by Key Stage 2 many (from an admittedly even longer list) fell below 70%, and at Key Stage 3 some units were well below 60%. A similar, if less extreme, pattern was evident for science across Key Stages 1, 2 and 3.
(Hover over the points to see corresponding values and sample sizes; click on the figure legend to show or hide individual subjects.)
Figure 5: Exit quiz performance by unit
Remembrance of things passed
The analysis above has focused on exit quiz performance as a measure of comprehension. It is also possible to investigate retention by comparing performance between the same questions presented in exit and intro quizzes.
Figure 6 shows the distributions of mean scores for questions that were presented in exit quizzes (blue columns) and the same questions presented in intro quizzes (red columns). Across all Key Stages and subjects, the mean score for any given question tended to be lower in intro quizzes, suggesting that pupils tended to forget some of the things they had previously been taught. Of course, this is to be expected and is a completely normal part of learning.
(Hover over the columns to see corresponding values; click on the figure legend to show or hide each quiz type.)
Figure 6: Distributions of mean scores for exit and intro questions
Figure 7 shows the distribution of the change in mean scores between questions posed in exit quizzes and the same questions used in intro quizzes. Performance typically dropped by around 5 percentage points, but sometimes by more. Very occasionally performance actually seemed to improve a bit between lessons.
(Hover over the columns in the figure to see corresponding values and counts.)
Figure 7: Change in mean score between exit and intro quizzes
It is appropriate to pause at this point and issue some health warnings. We cannot be certain that pupils always viewed lessons in the intended order, so the lower average performance in intro quizzes might be partly because they sometimes hadn't seen the relevant content. Indeed, we cannot even be sure that the same pupils answered both the exit and intro quiz versions of each question.
Even so, how can intro quiz performance ever be higher than exit quiz performance? There are at least four possibilities:
- Consecutive lessons, and associated quizzes, taken in quick succession
- Other complementary study activities between Oak lessons
- Natural knowledge consolidation between lessons
- Statistical effects, especially when sample sizes are relatively small
The main message here is that we should think of these retention results as broad indicators derived from a mass of pupil activity, not definitive proof obtained under controlled experimental conditions.
Figure 8 breaks down these putative retention measures by Key Stage and subject. Again, we invite you to explore these using the menu below, but given our focus on maths in the comprehension analysis above, it's interesting to note just how well this subject appears to perform in terms of retention across all Key Stages: 1, 2, 3 and 4.
Indeed, at Key Stage 4, intro quiz performance for maths was slightly better than exit quiz performance. Given the fact that these pupils were often being assessed for GCSE grades, this might have been caused at least in part by further complementary study between Oak lessons. In addition, Oak's Key Stage 4 maths lessons are consistently short (often around 15 minutes or less), so it also seems likely that pupils frequently took several lessons in succession.
In any case, it is striking to note that while older pupils appeared to struggle more with comprehending maths than most other subjects, they seemed to have much less trouble retaining that knowledge one acquired. By comparison, history and religious education, for example, did relatively poorly in terms of retention. Perhaps this is at least in part because maths tends to rely more on general abstract concepts, while humanities subjects make greater use of contingent facts.
(Use the menu below to switch between Key Stages: 1, 2, 3 and 4. Hover over the columns to see corresponding values and sample sizes.)
Figure 8: Change in mean score between exit and intro quizzes, by Key Stage and subject
Comprehensive perspective
Figure 9 summarises comprehension and retention scores for the five subjects that are present across all Key Stages. (A sixth subject, RSHE/PSHE was included in Figure 4, which showed comprehension scores, but is omitted here because intro quiz data were not available, so there are no retention scores.)
At Key Stage 1, retention seems to correlate with comprehension; in particular, comprehension and retention in maths (red circle) are both high. At Key Stage 2, activity (as indicated by the circle sizes) is greater, but the correlation between comprehension and retention has disappeared. By Key Stage 3, the number of responses in science (brown circle) is particularly large, but comprehension and retention in that subject are both relatively low. And at Key Stage 4 there is somewhat higher retention in certain subjects, but the overall pattern is broadly similar to Key Stage 3.
As we have already seen above, the trend for maths between Key Stage 1 and Key Stage 4 is for comprehension to fall but retention to stay very high. In other words, older pupils seem to understand less of what they are taught, but what they do understand they tend to remember well.
(Hover over the dots to see corresponding values and sample sizes; click on the figure legend to hide or display individual subjects.)
Figure 9: Comprehension and retention by subject
Exit questions
As well as delivering popular learning resources, Oak also provides a window into children's educational challenges and accomplishments. It is not a controlled experiment, so findings like those reported here should be considered indicative rather than conclusive, but they do provide a new way to assess educational effectiveness and to understand children's learning experiences.
In particular, it is useful to be able to compare different subjects – and different units or even lessons within each subject – across a common platform that is closely modelled on the English national curriculum. While some of the findings described here may prove, in time, to be specific to Oak and its particular lesson or quiz designs, many of them surely represent more general trends that can inform teaching practice during both the current educational recovery period and the 'new normal' that will presumably follow. Indeed, perhaps it would be useful to continue to track these patterns over time and, in this way, to provide not only a platform that supports the vital task of teaching, but also one that enables continuous learning about education itself.
As ever, we welcome your thoughts: [email protected].
Are 'elite' sixth forms engines of progress?
11th February 2022 by Timo Hannay [link]
Update 11th February 2022: See this coverage from Schools Week and this analysis by FFT Education Datalab.
The UK government's latest 'levelling up' plans declare, among many other things, a desire to improve sixth form provision in currently underserved areas:
"The UK Government will ensure that talented children from disadvantaged backgrounds have access to a post-16 provider with a track record of progress on to leading universities by opening new 16-19 free schools targeted in areas where they are most needed, such as high priority EIAs [Education Investment Areas]."
The accompanying press release talks of 'elite' sixth forms – ie, those that admit and turn out students who are on a strong academic track. This post will look at the evidence that more such institutions are needed, and the effect they have on student outcomes. To summarise:
- Out of 2,467 existing mainstream state sixth forms in England, we identify 438 institutions (17.6% of the total) that might be considered elite based mainly on the prior academic attainment of their students.
- These are very unevenly distributed. Almost 19% of students in London attend an elite sixth form, but only about 8% of those in the West Midlands do. The differences by local authority (LA) are even more stark: of 150 LAs with secondary provision, 34 have no students at such institutions and a further 46 have less than 10%. Meanwhile, 19 LAs have more than 33% of students at elite sixth forms.
- Among LAs with high proportions of students in elite sixth forms, the progression rates into higher education are much greater (around 51% versus 33%). This persists even after controlling for variations in deprivation, demographics and prior educational attainment. Proportionally, the differences in progression to top universities were even greater.
- Notwithstanding other potential effects of highly academic sixth forms on the education system as a whole, there does indeed appear to be a case to expand proivision into underserved areas in order to support greater progress into higher education as a whole.
Patchwork in progress
Figure 1 shows attainment and value-added (ie, progress) scores, where available, for all 2,467 mainstream state sixth forms in England for the last pre-pandemic year, 2019. The line shows a linear regression through the data. As you might expect, higher attainment generally corresponds to greater progress, but with some divergence because students' prior GCSE grades vary.
Of these sixth forms, 163 are associated with grammar schools (purple dots). These tend to show high levels of attainment, but lower-than-average progress because their students are so high-attaining to begin with (ie, they usually have very good GCSE grades). A further 171 are associated with high-attaining secondary schools (green dots) that have Attainment 8 scores1 greater than 55. These have a somewhat similar distribution to grammar schools.
Prior attainment data is not always available, especially for standalone sixth forms, but we can also infer those that have such students by identifying sixth forms that combine high A-level attainment (average points score across the best three subjects of greater than 35) with relatively low progress (falls below the regression line in the figure). This adds a further 102 sixth forms (red dots). We also bestow honourary membership of this latter group to two further sixth forms (King's College London Maths School and Exeter Mathematics School) that have such high attainment it would be perverse to omit them.
This make a total of 438 'elite' sixth forms, which is just under 18% of the total.
(Click on the figure legend below to turn different groups of sixth forms on or off. Hover over the dots to see corresponding values and institution names.)
Figure 1: A-level value-added against A-level attainment for mainstream state sixth forms in England (2019)
Key to the question of levelling up is where these institutions are located. This is shown in Figure 2.
Grammar schools (purple dots) are well known to be unevenly distributed around the country (see our previous analysis for more on this), but the same is true of sixth forms associated with high-attaining secondary schools (green dots), which are particularly concentrated in London. Other sixth forms with high implied prior attainment (red dots) are a bit more widely scattered, but also fewer in number. The overall distribution does suggest some geographical bias.
(Use the menu below to switch between groups of elite sixth forms. Hover over the dots to see institution names.)
Figure 2: Locations of elite sixth form colleges in England
Figure 3 provides a more quantitative analysis by region. Across London and the South East, around 15-20% of sixth form students attend elite institutions. In the South East, this is largely because of the preponderance of grammar schools, while in London it is mainly because there are so many high-attaining secondary schools. The lowest overall proportions are in the West Midlands, where only 8% of students attend elite sixth forms.
(Use the menu to switch between groups of sixth forms. Hover over the map to see corresponding values.)
Figure 3: Proportions of students at elite sixth forms by region
At the local authority (LA) level, the overall distribution is even patchier, as shown in Figure 4. This uses a cartogram in which each area is scaled according to the local pupil population, making small urban locations easier to see.
Fully 34 LAs have no students at elite sixth forms and a further 46 have less than 10%. Meanwhile, 19 LAs have more than 33% at elite sixth forms and in a few these students account for a majority. Compare the distributions of sixth forms associated with grammar schools and high-attaining secondary schools, as well as those with high implied prior attainment, which combine to create the overall distribution of elite sixth forms.
(Use the menu to switch between groups. Hover over the map to see corresponding values and LA names.)
Figure 4: Proportions of students at elite sixth forms by local authority
The effects of elitism
Even if the distributions of these institutions are uneven, what if any consequences does this have? Our previous analysis of grammar schools found, among other things, that these selective secondary schools have slightly depressive effects on the academic performance of nearby comprehensive schools, but also positive effects on overall rates of progression into higher education. The necessary data to analyse the effects of elite sixth forms on the academic performance of less selective 16-19 institutions are not readily available, but we can look at their effects on higher education destinations2.
These are shown in Figure 5, which compares elite LAs (those with more than 33% of students at elite sixth forms; red column) with non-elite LAs (those with less than 10% of students at elite sixth forms; dark blue column). We will come to the light blue central column in the moment.
By definition, elite LAs have many more students at elite sixth forms (42% versus 2.8%). They also have far higher proportions of students progressing into higher education, top-third universities, Russell Group universities and Oxbridge. However, they also have lower levels of disadvantage, different ethnic mixes and higher levels of urbanisation. They have higher GCSE attainment too: the average Attainment 8 score is 48.8 versus 45.0 for non-elite LAs. Could any of these account for some or all of the differences in destinations?
It seems not. Even if we limit our analysis to non-elite LAs that share broad socioeconomic, demographic and educational characteristics with elite LAs, the results are almost identical3. This is shown in the light-blue central column below. Overall, these LAs are much more similar in terms of disadvantage, ethnicity and urbanisation (though because it's impossible to control adequately for all parameters simultaneously, they differ somewhat in terms of language). They also have more similar GCSE attainment levels, with an average Attainment 8 score of 47.2. Yet the resulting destinations into higher education, top-third universities, Russell Group universities and Oxbridge are essentially indistinguishable from all non-elite LAs taken together.
(Use the menu below to select a characteristic. Hover over the columns to see corresponding values.)
Figure 5: Sixth-form student characteristics of elite and non-elite local authority areas
Up with levelling?
As usual, none of this is conclusive and there may be other factors at play. For example, the proportion of sixth form students attending independent (ie, private) schools and colleges is higher among elite LAs (12%) than non-elite LAs (5.7%) and otherwise similar non-elite LAs (3.3%). We also haven't taken into account the flow of students across LA boundaries when they move from secondary school to sixth form. Nevertheless, these results do support the idea that sixth form provision is very unevenly distributed around the country, and that improving access to academically oriented institutions could result in more evenly distributed participation in higher education.
We welcome your thoughts: [email protected].
Footnotes:
