The use of online maths learning resources during the COVID-19 lockdown

Hannah Owen (Nesta) and Timo Hannay (SchoolDash)

1st October 2020

Summary

We have analysed the usage of online maths platforms before and during the lockdown in England that began in March 2020. We find that:

  • Usage during lockdown increased considerably – typically between 2 and 7 times baseline levels, depending on the product and precise usage metric.
  • Before lockdown, it was common (though not universal) for these products to be used disproportionately by schools with fewer poor pupils, located in more affluent areas or with higher Ofsted ratings. To some degree, these imbalances were reduced, or even reversed, during lockdown.
  • However, these previously under-represented types of school often showed lower levels of student engagement during lockdown. As a result, gaps in student activity (as opposed to teacher activity), frequently widened.
  • Furthermore, students at these relatively disadvantaged schools were more likely to access these online learning services using phones (or sometimes other mobile devices) rather than computers (laptop or desktop).

 

Introduction

Nesta and SchoolDash, in collaboration with edtech industry partners, have analysed usage data for a group of online maths learning resources during the period from late February to early June 2020 – just before and during the COVID-19 lockdown. (School closures were announced on 18th March and came into effect from 23rd March, but some preparations in anticipation are evident as early as the week of 9th March.)

Four different products have been analysed. In order to respect confidentiality they are not named in this report, but have the following broad characteristics:
  • Product 1: Popular among primary schools
  • Product 2: A consumer-oriented service popular with individuals and families
  • Product 3: Used across all phases: primary, all-through and secondary
  • Product 4: A well-established service used mainly by secondary schools

A word about links in the text that follows: Purple links take you to other parts of the page, or to other web pages. Red links control the figures to show you what we're talking about.

 

Coverage

Across all three of the services that are sold to schools (Products 1, 3 and 4), 19% of mainstream primary and secondary schools in England made use of at least one of them during the period analysed. Because these schools are, on average, larger than most, they represent some 29% of all English schoolchildren.

Figure 1 shows how this sample breaks down by various school types. The biggest differences are by phase, with about 13% of primary schools and 42% of secondary schools included. There are also significant regional variations, as well as differences by Ofsted rating and academy status. Importantly for what follows, the differences by school and local deprivation levels are modest.

(Use the menus below to explore the coverage of either schools or students across these different groups.)

Figure 1: Coverage by school type
Notes: School deprivation figures based on students' eligibility for free school meals, with bands defined by the DfE. Local deprivation figures based on the mean IDACI of postcodes within a 4km radius of each school, with schools then divided into three roughly equally sized groups. Large multi-academy trusts means more than 10 schools. For EAL, low means less than 4% of students, while high means at least 15% of students. Urban, suburban and rural groups use ONS rural-urban categories applied to school postcodes.
Sources: Vendor data; SchoolDash analysis

 

Changes in usage

All of the services analysed showed substantially increased use during lockdown. Figure 2 shows how this developed over time, with each product's activity normalised to 100 during the weeks commencing 24th February and 2nd March – just after the half-term holiday, but before preparations for lockdown began in earnest.

Products 1 and 3 showed similar timecourses, with activity rising 4- to 7-fold during March, then dipping during the Easter holiday and rising again in the summer term. (Note that in both cases these were driven in part by special promotional campaigns, including periods of free usage.)

Product 4 showed a similar profile, but with more modest gains of roughly twofold. This is probably explained by the fact that it is a more established product and was not subject to special promotional campaigns during lockdown.

Product 2 also displayed increased usage, but consistent with its positioning as a consumer-facing service, this showed little correlation with school term dates.

(Click on the legend to turn individual lines on or off; double-click to display one line on its own. Hover over points to see corresponding data.)

Figure 2: Relative weekly usage
Note: Weeks shown start each Monday, though weekly data for Product 4 actually started on Sundays.
Sources: Vendor data; SchoolDash analysis

 

Primary schools

Use of these online services – as well as the increases seen during lockdown – were not evenly distributed among schools. Figure 3 shows relative levels of Product 1 and Product 3 usage by different types of primary school. The measure used here takes into account the different numbers of students in each group of schools. Furthermore, the national mean is set to zero, so groups with positive scores are disproportionately active users while those with negative scores are making less-than-average use of the service. The scale is a percentage, so a value of, say, +10 means that activity is 10% higher than might be expected if usage per student was uniform across all schools.

Looking first at Product 1 teacher activity, it is striking that usage among high-deprivation schools was very low before lockdown (purple columns), but caught up somewhat during lockdown (orange columns). A similar trend is evident when looking at local deprivation levels and Ofsted rating. The usage gap between state and independent schools also declined.

However, Product 1 student activity tells a different story. On this measure, more deprived schools actually fell further behind during lockdown. There was a similar pattern by local deprivation level, while trends by Ofsted rating were somewhat less clear-cut.

The story for Product 3 is arguably a tiny bit rosier. Teacher activity for more deprived schools showed a narrowing gap with other schools during lockdown. The picture for student activity is more mixed, with high-deprivation schools narrowing the gap, but medium-deprivation schools falling further behind and low-deprivation schools pulling further ahead.

(Use the menus to explore teacher and student activity across these two products and different school groups.)

Figure 3: Relative activity by primary school type
Notes: School deprivation figures based on students' eligibility for free school meals, with bands defined by the DfE. Local deprivation figures based on the mean IDACI of postcodes within a 2km radius of each school, with schools then divided into three roughly equally sized groups. Small schools have fewer than 200 students, large ones have more than 320. Large multi-academy trusts means more than 10 schools. For EAL, low means less than 4% of students, while high means at least 15% of students. Urban, suburban and rural groups use ONS rural-urban categories applied to school postcodes.
Sources: Vendor data; SchoolDash analysis

 

Secondary schools

What of secondary schools? Figure 4 shows relative usage of Products 3 and 4 among different types of secondary school. Product 3 teacher activity showed no clear pattern before lockdown (purple columns) in relation to school deprivation, but after lockdown (orange columns), schools with higher deprivation tended to show lower relative usage. The same was true – only more so – for Product 3 student activity.

In contrast, Product 4 teacher activity was actually higher pre-lockdown among schools with high levels of deprivation and this gap was largely sustained during lockdown. This appears reassuring until we see that Product 4 student activity was not similarly high. In other words, the schools seem to be engaged, but the students much less so.

(Use the menus to explore teacher and student activity across these two products and different school groups.)

Figure 4: Relative activity by secondary school type
Notes: School deprivation figures based on students' eligibility for free school meals, with bands defined by the DfE. Local deprivation figures based on the mean IDACI of postcodes within a 4km radius of each school, with schools then divided into three roughly equally sized groups. Small schools have fewer than 700 students, large ones have more than 1200. Large multi-academy trusts means more than 10 schools. For EAL, low means less than 4% of students, while high means at least 15% of students. Urban, suburban and rural groups use ONS rural-urban categories applied to school postcodes.
Sources: Vendor data; SchoolDash analysis

 

Student activity relative to teacher activity

Figure 5 shows the change in the ratio of student to teacher sessions for Products 1, 3 and 4. This can be taken as a rough indicator of relative student engagement.

Perhaps because they are more recently introduced services and that attracted lots of new schools, Products 1 (purple line) and 3 (yellow line) showed a steep declines in mid-March as preparations for lockdown got underway (because , although student activity increased, teacher activity went up by even more). They continued at this lower level for the rest of the period covered (though Product 3 showed a brief peak during the half-term break in late May). In contrast, Product 4 (orange line) showed a much smaller initial reduction that soon recovered, but then declined gradually between late March and the end of May.

(Click on the legend to turn individual lines on or off; double-click to display one line on its own. Hover over points to see corresponding data.)

Figure 5: Ratio of student sessions to teacher sessions by week
Sources: Vendor data; SchoolDash analysis

As might be expect from the results already shown in Figure 3, the reductions in the student:teacher activity ratios for Products 1 and 3 were not evenly distributed among primary schools, as shown in Figure 6.

For Product 1, there was relatively little difference by school deprivation level before lockdown (purple columns), but during lockdown (orange columns) a pattern emerged in which students at more deprived schools were less engaged. There were also corresponding trends by local deprivation, Ofsted rating and region. Independent schools had lower ratios than state schools, both before and during lockdown, but this is presumably because they tend to have fewer pupils per teacher.

Sample sizes for Product 3 were smaller, particularly among primary schools and before lockdown, making patterns here harder to discern. For example, while there appeared to be a similar trend by local deprivation level, at least during lockdown, there was no obvious pattern by school deprivation level.

Figure 6: Pupil sessions : teacher sessions by primary school type
Notes: The ratios shown here are slightly higher than those in Figure 5 because non-primary schools have been filtered out. See also notes for Figure 3.
Sources: Vendor data; SchoolDash analysis

Figure 7 shows a similar analysis for Products 3 and 4 in secondary schools.

For Product 4, the school deprivation gap was already present beforehand (purple columns), but seemed to grow during the course of lockdown (orange columns). There were corresponding trends by local deprivation, Ofsted rating and region, among other factors. As seen above for primary schools, independent secondary schools had lower ratios than their state counterpart, but perhaps only because they have fewer pupils per teacher.

Product 3 is harder to interpret because the sample sizes are somewhat smaller, but it showed broadly similar patterns by school deprivation, local deprivation and Ofsted rating.

(Use the menus above Figures 6 and 7 to explore these and other school groups.)

Figure 7: Pupil sessions : teacher sessions by secondary school type
Notes: See notes for Figure 3.
Sources: Vendor data; SchoolDash analysis

 

Device types

Another important question concerning online learning is the kind of device used, with computers usually considered preferable to mobile devices, especially phones. Figure 8 shows the proportion of activity accounted for by computer users for all four products, and how this changed during the course of lockdown.

Product 1 (used mainly by primary schools) showed a decline during lockdown from about 60% to 50%, along with further slight dips during the school holidays. Product 2 (used mostly by individuals and families) showed the highest levels overall, though with a sustained dip centred on the Easter holiday. Product 3 (used by primary and secondary schools) showed a similar overall pattern to Product 1, but with computer usage levels about 20 percentage points higher. Computer usage for Product 4 (in widespread use among secondary schools) actually increased during lockdown, perhaps reflecting the fact that its students are older and, on average, had readier access to technology. (To display all the lines again, click here.)

(Click on the legend to turn individual lines on or off; double-click to display one line on its own. Hover over points to see corresponding data.)

Figure 8: Proportion of activity accounted for by computers
Sources: Vendor data; SchoolDash analysis

Once again, however, these characteristics and changes were not evenly distributed among schools. Figure 9 shows the proportion of primary-school usage accounted for by computers for Products 1 and 3.

For Product 1, there was a clear tendency for the most deprived schools, and for those in northern and midland regions, to show lower levels of computer usage. It was also lower among state schools than independent schools.

Product 3 showed similar trends with respect to school deprivation. However, in this case the usual north-south disparity between regions was evident only during lockdown (during the baseline period, computer usage tended to be higher in the north and midlands, though the samples here are relatively small so may not be representative). Among state schools, computer usage fell slightly during lockdown while among independent schools it rose sharply, albeit from a lower base.

Figure 9: Proportion of activity involving computers, by primary school type
Notes: See notes for Figure 3.
Sources: Vendor data; SchoolDash analysis

Figure 10 shows the proportion of secondary-school usage accounted for by computers for Products 3 and 4.

As we saw in Figure 8, computer usage of Product 4 actually grew during lockdown. But even these changes only seemed to increase the deprivation gap. Computer usage at low-deprivation schools rose by more than that in high-deprivation schools, increasing the pre-existing disparity. There were similar trends by local deprivation, Ofsted rating. Computer usage was also consistently higher in rural areas, southern regions and independent schools.

Product 3 showed an overall reduction in computer usage during lockdown and the picture by school type is more nuanced. During lockdown (but not beforehand), southern regions showed higher computer usage, as did schools with higher Ofsted ratings. But there was no clear trend by school deprivation or local deprivation. Use of computers changed hardly at all among independent schools, but fell among state schools (probably at least in part because of changes in the types of state schools using this product).

(Use the menus above Figures 9 and 10 to explore the different products and school groups.)

Figure 10: Proportion of activity involving computers, by secondary school type
Notes: See notes for Figure 3.
Sources: Vendor data; SchoolDash analysis

 

Concluding remarks

The COVID-19 pandemic has created massive social disruption and forced the widespread adoption of virtual approaches, not least in education. The analysis presented here suggests that this has served to reduce previous disparities in the use of online educational resources, but mainly at the school level. At the student level, our results suggest continuing and even widening gaps in terms of engagement and access to technology.

On a more positive note, the same technologies also allow us to conduct this kind of analysis, and even to monitor educational activities and trends in real time. Never have schools been under greater pressure – but neither have we had such powerful tools as we possess today to understand and improve them.