The Covid pandemic has sped up the existing trend of social enterprises delivering education programmes online. These social enterprises are now able to draw on large amounts of data collected through their platforms to measure and improve the impact of their interventions. For small organisations who are new to online delivery, there is a lot of data to make sense of and harnessing the useful data can be tricky. We asked two very different social enterprises to talk to us about how they use data from their online services to drive impact.
Thrive is a digitally enabled healthcare service that carries out prevention, early detection and management of common mental health conditions. So far, Thrive has guided 30k people to recovery. We were interested in what a larger organisation from a different sector (healthcare) might be able to teach us.
Skills Builder Partnership supports the building of essential skills like problem solving and leadership for students and adults to help them succeed. All of their internal data analysis is done by members of the team who have taught themselves how to code.
Both organisations identified clear advantages in using online data to augment more traditional impact measurement techniques such as surveys:
1. Not inconveniencing the user
Service users want to spend time benefitting from the service, not helping to improve it. An online platform provides an opportunity to capture data as the service is delivered and reduces the need to survey participants. A behavioural measure like knowing how many modules have been completed is also a far more powerful metric than asking people to rate how useful a programme has been. Thrive haa a data team which tracks behavioural data from their online platform to correlate outputs (e.g. completing programmes) with impact targets (e.g. improving mental health).
2. Giving back to the user
Where data collection does require inputs from the user (and therefore a degree of inconvenience), it is important that users gain something from sharing their data. Thrive has put this at the heart of their data analytics work and share all the data they collect to measure their impact with their users. The users appear to appreciate this; Thrive has found that users spend the most time viewing their own progress metrics. Impact is not just a measure for improving your own service or reporting to funders. The service users want to understand and see their own improvements and supporting them to do this could help to deepen their engagement and therefore the impact made.
A picture of Thrive’s user metrics screen
3. Realtime data analytics
Rather than measuring impact annually or monthly, online platforms can measure impact in real-time. This makes it much easier to measure changes to the platform, and AB testing can be used to drive impact. For example, you can test different engagement techniques with different users to see which is most effective in encouraging users to complete your programme.
The range of data acquired means that impact measurement can take into account real life complexity. With so many data points, you can clean data more effectively. Skills Builder is currently trialing new ways of using algorithms to analyse progress on one of their platforms to continually refine the accuracy of impact measurement.
Can you do this without a team of programmers/data scientists?
The problem with processing a huge amount of data is finding what is relevant. Some of it will seem useful or interesting, like the number of people visiting the website, but is unlikely to translate into impact. Organisations need to identify the data points which are steps along an impact journey. Can a small organisation really understand its data without additional resources? Skills Builder is a great example of a team who bootstrapped this by teaching themselves to analyse their own data.
When teachers measure students’ progress against the Skills Builder framework on their Hub, that progress is captured on a database. This is collected in PgAdmin (a free database tool) and processed with Python before feeding back into Salesforce for client management purposes. Python code written by the Skills Builder team automatically generates the impact reporting used internally and the reports that are sent to individual schools. This is a great example of cost effective, automatic and highly effective impact reporting.
Will from Skills Builder shares their code
The Skills Builder example shows how an investment of time can build something really impressive. There can be a hesitancy in getting started with analysing data as it is assumed that complex skills are needed. In reality, the management team of an organisation are likely to understand their business best and know what information they need to extract; they are best placed to process their own data.
Education platforms and impact measurement
Thrive told us that impact measurement sat at the heart of everything they do because that is the standard approach in healthcare. A requirement for proof of impact before engaging with a service is not the standard approach in education, but we think it can be, and the sector is moving in that direction.
Some organisations can afford employees to analyse data to improve impact. Where organisations can’t do this, there is the option to spend some of the team’s time developing a system like Skills Builder’s. This initial investment may be costly in terms of resource and team time, but over the longer term we would expect it to yield dividends. Firstly in saving time compared to more manual impact measurement processes, and secondly through increasing engagement and thereby generating more impact. Finally more knowledge about how users react to the service can help drive further improvements and again lead to more impact.
This blog summarises a panel discussion in one of our Impact Measurement workshops – if you would like to attend workshops like this, please join our Innovation and Investment community here.
Skills Builder has recommended the following sources for upskilling:
- Code Academy (free sessions and some paid for courses)
- Trailhead for Salesforce (free)
- Harvard CS50 on EdX (free)
They also received a Data Analytics apprenticeship funded by UBS.
UBS provides grant funding and volunteering support to Young Academy education innovation team. To learn more about how UBS supports its communities, visit: https://www.ubs.com/global/en/ubs-society.html