Unpacking Data
By Lani Evans, Head of Foundation & Sustainability
This article was originally published in the Philanthropy New Zealand magazine. Lani Evans, Head of the Te Rourou, One Aotearoa Foundation asks funders to take a moment and consider how to use data safely, effectively, and critically. Lani explains the difference between qualitative and quantitative, and considers the role of systemic bias, governance, context and narrative.
I am not a data scientist, but I do love data. Data helps us to tell stories and measure change, it helps us to gather insights, see trends and make decisions that prepare us for the future. But data also has a shadow side. It can narrow our thinking about people and places, and it can send us down unhelpful pathways that simplify and strip the nuance from complex and complicated problems.
Philanthropy, right now, appears to be in the midst of a data revolution, one that could fuel a new wave of innovation and social change. But in order to use data in safe and effective ways, we need to understand how to apply a critical lens and have critical conversations about its role within our work. And I’m not sure we’re doing that yet.
So where do we start? What should we be looking for or thinking about when interacting with data? For me there are five initial considerations that can help us look beyond the dashboard: the type of data itself; the potential for bias; governance; the wider context; and the narratives that frame the data and its use.
The first thing to think about is why the data exists, its accuracy and what it is designed to tell us.
The most accessible data tends to be quantitative – data that is mathematically precise and measurable, and shows up in numbers, scales or yes/no answers. This data is often collected without us really thinking about it – like the administrative data that’s produced during our interactions with the Government, or when we use our loyalty card at the supermarket. Quantitative data is super useful, and can be incredibly accurate, but it can over-simplify. This sort of data is “great for observing broad patterns and trends, but can miss nuances that would be obvious to the human eye, and which form an important part of the stories of individuals and communities” (Thea Snow, Nesta).
Qualitative data, by comparison, tends to tell a more layered story, using techniques that uncover people’s emotions, stories and worldviews. However, by its nature, good qualitative data is labour intensive to collect and analyse. As a result, sample sizes for qualitative studies tend to be small, which means that the findings shouldn’t be generalised beyond the research context, especially where populations are diverse.
The second consideration is the role of bias. Systemic bias is present in many commonly used datasets and can result in errors of interpretation at any stage in its life cycle – during collection, or analysis, or conclusion – leading us to incorrect or incomplete outcomes. We need to understand and account for biases when we’re drawing conclusions from data – we can’t simply trust that the data is representative. A lot of surveys, for example, are still completed using landline numbers listed in the phone book. This data collection method introduces bias immediately by excluding much of the youth demographic, and anyone who has unstable access to housing.
Our third consideration is governance. Just as we would complete due diligence on the governance of non-profits we fund, we should also examine the governance structures of both organisations offering us data, and the governance of the datasets themselves. Those collecting, analysing and sharing datasets should have thought about, and be able to answer questions on ethical frameworks, data security and storage, and safeguarding mechanisms.
Fourth is the question of the context that surrounds the information. We need to unpack the broader ecosystem of influences that sit around data – and understand what this means for causation, and correlation of actions with outcomes. Imagine comparing the April 2019 and 2020 traffic infringement statistics without including the broader context (i.e. lockdown). You could make an incredibly compelling, and incredibly inaccurate statement about the efficacy of a speed reduction process. Or, a classic example of the difference between correlation and causation – the observation that when ice cream sales increase, so do drownings. Warmer weather is the lurking variable here.
And finally, there’s the question of narrative. Data often tells us stories of deficits and disadvantage, not because the data is inherently measuring deficit, but because we frame it that way. Educational attainment, for example, is a strengths-based data point until you use it to compare groups of young people. When we frame that data in terms of “good” outcomes (tertiary-level attainment) versus “bad” outcomes (NCEA Level 1) we shape the data from a deficit perspective. Data can be used in a more mana-enhancing way, by looking holistically at a broader range of aspects of a young person’s life.
To use data effectively, we need to examine it, to understand its provenance, its failings and its biases. We need to make active choices about how we govern it, frame it, and the narrative that we use it to create. And this is just the start – there are plenty of other questions to ask, like how do we democratise and de-privilege data? And how do we ensure that data doesn’t reduce our thinking about people and problems to a single story?
“The single story creates stereotypes, and the problem with stereotypes is not that they are untrue, but that they are incomplete. They make one story become the only story.” – Chimamanda Ngozi Adichie
“E koekoe te tūi, e ketekete te kākā, e kūkū te kererū” – the tūi squawks, the kākā chatters, the pigeon coos. They are all birds, but it’s the differences in their songs and stories that make them special. We need to understand if the data we’re using is telling us about birds, or about kererū. And then we need to go and spend time in the forest ourselves, and ask the kererū what it is they really want. If we don’t, we risk further entrenching the inequity that exists within Aotearoa.
