We tend to think that vast amounts of data are representative by definition, but that is not necessarily the case. Big data should also be subjected to stress tests. Three experts enlighten us on the Uber’s and Airbnb’s of online access panels, the rise of the robots and the promise of artificial intelligence (AI) in this article extracted from the ESOMAR 2019 Global Market Research report and edited for Research World.
With humans leaving an ever-increasing digital fingerprint of what they think, feel, say and do, computationally analysing increasing volumes of data to reveal key patterns, trends and associations has – undoubtedly – been key to delivering actionable insights at scale over the last few years. This observation, made by Caroline Frankum, Global CEO of Kantar’s Profiles Division, comes with an added warning though. “Following the GDPR’s implementation last May, and as data privacy legislation intensifies – e.g. California Consumer Privacy Act (CCPA) coming into effect in January 2020, and the growing uncertainty of third-party cookies – it is increasingly apparent that it is not the amount of data that’s important, but what organizations can do with data compliantly that matters.”
Even so, it is still often believed that vast amounts of data are representative by definition, notes Andrew Konya, CEO of Remesh, a platform that allows users to get qualitative insights at a quantitative scale to make better decisions. “We do encounter the common misconception that more data – bigger N – means higher confidence in results. However, most researchers seem very aware of how a non-representative sample of participants in quantitative research translates to lower quality results.”
Presidential poll
Pete Doe, Chief Research Officer at clypd, an audience-based sales platform for television advertising, observed earlier that online access panels have enabled cheap survey research to proliferate in the past decade. He now confirms that there is a natural tendency for people to think ‘bigger is better’, even among some professionals with some statistical training. “People are taught that margins of error reduce as sample sizes increase, but they aren’t always taught about biases in measurement. So yes, it is a fairly common misconception, but this is not a new problem.”
Honesty and transparency
Big data keeps revolutionising the insights industry at breakneck speed, almost as if it deliberately ignores the many red flags. But the representativity concerns are very real and need to be tackled. For Doe, the main question that any practical research user has to answer is: ‘Will I make a better decision if I use this data source?’ The answer to that question is not always clear, he stresses.
Some of the concerns about representativity are actually concerns about validity. “Inferring people’s behaviour from device activity is not simple, especially if the device is used by different people at different times.”
As for red flags, Doe thinks that users should be wary of companies that are unwilling or unable to explain their data sources. “No-one with a realistic outlook expects perfection in research, but we should expect honesty and transparency.”
Billions at risk
Whilst online access panels have enabled cheap survey research to proliferate in the past decade, the results obtained from these convenience panels can be biased and unrepresentative due to the sampling methods employed. Several checks and stress tests could be applied in order to diminish these effects.
Doe believes these checks need to reflect the use of the data and the money at risk. For media transaction data (ratings), oversight from Joint Industrial Councils (and the Media Rating Council in the USA) is worth the expenditure to assure quality when billions of dollars of ad spend ride on the data, he emphasises.
“And these services don’t use cheap online survey research data for that very reason. For research with a narrower scope and less money at risk, users should at least expect transparency around data sources, the recency of any classification data, details about the survey, response rates, sample sizes, data editing and projection methods.”
Uber’s and Airbnb’s
In order to illustrate the low-budget online panel trend, Frankum makes a telling comparison: “Tech-enabled entrants offering cheaper online access panels are like the Ubers and Airbnb’s of the panel world – they do not own any panellists but focus on renting opt-in panel assets to leverage their technical assets.”
“Clients are increasingly looking for richer profiles of consumers. This means compliantly matching behavioural data to proprietary profile attributes to create more addressable audiences, before a single survey question has been asked. This is something the cheaper online access panels are not set up for, or accredited to do. So, their data is more limited in how it can be used.”
Increased integration
The use of big data for gaining insights is still very much in development, and will see further changes over the next few years. Doe believes it will continue to grow, and there will be even cheaper options available. “But these will be unlikely to deliver quality findings. There could be attempts to harness AI to synthesize insights, for example by inferring behaviours and attitudes from online data, perhaps harnessing voice activated device data, within privacy constraints.”
The infrastructure and algorithms people refer to as big data will enable increased integration between passively collected data, across multiple channels, and first-person research data, predicts Konya. The result is likely to generate insights which are increasingly likened to behavioural impacts.
Profound shift
Doe stresses that data is not research. “Data is a raw material that needs to be refined before it can be used as part of a research study, and that requires research and statistical expertise.” As having lots of data becomes easier, Konya thinks people will shift their focus from ‘what is the N?’ to ‘how confident are we in this result?’ That shift will likely move the industry away from working to deliver the minimum cost per participant – or data point – to working to deliver the lowest cost per unit confidence.
At Kantar, Frankum and her people are currently learning a lot from working with AI. “AI is also something to keep a close eye on when it comes to making big data more representative for insights and market research in the future.”
This article is an excerpt of the original “Big data and representativity”, published in the 2019 Global Market Research report. Read the full content by accessing the report, here.
