Overcoming Data Quality Challenges

The demand for online survey takers is at an all-time high, which is a good sign that the market research industry is thriving. However, this demand creates a serious supply issue that makes feasibility challenging, especially for hard-to-reach populations.

When demand for sample is high, sample providers start scraping the bottom of the barrel to get clients as much sample as they can. Fraudsters thrive in this environment, often getting in and out of several surveys and earning their rewards before anyone has a chance to blacklist them. Because of this, having quality check measures in place is even more important when demand for sample is high.

Leaving quality checks up to the sample providers and thinking this will give us flawless sample is simply not realistic. It never has been, and it never will be. Every insights provider should only work with sample companies that deliver quality respondents on a consistent basis, but there are always some fraudsters who will find a way to cheat the system. Despite multiple sample providers claiming they provide better respondents with less cleaning needed, we still see fraudulent responses in every study. Depending on the population, this typically ranges from 5% to 30%.

Similarly, relying on third party fraud prevention solutions is also not good enough. Despite some hefty claims being made by the handful of major fraud prevention solutions, the fact is that they are always at least one step behind the “bad guys.” Even the Insights Association acknowledges that these fraud prevention solutions are not fully reliable saying the technology seems pretty nascent, pretty young, pretty immature at finding actual fraud and at staying current.

Sample providers and third-party solutions are pieces of the puzzle, but don’t completely solve the fraudster issue. There are a couple of very important steps that not everyone takes, because they believe the previously mentioned solutions are “good enough:”

• The first step is designing the questionnaire with multiple quality fails in place. These can include red herring questions, validating an earlier survey response, open-ended response review, among many others. There should always be a minimum of two data points for quality review.
• The second step is to have a process in place to carefully review the data and identify any fraudsters who slipped through the cracks. I would argue that these are the most important steps in the entire research process.

Insights are only as good as the data they are based on. We must keep fighting the battle on fraudsters for the sake of high-quality insights. The best way to do this in today’s world is to use a multi-pronged approach which must include skilled data managers using proven processes for identifying and eliminating bad data. Otherwise, we would find ourselves in a “garbage in, garbage out” situation which doesn’t benefit anyone.

Interested in CMB’s data quality measures? Contact us to learn more and work with us.