Up with the Exchanges
As readers may know, I am building a US consumer survey panel called Survey Diem. What you might not appreciate is how tough it can be to get a panel off the ground. You need to have clients with surveys in order to keep panelists happy, and you need happy panelists to respond to surveys and keep clients happy. While I do have prospective projects lined up, I don’t have an extensive pipeline, let’s say, which is why I was so excited about joining one or more survey sample exchanges.
Survey sample exchanges are online marketplaces where you can buy or sell access to respondents that meet the criteria for the projects in question. You can imagine that if you hang your shingle in an exchange and offer competitive pricing, you can feed on projects like a whale golloping krill. There’s just one problem. Pricing in these marketplaces are brutal.
A friend and colleague recently shared some average pricing data for one of the bigger exchanges and my mouth dropped. The average price for a fifteen-minute survey with a twenty percent incidence rate is below $X. Perhaps I’m naïve, but I believe respondents should get at least $Y for fifteen minutes of their time, and then there’s the issue of me getting something to meet my operating expenses and wanting a profit greater than, say, X cents. The pricing is so low that I have considered doing projects on the exchange at zero profit just to keep panelists engaged.
Of course, what I’ve seen is just average pricing. Some in the exchange are getting, perhaps, $0.5X for the survey described above, and maybe I could get away with $2X considering all of the data quality safeguards Survey Diem has in place.
When it comes to whether research buyers really are willing to pay more for quality data, the exchanges are where the rubber hits the road. People will have to go for the $2X option when there are $0.5X options for the same project. The $0.5X providers aren’t going away.
Brandon Olesh, a strategic insights manager at Stanley Black & Decker, shared an interesting idea on LinkedIn on Friday. He proposed that sample buyers get discounts when a certain percentage of the sample is flagged as fraudulent and pay a premium when the sample is actually clean. I am for anything that recognizes that there is a cost to fighting fraud in survey sample.
At heart in this proposal is the key issue—buyers say they’ll pay for higher quality data, but how exactly are you measuring data quality? What metrics do you use? Brandon floated the percentage of respondents flagged as bots and the percentage of respondents failing in-survey traps as key metrics. I would just add that some fraud is hard to ever catch, so you have to look at certain sine qua non panel attributes. In the case of Survey Diem, every panelist is verified as a US consumer by driver’s license or state-issued identity card. Let’s say that’s worth $X, as a starting point, and then you build price from there, perhaps adding dollars as your quality thresholds are met.
My main point is that the CPI for a quality panel is going to be an outlier in sample exchanges, clearly by quite a bit. The question is, will Survey Diem ever get picked for the dance? I know there are clients out there who will pay for good sample. What I would love to see, however, is the average price for a fifteen-minute, twenty percent incidence survey to go up. Think of it this way—the average prices in the exchange represent the soul of the industry. If nothing changes in the next year, I suspect the industry will have made little progress in resolving its massive data quality problems.
Prices going up wouldn’t necessarily mean quality is improving, but I think it would be a good indicator. I trust that with higher prices, most panels would spend more on quality controls and better panelist incentives. The ones that take this all as profit would be found out eventually. Again, is this naïve? Possibly. Does something need to change? Yes.
Is paying $X for a project instead of $0.5X going to seriously put research buyers out? Of course, it depends on how many projects you do in a year, but if you do a handful of studies, an extra $0.5X probably isn’t going to kill your marketing budget. And now let’s really get real…you are better doing off doing one $X project than two $0.5X projects because when you cut corners with this stuff, you get bad data that is worth less than zero. This is the new math in survey research. Cheap projects aren’t “good enough.” They have negative value.
It's time to support investments in quality, and I am happy to share my journey with you.
Thanks for reading.
-CW
I’m not actually too bothered if the bots & fakers are identified & screened before entering the survey. That’s the direct hit the sample supplier has to pay. What concerns me more are the ones we have to weed out once they’re in there (mainly in the form of bad open end answers). In any case, if we identify them, they are rejected. It’s a pain and we spend considerable time on it, but no sample source is perfect.
In my experience, B2B respondents are much more problematic than consumer. Paying $20-80 per respondent attracts the scammers more than the $2-3 for consumer work.
Perhaps someone’s investment in a side-by-side comparison of data from that 15-minute survey could make the case for data quality. What is data quality? Data that you trust to use to make decisions. Let’s say the $1 sample attracts many more bots and fakers than your $4 sample, but the data itself from the ending sample looks the same as yours. Well, that doesn’t support your case. If the data is different, how would you know which data set is the source of truth? I guess you’d need to sprinkle in some sort of factual questions or something where the outcome is already known in order to validate that your data is “better.” You’re fighting an uphill battle. I used to go to SampleCon, and they’d say how the entire ecosystem needs to change and improve, but nothing changes. And in some respects, they’ve worsened. I sincerely wish you the best in this endeavor.