I read an interesting article yesterday on the Practical eCommerce website called ‘When to Ignore a 50% Conversion Rate’
It discussed the issue of data validation - meaning that if you launch a new keyword in your pay-per-click campaign, then the first visitor from that keyword goes ahead and converts, then your conversion rate for that keyword is 100%. However, because you’re only looking at the behavior of a single visitor, you don’t have a large enough sample size for that data to be valid.
Initially what struck me about this article was how obvious that was. But as I’ve pondered it a little bit, it occurs to me that many people may be making the mistake of making decision on too few data (although I suspect most marketers don’t make decisions based on a single click).
So, how big does a sample size need to be? Now, if you ask a statistician, I’m sure that you’ll get a different answer, but for the typical marketer I’m assuming that we’re looking for a best guess. Here are a few thoughts that might help your guess be more accurate.
1. If you’re looking at impressions, look for at least 1,000. If you’re looking at clicks, look for at least 200.
2. When comparing conversion rates of two Google ads, I like to calculate what percentage would be expressed by one more conversion. Simply multiply the number of clicks by 0.01 (or multiply by 1%) - the answer is how many percentage points would be represented by a single conversion.
3. Our friends at Vertster also have a little tool that measures the difference between CTR’s and tells you if the sample size is big enough.
I’d be interested in your feedback, and any other techniques you might use for this sort of data validation…
