- Define N as "the number of trials."
For Hammy this is 8+4 = 12.
For the AdWords example this is 32+19 = 51.
- Define D as "half the difference between the 'winner' and the 'loser'."
For Hammy this is (8-4) ÷ 2 = 2.
For AdWords this is (32-19) ÷ 2 = 6.5.
- The test result is statistically significant if D2 is bigger than N.
For Hammy, D2 is 4, which is not bigger than 12, so it is not significant.
For AdWords, D2 is 42.25, which is not bigger than 51, so it is not significant.
Sunday, April 12, 2009
AdWords A/B testing
A/B test shows a statistically significant difference is: