Split testing introduction and general information

Posted by Jeff Rasmussen. Last updated: Fri, Oct 11, 2013

Split-testing consists of optimization experiments through statistical analysis. Why guess whether a green link text will attract more clicks than your normal color of blue. Start a split-test experiment and investigate it by yourself.

The split-test experiment contains two sample groups. The sample groups are:

A) The control group. This group should receive the normal version of the newsletter, i.e. the blue link text.

B) The experiment group. This group should receive the modified version of the newsletter, i.e. the green link text.

We could just test two different experiments but in order to stick to a fairly scientific approach we should evaluate a control version as well.

The next thing you have to consider is your criteria. The criteria consists of two settings: 'Operator' ('Most' or 'Fewest') and a variable (for example 'Views').

We have to be sure that the improvements we observe in the experiment groups are based on actual improvements and not just random occurrences. Consider this example:

A) 1276 views B) 1277 views

Criteria: Most views

You will never be able to say whether or not the extra view of B was caused by a random occurrence or due to the fact that you changed the color of the link text.

There is several static models that solves this problem dependent on how sure you wish to be about the conclusion. The model I will present below utilizes these two variables:

  • Criteria threshold
  • Test size

The criteria threshold gives you the possibility of defining your experiment sample group to perform better than the control experiment group with this threshold (measured in percent).

Thus this is the minimal value of improvement you will be able to observe from the experiment.

The sampel test size is the amount of receivers for each of your sample groups.

When the criteria threshold has been chosen you have to decide how certain you wish to be with the outcome.

You may use the table below in order to check our recommended test group sizes for three different confidence levels.

In order to find the total amount of receivers, that are necessary for you experiment, you have to multiply the test group size, in the table below, by 2 as to account for both A and B.

Remember that you ought to save room for the winner version (C). If one chooses a high confidence level, with a low threshold, on a small list, there is no room left for the winner.

Confidence level 90%
Threshold Recipients on list (population)
500 1000 2000 5000 10000 100000 500000
1 466 872 1544 2876 4036 6337 6675
2 387 629 917 1264 1447 1664 1686
3 301 430 547 654 700 747 751
4 230 298 350 390 406 422 423
5 176 214 239 257 264 270 271
10 60 64 66 67 68 68 68
15 29 30 30 30 30 31 31
20 17 17 17 17 17 17 17
25 11 11 11 11 11 11 11
30 8 8 8 8 8 8 8
Confidence level 95%
Threshold Recipients on list (population)
500 1000 2000 5000 10000 100000 500000
1 476 906 1656 3289 4900 8763 9424
2 414 707 1092 1623 1937 2345 2390
3 341 517 697 880 965 1056 1065
4 274 376 462 537 567 597 600
5 218 278 323 357 370 383 384
10 81 88 92 95 96 96 97
15 40 41 42 43 43 43 43
20 23 24 24 24 24 25 25
25 15 16 16 16 16 16 16
30 11 11 11 11 11 11 11
Confidence level 99%
Threshold Recipients on list (population)
500 1000 2000 5000 10000 100000 500000
1 486 944 1786 3845 6247 14267 16106
2 447 807 1351 2272 2939 3995 4126
3 394 650 962 1351 1561 1816 1843
4 338 511 685 862 943 1030 1038
5 286 400 500 588 625 662 665
10 126 143 154 162 164 167 167
15 65 69 72 73 74 74 74
20 39 40 41 42 42 42 42
25 26 26 27 27 27 27 27
30 18 19 19 19 19 19 19

The calculations are based upon Cochran's Formula For Calculating A Sample For Proportions: http://edis.ifas.ufl.edu/pdffiles/PD/PD00600.pdf


What next?

Read more articles on these subjects: Split test, Segmentation

Keywords

See other articles with these keywords:


Languages

See this article in:


About the author

Jeff Rasmussen
Communications Officer

Adept at strategic communication, especially external. Handles the Ubivox LinkedIn profile and marketing campaigns.


Further questions

Do not hesitate to contact us if you have any questions.

Start livechat
Write e-mail