Split testing introduction and general information

Split-testing consists of optimization experiments through statistical analysis. Why guess whether a green link text will attract more clicks than your normal color of blue. Start a split-test experiment and investigate it by yourself.

The split-test experiment contains two sample groups. The sample groups are:

A) The control group. This group should receive the normal version of the newsletter, i.e. the blue link text.

B) The experiment group. This group should receive the modified version of the newsletter, i.e. the green link text.

We could just test two different experiments but in order to stick to a fairly scientific approach we should evaluate a control version as well.

The next thing you have to consider is your criteria. The criteria consists of two settings: 'Operator' ('Most' or 'Fewest') and a variable (for example 'Views').

We have to be sure that the improvements we observe in the experiment groups are based on actual improvements and not just random occurrences. Consider this example:

A) 1276 views B) 1277 views

Criteria: Most views

You will never be able to say whether or not the extra view of B was caused by a random occurrence or due to the fact that you changed the color of the link text.

There is several static models that solves this problem dependent on how sure you wish to be about the conclusion. The model I will present below utilizes these two variables:

• Criteria threshold
• Test size

The criteria threshold gives you the possibility of defining your experiment sample group to perform better than the control experiment group with this threshold (measured in percent).

Thus this is the minimal value of improvement you will be able to observe from the experiment.

The sampel test size is the amount of receivers for each of your sample groups.

When the criteria threshold has been chosen you have to decide how certain you wish to be with the outcome.

You may use the table below in order to check our recommended test group sizes for three different confidence levels.

In order to find the total amount of receivers, that are necessary for you experiment, you have to multiply the test group size, in the table below, by 2 as to account for both A and B.

Remember that you ought to save room for the winner version (C). If one chooses a high confidence level, with a low threshold, on a small list, there is no room left for the winner.

The calculations are based upon Cochran's Formula For Calculating A Sample For Proportions: http://edis.ifas.ufl.edu/pdffiles/PD/PD00600.pdf