How to implement A / B tests?

Navigation
8

How can I implement A / B testing on a new system?

I am developing a tool and would like to compare the results of each modification, but I have no practical knowledge of how to implement this type of test.

    
asked by anonymous 28.04.2014 / 20:36

2 answers

9

How A / B testing works:

A / B testing requires a mass of data already collected to draw conclusions. Therefore, for a new system it is not possible to conclude anything immediately.

The process consists of showing slightly differentiated versions of the same thing to different users (at random), and checking what action is taken. Not to display the two versions at the same time.

So if you build a table where each record has:

  • The variation identifier displayed.

  • The action identifier taken by the user.

From there, you just have to do an analysis to see if the differentiation has any relevance or not.

Notes:

  • If the differentiation between versions is too large, it is difficult to know what really caused the behavioral difference, ie small differentiations make the result more precise .

  • The higher the mass of data, the more accurate the test will be.

    • The target audience must be well specified , as behavior variation may differ, for example, from one culture to another.

    >
  • It is necessary to set well defined what behaviors are being measured , and obviously how to obtain and record what was the action of the user ... the higher the accuracy of the measurement the better . An example is between clicking and not clicking on an advertisement. To register a non-click it is necessary to do this at the moment the advertisement is displayed, if the user clicks the record it must have its result changed to indicate that the click occurred.

  • You can change the proportion of random views, but the test result will have to be balanced with the same proportions

  • It is possible that the results change with the time of day, or with other seasonal variables, so you need to be careful about that, and determine if there are any external variables, and what they are >

    • 28.04.2014 / 20:53
    2

    The A / B test is used when you have two implementation options that are equally valid. For example: If during the development phase of a web page (e-commerce) your team believes that placing the [Buy] button would look as good next to the Product image as it is close to the Product description. In this case it would be interesting to use the A / B test to know in which of the two situations the best results are obtained. Two versions would be created and the results collected and compared. In this case you could leave the page for the same amount of time and check which resulted in more sales. This test is used when we have two good solutions and we are in doubt which one to adopt. Usually not done for all system modifications. If your team is in doubt about a modification or another, it's best to let users decide through this test.

        
    22.07.2016 / 23:41
    How to destroy a specific session? Best way to use and Instantiate the EntityManagerFactory