We conducted an A/B test of paywalls using manual traffic distribution for our client from the dating vertical. Based on the results, we used the best performing subscription screen.
Some time later, we fed historic data to the automatic distribution system — to see how the algorithm would distribute traffic.
Results: -Perfect option: 1,070 (100%). This is the number of purchases we could have generated if we guessed the best option at the very beginning and applied it without further testing.
-Historic: 745 purchases (69%). This is the number of purchases we generated with manual distribution.
-Thompson sampling: 935 (87%). This is the number of purchases that the automatic distribution algorithm could have delivered.
It turns out that the algorithm would have handled the task better. According to the test result, the option that we picked out during manual distribution, would be recognized the best by the automatic algorithm. But for the same money that was actually spent, the client could have generated more purchases at a lower CAC. In monetary terms, the project missed some $7,500 on the focus group for 5 days.
How we lost $7,500 on mobile app A/B tests but learned how to conduct them