A/B tests are extremely common, and their general concept is simple. However, as I’ve used this method more and more, things get more and more complicated. I wrote this article to talk about two mistakes I made along the way when running A/B tests.
Mistake 1. I didn’t estimate a test’s duration.
Reality: Estimating a test’s duration helps you use resources better
I still remember the very first AB test that I ran. It got a lot of attention from my company, and I often got this question “how long should we run this test for”. For a while, my answer was the same, “Till we get a significant result”. It makes sense right? But not knowing how long each test might take actually leads to poor timeline planning. You might end up running multiple tests at the same time. It’s not a good practice to do that because too many tests would just cause confusion.
With more experience, I learned to use some statistical methods to conclude the A/B testing duration. This allows me to say that there is 95% (or 85%, depends on what you choose) probability of getting statistically significant result after X number of days. I normally use the AB test duration calculator from VWO.
This calculator is a google sheet, and requires 4 pieces of input to get the result.
- The current conversion rate
- How much increment on conversion rate would you expect to see
- Number of variations to tests
- average number of daily visitors
With this estimation calculator, I was able to have a better understanding about the timeline. This helped me arrange all the AB tests and dev resources so much more effectively.
I also implemented this process into our AB testing flow so that I would push myself harder to think in advance. I highly recommend everyone make use of this practice and always compare the actual results vs. the predicted results.
Mistake 2. I waited too long for data.
Reality: For a startup, waiting for data = moving slow. Moving slow = Anti-growth.
I have seen many AB tests where the estimation of testing duration is over than 1 month. What? “1 month” is more like “1 year” for a startup that wants to grow fast. At the end of day, you can never have enough data to make decisions with 100% confidence. Lack of data is always a problem for startups. Does that mean data is useless? Heck NO. I believe growth is a combination of data and art. Data can give you some level of insights, but you have to understand how significant your data is. When it comes to decision making, you should be really clear on what drives that decision
Most people love data because you can almost never make wrong decisions when you have enough data. However, by the time you get all the data, your competitors might already move one step further. The key point is to use some data and take actions quickly and thoughtfully. It’s not easy, but that’s why it’s fun!