Introduction to Switchback Testing

Why Switchback testing

There are lots of situations that it is hard to set the classical AB testing. More precific, it is hard to randomize as two groups to do the experiments.

For example, in social network or message software, if Alice and Bob are in different groups, but have connection, they might find out the differet version between them, and this is called Network effect.

In other example, in two-sided platform, such as Uber or Airbnb, if we want to test the whether the different prices, the two sided users in a group might start to take more resources and push out the people in other groups.

In above cases, we cannot use classical AB testing, since there will be lots of effect make experiment not trustworthy. At this situation, instead of randomizing on users, we can randomize on time slot. The method is called backswicth testing.

How Switchback Test

Basically, we just split the time into several timeslots, and randomly assigned the groups in those timeslots.

How to split the time slot

Actually, there is no optimal answer about this question. Moreover, you need to be careful about the design of time slot. If the time slot too short, it might not capture the effect we want, such as setting 1 second as time slots for borrowing shared bikes; However, if the time slot is too long, it could have carryover effect and the number of time slots could be too less to make the power of experiment enough.

How to randomize the time slot

carryover effect means the user carry the pervious information to make the next decision. For example, if people want to borrow the shared bike again, but they see the price if higher than 5 minutes ago, they could rather decide other kinds of transporation than bikes.

If the experiment has the potential of carryover effect, then you need to optimize the randomization. Here, we define carryover duration, it means how long does the carryover effect happen. For example, when users borrow the bikes, they do not know the price until, maybe, next hour. During this time period, the carryover effect cannot be happened. Moreover, carryover order(m) equals carryover duration divide by time slot length. For example, if carryover duration is 4 hours, and time slot length is 0.5 hour, then $m$ = 8

$T$ means number of total time slots, and $n$ = $T$/$m$.

Suppose you have large carryover effect ($m$>1), and $n$ > 3, then the optimal randomize time slot is:

{1,2m+1,3m+1,...,(n-1)m+1}

which means, when you meet above time slot, you need to re-assign the groups. For example, if $m$ = 2 and $T$ = 12, then the optimal time slot is {1,5,7,9}. when I enter time slot 5, I need to assign different group to next time slot from before.

However, if your experiment has no carryover effect ($m$=0) or small effect ($m$=1), then just arbirary random the time slots.

Reference

https://www.statsig.com/blog/switchback-experiments

https://towardsdatascience.com/how-to-optimize-your-switchback-a-b-test-configuration-791a28bee678/

https://arxiv.org/pdf/2009.00148

https://medium.com/bolt-labs/tips-and-considerations-for-switchback-test-designs-d1bd7c493024

https://careersatdoordash.com/blog/experiment-rigor-for-switchback-experiment-analysis/