Yes, Players May Find Out They’re in an Experiment – So What?

In nearly every tech domain, from Amazon to airline pricing to Uber, users are in a variant. In fact, on LinkedIn, you may be in one right now. Experimentation isn’t just prevalent, it’s expected! Games seem to fall under a peculiar domain where experimentation is deemed “unfair.” As a result, the whole population suffers because we’re unable to improve an experience that applies to everyone.

There are certainly legitimate critiques of experimental design. For example, the Hawthorne effect is a bias that occurs when participants are aware they are part of an experiment, thereby altering their behavior in a manner they wouldn’t otherwise exhibit. Ironically, this is the problem with nearly all survey-based studies on loot boxes: crazed Redditors pushing an agenda, knowing what the experimenter is looking for. Testing first in non-English speaking regions is a key way to mitigate this and slow its spread onto Reddit.

If fairness is a concern, adopt King’s argument from the @Steve Levitt / John List paper: only run experiments that function as a discount. This is what economists scream at when we test price discrimination. It actually is consumer welfare-enhancing, not inhibiting.

For games that have population shards, the answer is similar. Test between shards rather than within a shard. Not only does this open up the experimentation playbook, but it also mitigates fairness concerns.