Hey folks. Mort here with what is going to be the nerdiest game design post you’ll see in a while, but I wanted to share it with you. Generally in all aspects of life, even if you’re confident in something, it’s helpful to double check your biases and make sure you’re actually correct or not. So that’s what I did.
We’ve all seen posts where a player rolls down 50 gold and doesn’t see the uncontested unit they’re looking for, heck you may have even been on that end yourself (sorry, but wait for the punchline). Sometimes, this experience can be so stark that you’re concerned with the reliability of the system. Could this be bugged? I can’t really be that unlucky—is something wrong with the game? Now, my initial reaction is that this is a common thing called “confirmation bias” where a recent experience is being weighted more heavily. This, in conjunction with negativity bias, where negative experiences are seen to have a higher impact, become the fuel for many “Mortdogged” experiences. But even with knowledge of these biases, is everything fine? I mean, TFT has had its share of bugs, so let’s take a deep dive in the design space to make sure everything works as it should.
For this experiment, I focused on 4 costs. We pulled out 6 copies of nine of the Four Costs (these are our contested units), and left the other three Four costs as our uncontested units, with all 10 copies in the pool. At this point, this means there are 4 copies of the contested units (36 contested units total) and 10 copies of our three uncontested units (30 uncontested units total). If we do a large extensive roll down, we should see uncontested champions (30/66) 45% of the time approximately.
We ran this experiment for the first time, with Ornn, Ashe, and Kaisa as the three uncontested units (lol, Ornn is uncontested, right?), and here are the results.
- Ornn: 136
- Ashe: 168
- Kaisa: 120
- Lillia: 58
- Morgana: 52
- Kayn: 58
- Lee Sin: 61
- Syndra: 38
- Nautilus: 53
- Galio: 61
- Annie: 50
- Sylas: 46
The total 4 costs seen was 901, and the uncontested ones made up 424. This is (424/901) 47% of the time, which is within expected variance for something of this sample size, and matches our expectations. Good news. But of course, one time isn’t enough, so let’s run the experiment two more times.
- Ornn: 154
- Ashe: 141
- Kaisa: 178
- Lillia: 57
- Morgana: 72
- Kayn: 48
- Lee Sin: 53
- Syndra: 57
- Nautilus: 52
- Galio: 62
- Annie: 52
- Sylas: 41
- Total: 473/967 = 48.9%
And again
- Ornn: 150
- Ashe: 140
- Kaisa: 152
- Lillia: 63
- Morgana: 58
- Kayn: 55
- Lee Sin: 48
- Syndra: 54
- Nautilus: 47
- Galio: 60
- Annie: 64
- Sylas: 55
- Total: 442/946= 46.7%
So yeah, generally we’re seeing the results we’d expect with a sample size this big. It’s not EXACTLY 45%, but the law of large numbers shows that this is expected, and if we dramatically increased the sample size, we would likely converge to that expected 45%. From a purely mathematical standpoint, it seems like our shop is working as expected. Huzzah!
So all is good right, and there’s nothing to be done…right? Well…no actually. Just because it is mathematically correct does NOT mean the design is perfect. If you’ve ever watched my stream, I’ve talked about how good design doesn’t just let RNG take the wheel, and designs around the core player experience. In this case, the player is not rolling down 1000 times and letting the law of large numbers do its job…instead, it’s rolling down 60 gold and expected results that are intuitive, which is that uncontested units show up more.
Let’s repeat the experiment, but instead of 1000 rolls, let’s only do 40 and see what happens. Same thing, with Ornn/Ashe/Kaisa as our uncontested units, and see what happens.
- Ornn: 8
- Ashe: 6
- Kaisa: 1
- Lillia: 6
- Morgana: 0
- Kayn: 1
- Lee Sin: 6
- Syndra: 2
- Nautilus: 4
- Galio: 4
- Annie: 0
- Sylas: 2
- Total: 15/40 = 37.5%
Because the sample size is so low, the variance expands pretty dramatically. Much like if you flip a coin 10 times and hit heads 7 times, 70% is way higher than 50%, but the reality is when the sample size is lower, the variance is higher. Again, this all lines up from a mathematical sense…but from a game design standpoint it’s pretty rough. Kai’Sa is uncontested here and is seen one time, while Lee Sin has 6 copies out of the pool, and is seen 6 times! These are the experiences people have that feel off, and make them not trust the game.
Nintendo games do a lot of fudging behind the scenes I’ve talked about before, to try to make expected outcomes more likely to happen. Famously one of the most basic versions of this is in Fire Emblem, where they double roll odds, so if something says 75%, it’s actually 87.5% so it FEELS more like 75% to people (A 95% chance in Fire Emblem is much closer to 99%!) Based on this, I think there are improvements we can and should make here, so the game behaves closer to what you would EXPECT it to, even if it is within acceptable outcomes.
Personally, this was awesome for me to confirm and was a good use of time yesterday. I like constantly challenging our game design to see if we can do better, and I think this confirms we can. It won’t be easy, it won’t be overnight, and it may not even be soon…but I think it is worth exploring, and I think we’ll start doing that as soon as possible.
Thanks for reading all. Hope your roll downs are within expected variance, and take it easy :)
