A couple weeks ago I wrote about how the first scenario for the software development simulation turned out to be boring. Very, very boring.
This week my friend Davis Frank and I got together (in person! yay vaccines!) to work on finding the fun within the mechanics of the game. Our plan was to adapt the simulation into a board game with the hope that a change in medium would help us explore dimensions of engagement. We covered his dining room table with cards, dice, chips, and other game pieces. Then we both just stared at the stacks of game paraphernalia, unsure where to start.
Davis noted that the boring version of the programmer-tester scenario didn’t have a goal for the player. The level is over when there’s no more work to do, but that’s not exactly a goal. It’s an end state. So we added a player goal: deliver a release with no known bugs within a time limit. In the context of a board game, that translated to game rounds. Maybe 7? Sure, 7 sounded like a good number.
We each took a character: Davis played the programmer; I played the tester. We rolled dice and moved chips representing features and bugs around our mocked up board, making rules up on the fly. It was like a game of Calvinball. When we shipped enough features within the allotted number of turns, we cheered. It was already more fun. But something was missing.
We were just moving chips around the board according to a roll of the dice. There were no decision points. Our cheers of celebration or groans of defeat were entirely in the hands of fate. We did not have any opportunities to make choices. Without choice, there could be no strategy.
What if there were an outcome from the dice roll that put a decision in the hands of the player? We adjusted the rules so there were three possibilities instead of two. A die roll could make a feature move forward, backward, or player’s choice. For the programmer, that third outcome was analogous to when a feature is close enough that it could be pushed into test (perhaps by cutting corners). For the tester, it was analogous to having indeterminate test results: there’s anomalous behavior, and it might be a bug, but it might not.
We played again. Choice helped, but there was still something missing. We realized that even though we had choice, it was an illusion. If the only goal in the game was to ship software, then there was only one rational choice to make on indeterminate outcomes: move forward. We needed a reason to make a different choice. There had to be a tradeoff.
The real world tradeoff for pushing forward a feature that is not quite ready is increased risk. However the game mechanic does not yet have a customer who would complain about low quality. So risk was difficult to model.
What if the tester had a bug quota? They could only win if both the software shipped AND they had filed enough bugs to meet their performance objective. That’s a very real tradeoff.
We now had three core elements in the scenario that made it more engaging:
- A goal representing an achievable but not guaranteed outcome.
- Multiple decision points where the player could make a choice and develop a strategy.
- A tradeoff where a given choice would only optimize for one dimension of success.
The resulting scenario is perhaps not ideal in that it involves a dysfunctional dynamic. The tester has to file enough bugs to satisfy management, but the more bugs they file the less likely the release will be done on time: a no-win situation. But as we played the game, we realized that the tradeoff felt all too realistic. I had seen it time and again in the real world in the 1990s and early 2000s with independent QA groups measured on defect detection efficiency.
Fortunately I think that happens less frequently now. However I fear that there are still organizations that measure tester effectiveness by counting bugs. If the larger goal of the simulation is to help folks learn how to make better decisions, this scenario should certainly help leaders understand why measuring testers on bug counts is a terrible idea.
I threw together the level with a text-based interface so Davis and I could play the digital version of the game. It’s not quite what I would call “fun” and I don’t recommend it as a party game. But it’s a HUGE improvement. It also gave us some knobs to tune: number of game rounds, probabilities for the three possible outcomes, the tester's quota, and the size of the release.
If you’d like to see the result, I made a video. In addition to showing Davis and me playing the scenario, the video also explains more about the underlying simulation mechanics and how running the game in batch mode enabled me to experiment with different permutations of the tuning parameters.
Next up: implementing the programmer-tester scenario from the developer’s perspective. Also, adding customers or end users into the simulation so there is a definitive source of feedback on quality and real consequences for choices that increase or decrease risk.
In the meantime if there’s enough interest in the programmer-tester scenario as currently implemented, I’ll put a web-based front end on it like I did for the Guess the Pattern game. Please let me know! Alternatively if you’d be interested in a different scenario, I’d love your suggestions. You can reach out in the comments on the video or send me an email at firstname.lastname@example.org.