2 min read

On Bugs and Customer Satisfaction

Working on the simulation has prompted me to reflect on the intricacies of feedback cycles. In previous weeks I was focused on programmers, testers, and bugs. More recently I’ve been working on adding the concept of deployments, customers, and customer satisfaction.

This begs the question: what is the relationship between the internal and external feedback loops? How should the simulation model the connection between bugs, delivered value, and customer satisfaction?

If a bug represents a discrepancy between expectations and experience, external users and internal stakeholders inevitably have different expectations. The way customers perceive a product is different from the way internal folks perceive it. What we count as a “bug” internally might not even register for a user. On the flip side, things we don’t think are important could turn out to be critical problems.

Consider my real-world experience on a project at a fast-paced Silicon Valley startup in the late 90s. We ended up with a ratio of 2 testers for every programmer and found over 1000 bugs in the last 6 months of a year-long project. At the time, I thought the project was an utter disaster. It certainly was chaotic. (It was one of the examples I drew from when writing a paper with Cem Kaner and Jennifer Smith-Brock on the ratio of testers to developers.) Years later I had the chance to talk with an actual user. He gushed about how much he loved that software and that that release in particular was the best one ever.

At another fast-paced Silicon Valley startup, we had worked hard to improve our internal testing. So we were shocked and disappointed when we had a particularly disastrous release. The root cause boiled down to the fact that our internal models of usage patterns did not match the way customers actually experienced the product. A release that worked fine in lab conditions failed abysmally out in the real world. Customers flooded our tech support line with complaints.

As I design this simulation I am attempting to abstract away as many details as possible while preserving the essential tensions and tradeoffs that make software development challenging. What I've come to realize over the last couple of weeks is that internal and external feedback loops need to be two distinct, independent things.  They need to operate just differently enough that there is a possibility of software passing all the internal tests yet still disappointing customers, and vice versa.

I suspect I will be contemplating the relationship between internal and external feedback loops for a while. In the meantime I’ll be working on tuning the number and size of bugs generated based on the software reliability. (As of right now, bugs generate at an astounding and almost comical rate.)

Stay Curious,