For quite a while, I was excited about the new videogame Cyberpunk 2077 being released. It looked to be quite an amazing game. When it was released, although it had many great aspects the game also had a lot of bugs and defects. One bug, in particular, made the rounds on social media, as video capture. It shows the main character walking around the city streets and coming across an NPC (a computer-controlled character) sitting in a wheelchair. If something exciting happens nearby, such as a fistfight or whatever, the person in the wheelchair stands up and runs away. This breaks the suspension-of-disbelief and immersion of that moment in the game. And this scene, of course, was the subject of much ridicule.

From a software engineering perspective, this scene was fascinating to me. What I suspect may have happened is, over time as the game was being developed, the wheelchair item was introduced into the game with its graphical assets, and was set as a sittable object type, similar to a bench or chair. Makes perfect sense, right? It’s an object in the game world that you can sit on. But then the emergent behavior of this object is that randomly generated NPCs then appear, and sit on this sittable object. But they are not programmed to be unable to walk, so they can just…get up and walk away from the wheelchair.

From a unit testing or even system testing perspective everything is working exactly as designed. The wheelchair appears in the game world and is sittable. NPCs can sit in the wheelchair and react to external events around them. All of the requirements and specifications of the game engine and system are verified and work correctly. But the actual user experience, which is not something that can necessarily be automated, is non-ideal. The requirements and specifications were not validated properly.

Validating your software, as opposed to verifying it, can be challenging. It is difficult to write a unit test or a system test for requirements such as “this piece of software must be useful” or “this piece of software must be fun”. I remember one time I was a software architect for a large and complex enterprise system with a large userbase and a complicated user interface. The software was having severe adoption issues. It was passing all the unit and system tests, being released, and then being just vilified by the users who claimed it was a buggy piece of junk. The software team was so confused…all of the unit tests were passing. Why was the software so poorly accepted? I remember digging in a little bit to all this and being surprised when many of the software developers had never even seen the finished product, and for that matter didn’t even really understand what the software did. They just wrote C# code to the requirements and user stories as directed. In retrospect, why was anyone surprised that the software was not well-received by the users? The software was verified but never validated.

Going back to Cyberpunk, maybe the answer is that the wheelchair is not a separate object, but part of the NPC object itself. But if you really think about it, that’s a LOT more work than just dropping a wheelchair in and flagging it as sittable.

These are valuable case studies for a software developer or any product engineer for that matter. You can’t just take a programmer and have them write code to meet unit tests and expect the software to truly excel. You have to spend the time and energy to understand your users, understand the real-world use cases, and ideally have that tight integration and feedback cycle with end-user representatives. Otherwise, you might end up with a software product that passes 100% of the unit tests but is worthless to the actual users.