One of the things that we’ve seen in the last two years as the global economy has shifted so rapidly is the importance of supply chains. One excellent example, and I have no idea if this is real or not, is that I’ve heard that there was a turkey shortage for Thanksgiving. Thinking about this, how can there be a turkey shortage? Are there plenty of turkeys at the farms if you go and check? So why is there a turkey shortage?
The above story makes for a fascinating thought experiment. Let’s walk through it. There are a great many things that have to happen for a turkey living on a farm to end up wrapped up in a refrigerated case at your local grocery store for you to buy. The turkeys must be slaughtered appropriately, packaged, cleaned, and frozen. The turkeys, and resultant turkey meat, have to be transported multiple times to multiple places. But wait, there’s more! You have to have suitable packaging with the proper sell-by dates and weights and the correct vendor branding and artwork. The turkeys all have to be sent to the applicable locations, warehouses, and stores all across the land. There are probably many different sorts of health inspections and quality inspections that have to happen.
And each of those individual items comes with its complexities. For example, I saw a news article recently that talked about pizza boxes. There was a local pizza chain with maybe 15 or 20 locations. This chain placed their quarterly order of pizza boxes from their supplier, with their unique branding and artwork. The supplier replied that since they were shorthanded, they would be unable to deliver the pizza boxes order in time. So the pizza company had a choice of either getting a smaller number of boxes with the right branding, or instead getting generic boxes without any branding at all. So now all of a sudden the pizza company has a challenge on how to deliver and sell. There might be plenty of demand, and the chain has the capacity to bake enough pizzas in the store. But because of one item in their logistics train, the pizza boxes, the supply chain falls apart, and suddenly there is an impact on this company’s ability to deliver.
The point of this story is to understand the logistics train of getting a turkey into the grocery store for you to buy. If any one of those things breaks down, it gets in the way of getting that turkey from the farm to your door. All these external dependencies and integrations happen that are probably beyond the scope of what you initially thought.
The same thing is true with software and with code. I think many software developers fall into the trap of only seeing what’s right in front of them. They blissfully write their code to run on a little mini server running on a container on their laptop. They’ve written unit tests, and done some quick manual verification, and everything looks fine and so they think everything is great.
Some development teams have pipelines and development environments already built out, and this can help. They’ll have a development environment where they can deploy the code with automated pipelines and run automated system testing. This can definitely lead to higher quality and a higher probability of avoiding issues because now you’ve made sure the code actually can deploy into a real environment.
One time I had an experience with a project that had major external dependency issues. The development environment did not have the same permissions boundaries and guardrails as the pre-production environment did. Everything worked fine in dev, but when the team deployed to the pre-prod environment, suddenly the pipelines didn’t work, the code wouldn’t deploy, and the solutions didn’t work. An experienced operations person might have easily foreseen this happening. However, an inexperienced development team that is used to working in a sandbox and development environments was caught off guard. Another example involved enterprise networking. The automated deployment required internet access because during bootstrapping, the system would download libraries from pypi and npm. But in the production environment, there’s no outbound access to the internet (whoops!).
The reality is, it takes a large team of multiple disciplines to work together to successfully launch a service into production. This is why you need to make sure that when you’re building and testing an enterprise deployment, it’s more than just the development team writing code delivering to the sprints. You need to bring in all these other stakeholders as early as you can to take a look at the whole system. Even better is push for the “fail early and fail fast” mantra, which means push for deployments as fast as you can. Don’t get caught up in spending weeks and weeks designing and planning on a whiteboard or with architecture diagrams. Instead, go ahead and try to get early test deployments out to production working as quickly as possible. Have manual exhaustive QA testing in the production environment for these initial deployments, and you’ll start to unearth these types of issues earlier in your development cycle. Addressing these issues farther to the left on your schedule ultimately results in more successful deployments and happier customers.