There is a strategic problem I’ve been wrestling with lately. It’s really just a question, albeit a complicated one. But I have really looked far and wide and been unable to find any good answers to the question, or even many people addressing it or even thinking about it.

So to understand the question, let’s start with this scenario (like I said, it’s complicated, so bear with me).

Let’s take a CI pipeline in a true DevOps environment. You’ve got an amazing setup built, where when you check in your code to the dev branch it’s automatically detected that a checkin occurred. Containers are built out and an automated unit test suite is run, along with security compliance testing. Detailed reporting is made available to a reporting server, maybe even a dashboard with a stoplight chart to make it very easy for management to see the status of the product.  (Since one of the tenets of Agile is to be in a constantly shippable state.)

Take it one step further, even better, you actually launch the server or container in the development/staging environment, deploy the code, run some external functional testing against the server (i.e. https requests), and do integration testing. Now not only do you have confidence in the codebase, you’ve also got confidence in the product itself, in integration, the operating system environment, the Web server, the latest patching on the host, the whole bit. (Because yes, I’ve seen patches to the operating system break software products, to the point were turning on automated patching in the production environment made me nervous. And well, that means you’re not patching prod, are you?)

If you’ve got the above set up, you are already way ahead of most IT organizations.  You’re not quite at that pure CI/CD agility phase, but you’re doing really darn good. The next step, the one that’s even more magical and will truly help you to reach enlightenment and nirvana, is when you have enough confidence in your development building and testing (including the operating system and everything) that you can automatically push that to production. Now you would have a full CI/CD pipeline from your developers straight to production. That’s true DevOps. That’s the way your business can be truly agile, because now requirements can go from business feedback to developers and to production in just a day or two, or even less!

Most tutorials and training focus on a pipeline like the one defined above.  If I’ve got a Ruby on Rails or Node.js product I can put it into the pipeline above, fill in all the pieces, and it works great.

OK, so here’s the question.

What about enterprise or commercial off-the-shelf systems?

Let’s say that I am running an internal Wikimedia server. How can I put a Wikimedia server into a CI pipeline like the one described above?

In other words, how many things do you DevOps? Do you only do your locally-developed products, your core business competencies?  Things where you own and write the source code? Or do you do all your products and systems, because all of those enterprise IT systems affect your business’s ability to operate? All of those things require IT operations, maintenance, and effort to take care of. So why would you not try to automate everything?

If I have Wikimedia deployed in my environment, and it’s used consistently every day by the technical staff, and it’s running on, say, CentOS, and I automatically run yum update every night, and one night the update breaks Wikimedia. So now I’ve broken a production system people rely on.

So I think this is a hole in a lot of organizations, even ones that are doing CI/CD and DevOps. They’ve got really great pipelines around their internal software products, but they’re not looking at all of their enterprise IT systems that drive their business. Your accounting systems or marketing/CRM tools are just as important, because if you take them away you impact your company‘s ability to execute. Think about payroll for example. If your accounting apartment is processing payroll on a commercial product that you support into the cloud, then you’d better take care of that system with the same level of maturity that you do your customer-facing product.

Now I think this is totally doable, but to be honest it takes a different skillset and mindset than the software CI pipeline. First of all, a lot of the pipeline work really revolves around coding and scripting. When you take a software development team, you just add to their requirements baseline the need to build an automated unit testing suite and deployment.  Then all of that pipeline work just appears in the product. Most developers want to build the pipeline because it makes their job easier anyways.

But what about enterprise IT? Especially IT staff who are typically trained to use user interfaces and understand how to build out things manually. Let’s face it, most vendor training and vendor documentation, although it might be really well done and define the processes and workflows, is usually focused around “click here, click that, type that in”. It’s not really that easy to script stuff like that. (Although I found Linux systems tend to be better than Windows systems, probably because most Linux system configuration is just text files.)

But it can be done. You can build a CI pipeline that automatically takes your server baseline, applies the latest operating system patches, installs and deploys Wikimedia, restores the database of the last good backup, deploys into the development environment, and does some basic functionality testing, like making sure port 443 is listening and not throwing a 500.

But in order to do this, you have to do some strategic thinking in your organization. You have to train or hire IT operations talent that knows how to script. You have to get operations folks with the right mindset. Software developers love automating their tasks, but many operations technicians focus their skillset and procedures around process-driven checklists. That can be a pretty big mindset to change.

But to be honest, this is huge. If you can identify those critical enterprise IT systems (not the internally developed ones) that are core to keeping your business running, then it really behooves your organization to get them into a CI pipeline as well. That way you can make sure that those enterprise IT systems are running on up-to-date operating system patches, and you have automated testing to ensure that the systems are running properly. And if there’s ever a problem you can easily just rebuild the server. This is what can truly help your company make an even greater step towards agility, not just in IT but in business.