kantver - Fotolia
Early on in my work life, I had a job as a quality control inspector for a cheese importer. My job was to make sure that the cheese being shipped met the customer’s needs. Now, I was no ordinary QA inspector. I was responsible for all the soft-ripening, French cheese that went out the door.
Soft-ripening French cheese is a gourmet product, one I gained expertise in working after school in a cheese store in New York City. Later, after graduating college with a music degree, I needed to pay the rent. The only real commercial skill I had was my cheese expertise. So, I went to work in the cheese industry. Later on I got into programming and things changed; but for a while, I earned a living working with soft-ripening French cheese.
Soft-ripening, French cheese is a tricky product. Unlike hard-ripening cheese, which gets harder as it ages (think cheddar), soft-ripening cheese gets softer over time, and eventually turns to liquid. The most common example of soft-ripening French cheese is brie. Soft-ripening cheese has a very short shelf life, 90 to 120 days, depending on the cheese. At the time, it was not unusual for importing companies to incur significant losses because customers refused to pay for cheese received too close to the end of its life. Other times loss was incurred because the importer could not sell the cheese at all. So the product sat in crates in the warehouse, slowly turning to mush.
This is where I came in. My job was to not only to make sure that the cheese was in good condition, which is hard enough to determine, but to also make sure we were shipping cheese that was good enough for the customer, while not leaving a lot unsold product in the warehouse. As strange as this may sound, in many ways this was a "rocket science" job. I had to know a lot about nearly 200 different types of soft-ripening cheese, and had to be able to determine where in its shelf life any one piece of cheese was at the time of inspection. As I said, rocket science. Go figure.
So, what does this have to do with DevOps QA?
A fundamental force in the DevOps mindset is automation. We deal with a lot to technology, most of which has surpassed the managing capabilities of a single person. The modern enterprise needs to automate in order to scale. And most modern businesses need to scale in order to survive.
A critical area of automation is quality assurance. We’ve automated many, if not most, aspects of DevOps QA. A lot of unit tests can be auto-generated. We write scripts for load testing, integration testing, UI testing, etc. We write test scripts to test our scripts. Given our accelerating rate of automation, it’s not unreasonable to say we are approaching total automation, the place where machines will replace humans.
But, I don’t think we’ll ever reach that point. We’ll get really close, but never arrive. Why? Because there will always be episodes when a critical decision needs to be made and you’ve just gotta go with your gut. Machines don’t have guts, humans do.
Going back to my days QA-ing the cheese, I had to manage a customer, the slow-ripening product, and the demands of my employer. Do I ship it or not? Depends. Is it Christmas time when the product will fly off the customer’s shelf due to holiday demand? Does my employer need to make room for a new shipment and is willing to take the loss on a complaint from a marginal customer? Opening up warehouse space is more valuable than keeping a marginal customer happy. If the product is subpar, ship it all the same. How do I know the product is subpar? I have to look at it and smell it, maybe handle it softly to sense how close to liquidity it is. I have to get a lot of information on a somewhat improvisational basis in order to make a decision.
The dynamic is analogous in DevOps. For example, imagine that there is some code in the CI/CD pipeline that continues to fail. The code is an implementation of a feature that is relevant to 10% of the user base. Dates are being missed due to the hold up of the code. Wall Street is going to not look kindly on the delay. Do we let our automation processes have their way? Or do we impose the human intervention, assess the situation and go with our gut?
More importantly, do we let a machine decide or do we hand it over to a human? Back in the days when I was watching the cheese go by I had no choice. Just about all decision making was done by humans. Today, the trend is the other way. And yet, there will always be mission critical edge cases that will test the limits of automation. This is the place where humans will be needed in the world of DevOps. As those of us who have been at it for a while have learned, at some point you just have to go with your gut. And as we know, it takes a lot of guts to work in DevOps.