patpitchaya - Fotolia
DevOps is a top priority in IT, as organizations realize that their ability to build and deliver software at scale is essential to success. Even as companies make big DevOps investments in terms of money and human capital, teams can't be always be sure that those investments will pay off.
CIOs are feeling the heat, too. Dynatrace's 2018 CIO report found that 74% of the 800 global CIOs polled said the absence of shared data and tools undermines their DevOps efforts, making it difficult for IT teams to see how DevOps really plays out -- their DevOps truth.
So what is the DevOps truth? How should we measure performance and scale it over time? What type of DevOps intelligence is most critical?
Let's take a look at some common DevOps measurement misconceptions, the outcomes you should focus on and how to use that intel to improve performance.
DevOps measurement misconceptions
The most common mistake I come across when helping DevOps teams measure success is their focus purely on outputs. For example, they're fixated on lines of code shipped or release velocity. Organizations like to make a certain amount of code the benchmark, because it's easy and tangible. It says nothing, though, about what that code accomplishes.
When you have to produce a designated amount of code a day, you really just bloat software. That means higher maintenance, more technical debt, a higher cost of change and more complications when training new hires. Your goal with DevOps should be to deliver value with the most efficient code, no matter how many lines of code it takes.
Meanwhile, we misconstrue velocity as a productivity tool when, in actuality, it is a capacity planning tool. Why doesn't velocity work as a DevOps measurement of productivity? For one, velocity is a relative measure, not an absolute one. Second, velocity focuses on individual team project completion at the expense of global collaboration. Teams are so focused on reaching their own goals that they cannot afford to share best practices or properly lend expertise to help other teams. Velocity breeds silos, which are the antithesis of successful DevOps.
Another DevOps measurement misconception is the need to adhere to maturity models. These just don't work in today's competitive business and agile technology environment. If your team is trapped in a maturity model, the competition will get better while you're left in the dust. What works well one year is not necessarily going to work well the next. It's time to move on from arbitrary metrics.
Shift the focus to outcomes
When you shift to an outcomes-based measurement approach, you ensure quality over quantity in software delivery. Target key outcomes across your entire organization, and you'll have a more accurate picture of how your software development lifecycle, or SDLC, is working. If you pit teams against each other under the pressure to hit metrics, it likely means you're focusing on bad measures.
Here are some global measures and outcomes that tell a much more complete story of your software delivery performance:
- Deployment frequency. This is not about how fast you can deliver software for the sake of it. Deployment frequency is about making sure you can deploy software to production often -- with speed and stability. You pull the trigger only when the business needs it. This is a business decision, not a technological restraint.
- Lead time for changes. This is a measure of the time from when the developer commits the code to when it gets deployed and is accessible. Lead time for changes is the most predictable and most highly automated portion of the software delivery pipeline. If you have manual process in place between code commitment and deployment, you need to automate that process.
- Mean time to recover. You need to expect failure. It's a given. Instead of measuring mean time between failures (MTBF), it's more valuable to track mean time to recover (MTTR) -- how quickly you can bounce back from a failure. Say a retail software delivery team has only one failure in a given year, but it happens on Black Friday and leads to both immediate and long-term revenue loss. Your MTBF would seem pretty good -- 364 days since the last accident -- because it ignores the magnitude of that one failure. Alternatively, MTTR is a DevOps measurement that gauges efficiency much better. If that same team has a failure on Black Friday but course corrected in a matter of minutes, then that's a pretty efficient team. Tracking MTTR provides teams with the confidence to innovate, because they know they can bounce back with minimal damage.
- Change fail rate. What is the likelihood your changes result in a service incident? With change fail rate, it's all about balancing a value add for customers while ensuring changes have been thought through and tested to avoid a service incident.
Become a high-performing team
Ultimately, you want to focus efforts on the right capabilities to improve software delivery performance. Research from the DevOps Research and Assessment team confirmed that the highest performing DevOps teams have the following:
- They have 46 times more frequent code deployments. That's the difference between multiple times per day and once a week or less.
- They have 440 times faster lead time from commit to deploy, the difference between less than an hour and more than a week.
- And they have 96 times faster MTTR. High performers recover in less than an hour instead of several days.
To start, identify the constraints holding you back the most. Then work to eliminate those constraints; re-evaluate your environment and system, as changes may create new constraints; and rinse and repeat until you're firing on all cylinders. Don't be overwhelmed or discouraged if you don't see significant results immediately. The key with DevOps gains is small improvements over time.
Because the practice is still new, DevOps measurement is a skill organizations need to cultivate. It's the quickest way to quantify ROI. If you focus on these performance metrics, your team will be better equipped to achieve commercial goals like productivity, profitability and customer growth, and noncommercial goals like quality of services, operating efficiency and customer satisfaction. When tied together, these factors have the power to drive business outcomes greater than ever imagined.