Do DevOps and SRE teams have customers? They must, right? I mean, don't they just fix the servers when something breaks?

To put it another way, DevOps successes are invisible, but their failures can be extremely disruptive. Kind of like plumbers.

How often do you think about plumbers? I would venture to guess that many people only ever interact with a plumber when their pipes freeze and break. At which point, they urgently need attention from a good plumber. But have you ever considered that plumbers, in fact, offer a service, and the complexity of that service is blissfully out-of-sight and out-of-mind?

For example, hospitals would not function if it weren't for the efforts of plumbers. They built everything from the pipes to deliver oxygen to the ICU, to the natural gas lines for the backup power generators.

DevOps and DevSecOps are similar in some ways to plumbing. It is sometimes hard to remember what they do day-to-day, and when something goes wrong, they show up to fix problems as quickly as they can. This is a really important part of their jobs.

But look beyond the on-demand support, and you will see the day-to-day services they provide are also essential to the smooth operation of so much of daily life. Just as it takes continuous effort to design, build, and maintain the pipes beneath our houses and roads, so too does it take continuous effort to design, build, and maintain the servers and software delivery pipelines that, for example, enable banks to provide checking accounts.

I would consider myself naive if I thought of my job as a DevOps engineer (a.k.a Digital Plumber) as simply responding to urgent calls to fix broken things. An effective DevOps engineer provides a valuable service to his customers that goes far beyond fixing broken build pipelines.

DevOps and DevSecOps responsibilities are best handled by collaborative teams of technical experts who complement each other's skills and knowledge. They also need to be empowered to experiment today in order to get ahead of the inevitable decay and "digital rot" that is a natural process of constant evolution in our highly connected world.

An effective DevOps team is customer oriented, risk aware, collaborative, security conscious, and pragmatic.

Thinking ahead is never easy, but it is an essential service that DevOps teams provide. With input from the business, DevOps teams can plot a path through the technology landscape that will help increase developer productivity, reduce downtime and outages, improve information security for an organization, tackle regulatory requirements, and even reduce stress on software delivery teams.

I would submit that a continuing mission of DevOps is to ensure the stable, day-to-day operation of secure software delivery over long periods of growth, opportunity, and digital evolution for businesses large and small.

The vision for DevOps teams can include ensuring business critical software-based services are available, performing well, appropriately secured, monitored, continuously updated, backed-up, and meeting the needs of their users.

An effective DevOps team is customer oriented, risk aware, collaborative, security conscious, and pragmatic.

So today, I'm thinking about how to be customer oriented. Assuming we can accept the premise that DevOps teams are providing a service that goes beyond being on-call, then we can also assume a customer oriented mindset is key to their success. But what does it take to have great customer service?

First, I think you need to know your customer: software developers. They are one type of customer that DevOps teams need to serve. And I think it's important to keep in mind that software teams also need to serve their users and customers. If problems and inefficiencies pop-up with their tools and environments, their users and customers are negatively affected. In order to meet the needs of this ever-evolving customer, DevOps teams need to think deeply about how to balance the needs of the business against the needs of the developers in order to provide top-shelf customer service.

A few years back, I had the good fortune to attend a talk by the one and only Frances Frei. She wrote a book called "Uncommon Service: How to Win by Putting Customers at the Core of Your Business". One of the themes Miss Frei emphasized in her talk was about being very deliberate and conscious of the trade-offs involved in making choices about customer service. The idea is that if you decide to pursue excellence for every single person who might use your services, then the quality of that service will be mediocre at best. But, if you look at all of your potential customers and make deliberate choices about the type of customer you want to serve, then you are enabled to provide excellent service for some, at the expense of poor or even zero service for others. This is a really powerful concept for the customer oriented team.

This mental model can help direct a team's efforts and maximize business impact. For example, let's say you want to design a DevOps team to serve your organization. What aspects of DevOps services are important to the business? You can imagine, for example, that if the business values flexibility very highly, then your customers (the dev teams) will need to be able to release code changes very quickly and potentially very often. You can also imagine that if you prioritize flexibility, that may increase risk when it comes to overall stability. In other words, the more code you push, the more work you add to related processes like code review, QA, security verification, and change approval processes. On the other hand, if you prioritize stability and risk management, then you can imagine that flexibility will be much more challenging to accommodate because of the necessary constraints that come with a higher level of scrutiny.

These kinds of tradeoffs vary according to the needs of the business being served. An engineering culture at a startup for example, needs to innovate quickly and likely values flexibility over stability, at least at first. Note that does not mean stability is unimportant, just that when faced with a design or implementation decision, if one improves flexibility but might be more risky, the choice becomes clearer.

However, engineering culture in a more strictly regulated industry like financial services values stability, security, and accountability over flexibility. Again, that does not mean flexibility is unimportant, just that if a change improves stability and security at the expense of flexibility, the choice becomes clearer.

The flexibility vs. stability tradeoff is somewhat abstract, but you get the idea. To make the concept of these trade-off decisions more concrete, I want to consider another example.

DevOps teams provide a valuable service to developers. They are directly involved in provisioning and maintaining the servers that allow developers to package up their software products and deliver them to end-users. When done well, this work reduces the cognitive load on developers by allowing them to delegate the work involved in setting up and troubleshooting their software delivery pipelines.

Dev teams, in collaboration with DevOps teams, also try to automate processes like deployment wherever they can. This eliminates the human error introduced by people performing build and test steps manually.

The software tools that enable all of this are constantly evolving. One dev team might start using an automated testing tool that over time, becomes more and more integral to their success. This is a good thing! Automated testing provides early and clear feedback to developers who are trying to produce high quality software and reduce the occurrence of bugs in their product. However, the testing tools themselves also change over time, and they can easily become incompatible with the CI/CD (Continuous Integration/Continuous Delivery is an important service offering for DevOps teams) pipelines for that project. The reasons for this are numerous, and a topic for another day. What is important for a DevOps team, is that our sophisticated customer has more than one need.

When we endeavor to serve both a growing business and its software development teams, we need to be deliberate about what our service is offering. To take a few examples, I submit the following: to the business, our service offers reliability, stability, security and accountability. To the dev teams, our service offers on-demand support and actionable feedback quickly and constantly.

What are the trade-offs we need to consider given these offerings? Do we accommodate every methodology and tool for automated testing ever built and thereby force ourselves into providing mediocre service to dev teams? Or do we do as Ms. Frei suggests and choose to be bad at some of them so that we can be great at others. That all depends on the circumstances of your customer.