We need to remove as much manual, repetitive, and low-return work as possible so we can focus on engineering-hard problems.Įliminating toil also helps accelerate remediation and recovery. For example, automated tests in your Continuous Integration pipelines will help proactively identify software bugs and automated infrastructure provisioning reduces the engineering time necessary to stand up environments. Well-designed and well-implemented automation helps remove toil from deploying your software.

Slowing down to find all potential issues before the release isn't a viable solution – slowing down releases sacrifices velocity. When issues occur (and they will, I guarantee it), IT Operations and software engineers need observability to identify the issue, remediate, and recover. However, there is no such thing as 100% reliable or 100% secure. When you deploy software, it should be reliable, secure, and observable. Toil is any engineering effort devoid of meaningful value. Toil is the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows. In the realm of operations, “cruft and bugs” can be described by one word: toil. We refer to this as “bug and cruft” free. When software engineers write code, they want it to be simple, fast, and reliable.

Over time, the efficiency of your operations should increase.

In general, this means that you remove inefficiencies in one component, so that other components may perform quantifiably better. SRE can feel opaque, but in practice, it is the essence of engineering. But… how do you treat operations like a software problem? Site Reliability Engineers (SREs) love to hate toil, but what exactly is toil? And why are SREs obsessed with removing toil? In a nutshell, Site Reliability Engineering is what happens when you treat IT operations like a software problem. What is Toil, and Why Are SREs Obsessed with It?

Toil & Trouble by Jessica Spotswood