Automation Over Norms

One of the hardest lessons for me to accept as I've progressed in becoming an individual contributor leader in technology organizations is that rules, norms, and policies are not nearly as effective at actually producing a desired result as I'd like them to be. To be totally transparent, this applies to myself as well.

At prior jobs, I have been the engineer called into meetings with the SOC2 auditors to explain (or attempt to explain) suspected deviations in our change control policy. I still, sometimes, did things incorrectly when the conditions were right to drive me that particular brand of frustrated. But hey, you try debugging a broken CI/CD pipeline for the entire engineering organization and have a correct JIRA ticket for every commit.

This has lead me to adopt a belief that roughly goes something like this:

If norms and rules are not backed by automation, they do not exist at all times. No amount of training, communication, or reiteration will change this.

To put this in software engineering terms: If you don't have a check enforcing that something is true on each pull request or deployment, you cannot, truly, know it is true.

This is a hard balance to strike, because perfection can't be the enemy of progress. You can't expect to just flip a switch and start enforcing that all pull requests have zero vulnerabilities if you have 200 vulnerabilities today - your product partners will rightly defenestrate you and go on about shipping what they want to ship. Likewise, the first versions of this automation will usually be somewhat naive.

Today we'll walk through some practical tools I've used for enforcing different kinds of norms, and some guiding principles for implementing new norms successfully at an organization that has not had these previously. Let's go!

Tools for Norm Automation

Over the course of my career I have happened across a few different tools that I've found really useful for enforcing norms, and occasionally written a few that I think are pretty useful as well.

Repolinter

Many linting tools strength is also their occasional weakness: they're designed with a specific language structure in mind. This is absolutely the right way to go for many of these tools. ESLint has been a godsend to me more than once. But when I was working at Mailchimp I found myself wanting a tool that could do some kind of meta-linting over a whole repository.

For example, how do I assert that all Gradle projects across a github org use a particular syntax for their Gradle file? Likewise, how would I ensure that all Dockerfiles in a github org specifically set a user other than root before the end of the Dockerfile?

It turns out that TODO Group's repolinter was the answer I was looking for here.

Repolinter allowed me to write a set of rules to apply to all repositories in a GitHub org, and even to apply them conditionally (we don't apply Docker rules if no Dockerfile is present). The beauty of repolinter is that it is incredibly extensible. The vast majority of rules we used were along the lines of "assert this file contains this content," but there are lots of other options in their toolchain for figuring out how to enforce norms.

SonarQube

The static analysis tool SonarQube has been one of the best code review sidekicks that I've seen in my career so far. I've been at multiple organizations where code quality became a heavy topic of conversation more than once. (Usually after some kind of incident the CEO noticed.)

In addition to basic analysis over test coverage (which correlates highly with code quality) SonarQube is also capable of identifying code smells in a dizzying number of languages, performing static analysis over codebases for security and complexity concerns, and enforce certain standards before allowing a pull request to be merged.

GitHub Actions

Finally, I'd be remiss if I didn't mention GitHub Actions itself. Most engineers know this tool for general CI/CD behaviors, but it's also really useful for enforcing specific standards on Pull Requests to push the kind of cultural and quality change you want to see. It's also really, really useful for enhancing human analysis.

Some creative uses of GitHub actions I've seen along these lines in the past include:

• Using a simple regex GitHub Action to assert that PR titles match the expected convention and / or include a reference to a ticket.
• A GitHub action that will render out a series of helm manifests, then diff the before/after to determine if SRE needed to manually review any changes. This allowed SRE to be a code owner on manifests, but GitHub actions could approve the most common, trivial changes without human interaction.

Principles for Norm Automation

As with most things, good tooling is less than half the problem. A screwdriver doesn't do you any good if you need to push a nail into a two-by-four. You have to know when and how to use the tool to use it effectively.

Organizations I've worked in have usually failed to automate their norms for one of a handful of different, specific reasons such as:

1. Trying to do too much too fast. See my earlier comments about not letting perfection be the enemy of better.
2. Trying to automate a norm before interrogating and communicating the reason for the norm to the wider team.
3. Incorrectly assuming that the leadership structure that would need to buy-in to enforcing the norm understands it the same way as the folks "on the ground."
4. Enforcing a norm in such a way that creates an "us vs them" mentality between the team doing the automating and everyone else that has to use it.

As a result of these experiences, I've adopted a few guiding principles for how I go about implementing the automation of norms in teams, especially if I'm doing so for the first time on a team that's not used to having these kinds of things automated.

Principle 1: Keep it Simple, Stupid. Start with the dumbest possible implementation of the automation that you can. Most of the time, folks only need a brief reminder to do the right thing. Only start making the automation more complicated when it becomes apparent that folks are intentionally circumventing it for expediency.

Principle 2: Communicate, communicate, communicate. Before implementing some new norm-enforcing automation, folks should (ideally) get a little bit tired of hearing you talk about it. What this means is going to be a bit different for every organization, but the goal is the same: not a single person should be surprised when the automation starts complaining at them.

Principle 3: Provide an escape hatch. Never ever ever ever implement some kind of norm-enforcing automation without the ability for someone to go "sudo make me a sandwich" and bypass it. The quickest way to have an automation killed is to have it identified as a blocker to something critical that needs to go to production now or else. Even if you think you'll never need it, make sure you know what the escape hatch is going to be ahead of time.

Tying it All Together

Today we've gone over some tools that are useful for automation enforcement, and some guiding principles that I use when I go about implementing these things for the first time. Hopefully this helps you if you're looking to do the same thing. Implementing automation to enforce norms is, indeed, more effective than just relying on communication and policies alone. However, you have to do it in a way that doesn't neglect the human or corporate constraints you're operating within if you want to be successful.

I'd love to hear from folks in the comments if you have other tools that have been useful for implementing these kinds of norms as automation, or if you have some guiding principles that you think I should have included. Let me know!