Measuring the maturity of a CD pipeline

In my proposed five-tier hierarchy of continuous delivery maturity, the name of each tier is simply the latest day in the working week on which you feel comfortable deploying.

Tier 1: "Friday"

It's extremely rare that anything goes wrong with a push to production. If something does go wrong, it'll be visible immediately, and the push can be rolled back trivially. You're calm and confident about all of these facts; you've seen the systems work correctly many times. Pushes are small.

You are comfortable pushing to production on a Friday.

Tier 2: "Thursday"

Pushing to production is a fairly standard operation, but even under ideal circumstances it can take hours; long enough that it can overrun past the end of the working day. And things have been known to go wrong. Rollback is a little fiddly.

You are comfortable pushing to production on a Thursday, but prefer not to do it on a Friday — that would be too close to the weekend.

Tier 3: "Wednesday"

The system is often unstable for a few days after a push, and requires nursing along. The availability of people who can fix those stability problems is limited. You don't like the chances of instability continuing into the weekend.

You are comfortable pushing to production on a Wednesday, but not on a Thursday or a Friday.

Tier 4: "Tuesday"

You routinely need multiple working days to shepherd a major push to production all the way to completion, fixing the inevitable problems along the way — or rolling them all back, which is difficult.

You are comfortable pushing to production on a Tuesday, but not a Wednesday or later in the week.

Tier 5: "Monday"

Deployments are routinely followed by days of downtime and firefighting. Deployments are occasionally postponed because the previous week's deployment is overrunning. Your entire life is spent pushing to production or handling the fallout from earlier pushes.

You are only comfortable pushing to production on a Monday.

Notes

Being honest, the two most common tiers found in reality are going to be the top tier, "Friday", and the bottom tier, "Monday".

There are plenty of possible tiers below "Monday". But at that point you're not even delivering weekly, which I argue is too slow a cadence to qualify as "continuous delivery". This is a scenario in which there's no time in the working week when deploying is considered safe. It's a scenario in which it takes more than a working week to get all of your ducks in a row, deploy, wait for everything to not fall over, do some manual testing and slowly back away. It can be a scenario in which you don't have a deployment procedure at all. Or in which you have no production system...

There are subtiers within "Friday": "first thing Friday", "Friday morning", "Friday afternoon", "2pm Friday", "4:30pm Friday", and so on, asymptotically approaching the weekend as the amount of expected fallout caused by a push shrinks from hours to minutes. The highest possible tier is "close of business, Friday".

There are secret unholy tiers "above" the "Friday" realm where for some reason you consider it acceptable to push to production on a Saturday or a Sunday. These tiers cannot be attained by improving your CD pipeline, nor should they be considered aspirational. Switch off, fool! Stop working on the weekend! You're creating a terrible precedent!

Discussion (6)

2020-01-14 01:28:21 by qntm:

Based on a brief Twitter thread.

2020-01-14 02:09:52 by Tim McCormack:

The main reason I've seen for not pushing on a Friday is not that the deployment is difficult or lengthy, but that it can take time for customers to notice a bug and for the report to make it allll the way to the engineers. ...but I think you're right that when you only consider the unpleasantness and instability of deploys, it breaks down into something approximating "never OK" and "always OK".

2020-01-14 03:05:37 by DSimon:

Speaking of deploy problems: why are Russian translations of the Ed stories suddenly occupying most of the "latest updates" area of the homepage?

2020-01-14 03:25:14 by qntm:

I have been upgrading the way in which qntm.org handles translated versions of my stuff. Instead of being /ad hoc/, separately hosted HTML files with varying resemblance to the rest of the site, translations are now fully-fledged pages in the main qntm.org content management system. That means recent additions now show up in the "Latest" section on the main page. Future work will group translations together so you can use a language switcher drop-down (or something similar) to select among different versions of a page, so I don't have to manually insert links in the text of the original English version.

2020-01-14 03:51:36 by a different sam:

Maybe a logarithmic scale works here, since we're looking at multiple orders of magnitude and there's that Friday asymptote: as t -> 17:00, engineers -> pub. It even comes with a ready-made name: the cedibel.

2020-01-14 16:45:38 by palfrey:

There is one way to achieve the above Friday tier without doing stupid things (e.g. weekend working): automation. Example scenario: 1) Vulnerability report comes out on a Saturday indicating my dependency X has a a horrible security bug, and needs upgrading to version Y 2) Automated tool (e.g. https://dependabot.com/) makes PR upgrading my tool. 3) Another automated tool (https://mergify.io/) sees we have a green build for that upgrade and merges the PR 4) Random other tooling goes "oh, you've got changes to master, let's redeploy" This exceeds the Friday tier, as no humans were involved. If you've also got tools like Flagger (https://flagger.app/) then you can in theory also cope with the site not working properly post-deploy.

New comment by :

Plain text only. Line breaks become <br/>

The square root of minus one: