Automating Processes with Software Is Hard

Exceptions or edge cases add additional states.

To fight the explosion of state count (and the intermediate states those generate), you have a couple powerful tools:

   1. Identifying and routing out divergent items (aka ensuring items get more similar as they progress through automation)
   2. Reunifying divergent paths, instead of building branches

If you want to go back and automate a class of work that's being routed out, write a new automation flow explicitly targeting it. Don't try and kludge into into some giant spaghetti monolith that can handle everything.

PS: This also has the side effect of simplifying and concluding discussions about "What should we do in this circumstance?" with other stakeholders. Which for more complex multi-type cases can be never-ending.

PPS: And for god's sake, never target automation of 100% of incoming workload. Ever. Iteratively approach it, but accept reaching it may be impossible.

My team and I once took on a very tricky automation project. At the time we had a complex software deployment done about once per month that involved a team of about a dozen people showing up at 4am to do it while traffic was low.

The deployment involved many manual steps and coordination of everybody involved. The person leading each deployment followed the documented list of steps and got each person to do their bit at the right time; people to run database migrations, people to install RPMs on particular servers, people to test and verify functionality. Mistakes and missed steps were not uncommon.

The very first thing we did was take the documentation and write a Jenkins job to post each step into a Slack channel specifically for coordinating the deployments. Someone clicked "go" and each step was posted as a message in that channel with a 'done' button to be clicked when that step was done. Clicking the button caused the next step to be posted.

The next release we did used that instead of one person reading the steps out of confluence. Everyone involved in the release could always see what step we were at, and when it was their turn to do their bit. This helped ensure no steps were ever missed too.

Over the following months we chipped away at that job a bit at a time. We'd pick a step in the process and automate just that step, starting with the low-hanging fruit first. The Slack message for that step went from "click to confirm you've done it" to "click to do it", with the result posted once it was done; followed by the next step to perform.

It was a long process, but it allowed the rest of the business (and us!) to gradually gain confidence in the automation, and lowered the risk of the project dramatically. Once several steps had been automated and battle-tested we removed the 'click to do' bits in between and the whole release became a couple of clicks followed by the odd bit of manual QA.

An approach like this seems to give more weight to the fact that just figuring out a way to document exactly what needs to be done is often the hardest part and if you get that right before you start writing automation code, it might make make the automation much more efficient.

[1] https://blog.danslimmon.com/2019/07/15/do-nothing-scripting-...

Automating a process makes it more standardized and legible, but takes away a lot of the nuance and of the resilience that human institutions tend to bake in. Do it too little, you get chaos; do it too much, you're destroying the very thing you were trying to nurture.

It's certainly deformation professionnelle, but the parallel with software is eerily relevant.

--- [0] https://en.wikipedia.org/wiki/Seeing_Like_a_State

http://www.complexcognition.co.uk/2021/06/ironies-of-automat...

Being flexible is good, but it comes at a cost. When that cost is too high, don’t do it. Realize that the customer who wants that workflow probably isn’t going to be your make-or-break moment. And in fact, they might have flexibility of their own. For example, you don’t accept Amex but they have a backup credit card or cash. It might be annoying to them if they are fussy, but it’s normal enough that the consequences are minimal. And yes, you may occasionally get a customer who doesn’t have that flexibility, but you shouldn’t be pinning your business on rare events. Figure out what’s most common among your target customers and support a few simple workflows. Say no to everything else until you have the resources to do it properly.

Another thing you can do is have a hybrid low tech/high tech solution. Automate structured inputs with software. Write down the rest as notes in a logbook. Over time you will probably see patterns in the logbook that you can automate.

Lastly, remember what Morpheus says in The Matrix, “Some [rules] can be bent. Others can be broken.” For example, you could simply pay the bill on behalf of the customer who wants cash on delivery. Now you assume some personal risk but the computer system doesn’t have to support their workflow. Is it worth it? Maybe, maybe not, but it’s a choice you can make.

How did you arrive at that conclusion? Most of those applications were written by the businesses themselves. IBM provided you a programming environment not a set of predefined business software. Which is why COBOL jobs still exist.

What changed with the mainframe was that instead of having a bunch of disparate and disconnected processes you now had a centralized set of dependent processes and a database that provided a single source of truth.

Businesses were going to want this capability regardless of how limited humans with green eyeshades were previously.

> Much of the early internet (and still most bank and insurance) look like HTML front ends to mainframe 3270 screens.

Well, precisely, those are custom applications. If they weren't we wouldn't have this issue. You talk about automation but you seem to have not noticed that mainframes have CICS, why it exists, and why businesses still use it.

The "old school" ways are actually still exceptionally powerful.

Some kinds of automation are absolutely essential. Aligning wafer stages during photolithography, shipping logistics, high frequency trading, etc. The business absolutely wouldn't work without it.

The other kinds of automation are more questionable. Developing E2E CI/CD automation for an internal tool that is redeployed once every quarter might be a more difficult sell to management. For these cases, even if the rate of manual process invocation is somewhat high (i.e. frustrating for some employees), the customer can't see it. They won't pay one additional cent for this.

There is also this entire meta time wasting game where the mere debate about "should we automate" takes up so much resources you could have manually invoked the thing more times than it would ever be feasibly invoked automatically during its projected lifetime. Alternatively, you could have just let your developers build the damn thing to get it out of their systems.

Premature automation can create more complexity in the process as edge cases are usually added in, or needing routing.

If you're building process management and automation from scratch as a way to learn, it's also a red flag of technical debt waiting.

What happens if you just say no? I have a feeling a lot of complexity stems from the fact that these exceptions are easy to handle when you’re doing things manually, so it’s hard to say no. But if you think about the business case I would not be surprised if the correct answer is no more often than not.

Hardware automation has had obvious boons to places it works. And often we will force the places we want it to be to change to accommodate the automation. Agriculture is an easy example of what I mean here. Take a look at some fields that are optimized for giant machinery. Very impressive and can manage a lot. My favorite recently was seeing the "kickout" effort to prevent bad fruit from entering a hopper.

To that end, is the main issue with software automation that it thinks it can be a total solution and often tries to start by not imposing changes on what is being automated?

Public reaction to automation has been mixed for obvious reasons, but also because when software is applied to structure- and/or determinism-resistant systems it fails to capture some essential components that ends up degrading the services that system can provide.

It always was the biggest issue I have seen “sorry cannot help you with that because our system says so”, which should never be the case. There should always be way around the system. Then of course way around the system needs approval from some higher up anyways.

Another gem from the article that I wanted to surface. It's yet-another-take on the general sentiment here, but it's very succinct. Automation is good, but automation is a tool that serves people (and not in a Soylent Green kind of way).

I start making a tool for myself to track a tedious or error prone process. Even after you've gotten the wiki to have the correct steps in the correct order, you can still run into human error. I use it and work on it until I stop burning my fingers. Then I share it with a few people who complain about the same thing, and see how they burn their fingers.

Then I roll it out to a wider team, wait for more problems, then suggest it for general use. Usually by this point we are seeing fewer problems with the tools than with humans doing things manually, so we escalate to teasing people to use the tool so they don't end up with an Incident. Then it becomes mandatory and anyone who still refuses to use it is now risking PIP-level trouble if they don't join the party.

People who try to skip steps make not only a lot of problems for themselves but also for anyone else who suggests automation tools. Running toward the finish line makes everything worse.

Anyway, one other insight I would add is that the issues tend to come up at the interface between the systems. Your automation has to get some input and send some output and both those are pain points. That’s why we sometimes prefer imperfect monolith software.

Imagine test automation on physical devices (ex-founder of a mobile app testing SaaS company here) where system0level events that can't be programmatically detected or dismissed can delay, interrupt or alter the outcome of your tests.

Dealing with exceptions to make execution outcomes more stable seem like an obvious choice, but what if it's not feasible due to the complexity of the environment? We found that accepting variance and extrapolating test results from multiple executions in the same (or identical) environment was the only way forward.

Medical diagnosis is interpretation, you can't predict all possible states involved in the diagnosis and outcome, it is inherently unsuitable to be treated in such a way.

and

https://xkcd.com/974/

Stay Healthy!