First of all, a small clarification. I am not against automation in itself.
Quite the contrary - I love to automate tasks and processes
Maybe because of this, I often get frustrated, when I see how automation projects are handled or planned. Today I want to lay down my own view and why I am at a point, that I often want to say “please don’t even start”.
The following paragraphs are a highly subjective view and only represent what I personally have seen or heard.
What exactly do I mean?
Over the last years automation in all forms became one of the the main factors why agile, “hyperperforming” teams are becoming more and more prevalent in the IT startup culture.
Many startup products seem to be more centered around customer need than competing solutions. Additionally changes and new functionalities are delivered way more frequent. The adaption time to new (market-)situations is only a fraction the the one of “established” players and so on. Reasons why IT startups are dominating the markets are abundant.
So it’s no surprise, that everyone wants to join the bandwagon and automate everything.
But in my opinion this only helps to widen the technological chasm between startup and corporate development cultures, mostly driven by the following reasons:
- having a cool (and competitive) tool stack
- overambitious Return-on-Invest expectations
- automation for its own sake
- trying to keep up with faster time-to-markets
I asked my colleagues (mostly QA-background) to send me a list of terms, they associate with automation. Their answers overall confirmed these points by mentioning a few tools and best practices (Test.io, Micro Focus UFT, DevOps), but also more abstract reasons like saving time, faster execution, ease up repeating tasks, quality assurance, …
Though these reasons are quite noble goals in itself, they do not address the main issue, that is making up the perceived discrepancy between young IT-focused companies and established corporations with decades of history: technical debt.
Keeping up with the startups
I assume most of the following points are already known to many product and project managers. Still there is a growing demand of automation-engineers in the market.
I my opinion this is due to the fact, that the fear of slow time-to-market is omnipresent. Everything need to be released as fast as possible.
For startups this is (relatively) easy. Their IT infrastructure was build on a green-field with end-to-end automation woven in the design.
Software and IT in “established” enterprises (especially with non-IT core business) is not build that way. Conditions 15-20 years ago, were quite different. Customers wanted stable and tested services.
I think this trend is most visible in game development. When I started playing video games, the software was shipped as-is. The internet existed, but wasn’t that widespread as today. Patches and updates did either not exist or were shipped as CDs / DVDs in magazines.
Nowadays there are so many titles in Early Access, with monthly, weekly or even daily updates. And even on tripple-A-titles, we can expect a day1-patch with 100% certainty.
At this point I do not buy games on the first day anymore. I am not expecting a well tested and ready to ship product anymore. We got to get software updates every day.
For startups, that freshly created their service, this is great. Nobody is expecting an error-free software on release.
But for legacy systems the customer-perception is different. We know the service since a few years, so it has to be (mainly) error-free. When enterprises redesign their software they always have to keep this mind. They do not have the cushion of a new software product. This need to always deliver highly functional products, while trying to redesign (or even reinvent) the solution and infrastructure behind those products, is what I meant with technical debt.
But still often the answer to solve this problem is: more automation.
Automation for its own sake
If I ask professionals why they automate their own work, there often is one common answers:
Because it’s cool. I have fun doing it.
I often find even myself in this mindset, too. From a technical point of view this is a perfectly valid reason to write scripts and to ease up my own workload - especially for repeating tasks.
This school of thought sometimes translates into corporate decisions to start automation initiatives. Another often heard reason on corporate level is
Because everyone is doing it. We can’t fall behind.
But, in my opinion, it is hard to transfer the process of writing developer automation scripts into enterprise ready frameworks.
Why I decide to write my own scripts
Writing personal scripts and automation solutions is often justified by some pain. In most cases it’s the pain of doing some repetitive, dull task (e.g. copying data from one spreadsheet to another). Doing this for more than a few minutes is way to painful (at least for me). So instead of torturing myself, I write a script. These scripts usually have to things in common
- they are specified to one, and only one well defined (normally easy) use case
(copying data from spreadsheet 1 to spreadsheet 2) - they are developed quite quick and dirty
(no data transformation, non-captured errors, …)
Why organizations decide to automate
This thinking is prevalent in many, if not all, developers minds. Management sees developers that all have written their own little scripts to do simple tasks for them (in the optimal case they even shared their scripts to other developers in the team or organization)
So why not introduce this mindset into the whole organization or processes? Here’s where one of the main problems is located. Within these easy scripts, the processes and tasks are clear and well defined. But as soon as an automation initiative is started a whole business (or technical) processes such as testing, deployment or infrastructure provisioning needs to be automated - with all the edge-cases that can occur.
All of a sudden this easy, all dull and repetitive tasks that each had their own script, need to be combined into the automation a complete process. Initially there was no (individual) pain to automate the whole process. It gets an automation because it can.I thought you have automation scripts for this? Why is it taking so long? Whats the problem?
For each individual dull task, that was automated by the developer the return-on-invest (ROI) calculation was done by the developer itself - because it was an easy automation and even if it fails, the task can still be done manual without to much of an impact.
Overambitious Return-on-Invest expectations
This brings me to my next point. Seeing that these simple scripts are saving time for the developers, management assumes that an enterprise automation initiative can produce the same (or a similar) ROI. But, as I already stated, the underlying processes and assumptions are different.
On the one hand, developer create their “helper-scripts” to do exactly that: to help them. On the other hand, organization management wants a stable and reproducible automation solution.
Stable automation solutions are not created over night
To achieve this, automation frameworks need to be developed the same way as any other software development initiative. This means the design needs to be discussed, use cases and stakeholders need to be identified, etc. This process is normally described as building up the automation framework.
Building stable frameworks requires a lot of time and effort. Putting these efforts into costs, any break even can only be achieved with scalable and repeatable use cases. Functionality needs to be reused to easily create new objects (e.g. test cases in test automation) or to run a lot more than before (e.g. continuous integration / continuous delivery).
Let me show what I mean, with an example:
A few years back I had the task to automate the smoke test set of my project at that time. The test set consisted of about 10 test cases that have been manually executed on a daily basis. The goal was to create a test automation solution, that
- has all these test cases defined in the test management system
- allows to define the input parameters of the test (e.g. different customer id)
- allows business test analysts to create new test cases if needed
- runs every morning without human intervention
- creates a report that can be interpreted by business test analysts
Overall these are more or less normale requirements for any test automation project.
The main problem with this initiative was, that these test test in question did cover all business critical paths. Every path tested it’s own business functionality and most of the steps were different.
Overall every test case had it’s own scripts. Economies of scale could only be achieved by the number of test execution, since scaling by reusability was not really an option. But, since there were no new deployments during the day, running this set more than once, did not results in new information.
The second problem, and thats why this example is located in this section, was the ROI-expectation. The project management expected an break even in the same release (about 6 months) as the test automation initiative.
Of course, with all the requirements in mind, this initiative failed. But I do not want to blame it all on the project management. I have to take some of the blame to myself too, for not speaking up early enough, for underestimating effort and so on. Doing the calculations now, I get why this never could have worked:
$$ 1 month = 20 d * 8 \cfrac{h}{d} = 160 h$$ $$ 160 h * 60 \cfrac{min}{h} =9600 min$$
This means, that setting up an automation framework with one full time resource over one month, would create an ROI after 9600 minutes of manual smoke tests (not considering changes in the application). With a manual smoke test time of about 10-15 minutes, the whole initiative would break even after
$$ \cfrac {9600 min}{15 \cfrac {min}{execution}} = 640 executions $$
Having one execution per day, this results in a break even after about two years - give or take.
Overall this project taught me, that the main objective in any automation initiative should concentrate on user instead of monetary values (or rather savings).
Riding the hype-train - automation with fancy tools
Another reason for rushed decisions to start automation initiatives is, what I like to call, riding the hype train, though this problem is mainly fueled by tech-oriented stakeholders.
Every now and then a new industry disrupting technology is introduced into the market. For example, starting with the introduction of Docker, the whole industry entered a stage of container-hype. Everything needs to be containerized nowadays.
But with containerization comes automation. Containerized services can only thrive, when they are automatically build, released and deployed. On the positive side, this hype accelerated great ideas like agile development, DevOps, product ownership etc.
On the other hand developers and engineers are flooded with fancy new tools.
Today every applications needs to be deployed into a Kubernetes cluster. Releasing and deploying new versions needs to be done in CI/CD Pipelines (Jenkins, Azure DevOps, GitLab CI, Github Actions, …), not because the projects really needs it, but because the tool is currently hyped and every developer and consultant wants to try out this hot stuff in a project.
If all you have is a hammer, everything looks like a nail.
Quite often I see content like “How to become a better DevOps engineer…”. In most cases this content is a list of tools or tool-stacks that a future DevOps engineer needs to know in order to survive this landscape.
We all read the jokes about Full Stack Developers being an entire IT department.
But next to describing necessary tools and technical requirements (you should know how to write code), these articles, videos or podcasts rarely tell you, how to change your mode of work, how to communicate, and what is necessary in order to really facilitate the usage of these tools: A streamlined process, over the whole organization.
The first publication, that made this clear to me, was the 2018 State of DevOps Report by puppet. They simply introduced 5 stages of DevOps evolution as a maturity model.
Automation (in their case of infrastructure delivery) is placed at stage 4 - so relatively late in this model. The evolutionary path before that was defined by manual tasks and their standardization.
My main takeaway was:
If everyone in your project or organization is doing the same processes over and over again, you can probably automate them.
If every team is doing it’s own thing, automation if deemed to fail.
I also think, this is one of the reasons why RPA-initiatives are (in my perception) more successful, then automation projects in the software development lifecycle. The automated processes are naturally grown to a point where everyone is doing the same steps. Analyzing and automating these processes is way easier, than writing automation code for use cases that could change within a day.
Conclusion
Even with all this critique on automation projects and initiatives, I am still a big fan of automation.
But over the years I learned to set my views and wishes into perspective.
Now I don’t like to create automation solutions just for their own sake, or because it’s cool. Instead I learned to ask myself a few question before even starting big projects…
Do I start on a green field?
If the answer is yes, I will always decide to automate as early and as much as possible. In this case, I can design my development- and delivery processes together with my automation. I can decide on tools and frameworks, that are up-to-date.
As a matter of fact, my current projects allowed exactly that. We were able to introduce agile, test driven development in a highly regulated organization (finance and insurance).
Never before, have I seen so many automated test cases and automated quality gates in a CI Pipeline in such a short time. This showed my, what is possible if you have a willing team and an even playground to start on.
If the answer to the question above is no, the decision to start automation becomes a bit more situational and diverse.
Is the process, that I want to automate, clear to everyone and hardened over time?
Are the automation solutions easy to understand and only a tool to help the user?
Answering one of these question with yes is a pretty good sign, that an automation initiative can succeed.
When starting these processes I think it’s important to first automate one use case in all it’s depth and then expand to other use cases.
Concentrate on one topic at a time, instead of trying to deliver a half-baked solution that tries to solve every automation problem.
Start with automated container builds, following a manual test stage.
After that widen your scope to automated releases and roll outs.
When all this is done, start to create small test automation scripts, or monitoring solutions.
Automation goals like
We want to save budget.
We want to more scope in the same time.
are, in my opinion, quite bad goals. Yes, automation will help to deliver faster and cheaper in the long-run.
But setting these goals as mission-statements will, as described above, most likely result in demotivated development teams and, long-term, a failed project.
Instead try to find engaging reasons why automation solutions are necessary and why they will help everyone involved.
- higher developer satisfaction by running dull and repeating tasks automated
- higher quality by running critical (and well defined) tasks
- higher customer satisfaction by delivering a stable solution
If your reasoning for introducing automation is
We want to become more agile, so we have to automate.
or
We want to become a DevOps organization, so we have to automate.
keep in mind, that you do not need to have automation to be agile or have a functioning early stage DevOps organization. Agile and DevOps are new ways how teams can work together and communicate. The main focus is on exactly that. Automation is just a tool that can help to facilitate and drive those ideas, but it is not a hard requirement.
Let’s all make software development more fun again and automation projects a little bit better, by not starting them without a good reason.
Thank you 😃
Closing words
In the end, I want to address everyone as customer of software.
This faster time-to-market mentality will eventually burn out developers and, in the long-run, result in mediocre software overall. Not every functionality that is fresh in the market, needs to be integrated into every solution or tool that we use.Maybe it’s time, that we as customers and users start to evaluate and prioritize quality of service instead of new functionality.
Do we really need a new feature every month? Shouldn’t we let the developers invest more time into security patches and quality assurance instead of the new and hip function nobody needs (or remembers) a year later anymore?
Buzzwords like Blockchain, AI, No-Code/Low-Code have their right to exist, but not in every little piece of software.
Just because I can shouldn’t be a reason to include a feature into a system.
That way everyone involved can concentrate on the important goal of software development: Delivering an awesome product, that helps to improve the world.