CI/CD — So close, Yet so far
Yes, the title explains the intent.
Continuous development and continuous integration is easy to achieve, but it is easier to find teams which don’t do it right. Honestly, I have seen more team doing it wrong, and boldly claim that they have CI/CD in place. So I thought of writing an article sharing the symptoms of this malady.
What is CI/CD?
Quoting Wikipedia
In software engineering, continuous integration (CI) is the practice of merging all developers’ working copies to a shared mainline several times a day.
Continuous delivery (CD) is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time.
The next obvious thought that comes to minds is “What’s the goal of CI/CD?”. I found a very precise answer on this blog post by Isaac Sacolick, which states:
The technical goal of CI is to establish a consistent and automated way to build, package, and test applications. With consistency in the integration process in place, teams are more likely to commit code changes more frequently, which leads to better collaboration and software quality.
Dead Giveaways that Your Team Got It Wrong!
Multiple develop branches for each major feature
In this case, the teams end up creating one branch for each major feature. The goal is to keep all changes isolated to a branch, till they have it working right.
Sounds good at the surface, till features start competing for environments. There are two major features — f1 and f2. Picture this — the QA team needs to sign off on both f1 and f2 in the QA environment. They either end up spinning up multiple QA environments, spending additional money on infra. Or, one of the features suffer rollout velocity, because the other feature gets priority of deployment in the unified QA environment.
At the end of the cycle, both f1 and f2 need to co-exist in the same codebase. At such a point the teams start experiencing ‘the merge conflict hell’ — the mere mention of which is enough to make developers jump out of their skins. This happens because both team branched off from the same trunk, kept on making changes in their isolated branches, practically altering the same files with diverse logic, and finally the need to unify all the changes. This typically ends in broken features, more bugs, night outs and slogging teams.
Simply put (as per Wikipedia)
Continuous integration (CI) is the practice of merging all developers’ working copies to a shared mainline several times a day.
Manually Triggered Builds
Teams combine multiple commits into branch, and manually trigger a build once the stars align. I would rather stop here. In 2021, this one should not even come up. If you are here, do your best to reach the state where a new build is generated upon each commit to the mainline branch.
Manual Deployments
The build is generated and needs to be deployed. Someone volunteers to do this manually for a while, ‘till we have more bandwidth to get this right’. Believe me, if teams do not take this as a P1 at the very beginning, it never gets prioritised. Honestly, this is pretty straightforward and easy. Almost all the CI/CD systems support automated deployments — be it Jenkins, Atlassian Bamboo, Azure DevOps, etc. Never underestimate the value of automated deployments and build promotion. The main value automated deployments bring in is less chances of application failure due to manual errors in deployments.
Manual testing upon deployments
This one is the most common issue, manual testing. Once manual testing has established its roots in a team, it results in lower release velocity, missed test case executions, ‘why was this not caught in lower environments?’, one-testable-build-per-day, etc. Witnessing such a situation, a wise leader once said …
If you want to reap the benefits of QA automation in future, you need to start investing in it now. Now!
And another wise leader once said …
If you have not tested it, it won’t work
The value automated QA brings in reflects in the quality of deliverables. Machines don’t make mistakes, they don’t feel bored testing the same functionality over and over again, day after day (or rather build after build, many times during the same day). Once programmed, they keep on testing all test cases with the same efficiency. Moreover, machines do it much faster than humans, thereby saving time and money.
Achieving solid regression test suites can be challenging though. The easiest way to achieve it is to make regression test cases part of the ‘code complete checklist’. The engineers who write code, need to write the test cases as well. This approach also helps engineers, as it makes them think of scenarios where code can break. An engineer who has written enough test cases, tends to write more stable code, naturally.
Generating a new build for each environment
Relatively rare, but this exists. In this scenario, the team ends up building a new build out of a branch (or a tag if they really wanted to get it right) for each environment.
That’s fine, but how do we set it right?
Setting it right it easier than it seems. Watch this space for link to the next article which will share the best practices to have a solid CI/CD in place.
