In the third part of this blog, I’ll talk about our route to market – the global release pipe-line.
Global release pipe-line
To ensure that projects do not encounter delays in getting code from development into production and for continuous delivery to work smoothly, a robust, repeatable, reliable, fault-tolerant deployment system is required.
Our global release pipe-line is a sequence of cloud based environments that we use to prove that all new products or code are fit for purpose, and quality assured ready to go live. The environments represent real world production environments. We currently have 4 cloud based environments (called Red, Mauve, Amber & Pre-prod) in the global release pipe-line.
The journey down the pipe-line begins when the team checks in code to the version control system. It ends when the change goes to production – in between, a lot can happen!
To declare a build to be ready to enter the global release pipe-line, it must pass basic functional tests, so all development teams work from the same code repository, write a test case for the functionality they develop, build and test their functionality locally, refactor, and verify—only then committing their changes (frequently) to the source code repository. At key stages of the pipeline, we tag our code commits to ensure we have a full audit trail and then send the build down-stream towards production and operations.
After a successful build, other tests are run, such as functional tests and performance tests, these are the quality gates. Fast feedback is an important part of continuous delivery, so at every stage, feedback is relayed to the project team. If a stage fails, someone needs to know right away because the pipeline stops until the problem is fixed. The job of the global release pipe-line is to detect and reject bad builds. In the last stage (pre-prod), the build is approved and goes live via a seamless, zero outage, blue/green switch. This gives us the confidence that we can deploy to a production grade environment, and then cut over, or cut back, quickly.
RED is a gateway Continuous Integration (CI) to harmonise all changes in scope with other changes being checked in.
MAUVE is development integration and contains builds which have been verified by developers. The test team are the gate-keepers and authorise promotion to the next stage in the release pipe-line.
AMBER (Test). This is where business stake-holders can review/sign off before go-live and represents the first point at which the Test team functional test the integrated code from all projects (in scope). However, sometimes code has to be de-bugged after being promoted to AMBER and it is important that it can be reversed via MAUVE and back to RED without encountering new code.
Pre-Prod/Production (Blue/Green) – Since the two production environments switch roles regularly, at any one time, the environments could be either “Pre-prod” or “Production” depending on the cycle. A final sanity test is done in pre-prod and then following a check-point meeting with stake-holders a blue/green switch pushes the new code to live.
“It’s not that we’ve never had a bad build, its’ just that we’ve never had a build that has negatively impacted customers or the website.”
We prioritise keeping the system releasable over doing new work, – this is a really important mind-set shift that all people in DevOps understand and adhere to, so when something goes wrong, all technical resources focus on the fix to get the pipe-line running again.
By automating the deployment as much as possible, whether a human click was needed for the actual deployment or not, the team is able to repeatedly deploy successfully. Our continuous delivery pipeline automatically tests the application, but keeps the deployment decision as a manual step. This enables us to deploy into production with a high level of confidence and lower risk(s)
In the next instalment, I’ll discuss some of the benefits and advantages of agile continuous delivery.