Are Monolith CI/CD pipelines killing the quality of your software?

ByLance T. Lee

Nov 8, 2022


When development and DevOps teams started adding testing to their CI/CD pipelines for monolithic applications, they unlocked huge benefits. Not only could they deploy faster, with fewer manual processes and interventions to go along with each merge, but they could also deploy with the certainty that they weren’t introducing bugs into their production environment.

But microservices have changed almost everything in CI/CD pipelines. With more complex microservices architectures, spanning multiple clusters and dozens or hundreds of individual nodes, one way for teams to keep up with the complexity of their cluster was to create extremely complex CI/CD pipelines. They tried to regain their confidence by their robustness.

In the end, they built CI/CD pipelines that resembled the monolithic applications that cloud native took us away from in the first place. Instead of smaller, independent processes for builds, code coverage, reviews, testing, and more, CI/CD pipelines were tightly coupled daisy-chains, where each step depended on the last to work properly.

Although monolithic pipelines have implications for every part of the application lifecycle, they pose the greatest threat to the business value of robust testing.

Are monolithic CI/CD pipelines suitable for testing?

Unfortunately, tests are often tightly tied to monolithic CI/CD pipelines in that they are entirely dependent on each other. You can’t run any tests without your CI/CD pipeline, and your pipeline can’t complete execution without being able to complete your tests and receive a pass/fail in return.

This creates murky and complex challenges both for developers trying to confidently push their commitments and for DevOps teams tasked with fine-tuning their pipeline to perfection:

  • You may need to restart your entire CI/CD workflow when you only need the result of one test. It slows down development. It is difficult to switch contexts for a few minutes to continue working on another branch while you wait for your result and excessive runs also result in a costly bill at the end of the month.
  • You cannot reuse test logic in multiple CI/CD tools or workflows. You can, of course, copy-paste your CI/CD logic into multiple stages of your pipeline, but you’ve also duplicated your maintenance efforts. Every time you make a significant change to one iteration of your test, you’ll need to do the same everywhere else.
  • You end up with inconsistent execution and reporting. As part of your test suite, you can use end-to-end testing (, load testing (k6), API testing (Postman and SoapUI), reliability testing (Artillery. io) and more. With all of these tests tightly integrated into a monolithic pipeline, you have half a dozen web services and applications to log into just to see the results of your last run. And forget about the ability to collate your organization’s test results into valuable reports without a ton of manual intervention.

This problem got us thinking: what if we build CI/CD pipelines that behave more like microservices than monoliths? That’s exactly what you get when you decouple your CI/CD pipeline from the way you create, manage, run, and display your tests. This allows your teams to work with the same agility that gave you access to cloud native and microservices.

Why should you decouple CI/CD and testing

When we talk about decoupling, we’re not talking about completely disconnecting CI/CD and testing. We’re talking about the ability for both to operate independently. A change in your CI/CD solution (GitHub Actions, Jenkins, ArgoCD, etc.) shouldn’t mean you need to change your test workflow or scripts, and vice versa.

In practical terms, this decoupling means that your CI/CD pipelines can trigger tests to run directly on your test environments through a sequence of simple API calls. The result is small, reusable, and composable CI/CD components that require no maintenance from their monolithic past. And because your test rig (and your testers!) can run specific tests regardless of the state of the CI/CD pipeline, it’s easy to relaunch them. this is you must pass to advance your pull request.

Decoupling offers immediate technical advantages:

  • You can reuse test-related steps across multiple CI/CD pipelines by making API calls to trigger the tests.
  • You can easily add or modify existing tests without worrying about inadvertently affecting your CI/CD pipeline.
  • Tests can be run whenever needed, both manually by your engineering team and by automation via CI/CD, API, etc.

The commercial benefits are perhaps less obvious, but they are just as important. Your decoupled CI/CD pipeline allows developers to work at their maximum speed by eliminating all the pain and time wasters of your monolithic past, giving them the confidence to work their magic without fear of interrupting production.

Because your tests are more editable and agile, it’s much easier to fine-tune them to perfect the end-user experience. It’s fewer bugs, better performance, and a more stable live application, all of which tie directly to your organization’s bottom line, whether your priority is reducing retention or maximizing revenue.

While we at Testkube have been thinking about CI/CD decoupling and testing for a while, it’s only just starting to make its way into the cloud native world. Based on the feedback we’re already getting from our community, this should make an extraordinary difference for DevOps and QA teams – a fast track to cloud-native testing that’s even more powerful than the monoliths of the past.

Using Testkube for cloud-native testing in your CI/CD pipeline

Testkube is an open source testing framework that lets you run, orchestrate, and visualize all your tests natively to Kubernetes. It uses custom resource definitions (CRDs) to define and configure tests, allowing you to run tests from inside your cluster through API calls instead of complex, tightly coupled CI/CD pipelines.

You can take two approaches when trying to decouple your CI/CD pipeline and your tests:

First, there’s the familiar synchronous deployment pipeline. In this case, you’re removing your tests from your CI/CD pipeline and replacing them with API calls to Testkube, which runs inside your cluster. If your tests fail, your pipeline shuts down in no time, preventing you from pursuing your bugs further into the production lifecycle. To determine where your code went wrong, you go to the Testkube UI to view your execution and its logs, and diagnose the problem.

The second approach, if you want a GitOps-like strategy to create a CI/CD pipeline, you can configure Testkube to trigger tests based on the changing state of your cluster or its resources. In this case, Testkube doesn’t just run independently and securely from inside your cluster, with testing decoupled from your CI/CD pipeline.

Your CI/CD and pipeline are two separate processes, with your test runs agnostically reporting only what it finds in your application. You can then connect Testkube to Slack, or via any webhook, to let your team know when a test fails and how.

Two viable ways to maximize the efficiency of your CI/CD pipeline in the cloud-native era, and as a bonus, Testkube not only helps you decouple your test pipelines, it also ensures that:

  • You are free to maintain test specific Docker images or many script files.
  • You leverage core Kubernetes functionality, plus the cluster compute you already manage and pay for, to scale test runs, minimizing your reliance on any CI/CD vendor.
  • You can use multiple testing tools (Postman, Cypress, k6, etc.) and view your results in a single dashboard.

Get started with Testkube

Ready to uncouple?

Install Testkube for Mac, Linux or Windows to create simpler, reusable and composable tests. Our plug-and-play architecture integrated with the most popular testing tools available like Cypress, Postman, k6, SoapUI, and more.

For advice from other DevOps and GitOps enthusiasts, check out our active Discord or find all of our open source project on GitHub! We are also on Twitter (@testkube_io)where we always share the latest news on our journey to become the most comprehensive and beloved Kubernetes native testing framework.

Band Created with Sketch.

Source link