Guide to Shift-left Test Approach in Kubernetes

Overview

In this article, we’ll cover the unique challenges that shift-left testing brings to the world of Kubernetes and microservices. We’ll then dive into some concrete strategies for implementing this type of testing in a microservices environment.

What is Shift Left Testing?

Shift-left testing is a set of practices that make test signal and feedback available early in the development lifecycle - as close as possible to code being written.

It’s well known that the later a bug is found in the SDLC (software development lifecycle), the longer and more expensive it takes to fix. Shifting left focuses your attention on prevention rather than focusing on detection after the fact.

Agile, DevOps and Shifting Left

In order to understand the benefits of shift-left, we can look back at the history of how we got here. In the beginning, there was the waterfall model of development that was made of highly specialized (and siloed) functions - design, development, QA and release. This model is well known as being slow, with explicit passing of responsibilities between specialized teams and long feedback loops between them.

Driven by the need to deliver software faster, the Agile methodology changed things significantly and made the entire development lifecycle more flexible and nimble in the process. It introduced the notion of sprints and optimized for shorter and more frequent iterations in a given period of time.

With competitive advantages becoming increasingly tied to time-to-market, the traditional approach of thinking of development and operations separately started to become a bottleneck. DevOps focuses on removing the silos between development and operations teams using tooling and automation. This has now grown to incorporate security (DevSecOps) as well.

We can empower development teams to think about not just building a feature but also ensuring that it is functionally correct, follows security best practices and performs well when it is released. Shift-left testing is an important tool that enables development teams to deal with this broad scope of responsibilities of operations and security. By making use of shift-left testing and test automation, which goes hand-in-hand, it becomes possible to continue to innovate and deliver software fast and with high quality.

Shift Left Testing & Microservices

Making developers care about operational and security considerations in addition to the core functionality that they are building certainly requires a shift in mindset and culture. However, this is not sufficient. We need the right tooling and automation before we can try and scale this practice across the organization.

From a process standpoint, there are 3 key challenges to solve when trying to implement shift-left testing practices across a microservices based stack.

1. Each microservices is built and managed separately which allows development to speed up but also requires that shift-left practices and processes be applied to each of these separate components.
2. Microservices allow developers to choose the right programming language and well suited service dependencies for the problem at hand. This added flexibility and decentralization comes at the cost of managing and implementing shift-left practices across different programming languages and dependencies.
3. Finally, testing each microservice in the context of the larger stack is also important. This usually involves several types of tests like integration tests, end-to-end tests, performance tests and so on to ensure that the cumulative effect of the microservices is as desired and expected. This is especially challenging in the microservices context because of the complexity of the overall application environment.

Different microservices & repos having divergent SDLC pipelines

Scaling Pipelines

To solve the first and second problem cited above, the most important consideration is choosing the right workflow system and DevOps pipeline implementation. The stages involved and the types of tests that will be run in those stages will vary between say - a Golang based backend and a JavaScript frontend that power different microservices. It is important that the process is easy to augment and manage for the development teams so that shift-left practices can be added in the manner that is specific to that particular microservice and programming language.

Modern pipeline tools such as GitHub Actions that make use of GitOps based practices are a step towards fulfilling this ideal - by providing configuration-based ways to configure the entire pipeline for each team’s microservice that can be managed and administered by the development team while leaving cross-cutting considerations like testing infrastructure to DevOps teams.

The kinds of tests that can be integrated and automated into this DevOps pipeline as a team implements the shift-left ideology includes static analysis and linters, unit tests, integration tests, performance tests, security tests, and so on. Given the variance in needs between different microservices, the pipeline system must make creating and managing pipelines easy, in a way that it can be managed primarily by development teams themselves while requiring only minimal guidance from DevOps & Platform Teams.

Test Environments

For several classes of tests mentioned above, such as integration tests, performance tests with microservices, the test environment is a key consideration. Environments are an isolated collection of microservices and configuration on which tests run. The nature and quality of environments has a direct impact on the shift-left testing practices and their efficacy.

At smaller scale (and when complexity is relatively low), it is possible to create test environments by cloning the entire production or staging environment and using that for testing. However, once an organization reaches a threshold number of microservices in the low tens, the infrastructure and maintenance cost of traditional test environments grows significantly. This leads to a setup where only some basic “smoke” tests and mock-based integration tests are done early in the SDLC with a lot of crucial testing being deferred for later. These do help in the short term but introduce new problems of test signal and validity which eventually shift the bottleneck to the staging environment or cause a drop in software quality. In the next section, we will look at a potential solution to this problem that can scale better with a microservice-based architecture.

Sandbox Environments

As noted before, one of the bottlenecks when an organization tries to scale shift-left testing is the availability of high-quality test environments. With traditional approaches, a few high fidelity “staging” environments that are used for testing signal end up as bottlenecks and become unstable over time, thereby slowing down the software development lifecycle.

With Signadot Sandboxes, the approach to environments is fundamentally different. As opposed to traditional environments, Sandboxes use application level multi-tenancy to share resources while ensuring isolation. Each environment can use external cloud dependencies, hosted databases, etc rather than mocked versions of them used in traditional environments.

They make use of the intuition that microservice test environments are used to test changes in a few services (typically under 3) against a much larger stack of services.

These sandbox environments spin up in seconds and scale to tens to thousands of microservices. Therefore, it becomes possible to have each pull request, branch, or even commit to get its own test environment, without the runaway infrastructure costs or operational burden even as you scale the number of microservices.

With high quality test environments that can spin up rapidly, even the smallest change can be tested in isolation to ensure performance, stability and correctness in the context of all its dependencies. Sandbox environments are an important tool that can help an organization in its journey towards realizing shift-left testing with microservices.

Conclusion

There are several new considerations that come into the picture when looking to scale shift-left testing across an organization that is making use of microservices. Ranging from the need to have flexibility to accommodate the needs of different microservices and different programming languages, to the need for high quality environments.

With the right set of tools, you can design and build systems and processes for shift-left testing that can scale across the entire organization with microservices and enable development teams to ship faster while improving software quality.

‍