Implementing assurance pipeline for Amazon EKS Platform

Organizations using Amazon Elastic Kubernetes Service (Amazon EKS) need to establish that their clusters are built-as-designed, production-ready, and follow Amazon EKS Best Practices. Although Amazon EKS manages the Kubernetes control plane, validating cluster configurations and establishing quality across infrastructure, applications, policies, and resilience remains a key responsibility for platform teams. This post details how platform engineering teams can build an assurance pipeline for Amazon EKS deployments, incorporating validation frameworks that verify configurations, test infrastructure as code (IaC), assess application resilience, and establish compliance with organizational standards.

This comprehensive validation approach complement the robust scalability capabilities of Amazon EKS, helping teams build confidence in their deployments and maintain high-quality Kubernetes environments that can handle the demands of large-scale operations.

Current pain points in validating EKS clusters

Organizations deploying applications on Amazon EKS face several validation challenges:

• Infrastructure validation gaps: Traditional testing often focuses on application code, neglecting IaC validation, and leading to misconfigurations and deployment failures.
• Siloed testing approaches: Teams often use disconnected testing methods across infrastructure, applications, and policies, creating blind spots in validation coverage.
• Limited policy enforcement testing: Organizations struggle to validate that their Kubernetes policies are correctly enforced, potentially exposing security vulnerabilities.
• Non-functional testing complexity: Load testing Kubernetes components such as CoreDNS requires specialized knowledge and tools that many teams lack.
• Resilience assessment challenges: Understanding how applications behave during infrastructure failures is difficult without through simulation and frameworks/tools supporting that simulation.
• Manual and time-consuming processes: Without automated validation frameworks, teams resort to manual validation, which are error-prone, limited in nature, and often lead to inefficient practices.

Solution overview

To address cluster validation challenges, we’ve developed an assurance pipeline that systematically validates Amazon EKS environments through six distinct frameworks, each serving a specific purpose in our validation process.

1. Infrastructure validation (Terraform test): Validates infrastructure before deployment by testing EKS cluster component modules and verifying compliance with Amazon Web Services (AWS) best practices. This early validation process helps detect and resolve infrastructure issues during the development phase rather than in production.
2. Behavioral testing (Pytest BDD): Validates cluster behavior through readable test scenarios that verify core operations such as pod scheduling and service discovery. The framework establishes proper component interactions and confirms that Kubernetes API operations respond as expected.
3. Package validation (Helm testing): Verifies Helm chart installations and cluster add-ons deployment while establishing proper resource creation. This validation step maintains consistency as code moves between different environments.
4. Policy compliance (Chainsaw): Tests admission controls, security policies, and network policies to establish that clusters adhere to organizational standards and compliance requirements. This comprehensive policy validation safeguards cluster security configurations.
5. Performance assessment (Locust): Evaluates cluster performance under various load conditions by measuring component response times and monitoring scaling behavior. This testing helps identify potential performance bottlenecks before they impact production workloads.
6. Resilience testing (AWS Tools): Uses AWS Resilience Hub and AWS Fault Injection Service (AWS FIS) to test failure recovery procedures and validate availability configurations. These tools help identify reliability improvements and establish robust cluster operations.

This pipeline gives us a clear view of our Amazon EKS environments, helping us catch issues before they affect our applications. Each framework adds a layer of validation, creating a practical approach to testing our Kubernetes infrastructure.

Prerequisites

The following prerequisites are necessary before continuing:

• AWS Command Line Interface (AWS CLI): For managing AWS services through command line
• kubectl: Command-line tool for managing Kubernetes clusters
• Terraform: IaC tool for provisioning AWS resources
• Helm: Kubernetes package manager for application deployment
• Python: Necessary for writing pytest BDD test cases

Furthermore, navigate to your GitLab project and configure the following:

1. Go to Settings > CI/CD > Variables.
2. Add the following variables:
   • AWS_ACCESS_KEY_ID: Your AWS access key
   • AWS_SECRET_ACCESS_KEY: Your AWS secret key
   • AWS_REGION: Your preferred AWS Region
   • CLUSTER_NAME: Your EKS cluster name

Walkthrough

In this walkthrough, you integrate the Amazon EKS validation framework in the GitLab CI/CD pipeline. Create a .gitlab-ci.yml file in your repository root with the following structure:

1. Unit testing with Terraform test

Unit testing your infrastructure code is crucial for catching configuration errors early in the development cycle. It helps:

• Validate that your infrastructure components are correctly defined
• Establish that resources have the expected properties and configurations
• Prevent costly mistakes before deploying to AWS
• Provide documentation of expected infrastructure behavior
• Enable refactoring with confidence

How to implement

To implement unit testing with Terraform’s native testing framework, you can use your existing Terraform repository and create a tests directory with an eks.tftest.hcl file. The eks.tftest.hcl file is a Terraform test configuration file used for unit testing Amazon EKS infrastructure code. It validates that your Amazon EKS infrastructure components are correctly defined before actual deployment to AWS, helping catch configuration errors early in the development cycle.In your existing Terraform project structure, create the following tests directory:

Sample run and expected output

When running the Terraform tests with the new native testing framework, the output looks like the following:

For more detailed output with verbose flag:

If there are any failures, the detailed error message looks like the following:

These outputs provide comprehensive validation that your infrastructure code is correctly defined and will create the expected resources when deployed.

2. Functional testing with Pytest BDD

Functional testing validates that your EKS cluster behaves as expected from an operational perspective. It’s essential because:

• It verifies that critical Kubernetes components are running correctly
• It establishes that cluster services are accessible and responding properly
• It validates that the cluster can perform its intended functions
• It catches integration issues that unit tests might miss
• It provides confidence that the cluster works for end users

How to implement

Create a tests/functional directory with your BDD tests:

Example requirements.txt (specifies the Python package dependencies needed to run the functional tests and establish consistent test environments across different systems):

Example cluster_validation.feature (behavior specifications written in Gherkin syntax, defines test scenarios in plain, human-readable language):

Example cluster_steps.py (the actual Python implementation of the test steps defined in the feature file):

Sample run and expected output

When running the functional tests, the output looks like the following:

The JUnit XML report (pytest-report.xml) contains structured test results like the following:

This output demonstrates:

• Successful execution of BDD scenarios
• Verification that all critical pods are running
• Confirmation that no errors are found in pod logs
• Test timing information
• Overall test summary showing all tests passed

The JUnit report can be integrated with CI/CD systems for reporting and tracking test results over time.

3. Helm testing

Helm testing establishes that your applications deploy correctly and function as expected within the Kubernetes environment. It’s important because:

• It validates that your Helm charts are correctly structured
• It establishes that deployed applications are accessible and functional
• It verifies that services can communicate with each other
• It catches configuration issues before they affect users
• It provides a standardized way to test application deployments

How to implement

Create a helm directory with your Helm charts and tests:

Example test-connection.yaml. This is a Helm test manifest that creates a temporary Pod to verify that the application’s service is accessible within the Kubernetes cluster by running a wget command):

Sample run and expected output

When running Helm tests, the output looks like the following:

This output confirms that:

• The Karpenter Helm chart was successfully deployed
• The Karpenter Controller pod was ready within the timeout period (120 seconds)
• All tests passed successfully

4. Kubernetes policy testing with Chainsaw

Policy testing establishes that your Kubernetes cluster enforces the security and compliance requirements that your organization needs. It’s critical because:

• It validates that security policies are correctly implemented
• It establishes that non-compliant resources are rejected
• It verifies that your governance controls are working
• It helps maintain compliance with industry standards and regulations
• It prevents security vulnerabilities from being introduced

How to implement

Create a policies directory with your Kyverno policies and Chainsaw tests:

Example test-require-labels.yaml. This is a Chainsaw test manifest that validates Kubernetes label policy enforcement by applying a policy, testing a valid deployment passes, and confirming that an invalid deployment without the required labels is properly rejected:

Sample run and expected output

When running policy tests with Chainsaw, the output looks like the following:

This output demonstrates:

• Successful application of the policy
• The necessary RBAC permissions are in place for Kyverno to perform its operations
• The policy for mutating deployments based on secret updates is correctly defined and active
• Overall test summary showing all tests passed

5. Non-functional testing with Locust

Non-functional testing evaluates the performance, scalability, and reliability of your EKS cluster under various conditions. It’s vital because:

• It identifies performance bottlenecks before they impact users
• It determines the maximum capacity of your cluster
• It validates that your cluster can handle expected load
• It helps optimize resource allocation and scaling configurations
• It establishes that critical services remain responsive under stress

How to implement

Create a tests/load directory with your Locust tests:

Example coredns_locustfile.py, This is a Locust load testing script that simulates DNS resolution stress on CoreDNS by dynamically creating Kubernetes services, querying their DNS records, and deleting them to measure DNS performance under load:

Sample run and expected output

When running the Locust load tests, the output looks like the following:

This output demonstrates how your CoreDNS service performs under load, showing metrics like the following:

• Average response time (345 ms)
• Minimum and maximum response times (78 ms to 1245 ms)
• Request throughput (1.8 requests per second)
• Error rate (0% in this example)

You can use these metrics to identify potential bottlenecks and establish that your cluster can handle the expected load before deploying to production.

6. Resilience assessment with Resilience Hub

Resilience assessment evaluates how well your EKS cluster can withstand and recover from failures. It’s essential because:

• It identifies single points of failure in your architecture
• It validates that your recovery mechanisms work as expected
• It establishes business continuity during disruptions
• It helps meet availability SLAs and compliance requirements
• It provides confidence that your cluster can handle real-world incidents

How to implement

Create a resilience directory with scripts for resilience assessment:

Example run-resilience-assessment.sh. This is a shell script that creates a Resilience Hub application for an EKS cluster, runs a resilience assessment to evaluate its disaster recovery capabilities, and saves the results to a JSON file:

Example run-fault-injection.sh. This is a shell script that creates and runs an AWS FIS experiment to test the resilience of an EKS cluster by simulating an availability zone outage and capturing the results:

Sample run and expected output

When running the resilience assessment scripts, the output looks like the following:

These outputs show a successful resilience assessment with a score of 85.0 and a completed fault injection experiment that simulated an Availability Zone failure. The assessment indicates that the cluster is policy compliant, and the fault injection experiment completed successfully, helping you identify how your cluster responds to failures.

Pipeline monitoring and visualization

You can view the pipeline execution in GitLab’s CI/CD interface, which provides a visual representation of each stage and its status. The pipeline generates reports and artifacts that can be reviewed to assess the quality of your Amazon EKS deployment. This implementation creates a complete quality assurance pipeline that validates all aspects of your EKS clusters throughout the development lifecycle.

Benefits of the Amazon EKS validation framework

Our Amazon EKS validation framework brings practical value to our Kubernetes operations through several key benefits. We test each part of our Amazon EKS setup to catch and fix issues before they reach production, leading to more stable services for our users. Our policy tests verify that security measures work as planned, giving us confidence in our cluster protection. Through load testing, we understand how our applications and infrastructure handle increased traffic, helping us prepare for busy periods and plan for growth. Tools such as Resilience Hub and AWS FIS teach us how our system reacts to failures so that we can improve recovery plans and reduce potential downtime. Moreover, the automation in our framework cuts down manual testing time so that we can focus more on building new features and responding quickly to changes. This approach of finding and fixing issues early in development saves costs when compared to addressing them in production. Our testing process also establishes that our Amazon EKS environment meets both regulatory standards and internal rules, clarifying audits and reviews. The framework is a practical tool that helps us build and maintain reliable Kubernetes infrastructure that serves our needs today and supports our growth for tomorrow.

Conclusion

A comprehensive quality assurance pipeline for Amazon EKS clusters helps establish that your Kubernetes environments are properly configured, secure, and ready for production workloads. You can implement the six validation components outlined in this post to verify correct infrastructure provisioning, establish functional correctness and security of applications, test the environment’s ability to handle expected load, and validate recovery from failures. This structured approach to validation builds confidence in your Amazon EKS deployments, reduces the risk of production issues, and maintains high-quality, production-ready Kubernetes environments. As containerized applications become increasingly critical to business operations, investing in comprehensive validation frameworks is essential. Organizations can maximize the benefits of Kubernetes while minimizing operational risks. You can adopt these validation frameworks to accelerate your journey toward reliable and performant Kubernetes deployments on Amazon EKS.

About the authors

Niall Thomson is a Principal Specialist Solutions Architect, Containers, at AWS where he helps customers who are building modern application platforms on AWS container services.

Ramesh Mathikumar is a Principal Consultant within the Global Financial Services practice. He has been working with Financial services customers over the last 25 years. At AWS, he helps customers succeed in their cloud journey by implementing AWS technologies every day.

Sundar Shanmugam is a Sr. Cloud Infrastructure Architect at AWS, specializing in solution architecture, workload migrations, and modernization. He focuses on developing innovative generative AI solutions and helps customers drive digital transformation while maximizing their AWS investment to achieve business objectives.