What if you can manage the deployment of all of your Kubernetes clusters with Kubernetes? What if you can provide a single control plane for all of your clusters and external resources? What if you could automatically enforce best practices to enable self-service? Following GitOps principals makes this possible. Leveraging the CNCF projects discussed in this article make this practical.

The Technology

• ArgoCD – ArgoCD is a contentious deployment tool for Kubernetes.
• Crossplane – Crossplane is a tool for managing external resources such as databases via the Kubernetes API.
• Kyverno – Kyverno is a tool for automatically enforcing policies in Kubernetes.
• Kubernetes – Kubernetes is a container orchestration platform.
• Terraform & Terragrunt (Optional) – While not CNCF projects, these tools are very useful for provisioning infrastructure and can be leveraged by Crossplane.

The Implementation

GitOps Workflow

Changes to environments happen via pull requests directly against the IaC repository monitored by ArgoCD. Additionally Argo Image Updater watches for new tags in the image repository. If a new tag matches Argo Image Updater’s version policy ArgoCD pulls the new tag and writes back to the IaC repository to record the change. Finally, Kyverno acts as a bouncer, ensuring that anything deployed to a given environment matches that environment’s constraints.

Using Argo Image Updater requires using Helm or Kustomize. Both are fine choices, requiring little more effort than employing plain YAML manifests. The benefits of either tool should not be ignored; encapsulation and automation. In multi-environment contexts it is critical that IaC can be reused in a modular fashion so that environments can be easily maintained and kept in sync.

By defining all IaC as Kubernetes API objects, it is possible to leverage ArgoCD as a multi-environment, and even multi-cloud control plane. This is accomplished using Crossplane to provision and manage all infrastructure.

Multi-Environment Cluster Topology

Hub Cluster

The hub cluster hosts ArgoCD and Crossplane, which control the creation and maintenance of all spoke clusters and external resources such as relational database services. Additionally, the hub cluster hosts Kyverno, which is leveraged to create automated policy enforcement for Crossplane. As an example, a Kyverno policy may be used to prevent a developer from deploying a database via Crossplane to a production environment that does not meet compliance requirements such as automated backups or high availability.

Spoke Cluster

A spoke cluster represents a self-contained environment. This cluster can serve many purposes. It could represent a development or production environment. A spoke may be created in a different VPC or even a different cloud provider than the spoke cluster. Spoke clusters can be copies of each other, or they can be standalone special purpose environments. A spoke cluster may serve as a Knative serverless environment, an ML workload or batch processing environment, or an environment dedicated to collecting logs and metrics.

A Word on Keeping things Ephemeral

You have probably heard the term pets vs cattle when it comes to container orchestration vs virtual machines. What about the clusters used by the container orchestration platform? Is it reasonable to expect those to live for ever? What if they go down? What is the recovery strategy?

If the containerized workloads are stateless it should be trivial to create a copy of a container orchestration environment and shift the workload to the new cluster without interruption. Stateful workloads like database servers are more challenging. Ephemeral clusters allow you to test major changes in isolation, create new environments, and provide a recovery strategy.

Provisioning the Hub Cluster

While the hub cluster is capable of provisioning other Kubernetes clusters, it cannot bootstrap itself. There are many options for provisioning the bootstrap cluster. A best practice is to encapsulate this process in a single command. My preferred method for this step is to use Terraform and Terragrunt. By leveraging Terragrunt to produce modules it is possible to create copies of the hub cluster, which is useful for testing changes and disaster recovery.

The command to create the hub cluster should accomplish the following:

• Create the supporting infrastructure for the hub cluster such as virtual networks and subnets.
• Deploy ArgoCD to the hub cluster.

An optional step is to allow Crossplane to “adopt” the hub cluster after Crossplane is deployed by ArgoCD. This makes it possible to perform ongoing maintenance via ArgoCD, allowing all operations to exist within the GitOps workflow. Crossplane can serve as a wrapper for Terraforn which allows organizations to leverage existing Terraform libraries.

Provisioning Spoke Clusters

After the hub cluster is created and ArgoCD and Crossplane have been installed to the hub cluster it is possible to create spoke clusters with Crossplane via ArgoCD. Because Crossplane resources are defined as Kubernetes API objects, it is possible to compose Crossplane and Kubernetes resources with common tools like Kustomize and Helm. Consider the possibility of creating helm charts which define a set of standards for creating spoke clusters including how the clusters are created and what is installed by default. Most likely each cluster should include some monitoring and log collection tools which can be packaged in a base cluster deployment chart. It may even be possible to abstract this concept across multiple cloud providers.

Closing Thoughts

The trouble with environments is that people always want more of them. Almost everyone can agree that one environment certainly is not enough. It is somewhere between one and infinity that people start to disagree about how many environments are necessary. Having an effective strategy for replicating environments is an essential feature of any mature internal developer platform. The inability to create new environments quickly can be a severe constraint on an organizations ability to deliver features and improvements to their products.

Leveraging tools like Helm and Kustomize, it is possible to create re-usable, modular, and composable environments. ArgoCD can be used as a single control plane for each of these environments, which reduces the overhead of managing many environments while enabling a GitOps based workflow for making changes to environments. Tools like Kyverno can set constraints on environments, ensuring that changes to environments can be made safe and self-service.