Testing Kubebuilder Operators

0_OhGPTI_JPEsTZdGm.png

This page third part of multi-part series. Before reading it, please make sure you read the following:

Kubernetes Controllers, Custom Resources, and Operators Explained

Building Kubernetes Operators

Testing Kubebuilder Operators (you’re here)

The full code of this page can be found here!

For any questions you have, you can reach me via Linkedin or Twitter.

If you like my content and want to improve your skills, please take a look at the Devops Culture Project and get all the resources you didn’t know you need.

In this part, we will write integration tests that will run in our local environment with no need for a Kubernetes cluster installation.

Goal

We are going to write a test that will do the following:

  • Run our controller in the background
  • Create a stateless AbstractWorkload
  • Check for a created Deployment
  • Delete the Deployment
  • Check for a created Deployment
  • Change the replica number
  • Check for an updated Deployment

Setting up the environment

Kubebuilder uses Kubernetes’ envtest package. With this package, we can run Kubernetes’ control plane locally, without the need to install a cluster.

Next to your controller go file, Kubebuilder generated another file: controllers/suite_test.go — this file is responsible for setting up the test environment and configuring and initializing anything needed before and after our tests.

The BeforeSuite is a function responsible for setting up the environment before the testing start to run. Let’s talk about what happens there:

First, it creates an envtest environment configuration. the CRDDirectoryPaths should contain a list of paths to custom CRD we want our control plane to know.

                testEnv = &envtest.Environment{
   CRDDirectoryPaths:     []string{filepath.Join("..", "config", "crd", "bases")},
   ErrorIfCRDPathMissing: true,
}
            

At the time of writing, Kubebuilder doesn’t have good support for remote CRDs. Sadly, I don’t have any recommendations for this problem at the moment (You can download the remote CRDs and store them locally, but that’s not a long-term solution).

Then we are starting our control plane, update our runtime scheme and initialize a Kubernetes client with that scheme — we will use it in our tests:

                cfg, err = testEnv.Start()
...
err = examplesv1alpha1.AddToScheme(scheme.Scheme)
...
k8sClient, err = client.New(cfg, client.Options{Scheme: scheme.Scheme})
            

The AfterSuite is a function responsible for tearing up the environment after the testing finish run. It is just stopping the envtest environment.

We’re missing one important thing: running our controller. We will need to add this part by ourselves, It's a good question why Kubebuilder didn’t generate a default behavior for us.

First of all, let’s make sure we all have a common ground for that one. You should have those variables at the top. If you don’t have, make sure you add them:

                var (
   cfg       *rest.Config
   k8sClient client.Client // You'll be using this client in your tests.
   testEnv   *envtest.Environment
   ctx       context.Context
   cancel    context.CancelFunc
)
            

Add this line at the start of the BeforeSuite function:

                ctx, cancel = context.WithCancel(context.TODO())
            

add this line at the start of the AfterSuite function:

                cancel()
            

Now we can initialize our manager, register our controller and run it in a new goroutine so it won’t block any cleanup. At the end of the BeforeSuite:

                k8sManager, err := ctrl.NewManager(cfg, ctrl.Options{
   Scheme: scheme.Scheme,
})
Expect(err).ToNot(HaveOccurred())

err = (&AbstractWorkloadReconciler{
   Client: k8sManager.GetClient(),
   Scheme: k8sManager.GetScheme(),
}).SetupWithManager(k8sManager)
Expect(err).ToNot(HaveOccurred())

go func() {
   defer GinkgoRecover()
   err = k8sManager.Start(ctx)
   Expect(err).ToNot(HaveOccurred(), "failed to run manager")
}()
            

We set up the environment, now we’re ready to start creating our test!

Testing the Controller

We will need to create a new test file for our controller: controllers/abstractworkload_controller_test.go

We’re going to use the Ginkgo framework which is yet another testing framework. Since Kubebuilder uses it in their documentation, I would suggest you use it.

I’m adding here the final imports for you to use since GoLand likes to delete unused imports:

                import (
   "context"
   . "github.com/onsi/ginkgo"
   . "github.com/onsi/gomega"
   "itamar.marom/abstractworkload/api/v1alpha1"
   v12 "k8s.io/api/apps/v1"
   metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
   "k8s.io/apimachinery/pkg/types"
   "time"
)
            

We will describe the testing topic we’re working on. Create some helpful variables and created the context for our testing. Then stating what should happen:

                var _ = Describe("CronJob controller", func() {

   // Define utility constants for object names and testing timeouts/durations and intervals.
   const (
      AbstractWorkloadName           = "test-stateless"
      AbstractWorkloadNamespace      = "default"
      AbstractWorkloadContainerImage = "nginx:latest"

      timeout  = time.Second * 10
      duration = time.Second * 10
      interval = time.Second * 5
   )
   var (
      deploymentLookupKey             = types.NamespacedName{Name: AbstractWorkloadName, Namespace: AbstractWorkloadNamespace}
      abstractWorkloadLookupKey       = types.NamespacedName{Namespace: AbstractWorkloadNamespace, Name: AbstractWorkloadName}
      AbstractWorkloadReplica   int32 = 2
      abstractWorkload                = &v1alpha1.AbstractWorkload{
         ObjectMeta: metav1.ObjectMeta{
            Name:      AbstractWorkloadName,
            Namespace: AbstractWorkloadNamespace,
         },
         Spec: v1alpha1.AbstractWorkloadSpec{
            Replicas:       &AbstractWorkloadReplica,
            ContainerImage: AbstractWorkloadContainerImage,
            WorkloadType:   v1alpha1.StrStateless,
         },
      }
   )

   Context("Lifecycle of stateless AbstractWorkload", func() {
      It("Should manage Deployment object and update the AbstractWorkload status", func() {
      })
   })
})
            

This is a very general test, in real life, you should be more descriptive and modular with your testing, checking just one thing at a time instead of the entire workflow.

we can now declare how we check this statement. Inside the It statement:

Create a stateless AbstractWorkload :

                By("By creating a new AbstractWorkload")
ctx := context.Background()
Expect(k8sClient.Create(ctx, abstractWorkload)).Should(Succeed())
            

Check a Deployment is created with the right spec:

                By("Checking the created Deployment")
createdDeployment := &v12.Deployment{}
Eventually(func() bool {
   err := k8sClient.Get(ctx, deploymentLookupKey, createdDeployment)
   if err != nil {
      return false
   }
   return true
}, timeout, interval).Should(BeTrue())

Expect(createdDeployment.Spec.Replicas).Should(Equal(abstractWorkload.Spec.Replicas))
Expect(createdDeployment.Spec.Template.Spec.Containers[0].Image).Should(Equal(AbstractWorkloadContainerImage))
            

Check if AbstractWorkload will create a new Deployment after it is deleted:

                By("Delete the Deployment and see that it being created again")
Expect(k8sClient.Delete(ctx, createdDeployment)).Should(Succeed())
newCreatedDeployment := &v12.Deployment{}
Eventually(func() bool {
   err := k8sClient.Get(ctx, deploymentLookupKey, newCreatedDeployment)
   if err != nil {
      return false
   }
   return true
}, timeout, interval).Should(BeTrue())
Expect(newCreatedDeployment.Spec.Replicas).Should(Equal(abstractWorkload.Spec.Replicas))
Expect(newCreatedDeployment.Spec.Template.Spec.Containers[0].Image).Should(Equal(AbstractWorkloadContainerImage))
            

Change the replica number and check for an updated Deployment

                By("Changing replicas number of AbstractWorkload and check Deployment")
replicaAbstractWorkload := &v1alpha1.AbstractWorkload{}
replicaDeployment := &v12.Deployment{}

Expect(k8sClient.Get(ctx, abstractWorkloadLookupKey, replicaAbstractWorkload)).Should(Succeed())
*replicaAbstractWorkload.Spec.Replicas = 1
Expect(k8sClient.Update(ctx, replicaAbstractWorkload)).Should(Succeed())

Eventually(func() (*int32, error) {
   err := k8sClient.Get(ctx, deploymentLookupKey, replicaDeployment)
   if err != nil {
      var badReplicas *int32
      *badReplicas = -1
      return badReplicas, err
   }
   return replicaDeployment.Spec.Replicas, nil
}, duration, interval).Should(Equal(replicaAbstractWorkload.Spec.Replicas))
            

After we have everything written and compiled, we can run our testing with a simple Makefile command:

                make test
            

If you followed everything and did it correctly, tests should pass.

Summary

We now know how to test our controller logic, and created confidence in our operator before deploying to production.

We learned how to set up our local integration testing environment using envtest, how to work with some testing frameworks, and how to interact with the API server using a Kubernetes client.

More tests can be added (check the OwnerReference implementation, check the AbstractWorkload status after creation).

WE ARE NOW READY TO DEPLOY OUR CONTROLLER TO A RUNNING KUBERNETES CLUSTER.

In the next parts, I will guide you through the controller’s deployment process and options. We will cover the cert-manager and webhooks options.

For any questions you have, you can reach me via Linkedin.

If you like my content and want to improve your skills, please take a look at the Devops Culture Project and get all the resources you didn’t know you need.


Only registered users can post comments. Please, login or signup.

Start blogging about your favorite technologies and get more readers

Join other developers and claim your FAUN account now!

Avatar

Itamar Marom

Platform Engineer, AppsFlyer

@itamar-marom
Platform Engineer @ AppsFlyer — Pursues the next world-changing idea and always doubts the current state of the world of applications.
Stats
8

Influence

379

Total Hits

1

Posts