This blog post is related to the blog post DEVELOP A SIMPLE OPERATOR TO DEPLOY A WEB APPLICATION USING THE GO OPERATOR SDK. In that last blog post we addressed the topic get a web frontend application running on Kubernetes using a GO Operator.

An important part in that scenario is also, how to manage the clean-up for an application instance and it’s related Kubernetes resources and objects created by the operator based on the Custom Resource Object, or:

How to delete services, secrets, and deployments related to a Custom Resource Object in a GO Operator?

Adam de Leeuw and I had an exchange on those topics which results in two different possible ways: an easy and one harder way ;-).

The blog post is structure in following major topics:

1. The easy way for the deletion of the related Kubernetes resources
2. A possible harder way for the deletion of the related Kubernetes resources
3. Setup of the example TenancyFrontend operator on the local machine

The easy way for the deletion of the related Kubernetes resources¶

With the SetControllerReference we can bind the existence of Kubernetes resource/objects we defined and created with our operator to a Custom Resource Object, with that binding the Kubernetes resource/objects will only exist as long as the related Custom Resource Object exists.

The image below shows a Custom Resource Object for the Custom Resource Definition TenancyFrontend.

“SetControllerReference sets owner as a Controller OwnerReference on controlled. This is used for garbage collection of the controlled object and for reconciling the owner object on changes to controlled (with a Watch + EnqueueRequestForOwner). Since only one OwnerReference can be a controller, it returns an error if there is another OwnerReference with Controller flag set.” Resource: GO controllerutil documentation

The gif below shows an example flow:

1. A Custom Resource Object will be created
2. The Operator creates the Kubernetes resources: deployment, services and secrets.
3. The SetControllerReference ensures that the deletion of the Custom Resource Object will result in the deletion of all related Kubernetes resources to that custom resource object.

The following source code shows the new function for the secret definition of the TenancyFrontend operator using the SetControllerReference function.

Therefor we need to …

1. … extend the function with r *TenancyFrontendReconciler, which mean it will extend to our reconciler object.
2. … add our custom resource object frontend *v1alpha1.TenancyFrontend as a parameter.

// Create Secret definition
func (r *TenancyFrontendReconciler) defineSecret(name string, namespace string, key string, value string, frontend *v1alpha1.TenancyFrontend) (*corev1.Secret, error) {
    secretdata := make(map[string]string)
    secretdata[key] = value

    sec := &corev1.Secret{
        TypeMeta:   metav1.TypeMeta{APIVersion: "v1", Kind: "Secret"},
        ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: namespace},
        Immutable:  new(bool),
        Data:       map[string][]byte{},
        StringData: secretdata,
        Type:       "Opaque",
    }

    // Used to ensure that the secret will be deleted when the custom resource object is removed
    ctrl.SetControllerReference(frontend, sec, r.Scheme)

    return sec, nil
}

The following code shows how the function defineSecret is invoked inside our reconcile function of the controller implementation.

...
targetSecretName := "appid.client-id-frontend"
    clientId := "b12a05c3-8164-45d9-a1b8-af1dedf8ccc3"
    targetSecret, err := r.defineSecret(targetSecretName, tenancyfrontend.Namespace, "VUE_APPID_CLIENT_ID", clientId, tenancyfrontend)
...

A possible harder way for the deletion of the related Kubernetes resources¶

The older defineSecret implementation, didn’t need the additional parameters, but with without the binding to the Custom Resource Object we need to manage the deletion by ourselves.

// Create Secret definition
func defineSecret(name string, namespace string, key string, value string, deploymentname string) (*corev1.Secret, error) {
    secretdata := make(map[string]string)
    secretdata[key] = value

    // Define map for the labels
    mlabel := make(map[string]string)
    key = "deployment"
    value = deploymentname
    mlabel[key] = value
    key = "tenancyfrontend"
    value = "yes"
    mlabel[key] = value

    return &corev1.Secret{
        TypeMeta:   metav1.TypeMeta{APIVersion: "v1", Kind: "Secret"},
        ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: namespace, Labels: mlabel},
        Immutable:  new(bool),
        Data:       map[string][]byte{},
        StringData: secretdata,
        Type:       "Opaque",
    }, nil
}

A deletion without a binding works but is harder to manage and for example this activity diagram is implemented in the Multi Tenancy Frontend Operator project in the delete-entries branch. A very useful blog post in that context is Generically working with Kubernetes objects in Go.

The following source code shows how to find and verify if a secret exists matches a specific label and secret name (if (secretname == item.Name) && (mlabel["tenancyfrontend"] == "yes")).

// Verify if a secret for the TenancyFrontend exists

func VerifySecretTenancyFrontend(clientset *kubernetes.Clientset, ctx context.Context,
    namespace string, secretname string) (*v1.Secret, error) {

    list, err := clientset.CoreV1().Secrets(namespace).List(ctx, metav1.ListOptions{})
    secret_items := list.Items

    if err != nil {
        helpers.CustomLogs(err.Error(), ctx, customLogger)
        return nil, err
    } else {
        for _, item := range secret_items {
            mlabel := item.Labels
            info := "Label app: [" + mlabel["tenancyfrontend"] + "] Name: [" + item.Name + "]"
            helpers.CustomLogs(info, ctx, customLogger)
            if (secretname == item.Name) && (mlabel["tenancyfrontend"] == "yes") {
                return &item, err
            }
        }
    }

    return nil, err
}

Setup of the example TenancyFrontend operator on the local machine¶

In that section use will use the example Multi Tenancy Frontend Operator project.

If you want to run the Operator SDK on your local machine follow the next steps. You need to install the Operator SDK and connect to an existing Kubernetes cluster. In that example setup we use free IBM Cloud Kubernetes cluster.

Step 1: Create a new folder¶

mkdir example
cd example

Step 2: Clone the operator code into new folder¶

git clone https://github.com/thomassuedbroecker/multi-tenancy-frontend-operator.git

Step 3: Navigate to the frontendOperator folder:¶

cd multi-tenancy-frontend-operator/frontendOperator

Note: If you want to checkout the harder way you only need to checkout the branch delete-entries.

git fetch
git branch -v -a
git switch delete-entries

Step 4: Execute following make commands¶

make generate
make manifests

Step 5: Run the operator locally¶

make install run

• Example output:

1.645016132941799e+09   INFO    controller-runtime.metrics      Metrics server is starting to listen        {"addr": ":8080"}
1.645016132942167e+09   INFO    setup   starting manager
1.645016132942423e+09   INFO    Starting server {"kind": "health probe", "addr": "[::]:8081"}
1.6450161329424288e+09  INFO    Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.645016132942475e+09   INFO    controller.tenancyfrontend      Starting EventSource    {"reconciler group": "multitenancy.example.net", "reconciler kind": "TenancyFrontend", "source": "kind source: *v1alpha1.TenancyFrontend"}
1.6450161329425201e+09  INFO    controller.tenancyfrontend      Starting Controller     {"reconciler group": "multitenancy.example.net", "reconciler kind": "TenancyFrontend"}
1.6450161330429811e+09  INFO    controller.tenancyfrontend      Starting workers        {"reconciler group": "multitenancy.example.net", "reconciler kind": "TenancyFrontend", "worker count": 1}

Inside the Kubernetes dashboard we see we don’t have anything deployed until now to our Kubernetes cluster, as you see in the image below.

Step 7: Deploy a custom resource to create a frontend application instance¶

Open a new terminal and apply the example definition of our example CRD.

kubectl apply -f config/samples/multitenancy_v1alpha1_tenancyfrontend.yaml 

Step 8: Verify the output in terminal of the running operator¶

Here we see for example that the secret appid.client-id-frontend is deployed.

Target secret appid.client-id-frontend exists show it is deployed.

1.6450307442006469e+09  INFO    controller.tenancyfrontend      Target secret appid.client-id-frontend exists, updating it now      {"reconciler group": "multitenancy.example.net", "reconciler kind": "TenancyFrontend", "name": "tenancyfrontend-sample", "namespace": "default"}
1.645030744206499e+09   INFO    controller.tenancyfrontend      Target secret appid.discovery-endpoint exists, updating it now      {"reconciler group": "multitenancy.example.net", "reconciler kind": "TenancyFrontend", "name": "tenancyfrontend-sample", "namespace": "default"}
1.6450307442127218e+09  INFO    controller.tenancyfrontend      Just return nil {"reconciler group": "multitenancy.example.net", "reconciler kind": "TenancyFrontend", "name": "tenancyfrontend-sample", "namespace": "default"}

In the following gif you see the created resources created by the operator.

Access the example frontend application¶

Step 1: Get the node port of the service-frontend¶

export NODEPORT=$(kubectl get svc service-frontend -o=jsonpath='{.spec.ports[0].nodePort}')
echo $NODEPORT

Step 2: Get the worker node IP of the Kubernetes cluster¶

WORKERNODEIP=$(ibmcloud ks workers --cluster YOUR_CLUSTER | awk '/Ready/ {print $2;exit;}')echo $WORKERNODEIP

Step 3: Open the example application in the browser¶

You will notice this will not work!

open http://$WORKERNODEIP:$NODEPORT

• Example output:

Step 4: Delete the example frontend application¶

Delete the example custom resource object for our example CRD.

kubectl delete -f config/samples/multitenancy_v1alpha1_tenancyfrontend.yaml 

Summary

Also, even for a simple deletion of Kubernetes resources/objects which we create within our GO operator for a Custom Resource Object, we have seen, there are always multiple ways to implement the same functionality in different ways. Surely the easy way is the best way for our example use case, because we don’t need to implement additional functionality, but the harder way shows how we can handle the Kubernetes resources/objects to achieve the same goal, delete the objects we created within our operator. That can help us in the future to manage other objects or resources we maybe wanna change.

I hope this was useful to you and let’s see what’s next?

Greetings,

Thomas

#operator, #go, #operatorsdk, #kubernetes, #buildlabs4saas, #operatorlearningjourney