This blog post is related to the blog post DEVELOP A SIMPLE OPERATOR TO DEPLOY A WEB APPLICATION USING THE GO OPERATOR SDK. In that last blog post we addressed the topic get a web frontend application running on Kubernetes using a GO Operator
.
An important part in that scenario is also, how to manage the clean-up for an application instance and it’s related Kubernetes resources and objects created by the operator based on the Custom Resource Object
, or:
How to delete services, secrets, and deployments related to a
Custom Resource Object
in aGO Operator
?
Adam de Leeuw and I had an exchange on those topics which results in two different possible ways: an easy and one harder way ;-).
The blog post is structure in following major topics:
- The easy way for the deletion of the related Kubernetes resources
- A possible harder way for the deletion of the related Kubernetes resources
- Setup of the example TenancyFrontend operator on the local machine
The easy way for the deletion of the related Kubernetes resources¶
With the SetControllerReference
we can bind the existence of Kubernetes resource/objects we defined and created with our operator to a Custom Resource Object
, with that binding the Kubernetes resource/objects will only exist as long as the related Custom Resource Object
exists.
The image below shows a Custom Resource Object
for the Custom Resource Definition
TenancyFrontend
.

“SetControllerReference sets owner as a Controller OwnerReference on controlled. This is used for garbage collection of the controlled object and for reconciling the owner object on changes to controlled (with a Watch + EnqueueRequestForOwner). Since only one OwnerReference can be a controller, it returns an error if there is another OwnerReference with Controller flag set.” Resource: GO
controllerutil
documentation
The gif below shows an example flow:
- A
Custom Resource Object
will be created - The Operator creates the Kubernetes resources: deployment, services and secrets.
- The
SetControllerReference
ensures that the deletion of theCustom Resource Object
will result in the deletion of all related Kubernetes resources to that custom resource object.

The following source code shows the new function for the secret definition of the TenancyFrontend
operator using the SetControllerReference
function.
Therefor we need to …
- … extend the function with
r *TenancyFrontend
Reconciler
, which mean it will extend to our reconciler object. - … add our custom resource object
frontend *v1alpha1.TenancyFrontend
as a parameter.
// Create Secret definition
func (r *TenancyFrontendReconciler) defineSecret(name string, namespace string, key string, value string, frontend *v1alpha1.TenancyFrontend) (*corev1.Secret, error) {
secretdata := make(map[string]string)
secretdata[key] = value
sec := &corev1.Secret{
TypeMeta: metav1.TypeMeta{APIVersion: "v1", Kind: "Secret"},
ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: namespace},
Immutable: new(bool),
Data: map[string][]byte{},
StringData: secretdata,
Type: "Opaque",
}
// Used to ensure that the secret will be deleted when the custom resource object is removed
ctrl.SetControllerReference(frontend, sec, r.Scheme)
return sec, nil
}
The following code shows how the function defineSecret
is invoked inside our reconcile function of the controller implementation.
...
targetSecretName := "appid.client-id-frontend"
clientId := "b12a05c3-8164-45d9-a1b8-af1dedf8ccc3"
targetSecret, err := r.defineSecret(targetSecretName, tenancyfrontend.Namespace, "VUE_APPID_CLIENT_ID", clientId, tenancyfrontend)
...
A possible harder way for the deletion of the related Kubernetes resources¶
The older defineSecret
implementation, didn’t need the additional parameters, but with without the binding to the Custom Resource Object
we need to manage the deletion by ourselves.
// Create Secret definition
func defineSecret(name string, namespace string, key string, value string, deploymentname string) (*corev1.Secret, error) {
secretdata := make(map[string]string)
secretdata[key] = value
// Define map for the labels
mlabel := make(map[string]string)
key = "deployment"
value = deploymentname
mlabel[key] = value
key = "tenancyfrontend"
value = "yes"
mlabel[key] = value
return &corev1.Secret{
TypeMeta: metav1.TypeMeta{APIVersion: "v1", Kind: "Secret"},
ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: namespace, Labels: mlabel},
Immutable: new(bool),
Data: map[string][]byte{},
StringData: secretdata,
Type: "Opaque",
}, nil
}
A deletion without a binding works but is harder to manage and for example this activity diagram is implemented in the Multi Tenancy Frontend Operator
project in the delete-entries
branch. A very useful blog post in that context is Generically working with Kubernetes objects in Go.

The following source code shows how to find and verify if a secret exists matches a specific label and secret name (if (secretname == item.Name) && (mlabel["tenancyfrontend"] == "yes")
).
// Verify if a secret for the TenancyFrontend exists
func VerifySecretTenancyFrontend(clientset *kubernetes.Clientset, ctx context.Context,
namespace string, secretname string) (*v1.Secret, error) {
list, err := clientset.CoreV1().Secrets(namespace).List(ctx, metav1.ListOptions{})
secret_items := list.Items
if err != nil {
helpers.CustomLogs(err.Error(), ctx, customLogger)
return nil, err
} else {
for _, item := range secret_items {
mlabel := item.Labels
info := "Label app: [" + mlabel["tenancyfrontend"] + "] Name: [" + item.Name + "]"
helpers.CustomLogs(info, ctx, customLogger)
if (secretname == item.Name) && (mlabel["tenancyfrontend"] == "yes") {
return &item, err
}
}
}
return nil, err
}
Setup of the example TenancyFrontend operator on the local machine¶
In that section use will use the example Multi Tenancy Frontend Operator
project.
If you want to run the Operator SDK on your local machine follow the next steps. You need to install the Operator SDK and connect to an existing Kubernetes cluster. In that example setup we use free IBM Cloud Kubernetes cluster.
Step 1: Create a new folder¶
mkdir example
cd example
Step 2: Clone the operator code into new folder¶
git clone https://github.com/thomassuedbroecker/multi-tenancy-frontend-operator.git
Step 3: Navigate to the frontendOperator
folder:¶
cd multi-tenancy-frontend-operator/frontendOperator
Note: If you want to checkout the harder way
you only need to checkout the branch delete-entries
.
git fetch
git branch -v -a
git switch delete-entries
Step 4: Execute following make commands¶
make generate
make manifests
Step 5: Run the operator locally¶
make install run
- Example output:
1.645016132941799e+09 INFO controller-runtime.metrics Metrics server is starting to listen {"addr": ":8080"}
1.645016132942167e+09 INFO setup starting manager
1.645016132942423e+09 INFO Starting server {"kind": "health probe", "addr": "[::]:8081"}
1.6450161329424288e+09 INFO Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.645016132942475e+09 INFO controller.tenancyfrontend Starting EventSource {"reconciler group": "multitenancy.example.net", "reconciler kind": "TenancyFrontend", "source": "kind source: *v1alpha1.TenancyFrontend"}
1.6450161329425201e+09 INFO controller.tenancyfrontend Starting Controller {"reconciler group": "multitenancy.example.net", "reconciler kind": "TenancyFrontend"}
1.6450161330429811e+09 INFO controller.tenancyfrontend Starting workers {"reconciler group": "multitenancy.example.net", "reconciler kind": "TenancyFrontend", "worker count": 1}
Inside the Kubernetes dashboard we see we don’t have anything deployed until now to our Kubernetes cluster, as you see in the image below.

Step 7: Deploy a custom resource to create a frontend application instance¶
Open a new terminal and apply the example definition of our example CRD.
kubectl apply -f config/samples/multitenancy_v1alpha1_tenancyfrontend.yaml
Step 8: Verify the output in terminal of the running operator¶
Here we see for example that the secret appid.client-id-frontend
is deployed.
Target secret appid.client-id-frontend exists
show it is deployed.
1.6450307442006469e+09 INFO controller.tenancyfrontend Target secret appid.client-id-frontend exists, updating it now {"reconciler group": "multitenancy.example.net", "reconciler kind": "TenancyFrontend", "name": "tenancyfrontend-sample", "namespace": "default"}
1.645030744206499e+09 INFO controller.tenancyfrontend Target secret appid.discovery-endpoint exists, updating it now {"reconciler group": "multitenancy.example.net", "reconciler kind": "TenancyFrontend", "name": "tenancyfrontend-sample", "namespace": "default"}
1.6450307442127218e+09 INFO controller.tenancyfrontend Just return nil {"reconciler group": "multitenancy.example.net", "reconciler kind": "TenancyFrontend", "name": "tenancyfrontend-sample", "namespace": "default"}
In the following gif you see the created resources created by the operator.

Access the example frontend application¶
Step 1: Get the node port of the service-frontend
¶
export NODEPORT=$(kubectl get svc service-frontend -o=jsonpath='{.spec.ports[0].nodePort}')
echo $NODEPORT
Step 2: Get the worker node IP of the Kubernetes cluster¶
WORKERNODEIP=$(ibmcloud ks workers --cluster YOUR_CLUSTER | awk '/Ready/ {print $2;exit;}')echo $WORKERNODEIP
Step 3: Open the example application in the browser¶
You will notice this will not work!
open http://$WORKERNODEIP:$NODEPORT
- Example output:

Step 4: Delete the example frontend application¶
Delete the example custom resource object for our example CRD.
kubectl delete -f config/samples/multitenancy_v1alpha1_tenancyfrontend.yaml
Summary
Also, even for a simple deletion of Kubernetes resources/objects which we create within our GO operator for a Custom Resource Object, we have seen, there are always multiple ways to implement the same functionality in different ways. Surely the easy way is the best way for our example use case, because we don’t need to implement additional functionality, but the harder way shows how we can handle the Kubernetes resources/objects to achieve the same goal, delete the objects we created within our operator. That can help us in the future to manage other objects or resources we maybe wanna change.
I hope this was useful to you and let’s see what’s next?
Greetings,
Thomas
#operator, #go, #operatorsdk, #kubernetes, #buildlabs4saas, #operatorlearningjourney
Leave a Reply