How to setup a mutation webhook in Kubernetes

This post will show the steps to configure a simple mutation webhook in the Kubernetes cluster. If you are not familiar with webhooks, you can read about them HERE. For this post, I will be using a python(flask) based webserver as a webhook for simplicity.

Objective/Example Usecase:

In this post, we will set up a webhook to ensure that if any namespace is labeled “env: prod,” then any deployment created in such namespace must have at least three replicas. This logic will be plugged into the Kubernetes cluster using the mutation webhook. As seen in the below snippet, the namespace “technekey” is labeled “env: prod.” So, all the deployments in this namespace must have at least three replicas.

kubectl get namespaces  --show-labels
NAME              STATUS   AGE     LABELS
default           Active   2d17h   kubernetes.io/metadata.name=default
kube-node-lease   Active   2d17h   kubernetes.io/metadata.name=kube-node-lease
kube-public       Active   2d17h   kubernetes.io/metadata.name=kube-public
kube-system       Active   2d17h   kubernetes.io/metadata.name=kube-system
technekey         Active   44h     kubernetes.io/metadata.name=technekey,env=prod

Notes:

  1. We are using self-signed certificates created using OpenSSL in this post. Feel free to use cert-manager if you can with CA Injection annotation.
  2. This post assumes that you already understand what a webhook is and what a mutation webhook is.
  3. Basic knowledge of JSON and Python(or GoLang) web frameworks is needed. This post uses a Python webserver.
  4. You can pull the image from This link. https://hub.docker.com/repository/docker/technekey/mutate-prod-deployment-replicas
  5. The code for this tutorial is located at https://github.com/technekey/mutation-webhook-example
  6. For following along the steps, you must have ssh access to controller node and sudo permissions to update the kube-api server conf. That is not possible for managed clusters.

Step-1: Setup the variable names

export DOMAIN=mutation-webhook                                           #webhook name
export NAMESPACE=custom-webhooks                                         #webhook namespace
export VALIDITY_DAYS=3650                                                #Validity of TLS certs
export KUBECONFIG_PATH_MASTER_NODE=/etc/kubernetes/webhook-pki/ #Master node path to hostPath   

step-2: create the webhook namespace

kubectl create ns "${NAMESPACE}"

step-3: Create CA KEY and CERT

openssl genrsa -out ${DOMAIN}_CA.key 4096 &>/dev/null

openssl req -new -x509 -days "${VALIDITY_DAYS}" -key ${DOMAIN}_CA.key -subj "/CN=${DOMAIN}.${NAMESPACE}.svc" -out ${DOMAIN}_CA.crt

step-4: create the CSR

openssl req -newkey rsa:4096 -nodes -keyout ${DOMAIN}.key -subj "/CN=${DOMAIN}.${NAMESPACE}.svc" -out ${DOMAIN}.csr

step-5: self sign the CSR

openssl x509 -req -extfile <(printf "subjectAltName=DNS:${DOMAIN}.${NAMESPACE}.svc,DNS:${DOMAIN}.${NAMESPACE}.svc.cluster.local") -days "${VALIDITY_DAYS}" -in ${DOMAIN}.csr -CA ${DOMAIN}_CA.crt -CAkey ${DOMAIN}_CA.key -CAcreateserial -out ${DOMAIN}.crt &>/dev/null

step-6: store the cert and key in a secret

kubectl create secret tls ${DOMAIN}-webhook-secret -n ${NAMESPACE} --cert "${DOMAIN}.crt" --key "${DOMAIN}.key"

step-7: generate the kubeconfig for webhook

This kubeconfig will be used by the webserver to authenticate to the API Server over TLS.

cat <<EOF | tee ${DOMAIN}_kubeconfig.yml &>/dev/null
apiVersion: v1
kind: Config
users:
- name: "${DOMAIN}.${NAMESPACE}.svc"
  user:
    client-certificate-data: "$(cat ${DOMAIN}.crt|base64 |tr -d '\n')"
    client-key-data: "$(cat ${DOMAIN}.key|base64 |tr -d '\n')"
EOF

step-8: generate the admission configuration

This file is supplied to the API Server using the –admission-control-config-file flag. Here, This file essentially tells the API Server the location of the Kube-config file of the plugin type MutatingAdmissionWebhook. We will enable this flag later in the procedure.

cat << EOF |tee ${DOMAIN}_AdmissionConfiguration.yml &>/dev/null
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: MutatingAdmissionWebhook
  configuration:
    apiVersion: apiserver.config.k8s.io/v1
    kind: WebhookAdmissionConfiguration
    kubeConfigFile: "${KUBECONFIG_PATH_MASTER_NODE}/${DOMAIN}/${DOMAIN}_kubeconfig.yml"
EOF

step-9: create the mutatingwebhookconfiguration

This object makes API Server aware of the Location(FQDN), port of the webhook, the route of the webhook to check, trigger, impacted resources, failure policy, namespace selector, etc. see comments.

cat << EOF | tee ${DOMAIN}_MutatingWebhookConfiguration.yml &>/dev/null
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: ${DOMAIN}
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    caBundle: "$(cat ${DOMAIN}_CA.crt  |base64 -w0)"
    service:
      name: ${DOMAIN}
      namespace: ${NAMESPACE}
      path: /mutate/deployments/replicas              # send the request to the  webhook at this path. Eg: example.com//mutate/deployments
      port: 443
  failurePolicy: Fail                        # if the webhook is unavailable, decision is to fail.
  matchPolicy: Equivalent
  name: ${DOMAIN}.${NAMESPACE}.svc
  namespaceSelector:
    matchLabels:
      env: prod                              # this webhook will called only when the operations are done at the namespaces with following labels. use {} for all.
  objectSelector: {}
  rules:
  - apiGroups:
    - apps
    apiVersions:
    - v1
    operations:           # send the request to webhook when the following operation is requested.
    - CREATE
    resources:
    - deployments         # The validation webhook is valid for the following type of resources.
    # * means:            for all namespaced and cluster scoped resources
    # Namespaced means:   for all namespaced resources
    # Cluster means:      for all the clustered scoped resources
    scope: Namespaced
  sideEffects: None
  timeoutSeconds: 5
EOF

step-10: Copy the required files to the master node(ROOT ACCESS NEEDED!)

# copy files to master node. I am copying the files in 2 steps. 1st I will copy them to user's home directory as I cannot directly move them to /etc/kubernetes/webhook-pki/

scp mutation-webhook_kubeconfig.yml mutation-webhook_AdmissionConfiguration.yml [email protected]:
mutation-webhook_kubeconfig.yml                                                                                                  100% 7170     8.0MB/s   00:00
mutation-webhook_AdmissionConfiguration.yml                                                                                      100%  310   421.7KB/s   00:00


# login to the SSH

ssh [email protected]


# switch to the root user

[email protected]:~$ sudo -i
[sudo] password for technekey:
[email protected]:~#


[email protected]:~# mkdir -p  /etc/kubernetes/webhook-pki/mutation-webhook/

[email protected]:~# mv ~technekey/mutation-webhook_kubeconfig.yml /etc/kubernetes/webhook-pki/mutation-webhook/
[email protected]:~#
[email protected]:~# mv ~technekey/mutation-webhook_AdmissionConfiguration.yml /etc/kubernetes/webhook-pki/mutation-webhook/
[email protected]:~#

step-11: Modify the API server to add admission config and hostPath volumes(see example in 2nd snippet)

#Add the following volume, make sure to replace variables with their values in the manifest file.  

- name: ${DOMAIN}-admission-plugins-config
    hostPath:
      path: "${KUBECONFIG_PATH_MASTER_NODE}/${DOMAIN}"

#Add the following MountPath

 - mountPath: "${KUBECONFIG_PATH_MASTER_NODE}/${DOMAIN}"
   name: ${DOMAIN}-admission-plugins-config
   readOnly: true
  #example: After variable interpolation. 

  volumes:
  - name: mutation-webhook-admission-plugins-config
    hostPath:
      path: /etc/kubernetes/webhook-pki/mutation-webhook

....
......

    - mountPath: /etc/kubernetes/webhook-pki/mutation-webhook
      name: mutation-webhook-admission-plugins-config
      readOnly: true

Now add the admissionConfiguration file location to the API Server as an argument.

    - --admission-control-config-file=/etc/kubernetes/webhook-pki/mutation-webhook/mutation-webhook_AdmissionConfiguration.yml

step-12: creating mutationwebhookconfiguration

kubectl create  -f ${DOMAIN}_MutatingWebhookConfiguration.yml

step-16: Deploy the webhook

cat << EOF |kubectl create  -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
  labels:
    app: ${DOMAIN}
  name: ${DOMAIN}
  namespace: ${NAMESPACE}
spec:
  selector:
    matchLabels:
      app: ${DOMAIN}
  template:
    metadata:
      labels:
        app: ${DOMAIN}
    spec:
      containers:
      - env:
        - name: CERT
          value: /etc/kubernetes/webhook-pki/mutation-webhook/tls.crt
        - name: KEY
          value: /etc/kubernetes/webhook-pki/mutation-webhook/tls.key
        - name: HOST
          value: 0.0.0.0
        - name: PORT
          value: "8080"
        - name: DEBUG
          value: "true"
        image: technekey/mutate-prod-deployment-replicas:latest
        imagePullPolicy: Always
        name: ${DOMAIN}
        volumeMounts:
        - mountPath: ${KUBECONFIG_PATH_MASTER_NODE}/${DOMAIN}
          name: ${DOMAIN}-admission-plugins-secret
          readOnly: true
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      volumes:
      - name: ${DOMAIN}-admission-plugins-secret
        secret:
          secretName: ${DOMAIN}-webhook-secret
EOF

step-17: expose the deployment(use the same port as step16)

 kubectl expose deployment  -n ${NAMESPACE}  ${DOMAIN}  --port 443 --target-port 8080


# validate that pod is running fine.

kubectl get all -n custom-webhooks
NAME                                    READY   STATUS    RESTARTS   AGE
pod/mutation-webhook-7c69469fb9-jqks4   1/1     Running   0          5m25s

NAME                       TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/mutation-webhook   ClusterIP   10.233.0.200   <none>        443/TCP   7s

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mutation-webhook   1/1     1            1           5m25s

NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/mutation-webhook-7c69469fb9   1         1         1       5m25s

Step-18: Enable the webhook configuration

kubectl create -f ${DOMAIN}_MutatingWebhookConfiguration.yml

step-19: Time to test

# Create a deployment in technekey namespace(labeled env=prod) with one replica

kubectl  create deployment  foobar --image=nginx --replicas 1 -n technekey
deployment.apps/foobar created

# notice the replicas are set to 3, but in original request it was set to 1.

kubectl get deployments.apps  -n technekey
NAME     READY   UP-TO-DATE   AVAILABLE   AGE
foobar   3/3     3            3           13s
Summary
Aggregate Rating
4 based on 3 votes

Leave a Comment

Your email address will not be published.

Scroll to Top