How to setup a validation webhook in Kubernetes?

Leave a Comment / Kubernetes, tech

This post describes how to write and configure a Kubernetes validation webhook in a Kubernetes cluster. The details about “what is a validation webhook” is described here. You can find the code used in this post on this  GitHub page.

Pre-requisites:

  • You must have admin privileges in the cluster.
  • Some knowledge of JSON.
  • Some knowledge of Python or GoLang. Depending on your choice of webhook implementation.

Objective: Example policy

  1. Allow the deployment creation only when it contains the “creator: SOME_VALUE” label. If this label is not present, fail the deployment creation.
  2. Limit this restriction to only a selected namespace. (Eg: technekey). This means enforcing the restriction provided in point#1 only in the technekey namespace.

A Webhook example:

To understand what are we doing, we need to see the webhook’s code first. Following is the webhook’s code, this code is written in Python, however, you can write your own implementation as per your choice.
In this flask program(webserver.py), the request object(A JSON) is checked for the “metadata.labels“, if the creator label is present, the program will return True, else it will return False(with some additional error message).

import sys
import os
from flask import Flask, request, jsonify
from pathlib import Path
import jsonpatch
import base64


admission_controller = Flask(__name__)

@admission_controller.before_request
def log_request():
    admission_controller.logger.debug("Request in JSON      %s\n", str(request.get_json()))
    admission_controller.logger.debug("Request data:        %s\n", request.data)
    admission_controller.logger.debug("Request args:        %s\n", request.args)
    admission_controller.logger.debug("Request form:        %s\n", request.form)
    admission_controller.logger.debug("Request ep  :        %s\n", request.endpoint)
    admission_controller.logger.debug("Request method:      %s\n", request.method)
    admission_controller.logger.debug("Request remote_addr: %s\n", request.remote_addr)
    return None

@admission_controller.route("/validate/checklabels", methods=["POST"])
def deployment_webhook():
    request_info = request.get_json()
    current_namespace = request_info["request"]["object"]["metadata"]["namespace"]

    if request_info["request"]["object"]["metadata"]["labels"].get("creator"):
        
        return_allowed = jsonify(
            {
                "apiVersion": request_info.get("apiVersion"),
                "kind": request_info.get("kind"),
                "response": {
                    "uid": request_info["request"].get("uid"),
                    "allowed": True,
                },
            }
          )
        admission_controller.logger.debug("response if allowed %s\n", str(return_allowed.get_json()))
        return return_allowed
    return_denied = jsonify(
        {
            "apiVersion": request_info.get("apiVersion"),
            "kind": request_info.get("kind"),
            "response": {
                "uid": request_info["request"].get("uid"),
                "allowed": False,
                "status": {"message": "Deployment must have 'creator' label to be admitted in this namespace({current_namespace})"},
            },
        }
      )
    admission_controller.logger.debug("response if denied %s\n",str(return_denied.get_json()))
    return return_denied 


if __name__ == "__main__":
    admission_controller.debug =  os.getenv("DEBUG", 'False').lower() in ('true', '1')
    admission_controller.run(
        host='0.0.0.0')

Setup Procedure:

Step-1: Plan where to deploy the webhook


we will deeply the webhook within the cluster with the following configuration, to follow along with the tutorial, I would encourage you to set these variables as per your requirements.

export DOMAIN=validation-webhook                               #name of the webhook service
export NAMESPACE=custom-webhooks                               #namespace of the webhook svc
export VALIDITY_DAYS=365                                       #validity of the certificates
export KUBECONFIG_PATH_MASTER_NODE=/etc/kubernetes/webhook-pki #hostPath for kube-api server

Step-2: Create the namespace

kubectl create ns "${NAMESPACE}"

Step-3: Create CA Key

openssl genrsa -out ${DOMAIN}_CA.key 4096 &>/dev/null

Step-4: Create CA Cert

openssl req -new -x509 -days "${VALIDITY_DAYS}" -key ${DOMAIN}_CA.key -subj "/CN=${DOMAIN}.${NAMESPACE}.svc" -out ${DOMAIN}_CA.crt

Step-5: Create the CSR for the webhook

openssl req -newkey rsa:4096 -nodes -keyout ${DOMAIN}.key -subj "/CN=${DOMAIN}.${NAMESPACE}.svc" -out ${DOMAIN}.csr

Step-6: Sign the CSR with CA cert/key

openssl x509 -req -extfile <(printf "subjectAltName=DNS:${DOMAIN}.${NAMESPACE}.svc,DNS:${DOMAIN}.${NAMESPACE}.svc.cluster.local") -days "${VALIDITY_DAYS}" -in ${DOMAIN}.csr -CA ${DOMAIN}_CA.crt -CAkey ${DOMAIN}_CA.key -CAcreateserial -out ${DOMAIN}.crt &>/dev/null

Step-7(a): Verify that the following files are created

ls -lrt 
total 20
-rw------- 1 technekey technekey 3268 May 9 10:29 validation-webhook_CA.key
-rw-rw-r-- 1 technekey technekey 1887 May 9 10:29 validation-webhook_CA.crt
-rw------- 1 technekey technekey 3272 May 9 10:34 validation-webhook.key
-rw-rw-r-- 1 technekey technekey 1622 May 9 10:34 validation-webhook.csr
-rw-rw-r-- 1 technekey technekey 2009 May 9 10:36 validation-webhook.crt

Step-7(b): Verify that SAN and CN are populated

openssl x509  -in validation-webhook.crt  -text -noout |grep -EA1  'CN =|ubject Alternative Name:'
        Issuer: CN = validation-webhook.custom-webhooks.svc
        Validity
--
        Subject: CN = validation-webhook.custom-webhooks.svc
        Subject Public Key Info:
--
            X509v3 Subject Alternative Name: 
                DNS:validation-webhook.custom-webhooks.svc, DNS:validation-webhook.custom-webhooks.svc.cluster.local

step-8: Store the cert/key in a TLS secret

kubectl create secret tls ${DOMAIN}-webhook-secret -n ${NAMESPACE} --cert "${DOMAIN}.crt" --key "${DOMAIN}.key"

step-9: Create a kubeconfig for webhook

Similar to any other user/client in Kubernetes, the webhook server needs to have its kubeconfig file to talk to the API Server. The following snippet will create the kubeconfig file called ${DOMAIN}_kubeconfig.yml

cat <<EOF | tee ${DOMAIN}_kubeconfig.yml &>/dev/null
apiVersion: v1
kind: Config
users:
- name: "${DOMAIN}.${NAMESPACE}.svc"
  user:
    client-certificate-data: "$(cat ${DOMAIN}.crt|base64 |tr -d '\n')"
    client-key-data: "$(cat ${DOMAIN}.key|base64 |tr -d '\n')"
EOF

step-10: Create an admission configuration file

From the name “AdmissionConfiguration”, the usefulness of this file may not be obvious. Simply said, this file is the way of making Kube-apiserver aware of the path of the kubeconfig file generated in the previous step. We will supply this file as an argument to the API Server using the –admission-control-config-file flag later in the procedure.

cat << EOF |tee ${DOMAIN}_AdmissionConfiguration.yml &>/dev/null
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: ValidatingAdmissionWebhook
  configuration:
    apiVersion: apiserver.config.k8s.io/v1
    kind: WebhookAdmissionConfiguration
    kubeConfigFile: "${KUBECONFIG_PATH_MASTER_NODE}/${DOMAIN}_kubeconfig.yml"
EOF

Step-11: Create validationWebhookConfiguration file

The object ValidatingWebhookConfiguration is used to configure the behavior of the webhook in the cluster. Here, we define the object type (Eg: deployment) that needs to be validated and the trigger-action(CREATE) on which the validation webhook needs to be called.
Here, we can also define the scope of the webhook, like the impacted namespace using namespaceSelector. failure policy means if the webhook itself fails to validate in a scenario like if the webhook is unreachable, then what to do with the REQUEST.

cat << EOF | tee ${DOMAIN}_ValidatingWebhookConfiguration.yml &>/dev/null
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: ${DOMAIN}
webhooks:
- admissionReviewVersions:
  - v1
  clientConfig:
    caBundle: "$(cat ${DOMAIN}_CA.crt  |base64 -w0)" 
    service:
      name: ${DOMAIN}
      namespace: ${NAMESPACE}
      path: /validate/checklabels          # send the request to the webhook at this path. Eg: example.com/validate/checklabels   
      port: 443
  failurePolicy: Fail                     # if the webhook is unavailable, decision is to fail. 
  matchPolicy: Equivalent
  name: ${DOMAIN}.${NAMESPACE}.svc
  namespaceSelector:
    matchLabels:
      webhook-enforced: ok                # this webhook will called only when the operations are done at the namespaces with following labels. use {} for all.
  objectSelector: {}
  rules:
  - apiGroups:
    - apps
    apiVersions:
    - v1
    operations:           # send the request to webhook when the following operation is requested. 
    - CREATE
    resources:
    - deployments         # The validation webhook is valid for the following type of resources.
    # scone * means:            for all namespaced and cluster scoped resources
    # scope Namespaced means:   for all namespaced resources
    # scope Cluster means:      for all the clustered scoped resources
    scope: Namespaced
  sideEffects: None
  timeoutSeconds: 5
EOF

step-12: Validate below files are created

ls -lrt 
total 32
-rw------- 1 technekey technekey 3268 May 9 10:29 validation-webhook_CA.key
-rw-rw-r-- 1 technekey technekey 1887 May 9 10:29 validation-webhook_CA.crt
-rw------- 1 technekey technekey 3272 May 9 10:34 validation-webhook.key
-rw-rw-r-- 1 technekey technekey 1622 May 9 10:34 validation-webhook.csr
-rw-rw-r-- 1 technekey technekey 2009 May 9 10:36 validation-webhook.crt
-rw-rw-r-- 1 technekey technekey 7192 May 9 11:10 validation-webhook_kubeconfig.yml
-rw-rw-r-- 1 technekey technekey  296 May 9 11:10 validation-webhook_AdmissionConfiguration.yml
-rw-rw-r-- 1 technekey technekey 3876 May 9 11:16 validation-webhook_ValidatingWebhookConfiguration.yml

step-13: transfer the kubeconfig and admissionconfiguration to the Master node(root access needed)

scp validation-webhook_kubeconfig.yml  validation-webhook_AdmissionConfiguration.yml  [email protected]:
validation-webhook_kubeconfig.yml                                                                                                                                                                  100% 7192     6.3MB/s   00:00    
validation-webhook_AdmissionConfiguration.yml                                                                                                                                                      100%  296   653.8KB/s   00:00

Now SSH to the master node, switch to the root user and move the files to ${KUBECONFIG_PATH_MASTER_NODE}

echo ${KUBECONFIG_PATH_MASTER_NODE}
/etc/kubernetes/webhook-pki

ssh [email protected]

 sudo -i
[sudo] password for technekey: 
root@kube-master:~# 

root@kube-master:/etc/kubernetes/webhook-pki# mv  ~technekey/validation-webhook_kubeconfig.yml /etc/kubernetes/webhook-pki/
root@kube-master:/etc/kubernetes/webhook-pki# mv ~technekey/validation-webhook_AdmissionConfiguration.yml  /etc/kubernetes/webhook-pki
root@kube-master:/etc/kubernetes/webhook-pki# 
root@kube-master:/etc/kubernetes/webhook-pki# ls -lrt
total 12
-rw-rw-r-- 1 technekey technekey 7192 May 31 16:29 validation-webhook_kubeconfig.yml
-rw-rw-r-- 1 technekey technekey  296 May 31 16:29 validation-webhook_AdmissionConfiguration.yml
root@kube-master:/etc/kubernetes/webhook-pki#

step-14: Add the following hostpath volumes to the API server

NOTE: You must Interpolate the variables to their respective values before adding them to API Server’s manifest file. After adding the below volume info, exit the API Server. It will restart after making the changes.
NOTE: Before making any changes to the API Server’s manifest file, take a backup of the file.

##Add the following volume in the kube-api server pod under the volume list:
 
  - name: admission-plugins-config
    hostPath:
      path: "${KUBECONFIG_PATH_MASTER_NODE}"

## Add the following volume in the VolumeMounts of the kube-api server pod:

    - mountPath: "${KUBECONFIG_PATH_MASTER_NODE}"
      name: admission-plugins-config
      readOnly: true

step-15: DEPLOY the webhook in the custom-webhooks namespace

I have used the image I built using the python webserver as described at the start of this tutorial. The git repo is here.

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: ${DOMAIN}
  name: ${DOMAIN}
  namespace: ${NAMESPACE}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ${DOMAIN}
  template:
    metadata:
      labels:
        app: ${DOMAIN}
    spec:
      containers:
      - name: ${DOMAIN}
        image: technekey/kubernetes-webhooks:latest
        env:
        - name: CERT
          value: ${KUBECONFIG_PATH_MASTER_NODE}/tls.crt
        - name: KEY
          value: ${KUBECONFIG_PATH_MASTER_NODE}/tls.key
        - name: HOST
          value: "0.0.0.0"
        - name: PORT
          value: "8080"
        volumeMounts:
        - mountPath: ${KUBECONFIG_PATH_MASTER_NODE}
          name: admission-plugins-secret
          readOnly: true
      volumes:
      - name: admission-plugins-secret
        secret:
          secretName: ${DOMAIN}-webhook-secret
          optional: false
EOF
deployment.apps/validation-webhook configured

step-16: expose the webhook deployment

kubectl expose deployment  -n ${NAMESPACE} ${DOMAIN} --port 443 --target-port 8080

Step-17: create validation configuration

kubectl create  -f ${DOMAIN}_ValidatingWebhookConfiguration.yml 
validatingwebhookconfiguration.admissionregistration.k8s.io/validation-webhook created

step-18: check the namespace labels

Notice, that the namespace “technekey” is labeled “monitored:ok”, if you check the validation configuration (step-11) object, we have used the namespace selector of “monitored: ok” to limit the effect of the validation webhook to the technekey namespace.

kubectl label ns technekey  monitored=ok --overwrite
kubectl get ns --show-labels 
NAME              STATUS   AGE   LABELS
custom-webhooks   Active   67m   kubernetes.io/metadata.name=custom-webhooks
default           Active   43h   kubernetes.io/metadata.name=default
kube-node-lease   Active   43h   kubernetes.io/metadata.name=kube-node-lease
kube-public       Active   43h   kubernetes.io/metadata.name=kube-public
kube-system       Active   43h   kubernetes.io/metadata.name=kube-system
technekey         Active   23h   kubernetes.io/metadata.name=technekey,monitored=ok

STEP-19: TIME TO TEST

Step-19A: Create a deployment in the default namespace(not selected as per namespaceSelector), with no creator label and working as expected

cat << EOF | kubectl create  -f - 
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: foobar
  name: foobar
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: foobar
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: foobar
    spec:
      containers:
      - image: nginx
        name: nginx
        resources: {}
status: {}
EOF
deployment.apps/foobar created

step-19B: Creating a deployment in the technekey namespace with no creator label, as expected failed, notice the failure message, coming from webhook

cat << EOF | kubectl create  -f - 
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: foobar
  name: foobar
  namespace: technekey
spec:
  replicas: 1
  selector:
    matchLabels:
      app: foobar
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: foobar
    spec:
      containers:
      - image: nginx
        name: nginx
        resources: {}
status: {}
EOF
Error from server: error when creating "STDIN": admission webhook "validation-webhook.custom-webhooks.svc" denied the request: Deployment must have 'creator' label to be admitted in this namespace

step19-c: creating a deployment in the technekey namespace with creator label, deployment is created fine

cat << EOF | kubectl create  -f - 
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: foobar
    creator: myname
  name: foobar
  namespace: technekey
spec:
  replicas: 1
  selector:
    matchLabels:
      app: foobar
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: foobar
    spec:
      containers:
      - image: nginx
        name: nginx
        resources: {}
status: {}
EOF
deployment.apps/foobar created

Related Posts

0 0 votes
Please, Rate this post
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x
Scroll to Top