This post will show the steps to configure a simple mutation webhook in the Kubernetes cluster. If you are not familiar with webhooks, you can read about them HERE. For this post, I will be using a python(flask) based webserver as a webhook for simplicity.
Objective/Example Usecase:
In this post, we will set up a webhook to ensure that if any namespace is labeled “env: prod,” then any deployment created in such namespace must have at least three replicas. This logic will be plugged into the Kubernetes cluster using the mutation webhook. As seen in the below snippet, the namespace “technekey” is labeled “env: prod.” So, all the deployments in this namespace must have at least three replicas.
kubectl get namespaces --show-labels
NAME STATUS AGE LABELS
default Active 2d17h kubernetes.io/metadata.name=default
kube-node-lease Active 2d17h kubernetes.io/metadata.name=kube-node-lease
kube-public Active 2d17h kubernetes.io/metadata.name=kube-public
kube-system Active 2d17h kubernetes.io/metadata.name=kube-system
technekey Active 44h kubernetes.io/metadata.name=technekey,env=prod
Notes:
- We are using self-signed certificates created using OpenSSL in this post. Feel free to use cert-manager if you can with CA Injection annotation.
- This post assumes that you already understand what a webhook is and what a mutation webhook is.
- Basic knowledge of JSON and Python(or GoLang) web frameworks is needed. This post uses a Python webserver.
- You can pull the image from This link. https://hub.docker.com/repository/docker/technekey/mutate-prod-deployment-replicas
- The code for this tutorial is located at https://github.com/technekey/mutation-webhook-example
- For following along the steps, you must have ssh access to controller node and sudo permissions to update the kube-api server conf. That is not possible for managed clusters.
Step-1: Setup the variable names
export DOMAIN=mutation-webhook #webhook name
export NAMESPACE=custom-webhooks #webhook namespace
export VALIDITY_DAYS=3650 #Validity of TLS certs
export KUBECONFIG_PATH_MASTER_NODE=/etc/kubernetes/webhook-pki/ #Master node path to hostPath step-2: create the webhook namespace
kubectl create ns "${NAMESPACE}"
step-3: Create CA KEY and CERT
openssl genrsa -out ${DOMAIN}_CA.key 4096 &>/dev/null
openssl req -new -x509 -days "${VALIDITY_DAYS}" -key ${DOMAIN}_CA.key -subj "/CN=${DOMAIN}.${NAMESPACE}.svc" -out ${DOMAIN}_CA.crt
step-4: create the CSR
openssl req -newkey rsa:4096 -nodes -keyout ${DOMAIN}.key -subj "/CN=${DOMAIN}.${NAMESPACE}.svc" -out ${DOMAIN}.csr
step-5: self sign the CSR
openssl x509 -req -extfile <(printf "subjectAltName=DNS:${DOMAIN}.${NAMESPACE}.svc,DNS:${DOMAIN}.${NAMESPACE}.svc.cluster.local") -days "${VALIDITY_DAYS}" -in ${DOMAIN}.csr -CA ${DOMAIN}_CA.crt -CAkey ${DOMAIN}_CA.key -CAcreateserial -out ${DOMAIN}.crt &>/dev/null
step-6: store the cert and key in a secret
kubectl create secret tls ${DOMAIN}-webhook-secret -n ${NAMESPACE} --cert "${DOMAIN}.crt" --key "${DOMAIN}.key"
step-7: generate the kubeconfig for webhook
This kubeconfig will be used by the webserver to authenticate to the API Server over TLS.
cat <<EOF | tee ${DOMAIN}_kubeconfig.yml &>/dev/null
apiVersion: v1
kind: Config
users:
- name: "${DOMAIN}.${NAMESPACE}.svc"
user:
client-certificate-data: "$(cat ${DOMAIN}.crt|base64 |tr -d '\n')"
client-key-data: "$(cat ${DOMAIN}.key|base64 |tr -d '\n')"
EOF
step-8: generate the admission configuration
This file is supplied to the API Server using the –admission-control-config-file flag. Here, This file essentially tells the API Server the location of the Kube-config file of the plugin type MutatingAdmissionWebhook. We will enable this flag later in the procedure.
cat << EOF |tee ${DOMAIN}_AdmissionConfiguration.yml &>/dev/null
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: MutatingAdmissionWebhook
configuration:
apiVersion: apiserver.config.k8s.io/v1
kind: WebhookAdmissionConfiguration
kubeConfigFile: "${KUBECONFIG_PATH_MASTER_NODE}/${DOMAIN}/${DOMAIN}_kubeconfig.yml"
EOF
step-9: create the mutatingwebhookconfiguration
This object makes API Server aware of the Location(FQDN), port of the webhook, the route of the webhook to check, trigger, impacted resources, failure policy, namespace selector, etc. see comments.
cat << EOF | tee ${DOMAIN}_MutatingWebhookConfiguration.yml &>/dev/null
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
name: ${DOMAIN}
webhooks:
- admissionReviewVersions:
- v1
clientConfig:
caBundle: "$(cat ${DOMAIN}_CA.crt |base64 -w0)"
service:
name: ${DOMAIN}
namespace: ${NAMESPACE}
path: /mutate/deployments/replicas # send the request to the webhook at this path. Eg: example.com//mutate/deployments
port: 443
failurePolicy: Fail # if the webhook is unavailable, decision is to fail.
matchPolicy: Equivalent
name: ${DOMAIN}.${NAMESPACE}.svc
namespaceSelector:
matchLabels:
env: prod # this webhook will called only when the operations are done at the namespaces with following labels. use {} for all.
objectSelector: {}
rules:
- apiGroups:
- apps
apiVersions:
- v1
operations: # send the request to webhook when the following operation is requested.
- CREATE
resources:
- deployments # The validation webhook is valid for the following type of resources.
# * means: for all namespaced and cluster scoped resources
# Namespaced means: for all namespaced resources
# Cluster means: for all the clustered scoped resources
scope: Namespaced
sideEffects: None
timeoutSeconds: 5
EOF
step-10: Copy the required files to the master node(ROOT ACCESS NEEDED!)
# copy files to master node. I am copying the files in 2 steps. 1st I will copy them to user's home directory as I cannot directly move them to /etc/kubernetes/webhook-pki/
scp mutation-webhook_kubeconfig.yml mutation-webhook_AdmissionConfiguration.yml [email protected]:
mutation-webhook_kubeconfig.yml 100% 7170 8.0MB/s 00:00
mutation-webhook_AdmissionConfiguration.yml 100% 310 421.7KB/s 00:00
# login to the SSH
ssh [email protected]
# switch to the root user
technekey@kube-master:~$ sudo -i
[sudo] password for technekey:
root@kube-master:~#
root@kube-master:~# mkdir -p /etc/kubernetes/webhook-pki/mutation-webhook/
root@kube-master:~# mv ~technekey/mutation-webhook_kubeconfig.yml /etc/kubernetes/webhook-pki/mutation-webhook/
root@kube-master:~#
root@kube-master:~# mv ~technekey/mutation-webhook_AdmissionConfiguration.yml /etc/kubernetes/webhook-pki/mutation-webhook/
root@kube-master:~#
step-11: Modify the API server to add admission config and hostPath volumes(see example in 2nd snippet)
#Add the following volume, make sure to replace variables with their values in the manifest file.
- name: ${DOMAIN}-admission-plugins-config
hostPath:
path: "${KUBECONFIG_PATH_MASTER_NODE}/${DOMAIN}"
#Add the following MountPath
- mountPath: "${KUBECONFIG_PATH_MASTER_NODE}/${DOMAIN}"
name: ${DOMAIN}-admission-plugins-config
readOnly: true
#example: After variable interpolation.
volumes:
- name: mutation-webhook-admission-plugins-config
hostPath:
path: /etc/kubernetes/webhook-pki/mutation-webhook
....
......
- mountPath: /etc/kubernetes/webhook-pki/mutation-webhook
name: mutation-webhook-admission-plugins-config
readOnly: true
Now add the admissionConfiguration file location to the API Server as an argument.
- --admission-control-config-file=/etc/kubernetes/webhook-pki/mutation-webhook/mutation-webhook_AdmissionConfiguration.yml
step-12: creating mutationwebhookconfiguration
kubectl create -f ${DOMAIN}_MutatingWebhookConfiguration.ymlstep-16: Deploy the webhook
cat << EOF |kubectl create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
labels:
app: ${DOMAIN}
name: ${DOMAIN}
namespace: ${NAMESPACE}
spec:
selector:
matchLabels:
app: ${DOMAIN}
template:
metadata:
labels:
app: ${DOMAIN}
spec:
containers:
- env:
- name: CERT
value: /etc/kubernetes/webhook-pki/mutation-webhook/tls.crt
- name: KEY
value: /etc/kubernetes/webhook-pki/mutation-webhook/tls.key
- name: HOST
value: 0.0.0.0
- name: PORT
value: "8080"
- name: DEBUG
value: "true"
image: technekey/mutate-prod-deployment-replicas:latest
imagePullPolicy: Always
name: ${DOMAIN}
volumeMounts:
- mountPath: ${KUBECONFIG_PATH_MASTER_NODE}/${DOMAIN}
name: ${DOMAIN}-admission-plugins-secret
readOnly: true
dnsPolicy: ClusterFirst
restartPolicy: Always
volumes:
- name: ${DOMAIN}-admission-plugins-secret
secret:
secretName: ${DOMAIN}-webhook-secret
EOF
step-17: expose the deployment(use the same port as step16)
kubectl expose deployment -n ${NAMESPACE} ${DOMAIN} --port 443 --target-port 8080
# validate that pod is running fine.
kubectl get all -n custom-webhooks
NAME READY STATUS RESTARTS AGE
pod/mutation-webhook-7c69469fb9-jqks4 1/1 Running 0 5m25s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/mutation-webhook ClusterIP 10.233.0.200 <none> 443/TCP 7s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/mutation-webhook 1/1 1 1 5m25s
NAME DESIRED CURRENT READY AGE
replicaset.apps/mutation-webhook-7c69469fb9 1 1 1 5m25s
Step-18: Enable the webhook configuration
kubectl create -f ${DOMAIN}_MutatingWebhookConfiguration.ymlstep-19: Time to test
# Create a deployment in technekey namespace(labeled env=prod) with one replica
kubectl create deployment foobar --image=nginx --replicas 1 -n technekey
deployment.apps/foobar created
# notice the replicas are set to 3, but in original request it was set to 1.
kubectl get deployments.apps -n technekey
NAME READY UP-TO-DATE AVAILABLE AGE
foobar 3/3 3 3 13s
Summary
Aggregate Rating
4.5 based on votes