This post will provide the procedure and examples to enable the automatic namespace creation in the Kubernetes cluster. By default, if a user requests an object creation in any non-existing namespace, the request will fail. However, this default behavior of the API-Server(Kubernetes) can be altered by an admission plugin called NamesapceAutoProvision.
The Default Behaviour:
list the existing namespaces in the cluster
kubectl get ns
NAME STATUS AGE
default Active 33d
falco Active 32d
jenkins Active 5d3h
kube-node-lease Active 33d
kube-public Active 33d
kube-system Active 33d
nfs-prov Active 33d
Creating a pod in a non-existing namespace(before enabling admission plugin)
Note that the namespace “technekey” is not present in the above list of namespaces
kubectl run foo --image=nginx --namespace technekey
Error from server (NotFound): namespaces "technekey" not foundEnabling the NamesapceAutoProvision Admission plugin
This admission controller examines all incoming requests on namespaced resources and checks if the referenced namespace does exist. It creates a namespace if it cannot be found. This admission controller is useful in deployments that do not want to restrict the creation of a namespace prior to its usage. You can find more about admission plugins here. To enable this admission plugin, you must have enough permissions to edit the API-server manifest file. It would help if you waited for ~5-10 minutes before starting to create pods.
vi /etc/kubernetes/manifests/kube-apiserver.yaml
- - --enable-admission-plugins=NodeRestriction
+ - --enable-admission-plugins=NodeRestriction,NamespaceAutoProvision
Creating a pod in a non-existing namespace(after enabling admission-plugin)
kubectl run foo --image=nginx --namespace technekey
pod/foo created
List the Pod status
kubectl get pod,secret,sa -n technekey
NAME READY STATUS RESTARTS AGE
pod/foo 0/1 ContainerCreating 0 1s
NAME TYPE DATA AGE
secret/default-token-4snp5 kubernetes.io/service-account-token 3 1s
NAME SECRETS AGE
serviceaccount/default 1 1s
The exact sequence is also noted in the audit log(filtered snippet below); you may note that resource quota, service account, and secret are auto-created.
{
"kind": "Event",
"apiVersion": "audit.k8s.io/v1",
"level": "Metadata",
"auditID": "9d50a431-4480-4980-8e28-4ef86b061845",
"stage": "ResponseComplete",
"requestURI": "/api/v1/namespaces",
"verb": "create",
"user": {
"username": "system:apiserver",
"uid": "267b5496-1d31-4473-a9dc-f7b5624386e1",
"groups": [
"system:masters"
]
},
"sourceIPs": [
"::1"
],
"userAgent": "Go-http-client/2.0",
"objectRef": {
"resource": "namespaces",
"name": "technekey",
"apiVersion": "v1"
},
"responseStatus": {
"metadata": {},
"code": 201
},
"requestReceivedTimestamp": "2022-05-02T22:32:56.988401Z",
"stageTimestamp": "2022-05-02T22:32:56.991825Z",
"annotations": {
"authorization.k8s.io/decision": "allow",
"authorization.k8s.io/reason": ""
}
}
{
"kind": "Event",
"apiVersion": "audit.k8s.io/v1",
"level": "Metadata",
"auditID": "048f7d30-21f0-416f-8e85-05dbc3c24e3f",
"stage": "ResponseComplete",
"requestURI": "/api/v1/namespaces/technekey/serviceaccounts",
"verb": "create",
"user": {
"username": "system:serviceaccount:kube-system:service-account-controller",
"uid": "4a72d06e-b451-485e-91f5-355a8ed4b67a",
"groups": [
"system:serviceaccounts",
"system:serviceaccounts:kube-system",
"system:authenticated"
]
},
"sourceIPs": [
"127.0.0.1"
],
"userAgent": "kube-controller-manager/v1.23.3 (linux/amd64) kubernetes/816c97a/system:serviceaccount:kube-system:service-account-controller",
"objectRef": {
"resource": "serviceaccounts",
"namespace": "technekey",
"name": "default",
"apiVersion": "v1"
},
"responseStatus": {
"metadata": {},
"code": 201
},
"requestReceivedTimestamp": "2022-05-02T22:32:57.011600Z",
"stageTimestamp": "2022-05-02T22:32:57.014688Z",
"annotations": {
"authorization.k8s.io/decision": "allow",
"authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"system:controller:service-account-controller\" of ClusterRole \"system:controller:service-account-controller\" to ServiceAccount \"service-account-controller/kube-system\""
}
}
{
"kind": "Event",
"apiVersion": "audit.k8s.io/v1",
"level": "Metadata",
"auditID": "0b0968d8-d7c1-40a0-8451-9258f970d798",
"stage": "ResponseComplete",
"requestURI": "/api/v1/namespaces/technekey/configmaps",
"verb": "create",
"user": {
"username": "system:serviceaccount:kube-system:root-ca-cert-publisher",
"uid": "844fdd2f-7029-4b44-bd50-68857fa7a113",
"groups": [
"system:serviceaccounts",
"system:serviceaccounts:kube-system",
"system:authenticated"
]
},
"sourceIPs": [
"127.0.0.1"
],
"userAgent": "kube-controller-manager/v1.23.3 (linux/amd64) kubernetes/816c97a/system:serviceaccount:kube-system:root-ca-cert-publisher",
"objectRef": {
"resource": "configmaps",
"namespace": "technekey",
"name": "kube-root-ca.crt",
"apiVersion": "v1"
},
"responseStatus": {
"metadata": {},
"code": 201
},
"requestReceivedTimestamp": "2022-05-02T22:32:57.012287Z",
"stageTimestamp": "2022-05-02T22:32:57.015934Z",
"annotations": {
"authorization.k8s.io/decision": "allow",
"authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"system:controller:root-ca-cert-publisher\" of ClusterRole \"system:controller:root-ca-cert-publisher\" to ServiceAccount \"root-ca-cert-publisher/kube-system\""
}
}
{
"kind": "Event",
"apiVersion": "audit.k8s.io/v1",
"level": "Metadata",
"auditID": "4205b4ea-3893-4f0e-9692-ee1a4362ec06",
"stage": "ResponseComplete",
"requestURI": "/api/v1/namespaces/technekey/secrets",
"verb": "create",
"user": {
"username": "system:kube-controller-manager",
"groups": [
"system:authenticated"
]
},
"sourceIPs": [
"127.0.0.1"
],
"userAgent": "kube-controller-manager/v1.23.3 (linux/amd64) kubernetes/816c97a/tokens-controller",
"objectRef": {
"resource": "secrets",
"namespace": "technekey",
"name": "default-token-4snp5",
"apiVersion": "v1"
},
"responseStatus": {
"metadata": {},
"code": 201
},
"requestReceivedTimestamp": "2022-05-02T22:32:57.020885Z",
"stageTimestamp": "2022-05-02T22:32:57.023019Z",
"annotations": {
"authorization.k8s.io/decision": "allow",
"authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"system:kube-controller-manager\" of ClusterRole \"system:kube-controller-manager\" to User \"system:kube-controller-manager\""
}
}
Important Note:
Immediately after enabling the admission plugin, you will encounter the following error if you attempt to create a pod. I tried the exact five times, and I faced the same error every time. So the solution is to wait for ~5 minutes after enabling the admission plugin before creating any pod/object.
kubectl run foo --image=nginx --namespace non-exiting-pod
Error from server (Forbidden): pods "foo" is forbidden: error looking up service account non-existing-pod/default: serviceaccount "default" not found