Automatically create Namespace in Kubernetes

This post will provide the procedure and examples to enable the automatic namespace creation in the Kubernetes cluster. By default, if a user requests an object creation in any non-existing namespace, the request will fail. However, this default behavior of the API-Server(Kubernetes) can be altered by an admission plugin called NamesapceAutoProvision.

The Default Behaviour:

list the existing namespaces in the cluster

kubectl get ns
NAME              STATUS   AGE
default           Active   33d
falco             Active   32d
jenkins           Active   5d3h
kube-node-lease   Active   33d
kube-public       Active   33d
kube-system       Active   33d
nfs-prov          Active   33d

Creating a pod in a non-existing namespace(before enabling admission plugin)

Note that the namespace “technekey” is not present in the above list of namespaces

kubectl run foo --image=nginx --namespace technekey
Error from server (NotFound): namespaces "technekey" not found

Enabling the NamesapceAutoProvision Admission plugin

This admission controller examines all incoming requests on namespaced resources and checks if the referenced namespace does exist. It creates a namespace if it cannot be found. This admission controller is useful in deployments that do not want to restrict the creation of a namespace prior to its usage. You can find more about admission plugins here. To enable this admission plugin, you must have enough permissions to edit the API-server manifest file. It would help if you waited for ~5-10 minutes before starting to create pods.

vi /etc/kubernetes/manifests/kube-apiserver.yaml

- - --enable-admission-plugins=NodeRestriction
+ - --enable-admission-plugins=NodeRestriction,NamespaceAutoProvision

Creating a pod in a non-existing namespace(after enabling admission-plugin)

kubectl run foo --image=nginx --namespace technekey
pod/foo created

List the Pod status

kubectl get pod,secret,sa -n technekey
NAME      READY   STATUS              RESTARTS   AGE
pod/foo   0/1     ContainerCreating   0          1s

NAME                         TYPE                                  DATA   AGE
secret/default-token-4snp5   kubernetes.io/service-account-token   3      1s

NAME                     SECRETS   AGE
serviceaccount/default   1         1s

The exact sequence is also noted in the audit log(filtered snippet below); you may note that resource quota, service account, and secret are auto-created. 

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "9d50a431-4480-4980-8e28-4ef86b061845",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces",
  "verb": "create",
  "user": {
    "username": "system:apiserver",
    "uid": "267b5496-1d31-4473-a9dc-f7b5624386e1",
    "groups": [
      "system:masters"
    ]
  },
  "sourceIPs": [
    "::1"
  ],
  "userAgent": "Go-http-client/2.0",
  "objectRef": {
    "resource": "namespaces",
    "name": "technekey",
    "apiVersion": "v1"
  },
  "responseStatus": {
    "metadata": {},
    "code": 201
  },
  "requestReceivedTimestamp": "2022-05-02T22:32:56.988401Z",
  "stageTimestamp": "2022-05-02T22:32:56.991825Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": ""
  }
}

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "048f7d30-21f0-416f-8e85-05dbc3c24e3f",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/technekey/serviceaccounts",
  "verb": "create",
  "user": {
    "username": "system:serviceaccount:kube-system:service-account-controller",
    "uid": "4a72d06e-b451-485e-91f5-355a8ed4b67a",
    "groups": [
      "system:serviceaccounts",
      "system:serviceaccounts:kube-system",
      "system:authenticated"
    ]
  },
  "sourceIPs": [
    "127.0.0.1"
  ],
  "userAgent": "kube-controller-manager/v1.23.3 (linux/amd64) kubernetes/816c97a/system:serviceaccount:kube-system:service-account-controller",
  "objectRef": {
    "resource": "serviceaccounts",
    "namespace": "technekey",
    "name": "default",
    "apiVersion": "v1"
  },
  "responseStatus": {
    "metadata": {},
    "code": 201
  },
  "requestReceivedTimestamp": "2022-05-02T22:32:57.011600Z",
  "stageTimestamp": "2022-05-02T22:32:57.014688Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"system:controller:service-account-controller\" of ClusterRole \"system:controller:service-account-controller\" to ServiceAccount \"service-account-controller/kube-system\""
  }
}
{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "0b0968d8-d7c1-40a0-8451-9258f970d798",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/technekey/configmaps",
  "verb": "create",
  "user": {
    "username": "system:serviceaccount:kube-system:root-ca-cert-publisher",
    "uid": "844fdd2f-7029-4b44-bd50-68857fa7a113",
    "groups": [
      "system:serviceaccounts",
      "system:serviceaccounts:kube-system",
      "system:authenticated"
    ]
  },
  "sourceIPs": [
    "127.0.0.1"
  ],
  "userAgent": "kube-controller-manager/v1.23.3 (linux/amd64) kubernetes/816c97a/system:serviceaccount:kube-system:root-ca-cert-publisher",
  "objectRef": {
    "resource": "configmaps",
    "namespace": "technekey",
    "name": "kube-root-ca.crt",
    "apiVersion": "v1"
  },
  "responseStatus": {
    "metadata": {},
    "code": 201
  },
  "requestReceivedTimestamp": "2022-05-02T22:32:57.012287Z",
  "stageTimestamp": "2022-05-02T22:32:57.015934Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"system:controller:root-ca-cert-publisher\" of ClusterRole \"system:controller:root-ca-cert-publisher\" to ServiceAccount \"root-ca-cert-publisher/kube-system\""
  }
}
{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "4205b4ea-3893-4f0e-9692-ee1a4362ec06",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/technekey/secrets",
  "verb": "create",
  "user": {
    "username": "system:kube-controller-manager",
    "groups": [
      "system:authenticated"
    ]
  },
  "sourceIPs": [
    "127.0.0.1"
  ],
  "userAgent": "kube-controller-manager/v1.23.3 (linux/amd64) kubernetes/816c97a/tokens-controller",
  "objectRef": {
    "resource": "secrets",
    "namespace": "technekey",
    "name": "default-token-4snp5",
    "apiVersion": "v1"
  },
  "responseStatus": {
    "metadata": {},
    "code": 201
  },
  "requestReceivedTimestamp": "2022-05-02T22:32:57.020885Z",
  "stageTimestamp": "2022-05-02T22:32:57.023019Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"system:kube-controller-manager\" of ClusterRole \"system:kube-controller-manager\" to User \"system:kube-controller-manager\""
  }
}

Important Note:

Immediately after enabling the admission plugin, you will encounter the following error if you attempt to create a pod. I tried the exact five times, and I faced the same error every time. So the solution is to wait for ~5 minutes after enabling the admission plugin before creating any pod/object.

kubectl  run foo --image=nginx --namespace non-exiting-pod
Error from server (Forbidden): pods "foo" is forbidden: error looking up service account non-existing-pod/default: serviceaccount "default" not found
Summary
Aggregate Rating
4.5 based on 3 votes

Leave a Comment

Your email address will not be published.

Scroll to Top