Enforcing the Container Image Download Behavior in Kubernetes

In this post, we will see a few ways to control and enforce the container image download behavior of Kubernetes(kubelet). We will also see the various examples with different Image Pull policies used.

The Default behavior

When a pod creation is requested, it will be scheduled on the most suitable node by the Kube-scheduler—the kubelet running on that node would query the container image registry to resolve the name to an image digest. If the kubelet has a container image with that exact digest cached locally, the kubelet uses its cached image; otherwise, the kubelet pulls the image with the resolved digest and uses it to launch the container.

Changing the default behavior

You may change the default behavior by modifying the Pod manifest file’s .spec.contianers[*].imagePullPolicy field with one of the following three values in column 1 of the below table:

Image Pull PolicyIs this Default PolicyBehavior
AlwaysYESWhen a pod creation is requested, it will be scheduled on a node by default. Then, the kubelet running on that node queries the container image registry to resolve the name to an image digest. If the kubelet has a container image with that exact digest cached locally, the kubelet uses its cached image; otherwise, the kubelet pulls the image with the resolved digest and uses it to launch the container.
IfNotPresentNOThe image is pulled only if it is not already present locally.
NeverNOThe kubelet does not try fetching the image. If the image is somehow already present locally, the kubelet attempts to start the container; otherwise, the startup fails. See pre-pulled images for more details.
Image Pull Policies and their behavior

The following examples demonstrate how changing the image pull policy would change the way image downloading would work in Kubernetes.

Example-1: Using ‘Never’ as an image pull policy when an image is absent on the scheduled node

POD-MANIFESTInitial State of the image on the Scheduled node
(Eg: kube-worker-1)
Image state after on the scheduled node
after pod creation request(Eg: kube-worker-1)
Pod Status after Creation request
apiVersion: v1
kind: Pod
metadata:
labels:
run: my-pod
name: my-pod
spec:
containers:
- image: nginx

name: my-pod
imagePullPolicy: Never
resources: {}
dnsPolicy: ClusterFirst
restartPolicy: Always
nodeName: kube-worker-1
IMAEGE IS NOT PRESENT ON THE NODE IMAGE IS NOT PRESENT ON THE NODENAME READY STATUS RESTARTS AGE
my-pod 0/1 ErrImageNeverPull 0 2s



sudo crictl images |grep -i nginx
sudo crictl images |grep -i nginx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ErrImageNeverPull 10s (x3 over 25s) kubelet Container image "nginx" is not present with pull policy of Never
Warning Failed 10s (x3 over 25s) kubelet Error: ErrImageNeverPull

Example-2: Using ‘Never’ as a pull policy when an image is already present on the scheduled node

POD-MANIFESTInitial State of the image
on the Scheduled node
(Eg: kube-worker-1)
Image state on the scheduled
node after pod creation
request(Eg: kube-worker-1)
Pod Status after Creation request
apiVersion: v1
kind: Pod
metadata:
labels:
run: my-pod
name: my-pod
spec:
containers:
- image: nginx

name: my-pod
imagePullPolicy: Never
resources: {}
dnsPolicy: ClusterFirst
restartPolicy: Always
nodeName: kube-worker-1
IMAEGE IS PRESENT ON THE NODE IMAGE IS PRESENT ON THE NODE




kubectl get pod
NAME READY STATUS RESTARTS AGE
my-pod 1/1 Running 0 58s



sudo crictl images |grep -i nginx
docker.io/library/nginx latest fa5269854a5e6 56.7MB
sudo crictl images |grep -i nginx
docker.io/library/nginx latest fa5269854a5e6 56.7MB
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulled 11s kubelet Container image "nginx" already present on machine
Normal Created 11s kubelet Created container my-pod
Normal Started 11s kubelet Started container my-pod

Example-3: Using IfNotPresent as an image pull policy

POD-MANIFESTInitial State of the image on the Scheduled node
(Eg: kube-worker-1)
Image state after on the scheduled node
after pod creation request(Eg: kube-worker-1)
Pod Status after Creation request
apiVersion: v1
kind: Pod
metadata:
labels:
run: my-pod
name: my-pod
spec:
containers:
- image: nginx

name: my-pod
imagePullPolicy: IfNotPresent
resources: {}
dnsPolicy: ClusterFirst
restartPolicy: Always
nodeName: kube-worker-1
IMAEGE IS NOT PRESENT ON THE NODE IMAGE IS PRESENT ON THE NODEkubectl get pod
NAME READY STATUS RESTARTS AGE
my-pod 1/1 Running 0 2m18s
sudo crictl images |grep -i nginx
sudo crictl images |grep -i nginx
docker.io/library/nginx latest fa5269854a5e6 56.7MB
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulling 6s kubelet Pulling image "nginx"
Normal Pulled 1s kubelet Successfully pulled image "nginx" in 5.133240384s
Normal Created 1s kubelet Created container my-pod
Normal Started 0s kubelet Started container my-pod

Example-4: Using Always as an image pull policy

POD-MANIFESTInitial State of the image on the Scheduled node
(Eg: kube-worker-1)
Image state after on the scheduled node
after pod creation request(Eg: kube-worker-1)
Pod Status after Creation request
apiVersion: v1
kind: Pod
metadata:
labels:
run: my-pod
name: my-pod
spec:
containers:
- image: nginx

name: my-pod
imagePullPolicy: Always
resources: {}
dnsPolicy: ClusterFirst
restartPolicy: Always
nodeName: kube-worker-1
IMAEGE IS PRESENT ON THE NODE IMAGE IS PRESENT ON THE NODEkubectl get pod
NAME READY STATUS RESTARTS AGE
my-pod 1/1 Running 0 114s
sudo crictl images |grep -i nginx
docker.io/library/nginx latest fa5269854a5e6 56.7MB
sudo crictl images |grep -i nginx
docker.io/library/nginx latest fa5269854a5e6 56.7MB
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulling 7s kubelet Pulling image "nginx"
Normal Pulled 6s kubelet Successfully pulled image "nginx" in 505.79813ms

Normal Created 6s kubelet Created container my-pod
Normal Started 6s kubelet Started container my-pod

Pre-downloaded images vs downloading from the registry

Using the image from the RegistryUsing locally stored images
1. It will be our responsibility to secure and manage the authentication to the registry. This is done via imagePullSecrets.
2. Downloading the image from the registry would consume some time(ms to minutes)
3. Users/admin does not(at least lesser management) need to manage the images on local nodes. If an image is missing, it will be downloaded.
4. Extremely suited for CI/CD environment.
1. No need to manage the authentication to the registry.
2. Image downloading time is saved.
3. The worst part is managing the images locally, keeping them updated and secured.
4. Doing the cleanup and upload of the image need to be done manually. Not very compatible with CI/CD.

The disadvantage of the above policies?

  • The cluster Admin is not able to control what imagePullPolicy a rogue developer is using while creating the pod.
  • The different developers may use different imagePullPolicy, bad for consistency.

Enforcing imagePullPolicy ?

To maintain the consistency in the cluster and ensure that every developer is pulling the image from the registry and not using the locally cached image we can enable the AlwaysPullImage admission plugin. This plugin will overwrite the ImagePullPolicy present in the request and mutate it to “Always” download the image before processing the request. You must have admin privileges to enable any admission plugin. Add the “AlwaysPullImages” to your list of admission plugins in the API Server manifest file.

vim /etc/kubernetes/manifests/kube-apiserver.yaml
- - --enable-admission-plugins=NodeRestriction,NamespaceAutoProvision
+ - --enable-admission-plugins=NodeRestriction,NamespaceAutoProvision,AlwaysPullImages
POD-MANIFESTInitial State of the image
on the Scheduled node
(Eg: kube-worker-1)
Image state on the scheduled
node after pod creation
request(Eg: kube-worker-1)
Pod Status after Creation request
apiVersion: v1
kind: Pod
metadata:
labels:
run: my-pod
name: my-pod
spec:
containers:
- image: nginx

name: my-pod
imagePullPolicy: Never
resources: {}
dnsPolicy: ClusterFirst
restartPolicy: Always
nodeName: kube-worker-1
IMAEGE IS NOT PRESENT ON THE NODE IMAGE IS PRESENT ON THE NODE




kubectl get pod
NAME READY STATUS RESTARTS AGE
my-pod 1/1 Running 0 28s



sudo crictl images |grep -i nginx
sudo crictl images |grep -i nginx
docker.io/library/nginx latest fa5269854a5e6 56.7MB
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulling 10m kubelet Pulling image "nginx"
Normal Pulled 10m kubelet Successfully pulled image "nginx" in 5.124016494s

Normal Created 10m kubelet Created container my-pod
Normal Started 10m kubelet Started container my-pod

You may also check the ImagePullPolicy of the POD, it will be set to always regardless of being set to Never in the manifest at the time of creation.

kubectl get pod my-pod  -o jsonpath='{.spec.containers[0].imagePullPolicy}{"\n"}'
Always
Summary
Aggregate Rating
5 based on 1 votes

Leave a Comment

Your email address will not be published.

Scroll to Top