How to provision Dynamic Volumes(NFS) on a home lab?

On the NFS-Server node:

Create a directory where all the NFS-clients will be storing their data.

sudo mkdir -p "/nfs_exported"
export export_dir="/nfs_exported"

Now install and start NFS-server on the NFS-server node:

sudo apt update
sudo apt install nfs-kernel-server
sudo systemctl start nfs-kernel-server.service
sudo exportfs -ar

Create an Array of NFS-clients

This is a list of nodes I want to have access to the NFS server.

IPS=(192.168.1.132 192.168.1.133 192.168.1.134)

#Alternatively, Here is a hack to get the $IPS array populated automatically, unless you want to do it manually
IFS=$' ' read -r -d '' -a IPS < <( kubectl get nodes -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}' && printf '\0' )

Add the client details in /etc/exports

for node in "${IPS[@]}";do
echo "${export_dir} ${node}(rw,no_subtree_check,no_root_squash)" >> /etc/exports
done

On the NFS-Clients node:

Install the NFS client on all the nfs client nodes:

sudo apt install nfs-common

From any node having kubectl/helm access to kubernetes:

Install helm if not already present:

$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
$ chmod 700 get_helm.sh
$ ./get_helm.sh

Add the nfs-subdir-external-provisioner helm chart:

helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
"nfs-subdir-external-provisioner" has been added to your repositories

Installing the nfs-subdir-external-provisioner chart:

helm install nfs-subdir-external-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
 --set nfs.server=192.168.1.66 \
 --set nfs.path=/nfs_exported \
 --create-namespace \
 --namespace nfs-prov
NAME: nfs-subdir-external-provisioner
LAST DEPLOYED: Tue Feb 15 14:12:14 2022
NAMESPACE: nfs-prov
STATUS: deployed
REVISION: 1
TEST SUITE: None

Verify the installation status:

helm list -n nfs-prov
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
nfs-subdir-external-provisioner nfs-prov 1 2022-02-15 14:12:14.804699104 -0600 CST deployed nfs-subdir-external-provisioner-4.0.16 4.0.2

kubectl get pod -n nfs-prov 
NAME READY STATUS RESTARTS AGE
nfs-subdir-external-provisioner-58d8d9bd6c-gv4w4 1/1 Running 0 65s

kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
nfs-client cluster.local/nfs-subdir-external-provisioner Delete Immediate true 23m

You can verify that the exported directory is present on the NFS-server and also in the POD. Run the following command in the NFS-Server. Notice the prefix “Archived” in the directory name, the volume in the NFS server will be renamed with prefix “Archived” when the pod is deleted. This is done to get the persistence. 

ls -lrt /nfs_exported/
total 8
drwxrwxrwx 2 root root 4096 Mar 30 11:15 archived-default-myclaim-pvc-8749df7f-037e-4987-9a24-6a6ab3b3ae40
drwxrwxrwx 2 root root 4096 Mar 30 11:30 default-myclaim-pvc-618e1b0b-d894-4cb5-b9d6-d3b091f2c713

You can now run another test(optional) to check the sync between the NFS-Server’s content and the client(pod). Here we are writing some text to a shared mountpoint within the container. 

kubectl exec -it mypod -- bash -c 'echo "this is written inside pod" > /var/www/html/foo'

Now, verify that the content is synced in the NFS-Server. Read the file from the NFW-Server. 

cat default-myclaim-pvc-618e1b0b-d894-4cb5-b9d6-d3b091f2c713/foo
this is written inside pod