Kubernetes Notes - Volume

Containers in Kubernetes are ephemeral by default. This means that when a container stops or a Pod is deleted, any data stored inside the container filesystem is lost.

To solve this problem, Kubernetes provides Volumes, which allow data to persist and be shared between containers in a Pod.

Volumes are mounted into containers and provide persistent or shared storage depending on the type used.

How Volume Works

Remember the first example we make, this time we will mount with different type of Kubernetes Volumes.

First let’s demostrate how we declare a volume.

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: nginx
 5spec:
 6  containers:
 7  - name: app
 8    image: nginx
 9    volumeMounts:
10    - mountPath: /usr/share/nginx/html
11      name: web-data
12
13  volumes:
14  - name: web-data
15    emptyDir: {}

volumes defines the storage resource
volumeMounts attaches the volume to the container
/usr/share/nginx/html becomes backed by the volume

Volume Types

emptyDir

emptyDir is a temporary volume created when a Pod starts. This is useful for caching and temporary files.

exists as long as the Pod runs
deleted when the Pod is removed

1volumes:
2- name: cache-volume
3  emptyDir: {}

hostPath

hostPath mounts a directory from the host node filesystem into the Pod. Since it directly mount from the node the pod deployed to, the volume is not shared across pods/nodes.

Use this for development and testing.

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: nginx
 5spec:
 6  containers:
 7  - name: app
 8    image: nginx
 9    volumeMounts:
10    - mountPath: /usr/share/nginx/html
11      name: web-data
12    command: ["/bin/sh"]
13      args:
14        - -c
15        - |
16          echo "<html>\n<!-- <h2>  </h2> -->\n<h3>my first website</h3>\n<p>look at me mom i'm a devops.</p>" > /usr/share/nginx/html/index.html
17          sleep 3600
18
19  volumes:
20  - name: web-data
21    hostPath:
22      path: /srv/k8s/host/

Check the node the pod deployed to and ssh.

1NAME               READY   STATUS    RESTARTS   AGE    IP                NODE       NOMINATED NODE   READINESS GATES
2nginx              1/1     Running   0          12s    172.16.235.62     master03   <none>           <none>
3
4ssh master03
5[mcbtaguiad@master03 ~]$ cat /srv/k8s/host/index.html 
6<html>
7<!-- <h2>  </h2> -->
8<h3>my first website</h3>
9<p>look at me mom i'm a devops.</p>

PersistentVolume (PV)

This is a cluster wide volume which persist even if the pod are deleted or restarted - unless intentionally deleted, becareful doing this in production. Unlike hostPath, PV is shared by pods bound to this volume. You’ll need to create PersistentVolumeClaim to make pod/container consume the PV.

PVs provide an abstraction layer over the actual underlying distributed storage systems - using NFS-CSI, cloud provider or self-hosted solutions like Longhorn or Rook-Ceph (search more on this topic). I’ll briefly explain this in the following section.

Example PV:

1apiVersion: v1
2kind: PersistentVolume
3metadata:
4  name: pv-storage
5spec:
6  capacity:
7    storage: 5Gi
8  accessModes:
9    - ReadWriteOnce

This is automatically created if you have deployed already a Container Storage Interface (CSI). This will be requested and created by the StorageClass you defined or automatically created by the distributed storage system.

PersistentVolumeClaim (PVC)

A PersistentVolumeClaim (PVC) is a request for storage by a Pod. PVC will consume the PV resource created.

Instead of referencing a volume directly, Pods request storage through a claim.

For this example I’m using Rook-Ceph with StorageClass - rook-cephfs.

1kubectl get storageclass
2NAME                   PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
3rook-cephfs            rook-ceph.cephfs.csi.ceph.com   Delete          Immediate              true                   46s

nginx.yaml

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4  name: nginx
 5spec:
 6  containers:
 7  - name: app
 8    image: nginx
 9    volumeMounts:
10    - mountPath: /usr/share/nginx/html
11      name: web-data
12    command: ["/bin/sh"]
13    args:
14      - -c
15      - |
16        echo "<html>\n<!-- <h2>  </h2> -->\n<h3>my first website</h3>\n<p>look at me mom i'm a devops.</p>" > /usr/share/nginx/html/index.html
17        sleep 3600
18
19  volumes:
20  - name: web-data
21    persistentVolumeClaim:
22      claimName: nginx-pvc

The pod will stay in Pending state until the pvc manifest is created.

pvc.yaml

 1apiVersion: v1
 2kind: PersistentVolumeClaim
 3metadata:
 4  name: nginx-pvc
 5spec: 
 6  storageClassName: rook-cephfs
 7  accessModes:
 8  - ReadWriteMany
 9  resources:
10    requests:
11      storage: 1Gi

Verify PVC and PV.

1kubectl get pvc -n demo
2NAME                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
3nginx-pvc             Bound    pvc-1d381897-aa46-4475-88ce-f9c222613383   1Gi        RWX            rook-cephfs    <unset>                 7m41s

1kubectl get pv 
2NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                      STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
3pvc-1d381897-aa46-4475-88ce-f9c222613383   1Gi        RWX            Delete           Bound    demo/nginx-pvc             rook-cephfs    <unset>                          3m41s

PVC is bound to the PV.

Persistent Storage Architecture

1Pod
2 ↓
3PersistentVolumeClaim
4 ↓
5PersistentVolume
6 ↓
7Storage Backend (Disk / Cloud Storage / NFS)

Container Storag Interface

Kubernetes Container Storage Interface (CSI) is a standardized interface enabling external storage systems to connect with Kubernetes, allowing for dynamic provisioning, volume snapshots, and mounting of persistent storage.

Let’s just deploy a simple CSI for this example - NFS-CSI. Unlike other CSI, this does not have backup and replication, Kubernetes access and manage NFS server for persistend storage. Don’t use this in production.

Install NFS-CSI, check this repo for more deployment option (helm).

 1curl -skSL https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/deploy/install-driver.sh | bash -s master --
 2Installing NFS CSI driver, version: master ...
 3serviceaccount/csi-nfs-controller-sa created
 4serviceaccount/csi-nfs-node-sa created
 5clusterrole.rbac.authorization.k8s.io/nfs-external-provisioner-role created
 6clusterrolebinding.rbac.authorization.k8s.io/nfs-csi-provisioner-binding created
 7clusterrole.rbac.authorization.k8s.io/nfs-external-resizer-role created
 8clusterrolebinding.rbac.authorization.k8s.io/nfs-csi-resizer-role created
 9csidriver.storage.k8s.io/nfs.csi.k8s.io created
10deployment.apps/csi-nfs-controller created
11daemonset.apps/csi-nfs-node created
12NFS CSI driver installed successfully.
13
14kubectl get pods -A | grep csi
15kube-system      csi-nfs-controller-6c45446444-r5xc9                         5/5     Running     0               38s
16kube-system      csi-nfs-node-bvfd5                                          3/3     Running     0               37s
17kube-system      csi-nfs-node-pk6p9                                          3/3     Running     0               37s
18kube-system      csi-nfs-node-qlzwd                                          3/3     Running     0               37s

Create StorageClass.

 1apiVersion: storage.k8s.io/v1
 2kind: StorageClass
 3metadata:
 4  name: nfs-csi
 5provisioner: nfs.csi.k8s.io
 6parameters:
 7  server: 192.168.254.205
 8  share: /srv/nfs
 9reclaimPolicy: Delete
10allowVolumeExpansion: true
11volumeBindingMode: Immediate
12mountOptions:
13  - hard
14  - nfsvers=4.1

reclaimPolicy type:

Retain - Keeps the volume after PVC deletion. Admin must manually delete or reuse it. Data is preserved.
Delete - Automatically deletes the underlying storage (e.g., cloud disk) when PVC is deleted.

A NFS server is running in IP 192.168.254.205, make sure the K8S nodes are added in /etc/exports.

Using the example earlier just change StorageClassName to nfs-csi.

 1apiVersion: v1
 2kind: PersistentVolumeClaim
 3metadata:
 4  name: nginx-pvc
 5spec: 
 6  storageClassName: nfs-csi
 7  accessModes:
 8  - ReadWriteMany
 9  resources:
10    requests:
11      storage: 1Gi

marktaguiad.dev