1. Home
  2. How to deploy Longhorn on top of a Bright cluster

How to deploy Longhorn on top of a Bright cluster

This document is tested on BCM10 with Ubuntu 22.04
1. We will start with deploying kubernetes latest version, using cm-kubernetes-setup utility, with the following components
  • containerd
  • calico as the network plugin
  • No keyverno
  • Permission manager
  • No additional operators
2. Install open-iscsi in the software image
root@longhorn:~# cm-chroot-sw-img /cm/images/default-image/
root@default-image:/# apt update
root@default-image:/# apt install open-iscsi
3. Disable multipathd in the software image
root@default-image:/# systemctl disable multipathd
4. Add iscsi_tcp kernel module on startup
root@longhorn:~# cat /cm/images/default-image/etc/modules
# /etc/modules: kernel modules to load at boot time.
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
5. Enable iscsid
root@longhorn:~# cm-chroot-sw-img /cm/images/default-image/
root@default-image:/# systemctl enable iscsid
root@default-image:/# exit
root@longhorn:~# pdsh -g computenode 'echo "InitiatorName=$(/sbin/iscsi-iname)" > /etc/iscsi/initiatorname.iscsi'
6. Exclude /etc/iscsi/initiatorname.iscsi in the update/syncinstall lists
root@longhorn:~# cmsh
[longhorn]% category use default 
[longhorn->category[default]]% set excludelistsyncinstall 
#(add the following line; save and exit
- /etc/iscsi/initiatorname.iscsi 
[longhorn->category*[default*]]% commit
[longhorn->category[default]]% set excludelistupdate 
#(add the following line; save and exit
- /etc/iscsi/initiatorname.iscsi 
[longhorn->category*[default*]]% commit
7. Reboot compute nodes
root@longhorn:~# cmsh
[longhorn]% device
[longhorn->device]% reboot -n node001..node003
8. Verify Longhorn prerequisites
root@longhorn:~# module load kubernetes/
root@longhorn:~# curl -sSfL https://raw.githubusercontent.com/longhorn/longhorn/v1.5.3/scripts/environment_check.sh | bash
[INFO]  Required dependencies 'kubectl jq mktemp sort printf' are installed.
[INFO]  All nodes have unique hostnames.
[INFO]  Waiting for longhorn-environment-check pods to become ready (0/3)...
[INFO]  Waiting for longhorn-environment-check pods to become ready (0/3)...
[INFO]  All longhorn-environment-check pods are ready (3/3).
[INFO]  MountPropagation is enabled
[INFO]  Checking kernel release...
[INFO]  Checking iscsid...
[INFO]  Checking multipathd...
[INFO]  Checking packages...
[INFO]  Checking nfs client...
[INFO]  Cleaning up longhorn-environment-check pods...
[INFO]  Cleanup completed.

Note On Compute nodes with multiple disks

By default the long horn will use /var/lib/longhorn path to store the volumes and the replicas. For compute nodes with multiple disks that can be used for longhorn volumes, we would suggest to create a software RAID with all the available disks and mount it on /var/lib/longhorn via the disksetup. If you’d like to keep using separate disks, then each disk has to be mounted separately, such as /mnt/disk1, /mnt/disk2, ..etc and then add the disks for each node from the longhorn-frontend after the initial installation longhorn.

Installing longhorn
1. Add longhorn Helm repository:
root@longhorn:~# helm repo add longhorn https://charts.longhorn.io
"longhorn" has been added to your repositories

root@longhorn:~#  helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "metallb" chart repository
...Successfully got an update from the "longhorn" chart repository
Update Complete. ⎈Happy Helming!⎈


2. Install longhorn Helm Chart
root@longhorn:~# helm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace --version 1.5.3
NAME: longhorn
LAST DEPLOYED: Tue Nov 28 16:13:03 2023
NAMESPACE: longhorn-system
STATUS: deployed
Longhorn is now installed on the cluster!

Please wait a few minutes for other Longhorn components such as CSI deployments, Engine Images, and Instance Managers to be initialized.

Visit our documentation at https://longhorn.io/docs/


3. Check the status of the installed longhorn:
root@longhorn:~# kubectl get pods -n longhorn-system
NAME                                                READY   STATUS    RESTARTS   AGE
csi-attacher-f98ff75fb-46h2x                        1/1     Running   0          2m20s
csi-attacher-f98ff75fb-sqxc4                        1/1     Running   0          2m20s
csi-attacher-f98ff75fb-tvb86                        1/1     Running   0          2m20s
csi-provisioner-65cb5cc4ff-db46z                    1/1     Running   0          2m20s
csi-provisioner-65cb5cc4ff-hbxdw                    1/1     Running   0          2m20s
csi-provisioner-65cb5cc4ff-vl2xt                    1/1     Running   0          2m20s
csi-resizer-77cddbfc7c-brrx8                        1/1     Running   0          2m20s
csi-resizer-77cddbfc7c-d4nq7                        1/1     Running   0          2m20s
csi-resizer-77cddbfc7c-pqs6v                        1/1     Running   0          2m20s
csi-snapshotter-78cb65cc4f-9whwc                    1/1     Running   0          2m20s
csi-snapshotter-78cb65cc4f-nhb5c                    1/1     Running   0          2m20s
csi-snapshotter-78cb65cc4f-slbzn                    1/1     Running   0          2m20s
engine-image-ei-68f17757-9rbzp                      1/1     Running   0          2m36s
engine-image-ei-68f17757-hj7rw                      1/1     Running   0          2m36s
engine-image-ei-68f17757-qrf4t                      1/1     Running   0          2m36s
instance-manager-0030104d0ef614616bde207391756aa4   1/1     Running   0          2m35s
instance-manager-76f7fbb4c54feea1f267aaa5dfcccac0   1/1     Running   0          2m36s
instance-manager-80b6a95c946519d21e4e65142b0646f5   1/1     Running   0          2m29s
longhorn-csi-plugin-859zw                           3/3     Running   0          2m19s
longhorn-csi-plugin-f2dtv                           3/3     Running   0          2m19s
longhorn-csi-plugin-h2fz2                           3/3     Running   0          2m19s
longhorn-driver-deployer-569566d59f-7pfbt           1/1     Running   0          2m52s
longhorn-manager-t4hdp                              1/1     Running   0          2m52s
longhorn-manager-wcq2g                              1/1     Running   0          2m52s
longhorn-manager-zh48s                              1/1     Running   0          2m52s
longhorn-ui-6549ffb4d7-9dv28                        1/1     Running   0          2m52s
longhorn-ui-6549ffb4d7-xq249                        1/1     Running   0          2m52s
4. Create basic authentication for longhorn frontend
root@longhorn:~# USER=root; PASSWORD=system; echo "${USER}:$(openssl passwd -stdin -apr1 <<< ${PASSWORD})" >> auth

root@longhorn:~# kubectl -n longhorn-system create secret generic basic-auth --from-file=auth

root@longhorn:~# cat > longhorn-ingress.yml
apiVersion: networking.k8s.io/v1
kind: Ingress
  name: longhorn-ingress
  namespace: longhorn-system
    # type of authentication
    nginx.ingress.kubernetes.io/auth-type: basic
    # prevent the controller from redirecting (308) to HTTPS
    nginx.ingress.kubernetes.io/ssl-redirect: 'false'
    # name of the secret that contains the user/password definitions
    nginx.ingress.kubernetes.io/auth-secret: basic-auth
    # message to display with an appropriate context why the authentication is required
    nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required '
    # custom max body size for file uploading like backing image uploading
    nginx.ingress.kubernetes.io/proxy-body-size: 10000m
  - http:
      - pathType: Prefix
        path: "/"
            name: longhorn-frontend
              number: 80

root@longhorn:~# kubectl -n longhorn-system apply -f longhorn-ingress.yml
ingress.networking.k8s.io/longhorn-ingress created
5. Access longhorn UI

  a. Check the service IP of the longhorn-frontend

root@longhorn:~# kubectl -n longhorn-system get svc
NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
longhorn-admission-webhook    ClusterIP   <none>        9502/TCP   5m28s
longhorn-backend              ClusterIP    <none>        9500/TCP   5m28s
longhorn-conversion-webhook   ClusterIP    <none>        9501/TCP   5m28s
longhorn-engine-manager       ClusterIP   None             <none>        <none>     5m28s
longhorn-frontend             ClusterIP    <none>        80/TCP     5m28s
longhorn-recovery-backend     ClusterIP    <none>        9503/TCP   5m28s
longhorn-replica-manager      ClusterIP   None             <none>        <none>     5m28s

  b. Navigate to the IP of longhorn-frontend in your browser

Testing Longhorn

1. Create storage class


root@longhorn:~# kubectl create -f https://raw.githubusercontent.com/longhorn/longhorn/v1.5.3/examples/storageclass.yaml
storageclass.storage.k8s.io/longhorn-test created
2. Create a Pod that uses Longhorn volumes by running this command
root@longhorn:~# kubectl create -f https://raw.githubusercontent.com/longhorn/longhorn/v1.5.3/examples/pod_with_pvc.yaml
persistentvolumeclaim/longhorn-volv-pvc created
pod/volume-test created

root@longhorn:~# kubectl get pods
volume-test   1/1     Running   0          37s

root@longhorn:~# kubectl get pvc
NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
longhorn-volv-pvc   Bound    pvc-db60d9fa-9ebb-41ef-911b-b79214bcca5a   2Gi        RWO            longhorn       59s

root@longhorn:~# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                       STORAGECLASS   REASON   AGE
pvc-db60d9fa-9ebb-41ef-911b-b79214bcca5a   2Gi        RWO            Delete           Bound    default/longhorn-volv-pvc   longhorn                59s
3. Check the mounted volume inside the pod:
root@longhorn:~# kubectl exec -it volume-test -- /bin/sh
/ # df -hT
Filesystem           Type            Size      Used Available Use% Mounted on
overlay              overlay       100.0G     13.8G     86.2G  14% /
tmpfs                tmpfs          64.0M         0     64.0M   0% /dev
                     ext4            1.9G     24.0K      1.9G   0% /data
/dev/vdb1            xfs           100.0G     13.8G     86.2G  14% /etc/hosts
/dev/vdb1            xfs           100.0G     13.8G     86.2G  14% /dev/termination-log
/dev/vdb1            xfs           100.0G     13.8G     86.2G  14% /etc/hostname
/dev/vdb1            xfs           100.0G     13.8G     86.2G  14% /etc/resolv.conf
shm                  tmpfs          64.0M         0     64.0M   0% /dev/shm
tmpfs                tmpfs           7.7G     12.0K      7.7G   0% /run/secrets/kubernetes.io/serviceaccount
tmpfs                tmpfs           3.9G         0      3.9G   0% /proc/acpi
tmpfs                tmpfs          64.0M         0     64.0M   0% /proc/kcore
tmpfs                tmpfs          64.0M         0     64.0M   0% /proc/keys
tmpfs                tmpfs          64.0M         0     64.0M   0% /proc/timer_list
tmpfs                tmpfs           3.9G         0      3.9G   0% /proc/scsi
tmpfs                tmpfs           3.9G         0      3.9G   0% /sys/firmware
/ # touch /data/adel
/ # ls -l /data/
total 16
-rw-r--r--    1 root     root             0 Nov 28 15:23 adel
drwx------    2 root     root         16384 Nov 28 15:21 lost+found







Updated on December 4, 2023