12.kubernetes筆記 Volume存儲卷(三) Longhorn Storageclass 動態(tài)預(yù)配存儲空間

目錄
Storageclass 簡介
常用字段
Longhorn Storageclass插件
安裝Longhorn
示例1:測試創(chuàng)建PVC 自動創(chuàng)建PV
longhorn數(shù)據(jù)存儲位置及自定義資源

前言:

前面介紹的PV、PVC完成了存儲類型和Pod掛載的解耦,開發(fā)人員直接使用PVC,而Volume存儲系統(tǒng)的PV由管理員維護(hù),但隨之帶來的問題,開發(fā)人員使用PVC之前,需要系統(tǒng)管理員提前手動創(chuàng)建PV,而隨著pod的不斷增多,系統(tǒng)管理員的工作會變得重復(fù)和繁瑣,靜態(tài)供給方式這并不符合自動化運(yùn)維的趨勢,隨之應(yīng)運(yùn)而生的Storageclass 動態(tài)預(yù)配存儲空間,它的作用就是創(chuàng)建PV的模板,解決了PV的自動創(chuàng)建,備份等擴(kuò)展功能.

Storageclass 簡介

簡稱SC;PV和PVC都可屬于某個(gè)特定的SC;
創(chuàng)建PV的模板:可以將某個(gè)存儲服務(wù)與SC關(guān)聯(lián)起來,并且將該存儲服務(wù)的管理接口提供給SC,從而讓SC能夠在存儲服務(wù)上CRUD(create、Read.Update和Delete)存儲單元;因而,在同一個(gè)SC上聲明PVC時(shí),若無現(xiàn)存可匹配的PV,則SC能夠調(diào)用管理接口直接創(chuàng)建出一個(gè)符合PVC聲明的需求的PV來。這種PV的提供機(jī)制,就稱為Dynamic Provision.

具體來說,StorageClass會定義一下兩部分:

  1. PV的屬性 ,比如存儲的大小、類型等;
  2. 創(chuàng)建這種PV需要使用到的存儲插件,比如Ceph等;

有了這兩部分信息,Kubernetes就能夠根據(jù)用戶提交的PVC,找到對應(yīng)的StorageClass,然后Kubernetes就會調(diào)用 StorageClass聲明的存儲插件,創(chuàng)建出需要的PV。

這里需要注意的是這么多種存儲系統(tǒng)并不是所有的存儲系統(tǒng)都支持StorageClass,它需要你的存儲系統(tǒng)提供某種接口來讓controller可以調(diào)用并傳遞進(jìn)去PVC的參數(shù)去創(chuàng)建PV

常用字段:

StorageClass資源的期望狀態(tài)直接與apiversion、kind和metadata定義于同一級別而無須嵌套于spec字段中,它支持使用的字段包括如下幾個(gè)

  • allowVolumeExpansion <boolean>:是否支持存儲卷空間擴(kuò)展功能;
  • allowedTopologies <[]0bject>:定義可以動態(tài)配置存儲卷的節(jié)點(diǎn)拓?fù)?,僅啟用了卷調(diào)度功能的服務(wù)器才會用到該字段;每個(gè)卷插件都有自己支持的拓?fù)湟?guī)范,空的拓?fù)溥x擇器表示無拓?fù)湎拗?
  • provisioner <strinp>:必選字段,用于指定存儲服務(wù)方(provisioner,或稱為預(yù)備器),存儲類要依賴該字段值來判定要使用的存儲插件以便適配到目標(biāo)存儲系統(tǒng);kubernetes內(nèi)建支持許多的Provisioner,它們的名字都以kubernetes.io/為前綴,例如kubernetes.io/glusterfs等;
  • parameters <map[string]stringp: 定義連接至指定的Provisioner類別下的某特定存儲時(shí)需要使用的各相關(guān)參數(shù);不同Provisioner的可用的參數(shù)各不相同;
  • reclaimPolicy <strinp:由當(dāng)前存儲類動態(tài)創(chuàng)建約PV資源的默認(rèn)回收策路,可用值為Delete(默認(rèn))和Retain兩個(gè);但那些靜態(tài)FV的回收策略則取決于它們自身的定義;
  • volumeBindingMlode <string>:定義如何為PVC完成預(yù)配和綁定,默認(rèn)值為VolumeBindingImmediate;該字段僅在啟用了存儲卷調(diào)度功能時(shí)才能生效;
  • mountoptions<[]string>:由當(dāng)前類動態(tài)創(chuàng)建的PV資源的默認(rèn)掛載選項(xiàng)列表。
[root@k8s-master storage]# cat Storageclass-rdb-demo.yaml 
apiversion: storage.k8s.io/v1 
kind: Storageclass
metadata:
  name: fast-rbd
provisioner: kubernetes.io/rbd  #不同的存儲后端 parameters模板各不一樣 需要查閱官方文檔 這里是ceph的 也不是所有的存儲類型都是支持Storageclass
parameters:
  monitors: ceph01.ilinux.io:6789,ceph02.ilinux.io:6789,ceph03.ilinux.io:6789
  adminId: admin
  adminSecretName: ceph-admin-secret
  adminSecretNamespace: kube-system
  pool: kube
  userId: kube
  userSecretName: ceph-kube-secret
  userSecretNamespace: kube-system
  fsType: ext4
  imageFormat: "2"
  imageFeatures: "layering"
reclaimPolicy: Retain  #回收策略
Longhorn Storageclass插件 提供更多的擴(kuò)展功能

官方文檔: https://longhorn.io

Longhorn極大地提升了開發(fā)人員和ITOps的效率,僅需點(diǎn)擊一下鼠標(biāo),即可輕松實(shí)現(xiàn)持久化存儲,并且無需為專有解決方案支付昂貴的費(fèi)用。除此之外,Longhorn減少了管理數(shù)據(jù)及操作環(huán)境所需的資源,從而幫助企業(yè)更加專注且快速地交付代碼及應(yīng)用程序。

Longhorn依舊秉承Rancher 100%開源的產(chǎn)品理念,它是一個(gè)使用微服務(wù)構(gòu)建的分布式塊存儲項(xiàng)目。2019年,Longhorn發(fā)布Beta版本,并于同年10月作為沙箱(Sandbox)項(xiàng)目捐獻(xiàn)給CNCF。Longhorn受到了開發(fā)者們的廣泛關(guān)注,成千上萬名用戶對其進(jìn)行了壓力測試,并提供了極為寶貴的反饋意見。

Longhorn的GA版本提供了一系列功能豐富的企業(yè)級存儲功能,包括:

  • 自動配置,快照,備份和恢復(fù)
  • 零中斷卷擴(kuò)容
  • 具有定義的RTO和RPO的跨集群災(zāi)難恢復(fù)卷
  • 在不影響卷的情況下實(shí)時(shí)升級
  • 全功能的Kubernetes CLI集成和獨(dú)立的用戶界面
安裝Longhorn

安裝準(zhǔn)備:節(jié)點(diǎn)數(shù)至少為3臺 要選擇leader

[root@k8s-master storage]#  yum -y install iscsi-initiator-utils  #安裝前需要安裝ISCSI
[root@k8s-master storage]# kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/v1.1.2/deploy/longhorn.yaml  #安裝 因?yàn)橐螺d鏡像 需要等待一些時(shí)間

[root@k8s-master storage]# kubectl get pods -n longhorn-system --watch  #等待所有pod就緒

[root@k8s-master storage]# kubectl get pods --namespace longhorn-system -o wide   #所有pod就緒
NAME                                        READY   STATUS    RESTARTS   AGE     IP             NODE        NOMINATED NODE   READINESS GATES
csi-attacher-54c7586574-fvv4p               1/1     Running   0          37m     10.244.3.16    k8s-node3   <none>           <none>
csi-attacher-54c7586574-swdsr               1/1     Running   0          43m     10.244.2.111   k8s-node2   <none>           <none>
csi-attacher-54c7586574-zkzrg               1/1     Running   0          43m     10.244.3.10    k8s-node3   <none>           <none>
csi-provisioner-5ff5bd6b88-bs687            1/1     Running   0          37m     10.244.3.17    k8s-node3   <none>           <none>
csi-provisioner-5ff5bd6b88-gl4xn            1/1     Running   0          43m     10.244.2.112   k8s-node2   <none>           <none>
csi-provisioner-5ff5bd6b88-qkzt4            1/1     Running   0          43m     10.244.3.11    k8s-node3   <none>           <none>
csi-resizer-7699cdfc4-4w49w                 1/1     Running   0          37m     10.244.3.15    k8s-node3   <none>           <none>
csi-resizer-7699cdfc4-l2j49                 1/1     Running   0          43m     10.244.3.12    k8s-node3   <none>           <none>
csi-resizer-7699cdfc4-sndlm                 1/1     Running   0          43m     10.244.2.113   k8s-node2   <none>           <none>
csi-snapshotter-8f58f46b4-6s89m             1/1     Running   0          37m     10.244.2.119   k8s-node2   <none>           <none>
csi-snapshotter-8f58f46b4-qgv5r             1/1     Running   0          43m     10.244.3.13    k8s-node3   <none>           <none>
csi-snapshotter-8f58f46b4-tf5ls             1/1     Running   0          43m     10.244.2.115   k8s-node2   <none>           <none>
engine-image-ei-a5a44787-5ntlm              1/1     Running   0          44m     10.244.1.146   k8s-node1   <none>           <none>
engine-image-ei-a5a44787-h45hr              1/1     Running   0          44m     10.244.3.6     k8s-node3   <none>           <none>
engine-image-ei-a5a44787-phnjf              1/1     Running   0          44m     10.244.2.108   k8s-node2   <none>           <none>
instance-manager-e-4384d6f1                 1/1     Running   0          44m     10.244.2.110   k8s-node2   <none>           <none>
instance-manager-e-54f46256                 1/1     Running   0          34m     10.244.1.148   k8s-node1   <none>           <none>
instance-manager-e-e008dd8a                 1/1     Running   0          44m     10.244.3.7     k8s-node3   <none>           <none>
instance-manager-r-0ad3175d                 1/1     Running   0          44m     10.244.3.8     k8s-node3   <none>           <none>
instance-manager-r-61277092                 1/1     Running   0          44m     10.244.2.109   k8s-node2   <none>           <none>
instance-manager-r-d8a9eb0e                 1/1     Running   0          34m     10.244.1.149   k8s-node1   <none>           <none>
longhorn-csi-plugin-5htsd                   2/2     Running   0          7m41s   10.244.2.123   k8s-node2   <none>           <none>
longhorn-csi-plugin-hpjgl                   2/2     Running   0          16s     10.244.1.151   k8s-node1   <none>           <none>
longhorn-csi-plugin-wtkcj                   2/2     Running   0          43m     10.244.3.14    k8s-node3   <none>           <none>
longhorn-driver-deployer-5479f45d86-l4fpq   1/1     Running   0          57m     10.244.3.4     k8s-node3   <none>           <none>
longhorn-manager-dgk4d                      1/1     Running   1          57m     10.244.1.145   k8s-node1   <none>           <none>
longhorn-manager-hb7cl                      1/1     Running   0          57m     10.244.2.107   k8s-node2   <none>           <none>
longhorn-manager-xrxll                      1/1     Running   0          57m     10.244.3.3     k8s-node3   <none>           <none>
longhorn-ui-79f8976fbf-sb79r                1/1     Running   0          57m     10.244.3.5     k8s-node3   <none>           <none>

示例1:測試創(chuàng)建PVC 自動創(chuàng)建PV

[root@k8s-master storage]# cat  pvc-dyn-longhorn-demo.yaml   
apiVersion: v1
kind: PersistentVolumeClaim   #資源類型
metadata:
  name: pvc-dyn-longhorn-demo
  namespace: default
spec:
  accessModes: ["ReadWriteOnce"]
  volumeMode: Filesystem
  resources:
    requests:
      storage: 2Gi   #新的PV會以最低容量創(chuàng)建PV
    limits:
      storage: 10Gi
  storageClassName: longhorn  #選擇longhorn創(chuàng)造

[root@k8s-master storage]# kubectl apply -f longhorn.yaml 
  • 修改回收策略及副本數(shù) 非必要操作 根據(jù)實(shí)際需要修改
[root@k8s-master storage]# kubectl get sc
NAME       PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
longhorn   driver.longhorn.io   Delete(注意回收策略 是刪除會有風(fēng)險(xiǎn))  Immediate   true      48s

[root@k8s-master storage]# vim longhorn.yaml  #修改回收策略 修改yaml文件
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: longhorn-storageclass
  namespace: longhorn-system
data:
  storageclass.yaml: |
    kind: StorageClass
    apiVersion: storage.k8s.io/v1
    metadata:
      name: longhorn
    provisioner: driver.longhorn.io   #這個(gè)是longhorn自定義的provisioner 存儲在本地,如果有網(wǎng)絡(luò)存儲按自己需求修改
    allowVolumeExpansion: true
    reclaimPolicy: Delete
    volumeBindingMode: Immediate
    parameters:
      numberOfReplicas: "3"   #數(shù)據(jù)副本數(shù) 副本數(shù)越多數(shù)據(jù)當(dāng)然會越安全,但同時(shí)的磁盤容量和性能要求也會更高
      staleReplicaTimeout: "2880"
      fromBackup: ""
    reclaimPolicy: Retain   #添加字段 修改回收策略
...
---
[root@k8s-master storage]# kubectl apply -f longhorn.yaml  #重新應(yīng)用配置
[root@k8s-master storage]# kubectl delete  -f  pvc-dyn-longhorn-demo.yaml
[root@k8s-master storage]# kubectl apply   -f  pvc-dyn-longhorn-demo.yaml  #重啟創(chuàng)建Pod

[root@k8s-master storage]# kubectl get sc  #修改成功
NAME       PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
longhorn   driver.longhorn.io   Retain          Immediate           true                   10m

[root@k8s-master storage]# kubectl get pv #自動創(chuàng)建PV
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                           STORAGECLASS   REASON   AGE
pv-nfs-demo002                             10Gi       RWX            Retain           Available                                                           2d5h
pv-nfs-demo003                             1Gi        RWO            Retain           Available                                                           2d5h
pvc-dbcbe588-a088-45b4-9972-74b0e6ca0b58   2Gi        RWO            Retain           Bound       default/pvc-dyn-longhorn-demo   longhorn                118s

[root@k8s-master storage]# kubectl get pvc #自動創(chuàng)建PVC
NAME                    STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-dyn-longhorn-demo   Bound     pvc-dbcbe588-a088-45b4-9972-74b0e6ca0b58   2Gi        RWO            longhorn       2m19s
修改svc 打開longhorn UI
[root@k8s-master ~]# kubectl get svc --namespace longhorn-system -o wide
NAME                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE   SELECTOR
csi-attacher        ClusterIP   10.102.194.8     <none>        12345/TCP   17h   app=csi-attacher
csi-provisioner     ClusterIP   10.99.37.10      <none>        12345/TCP   17h   app=csi-provisioner
csi-resizer         ClusterIP   10.111.56.226    <none>        12345/TCP   17h   app=csi-resizer
csi-snapshotter     ClusterIP   10.110.198.133   <none>        12345/TCP   17h   app=csi-snapshotter
longhorn-backend    ClusterIP   10.106.163.23    <none>        9500/TCP    17h   app=longhorn-manager
longhorn-frontend   ClusterIP   10.111.219.113   <none>        80/TCP      17h   app=longhorn-ui  #修改SVC為NodePort

[root@k8s-master ~]# kubectl edit  svc longhorn-frontend  --namespace longhorn-system 
service/longhorn-frontend edited

[root@k8s-master ~]# kubectl get svc --namespace longhorn-system -o wide
NAME                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE   SELECTOR
csi-attacher        ClusterIP   10.102.194.8     <none>        12345/TCP      17h   app=csi-attacher
csi-provisioner     ClusterIP   10.99.37.10      <none>        12345/TCP      17h   app=csi-provisioner
csi-resizer         ClusterIP   10.111.56.226    <none>        12345/TCP      17h   app=csi-resizer
csi-snapshotter     ClusterIP   10.110.198.133   <none>        12345/TCP      17h   app=csi-snapshotter
longhorn-backend    ClusterIP   10.106.163.23    <none>        9500/TCP       17h   app=longhorn-manager
longhorn-frontend   NodePort    10.111.219.113   <none>        80:30745/TCP   17h   app=longhorn-ui   #使用節(jié)點(diǎn)IP打開 節(jié)點(diǎn):30745

可以打開UI 查看到現(xiàn)有的卷、創(chuàng)建新卷、查看節(jié)點(diǎn)信息、備份等


  • 創(chuàng)建redis Pod綁定PVC
[root@k8s-master storage]# cat volumes-pvc-longhorn-demo.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: volumes-pvc-longhorn-demo
  namespace: default
spec:
  containers:
  - name: redis
    image: redis:alpine
    imagePullPolicy: IfNotPresent
    ports:
    - containerPort: 6379
      name: redisport
    volumeMounts:
    - mountPath: /data
      name: redis-data-vol
  volumes:
    - name: redis-data-vol
      persistentVolumeClaim:
        claimName: pvc-dyn-longhorn-demo  #使用sc創(chuàng)建的pvc

[root@k8s-master storage]# kubectl apply -f volumes-pvc-longhorn-demo.yaml

[root@k8s-master storage]# kubectl get pod -o wide
NAME                                 READY   STATUS    RESTARTS   AGE     IP             NODE        NOMINATED NODE   READINESS GATES
centos-deployment-66d8cd5f8b-95brg   1/1     Running   0          18h     10.244.2.117   k8s-node2   <none>           <none>
my-grafana-7d788c5479-bpztz          1/1     Running   0          18h     10.244.2.120   k8s-node2   <none>           <none>
volumes-pvc-longhorn-demo            1/1     Running   0          2m30s   10.244.1.172   k8s-node1   <none>           <none>

[root@k8s-master storage]# kubectl exec volumes-pvc-longhorn-demo -it -- /bin/sh
/data # redis-cli
127.0.0.1:6379> set mykey www.qq.com
OK
127.0.0.1:6379> bgsabe
(error) ERR unknown command `bgsabe`, with args beginning with: 
127.0.0.1:6379> bgsave
Background saving started
127.0.0.1:6379> exit
/data # ls
dump.rdb    lost+found
/data # exit

[root@k8s-master storage]# kubectl delete -f  volumes-pvc-longhorn-demo.yaml 
pod "volumes-pvc-longhorn-demo" deleted

[root@k8s-master storage]# cat volumes-pvc-longhorn-demo.yaml
apiVersion: v1
kind: Pod
metadata:
  name: volumes-pvc-longhorn-demo
  namespace: default
spec:
  nodeName: k8s-node2  #指定運(yùn)行節(jié)點(diǎn)
  containers:
  - name: redis
    image: redis:alpine
    imagePullPolicy: IfNotPresent
    ports:
    - containerPort: 6379
      name: redisport
    volumeMounts:
    - mountPath: /data
      name: redis-data-vol
  volumes:
    - name: redis-data-vol
      persistentVolumeClaim:
        claimName: pvc-dyn-longhorn-demo

[root@k8s-master storage]# kubectl get pod -o wide -w
NAME                                 READY   STATUS              RESTARTS   AGE   IP             NODE        NOMINATED NODE   READINESS GATES
centos-deployment-66d8cd5f8b-95brg   1/1     Running             0          18h   10.244.2.117   k8s-node2   <none>           <none>
my-grafana-7d788c5479-bpztz          1/1     Running             0          18h   10.244.2.120   k8s-node2   <none>           <none>
volumes-pvc-longhorn-demo            1/1     Running             0          68s   10.244.2.124   k8s-node2   <none>           <none>

[root@k8s-master storage]# kubectl exec volumes-pvc-longhorn-demo -it -- /bin/sh #查看pvc數(shù)據(jù)
/data # redis-cli
127.0.0.1:6379> get mykey
"www.qq.com"
127.0.0.1:6379> exit
/data # exit

longhorn數(shù)據(jù)存儲位置及自定義資源
  • 因?yàn)橹案北緮?shù),默認(rèn)設(shè)置為3,所以3個(gè)節(jié)點(diǎn),每個(gè)節(jié)點(diǎn)都一份數(shù)據(jù)保存
[root@k8s-node1 ~]# ls /var/lib/longhorn/replicas/ #默認(rèn)的數(shù)據(jù)保存路徑 
pvc-dbcbe588-a088-45b4-9972-74b0e6ca0b58-da2330a2
[root@k8s-node1 ~]# ls /var/lib/longhorn/replicas/pvc-dbcbe588-a088-45b4-9972-74b0e6ca0b58-da2330a2/
revision.counter  volume-head-000.img  volume-head-000.img.meta  volume.meta

[root@k8s-node2 ~]# ls /var/lib/longhorn/replicas/ 
pvc-dbcbe588-a088-45b4-9972-74b0e6ca0b58-83f7f58c
[root@k8s-node2 ~]# ls /var/lib/longhorn/replicas/pvc-dbcbe588-a088-45b4-9972-74b0e6ca0b58-83f7f58c/
revision.counter  volume-head-000.img  volume-head-000.img.meta  volume.meta

[root@k8s-node3 ~]# ls /var/lib/longhorn/replicas/
pvc-dbcbe588-a088-45b4-9972-74b0e6ca0b58-c90965c0
[root@k8s-node3 ~]# ls /var/lib/longhorn/replicas/pvc-dbcbe588-a088-45b4-9972-74b0e6ca0b58-c90965c0/
revision.counter  volume-head-000.img  volume-head-000.img.meta  volume.meta

[root@k8s-master storage]# kubectl api-resources --api-group=longhorn.io   #longhorn自定義的資源類型
NAME                   SHORTNAMES   APIGROUP      NAMESPACED   KIND
backingimagemanagers   lhbim        longhorn.io   true         BackingImageManager
backingimages          lhbi         longhorn.io   true         BackingImage
engineimages           lhei         longhorn.io   true         EngineImage
engines                lhe          longhorn.io   true         Engine
instancemanagers       lhim         longhorn.io   true         InstanceManager
nodes                  lhn          longhorn.io   true         Node
replicas               lhr          longhorn.io   true         Replica
settings               lhs          longhorn.io   true         Setting
sharemanagers          lhsm         longhorn.io   true         ShareManager
volumes                lhv          longhorn.io   true         Volume

[root@k8s-master storage]# kubectl get replicas -n longhorn-system
NAME                                                  STATE     NODE        DISK                                   INSTANCEMANAGER               IMAGE                               AGE
pvc-dbcbe588-a088-45b4-9972-74b0e6ca0b58-r-08f28c21   running   k8s-node3   49846d34-b5a8-4a86-96f4-f0d7ca191f2a   instance-manager-r-0ad3175d   longhornio/longhorn-engine:v1.1.2   3h54m
pvc-dbcbe588-a088-45b4-9972-74b0e6ca0b58-r-6694fc48   running   k8s-node2   ce1ed80b-43c9-4fc9-8266-cedb736bacaa   instance-manager-r-61277092   longhornio/longhorn-engine:v1.1.2   3h54m
pvc-dbcbe588-a088-45b4-9972-74b0e6ca0b58-r-86a35cd3   running   k8s-node1   3d40b18a-c0e9-459c-b37e-d878152d1261   instance-manager-r-d8a9eb0e   longhornio/longhorn-engine:v1.1.2   3h54m

?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成,瀏覽時(shí)請結(jié)合常識與多方信息審慎甄別。
平臺聲明:文章內(nèi)容(如有圖片或視頻亦包括在內(nèi))由作者上傳并發(fā)布,文章內(nèi)容僅代表作者本人觀點(diǎn),簡書系信息發(fā)布平臺,僅提供信息存儲服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容