kublet 運行在每個 worker 節(jié)點上，接收 kube-apiserver 發(fā)送的請求，管理 Pod 容器，執(zhí)行交互式命令，如 exec、run、logs 等。

kublet 啟動時自動向 kube-apiserver 注冊節(jié)點信息，內(nèi)置的 cadvisor 統(tǒng)計和監(jiān)控節(jié)點的資源使用情況。

為確保安全，本文檔只開啟接收 https 請求的安全端口，對請求進行認證和授權(quán)，拒絕未授權(quán)的訪問 (如 apiserver、heapster)。

下載分發(fā)，在部署 master 節(jié)點時已經(jīng)做過，依賴包也在 work 節(jié)點當中安裝過，現(xiàn)在直接進入配置

1、創(chuàng)建 kubelet bootstrap kubeconfig 文件

cat > magic57_create_kubeconfig_file.sh << "EOF"
#!/bin/bash
# 創(chuàng)建 kubelet bootstrap kubeconfig 文件
source /opt/k8s/bin/environment.sh
for node_name in ${NODE_NAMES[@]}
do
    echo ">>> ${node_name}"
    # 創(chuàng)建 token
    export BOOTSTRAP_TOKEN=$(kubeadm token create \
      --description kubelet-bootstrap-token \
      --groups system:bootstrappers:${node_name} \
      --kubeconfig ~/.kube/config)
    # 設(shè)置集群參數(shù)
    kubectl config set-cluster kubernetes \
      --certificate-authority=/etc/kubernetes/cert/ca.pem \
      --embed-certs=true \
      --server=${KUBE_APISERVER} \
      --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig
    # 設(shè)置客戶端認證參數(shù)
    kubectl config set-credentials kubelet-bootstrap \
      --token=${BOOTSTRAP_TOKEN} \
      --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig
    # 設(shè)置上下文參數(shù)
    kubectl config set-context default \
      --cluster=kubernetes \
      --user=kubelet-bootstrap \
      --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig
    # 設(shè)置默認上下文
    kubectl config use-context default --kubeconfig=kubelet-bootstrap-${node_name}.kubeconfig
done
EOF

• 證書中寫入 Token 而非證書，證書后續(xù)由 controller-manager 創(chuàng)建

查看 kubeadm 為各節(jié)點創(chuàng)建的 token：

kubeadm token list --kubeconfig ~/.kube/config
TOKEN   TTL    EXPIRES                     USAGES                   DESCRIPTION               EXTRA GROUPS
XX.XX   XXh    20XX-XX-XXTXX:XX:XX+XX:XX   authentication,signing   kubelet-bootstrap-token   system:bootstrappers:kube-node2
XX.XX   XXh    20XX-XX-XXTXX:XX:XX+XX:XX   authentication,signing   kubelet-bootstrap-token   system:bootstrappers:kube-node3
XX.XX   XXh    20XX-XX-XXTXX:XX:XX+XX:XX   authentication,signing   kubelet-bootstrap-token   system:bootstrappers:kube-node1

• 創(chuàng)建的 token 有效期為 1 天，超期后將不能再被使用，且會被 kube-controller-manager 的 tokencleaner 清理 (如果啟用該 controller 的話)；
• kube-apiserver 接收 kubelet 的 bootstrap token 后，將請求的 user 設(shè)置為 system:bootstrap:，group 設(shè)置為 system:bootstrappers；

各 token 關(guān)聯(lián)的 Secret：

kubectl get secrets  -n kube-system
NAME                                     TYPE                                  DATA      AGE
bootstrap-token-3v99js                   bootstrap.kubernetes.io/token         7         6h
bootstrap-token-904crw                   bootstrap.kubernetes.io/token         7         6h
bootstrap-token-v75f7h                   bootstrap.kubernetes.io/token         7         6h
default-token-sqk8s                      kubernetes.io/service-account-token   3         6h

2、分發(fā) bootstrap kubeconfig 文件到所有 worker 節(jié)點

cat > magic58_distribute_kubeconfig_file.sh << "EOF"
#!/bin/bash
# 分發(fā) bootstrap kubeconfig 文件到所有 worker 節(jié)點
source /opt/k8s/bin/environment.sh
for node_name in ${NODE_NAMES[@]}
do
    echo ">>> ${node_name}" 
    scp kubelet-bootstrap-${node_name}.kubeconfig k8s@${node_name}:/etc/kubernetes/kubelet-bootstrap.kubeconfig
done
EOF

3、創(chuàng)建和分發(fā) kubelet 參數(shù)配置文件

創(chuàng)建 kubelet 參數(shù)配置模板文件：

source /opt/k8s/bin/environment.sh

cat > kubelet.config.json.template <<EOF
{
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  "authentication": {
    "x509": {
      "clientCAFile": "/etc/kubernetes/cert/ca.pem"
    },
    "webhook": {
      "enabled": true,
      "cacheTTL": "2m0s"
    },
    "anonymous": {
      "enabled": false
    }
  },
  "authorization": {
    "mode": "Webhook",
    "webhook": {
      "cacheAuthorizedTTL": "5m0s",
      "cacheUnauthorizedTTL": "30s"
    }
  },
  "address": "##NODE_IP##",
  "port": 10250,
  "readOnlyPort": 0,
  "cgroupDriver": "cgroupfs",
  "hairpinMode": "promiscuous-bridge",
  "serializeImagePulls": false,
  "featureGates": {
    "RotateKubeletClientCertificate": true,
    "RotateKubeletServerCertificate": true
  },
  "clusterDomain": "${CLUSTER_DNS_DOMAIN}",
  "clusterDNS": ["${CLUSTER_DNS_SVC_IP}"]
}
EOF

• address：API 監(jiān)聽地址，不能為 127.0.0.1，否則 kube-apiserver、heapster 等不能調(diào)用 kubelet 的 API；
• readOnlyPort=0：關(guān)閉只讀端口 (默認 10255)，等效為未指定；
• authentication.anonymous.enabled：設(shè)置為 false，不允許匿名訪問 10250 端口；
• authentication.x509.clientCAFile：指定簽名客戶端證書的 CA 證書，開啟 HTTP 證書認證；
• authentication.webhook.enabled=true：開啟 HTTPs bearer token 認證；
• 對于未通過 x509 證書和 webhook 認證的請求 (kube-apiserver 或其他客戶端)，將被拒絕，提示 Unauthorized；
• authroization.mode=Webhook：kubelet 使用 SubjectAccessReview API 查詢 kube-apiserver 某 user、group 是否具有操作資源的權(quán)限 (RBAC)；
• featureGates.RotateKubeletClientCertificate、featureGates.RotateKubeletServerCertificate：自動 rotate 證書，證書的有效期取決于 kube-controller-manager 的 –experimental-cluster-signing-duration 參數(shù)；
• 需要 root 賬戶運行；

為各節(jié)點創(chuàng)建和分發(fā) kubelet 配置文件：

cat > magic59_createNode_distribute_file.sh << "EOF"
#!/bin/bash
# 為各節(jié)點創(chuàng)建和分發(fā) kubelet 配置文件
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}" 
    sed -e "s/##NODE_IP##/${node_ip}/" kubelet.config.json.template > kubelet.config-${node_ip}.json
    scp kubelet.config-${node_ip}.json root@${node_ip}:/etc/kubernetes/kubelet.config.json
done
EOF

4、創(chuàng)建和分發(fā) kubelet systemd unit 文件

創(chuàng)建 kubelet systemd unit 文件模板：

cat > kubelet.service.template <<EOF
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service
[Service]
WorkingDirectory=/var/lib/kubelet
ExecStart=/opt/k8s/bin/kubelet \\
  --bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \\
  --cert-dir=/etc/kubernetes/cert \\
  --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\
  --config=/etc/kubernetes/kubelet.config.json \\
  --hostname-override=##NODE_NAME## \\
  --pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest \\
  --allow-privileged=true \\
  --alsologtostderr=true \\
  --logtostderr=false \\
  --log-dir=/var/log/kubernetes \\
  --v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF

• 如果設(shè)置了 --hostname-override 選項，則 kube-proxy 也需要設(shè)置該選項，否則會出現(xiàn)找不到 Node 的情況；
• --bootstrap-kubeconfig：指向 bootstrap kubeconfig 文件，kubelet 使用該文件中的用戶名和 token 向 kube-apiserver 發(fā)送 TLS Bootstrapping 請求；
• K8S approve kubelet 的 csr 請求后，在 --cert-dir 目錄創(chuàng)建證書和私鑰文件，然后寫入 --kubeconfig 文件

為各節(jié)點創(chuàng)建和分發(fā) kubelet systemd unit 文件：

cat > magic60_create_distribute_kubelet_file.sh << "EOF"
#!/bin/bash
# 為各節(jié)點創(chuàng)建和分發(fā) kubelet systemd unit 文件
source /opt/k8s/bin/environment.sh
for node_name in ${NODE_NAMES[@]}
do 
    echo ">>> ${node_name}"
    sed -e "s/##NODE_NAME##/${node_name}/" kubelet.service.template > kubelet-${node_name}.service
    scp kubelet-${node_name}.service root@${node_name}:/etc/systemd/system/kubelet.service
    scp kubelet-bootstrap-${node_name}.kubeconfig root@${node_name}:/etc/kubernetes/kubelet-bootstrap.kubeconfig
done
EOF

5、Bootstrap Token Auth 和授予權(quán)限

kublet 啟動時查找配置的 –kubeletconfig 文件是否存在，如果不存在則使用 --bootstrap-kubeconfig 向 kube-apiserver 發(fā)送證書簽名請求 (CSR)。

kube-apiserver 收到 CSR 請求后，對其中的 Token 進行認證（事先使用 kubeadm 創(chuàng)建的 token），認證通過后將請求的 user 設(shè)置為 system:bootstrap:，group 設(shè)置為 system:bootstrappers，這一過程稱為 Bootstrap Token Auth。

默認情況下，這個 user 和 group 沒有創(chuàng)建 CSR 的權(quán)限，kubelet 啟動失敗，錯誤日志如下：

sudo journalctl -u kubelet -a |grep -A 2 'certificatesigningrequests'
XX XX XX:XX:XX kube-node1 kubelet[26986]: F0506 XX:XX:XX.314378   26986 server.go:233] failed to run Kubelet: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:bootstrap:lemy40" cannot create certificatesigningrequests.certificates.k8s.io at the cluster scope
XX XX XX:XX:XX kube-node1 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
XX XX XX:XX:XX kube-node1 systemd[1]: kubelet.service: Failed with result 'exit-code'.

解決：創(chuàng)建一個 clusterrolebinding 將 group system:bootstrappers 和 clusterrole system:node-bootstrapper 綁定

kubectl create clusterrolebinding kubelet-bootstrap \
  --clusterrole=system:node-bootstrapper \
  --group=system:bootstrappers

6、啟動 kubelet 服務

cat > magic61_start_kubelet_server.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}" 
    ssh root@${node_ip} "mkdir -p /var/lib/kubelet"
    ssh root@${node_ip} "/usr/sbin/swapoff -a"
    ssh root@${node_ip} "mkdir -p /var/log/kubernetes && chown -R k8s /var/log/kubernetes"
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kubelet && systemctl restart kubelet"
done
EOF

• 關(guān)閉 swap 分區(qū)，否則 kubelet 會啟動失?。?/li>
• 必須先創(chuàng)建工作和日志目錄；

journalctl -u kubelet |tail
XX XX XX:XX:XX kube-node1 kubelet[17331]: I1124 XX:XX:XX.255251   17331 container_manager_linux.go:427] [ContainerManager]: Discovered runtime cgroups name: /system.slice/docker.service
XX XX XX:XX:XX kube-node1 kubelet[17331]: I1124 XX:XX:XX.255711   17331 container_manager_linux.go:427] [ContainerManager]: Discovered runtime cgroups name: 
...........

kubelet 啟動后使用 --bootstrap-kubeconfig 向 kube-apiserver 發(fā)送 CSR 請求，當這個 CSR 被 approve 后，kube-controller-manager 為 kubelet 創(chuàng)建 TLS 客戶端證書、私鑰和 --kubeletconfig 文件。
注意：kube-controller-manager 需要配置 --cluster-signing-cert-file 和 --cluster-signing-key-file 參數(shù)，才會為 TLS Bootstrap 創(chuàng)建證書和私鑰

kubectl get csr
NAME                                                   AGE       REQUESTOR                 CONDITION
node-csr-QzuuQiuUfcSdp3j5W4B2UOuvQ_n9aTNHAlrLzVFiqrk   43s       system:bootstrap:zkiem5   Pending
node-csr-oVbPmU-ikVknpynwu0Ckz_MvkAO_F1j0hmbcDa__sGA   27s       system:bootstrap:mkus5s   Pending
node-csr-u0E1-ugxgotO_9FiGXo8DkD6a7-ew8sX2qPE6KPS2IY   13m       system:bootstrap:k0s2bj   Pending

kubectl get nodes
  No resources found.

• 三個 work 節(jié)點的 csr 均處于 pending 狀態(tài)

1）approve kubelet CSR 請求
可以手動或自動 approve CSR 請求。推薦使用自動的方式，因為從 v1.8 版本開始，可以自動輪轉(zhuǎn) approve csr 后生成的證書

2）手動 approve CSR 請求
查看 CSR 列表：

kubectl get csr
NAME                                                   AGE       REQUESTOR                 CONDITION
node-csr-QzuuQiuUfcSdp3j5W4B2UOuvQ_n9aTNHAlrLzVFiqrk   43s       system:bootstrap:zkiem5   Pending
node-csr-oVbPmU-ikVknpynwu0Ckz_MvkAO_F1j0hmbcDa__sGA   27s       system:bootstrap:mkus5s   Pending
node-csr-u0E1-ugxgotO_9FiGXo8DkD6a7-ew8sX2qPE6KPS2IY   13m       system:bootstrap:k0s2bj   Pending

approve CSR：

kubectl certificate approve node-csr-QzuuQiuUfcSdp3j5W4B2UOuvQ_n9aTNHAlrLzVFiqrk
certificatesigningrequest.certificates.k8s.io "node-csr-QzuuQiuUfcSdp3j5W4B2UOuvQ_n9aTNHAlrLzVFiqrk" approved

查看 Approve 結(jié)果：

kubectl describe  csr node-csr-QzuuQiuUfcSdp3j5W4B2UOuvQ_n9aTNHAlrLzVFiqrk
Name:               node-csr-QzuuQiuUfcSdp3j5W4B2UOuvQ_n9aTNHAlrLzVFiqrk
Labels:             <none>
Annotations:        <none>
CreationTimestamp:  Wed, XX XX 20XX XX:XX:XX +0800
Requesting User:    system:bootstrap:zkiem5
Status:             Approved
Subject:
         Common Name:    system:node:kube-node2
         Serial Number:
         Organization:   system:nodes
Events:  <none>

• Requesting User：請求 CSR 的用戶，kube-apiserver 對它進行認證和授權(quán)；
• Subject：請求簽名的證書信息；
• 證書的 CN 是 system:node:kube-node2， Organization 是 system:nodes，kube-apiserver 的 Node 授權(quán)模式會授予該證書的相關(guān)權(quán)限；

3）自動 approve CSR 請求
創(chuàng)建三個 ClusterRoleBinding，分別用于自動 approve client、renew client、renew server 證書：

cat > csr-crb.yaml <<EOF
 # Approve all CSRs for the group "system:bootstrappers"
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: auto-approve-csrs-for-group
 subjects:
 - kind: Group
   name: system:bootstrappers
   apiGroup: rbac.authorization.k8s.io
 roleRef:
   kind: ClusterRole
   name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
   apiGroup: rbac.authorization.k8s.io
---
 # To let a node of the group "system:nodes" renew its own credentials
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: node-client-cert-renewal
 subjects:
 - kind: Group
   name: system:nodes
   apiGroup: rbac.authorization.k8s.io
 roleRef:
   kind: ClusterRole
   name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
   apiGroup: rbac.authorization.k8s.io
---
# A ClusterRole which instructs the CSR approver to approve a node requesting a
# serving cert matching its client cert.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: approve-node-server-renewal-csr
rules:
- apiGroups: ["certificates.k8s.io"]
  resources: ["certificatesigningrequests/selfnodeserver"]
  verbs: ["create"]
---
 # To let a node of the group "system:nodes" renew its own server credentials
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: node-server-cert-renewal
 subjects:
 - kind: Group
   name: system:nodes
   apiGroup: rbac.authorization.k8s.io
 roleRef:
   kind: ClusterRole
   name: approve-node-server-renewal-csr
   apiGroup: rbac.authorization.k8s.io
EOF

• auto-approve-csrs-for-group：自動 approve node 的第一次 CSR； 注意第一次 CSR 時，請求的 Group 為 system:bootstrappers；
• node-client-cert-renewal：自動 approve node 后續(xù)過期的 client 證書，自動生成的證書 Group 為 system:nodes;
• node-server-cert-renewal：自動 approve node 后續(xù)過期的 server 證書，自動生成的證書 Group 為 system:nodes;

生效配置：

kubectl apply -f csr-crb.yaml

7、查看 kublet 的情況

等待一段時間 (1-10 分鐘)，三個節(jié)點的 CSR 都被自動 approve：

kubectl get csr
NAME                                                   AGE       REQUESTOR                 CONDITION
csr-98h25                                              6m        system:node:kube-node2    Approved,Issued
csr-lb5c9                                              7m        system:node:kube-node3    Approved,Issued
csr-m2hn4                                              14m       system:node:kube-node1    Approved,Issued
node-csr-7q7i0q4MF_K2TSEJj16At4CJFLlJkHIqei6nMIAaJCU   28m       system:bootstrap:k0s2bj   Approved,Issued
node-csr-ND77wk2P8k2lHBtgBaObiyYw0uz1Um7g2pRvveMF-c4   35m       system:bootstrap:mkus5s   Approved,Issued
node-csr-Nysmrw55nnM48NKwEJuiuCGmZoxouK4N8jiEHBtLQso   6m        system:bootstrap:zkiem5   Approved,Issued
node-csr-QzuuQiuUfcSdp3j5W4B2UOuvQ_n9aTNHAlrLzVFiqrk   1h        system:bootstrap:zkiem5   Approved,Issued
node-csr-oVbPmU-ikVknpynwu0Ckz_MvkAO_F1j0hmbcDa__sGA   1h        system:bootstrap:mkus5s   Approved,Issued
node-csr-u0E1-ugxgotO_9FiGXo8DkD6a7-ew8sX2qPE6KPS2IY   1h        system:bootstrap:k0s2bj   Approved,Issued

所有節(jié)點均 ready：

kubectl get node
NAME         STATUS    ROLES     AGE       VERSION
kube-node1   Ready     <none>    6h        v1.10.4
kube-node2   Ready     <none>    6h        v1.10.4
kube-node3   Ready     <none>    6h        v1.10.4

kube-controller-manager 為各 node 生成了 kubeconfig 文件和公私鑰：

ls -l /etc/kubernetes/kubelet.kubeconfig
-rw------- 1 root root 2296 XX XX XX:XX /etc/kubernetes/kubelet.kubeconfig

ls -l /etc/kubernetes/cert/|grep kubelet
-rw-r--r-- 1 root root 1046 XX XX XX:XX kubelet-client.crt
-rw------- 1 root root  227 XX XX XX:XX kubelet-client.key
-rw------- 1 root root 1334 XX XX XX:XX kubelet-server-20XX-XX-XX-XX-XX-XX.pem
lrwxrwxrwx 1 root root   59 XX XX XX:XX kubelet-server-current.pem -> /etc/kubernetes/cert/kubelet-server-20XX-XX-XX-XX-XX-XX.pem

• kubelet-server 證書會周期輪轉(zhuǎn)；

8、kubelet 提供的 API 接口

kublet 啟動后監(jiān)聽多個端口，用于接收 kube-apiserver 或其它組件發(fā)送的請求：

sudo netstat -lnpt|grep kubelet
tcp        0      0 172.68.96.101:10250     0.0.0.0:*               LISTEN      17331/kubelet
tcp        0      0 172.68.96.101:4194      0.0.0.0:*               LISTEN      17331/kubelet
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      17331/kubelet

• 4194: cadvisor http 服務；
• 10248: healthz http 服務；
• 10250: https API 服務；注意：未開啟只讀端口 10255；

例如執(zhí)行 kubectl exec -it nginx-ds-5rmws -- sh 命令時，kube-apiserver 會向 kubelet 發(fā)送如下請求：

POST /exec/default/nginx-ds-5rmws/my-nginx?command=sh&input=1&output=1&tty=1

kubelet 接收 10250 端口的 https 請求：

• /pods、/runningpods
• /metrics、/metrics/cadvisor、/metrics/probes
• /spec
• /stats、/stats/container
• /logs
• /run/、”/exec/”, “/attach/”, “/portForward/”, “/containerLogs/” 等管理

詳情參考：https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/server/server.go#L434:3

由于關(guān)閉了匿名認證，同時開啟了 webhook 授權(quán)，所有訪問 10250 端口 https API 的請求都需要被認證和授權(quán)。

預定義的 ClusterRole system:kubelet-api-admin 授予訪問 kubelet 所有 API 的權(quán)限：

kubectl describe clusterrole system:kubelet-api-admin
Name:         system:kubelet-api-admin
Labels:       kubernetes.io/bootstrapping=rbac-defaults
Annotations:  rbac.authorization.kubernetes.io/autoupdate=true
PolicyRule:
  Resources      Non-Resource URLs  Resource Names  Verbs
  ---------      -----------------  --------------  -----
  nodes          []                 []              [get list watch proxy]
  nodes/log      []                 []              [*]
  nodes/metrics  []                 []              [*]
  nodes/proxy    []                 []              [*]
  nodes/spec     []                 []              [*]
  nodes/stats    []                 []              [*]

9、kublet api 認證和授權(quán)

kublet 配置了如下認證參數(shù)：

• authentication.anonymous.enabled：設(shè)置為 false，不允許匿名訪問 10250 端口
• authentication.x509.clientCAFile：指定簽名客戶端證書的 CA 證書，開啟 HTTPs 證書認證
• authentication.webhook.enabled=true：開啟 HTTPs bearer token 認證；

同時配置了如下授權(quán)參數(shù)：

• authroization.mode=Webhook：開啟 RBAC 授權(quán)；

kubelet 收到請求后，使用 clientCAFile 對證書簽名進行認證，或者查詢 bearer token 是否有效。如果兩者都沒通過，則拒絕請求，提示 Unauthorized

curl -s --cacert /etc/kubernetes/cert/ca.pem https://172.68.96.103:10250/metrics
Unauthorized

curl -s --cacert /etc/kubernetes/cert/ca.pem -H "Authorization: Bearer 123456" https://172.68.96.103:10250/metrics
Unauthorized

通過認證后，kubelet 使用 SubjectAccessReview API 向 kube-apiserver 發(fā)送請求，查詢證書或 token 對應的 user、group 是否有操作資源的權(quán)限 (RBAC)；

證書認證和授權(quán)：

# 權(quán)限不足的證書；
curl -s --cacert /etc/kubernetes/cert/ca.pem --cert /etc/kubernetes/cert/kube-controller-manager.pem --key /etc/kubernetes/cert/kube-controller-manager-key.pem https://172.68.96.103:10250/metrics
Forbidden (user=system:kube-controller-manager, verb=get, resource=nodes, subresource=metrics)

# 使用部署 kubectl 命令行工具時創(chuàng)建的、具有最高權(quán)限的 admin 證書；
curl -s --cacert /etc/kubernetes/cert/ca.pem --cert ./admin.pem --key ./admin-key.pem https://172.68.96.103:10250/metrics|head
# HELP apiserver_client_certificate_expiration_seconds Distribution of the remaining lifetime on the certificate used to authenticate a request.
# TYPE apiserver_client_certificate_expiration_seconds histogram
apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="21600"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="43200"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="86400"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="172800"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="345600"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="604800"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="2.592e+06"} 0

• --cacert、--cert、--key 的參數(shù)值必須是文件路徑，如上面的 ./admin.pem 不能省略 ./，否則返回 401 Unauthorized；

bear token 認證和授權(quán)：

創(chuàng)建一個 ServiceAccount，將它和 ClusterRole system:kubelet-api-admin 綁定，從而具有調(diào)用 kubelet API 的權(quán)限：

kubectl create sa kubelet-api-test
kubectl create clusterrolebinding kubelet-api-test --$clusterrole=system:kubelet-api-admin --

serviceaccount=default:kubelet-api-test
SECRET=$(kubectl get secrets | grep kubelet-api-test | awk '{print $1}')
TOKEN=$(kubectl describe secret ${SECRET} | grep -E '^token' | awk '{print $2}')
echo ${TOKEN}

curl -s --cacert /etc/kubernetes/cert/ca.pem -H "Authorization: Bearer ${TOKEN}" https://172.27.129.111:10250/metrics|head
# HELP apiserver_client_certificate_expiration_seconds Distribution of the remaining lifetime on the certificate used to authenticate a request.
# TYPE apiserver_client_certificate_expiration_seconds histogram
apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="21600"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="43200"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="86400"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="172800"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="345600"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="604800"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="2.592e+06"} 0

10，cadvisor 和 metrics

cadvisor 統(tǒng)計所在節(jié)點各容器的資源 (CPU、內(nèi)存、磁盤、網(wǎng)卡) 使用情況，分別在自己的 http web 頁面 (4194 端口) 和 10250 以 promehteus metrics 的形式輸出

瀏覽器訪問 http://172.68.96.103:4194/containers/ 可以查看到 cadvisor 的監(jiān)控頁面：

image.png

11、獲取 kublet 的配置

從 kube-apiserver 獲取各 node 的配置：

source /opt/k8s/bin/environment.sh

# 使用部署 kubectl 命令行工具時創(chuàng)建的、具有最高權(quán)限的 admin 證書；
curl -sSL --cacert /etc/kubernetes/cert/ca.pem --cert ./admin.pem --key ./admin-key.pem ${KUBE_APISERVER}/api/v1/nodes/kube-node1/proxy/configz | jq \
  '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"'
{
  "syncFrequency": "1m0s",
  "fileCheckFrequency": "20s",
  "httpCheckFrequency": "20s",
  "address": "172.27.129.80",
  "port": 10250,
  "readOnlyPort": 10255,
  "authentication": {
    "x509": {},
    "webhook": {
      "enabled": false,
      "cacheTTL": "2m0s"
    },
    "anonymous": {
      "enabled": true
    }
  },
  "authorization": {
    "mode": "AlwaysAllow",
    "webhook": {
      "cacheAuthorizedTTL": "5m0s",
      "cacheUnauthorizedTTL": "30s"
    }
  },
  "registryPullQPS": 5,
  "registryBurst": 10,
  "eventRecordQPS": 5,
  "eventBurst": 10,
  "enableDebuggingHandlers": true,
  "healthzPort": 10248,
  "healthzBindAddress": "127.0.0.1",
  "oomScoreAdj": -999,
  "clusterDomain": "cluster.local.",
  "clusterDNS": [
    "10.254.0.2"
  ],
  "streamingConnectionIdleTimeout": "4h0m0s",
  "nodeStatusUpdateFrequency": "10s",
  "imageMinimumGCAge": "2m0s",
  "imageGCHighThresholdPercent": 85,
  "imageGCLowThresholdPercent": 80,
  "volumeStatsAggPeriod": "1m0s",
  "cgroupsPerQOS": true,
  "cgroupDriver": "cgroupfs",
  "cpuManagerPolicy": "none",
  "cpuManagerReconcilePeriod": "10s",
  "runtimeRequestTimeout": "2m0s",
  "hairpinMode": "promiscuous-bridge",
  "maxPods": 110,
  "podPidsLimit": -1,
  "resolvConf": "/etc/resolv.conf",
  "cpuCFSQuota": true,
  "maxOpenFiles": 1000000,
  "contentType": "application/vnd.kubernetes.protobuf",
  "kubeAPIQPS": 5,
  "kubeAPIBurst": 10,
  "serializeImagePulls": false,
  "evictionHard": {
    "imagefs.available": "15%",
    "memory.available": "100Mi",
    "nodefs.available": "10%",
    "nodefs.inodesFree": "5%"
  },
  "evictionPressureTransitionPeriod": "5m0s",
  "enableControllerAttachDetach": true,
  "makeIPTablesUtilChains": true,
  "iptablesMasqueradeBit": 14,
  "iptablesDropBit": 15,
  "featureGates": {
    "RotateKubeletClientCertificate": true,
    "RotateKubeletServerCertificate": true
  },
  "failSwapOn": true,
  "containerLogMaxSize": "10Mi",
  "containerLogMaxFiles": 5,
  "enforceNodeAllocatable": [
    "pods"
  ],
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1"
}

或者參考代碼中的注釋：https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/apis/kubeletconfig/v1beta1/types.go

12、參考

kubelet 認證和授權(quán)：https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-authentication-authorization/