kubenetes集群版本升级方式
作者：小马运维的一天
本文详细介绍了使用kubeadm和二进制方式搭建及升级Kubernetes集群的方法,介绍了版本控制、升级步骤、备份ETCD数据、升级各节点组件等关键操作,并提供了操作示例和注意事项,帮助理解和实施Kubernetes集群的搭建和升级过程
概述
搭建kubernetes集群的方式有很多种，比如二进制，kubeadm，RKE（Rancher）等，k8s集群升级方式也各有千秋，本文主要介绍使用kubeadm方式搭建的k8s集群和二进制搭建的集群升级方法。
Kubernetes 版本以 x.y.z 表示，其中 x 是主要版本， y 是次要版本，z 是补丁版本，遵循语义版本控制术语。升级版本时不建议跨多个次要版本升级，根据官方建议以跨1个次要版本升级或者跨多个补丁版本升级。例如：从1.20.1升级至1.20.15、从1.20.X升级1.21.X。
kubeadm方式升级
升级顺序如下步骤：
- 升级kubeadm版本。
- 备份ETCD数据。
- 升级master节点各组件。
- 升级node节点组件。
- 升级完成集群验证
1.集群概况
节点名称 | 节点IP | 版本 | 升级版本 | 备注 |
k8s-master | 192.168.60.20 | v1.24.17 | v1.25.14 | master节点 |
k8s-node01 | 192.168.60.21 | v1.24.17 | v1.25.14 | node节点 |
k8s-node02 | 192.168.60.20 | v1.24.17 | v1.25.14 | node节点 |
2.升级kubeadm版本
查看当前可升级的版本
yum list --showduplicates kubeadm --disableexcludes=kubernetes
查看当前kubeadm版本
[root@k8s-master kafka]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.17", GitCommit:"22a9682c8fe855c321be75c5faacde343f909b04", GitTreeState:"clean", BuildDate:"2023-08-23T23:43:11Z", GoVersion:"go1.20.7", Compiler:"gc", Platform:"linux/amd64"}升级kubeadm版本
[root@k8s-master kafka]# yum install -y kubeadm-1.25.14-0 --disableexcludes=kubernetes
验证
[root@k8s-master kafka]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.14", GitCommit:"a5967a3c4d0f33469b7e7798c9ee548f71455222", GitTreeState:"clean", BuildDate:"2023-09-13T09:10:47Z", GoVersion:"go1.20.8", Compiler:"gc", Platform:"linux/amd64"}3.ETCD备份
etcdctl snapshot save etcd.db
4.升级master节点组件
4.1验证升级计划，检查当前集群是否可被升级
[root@k8s-master kafka]# kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: v1.24.17
[upgrade/versions] kubeadm version: v1.25.14
I0907 01:19:06.634937 89546 version.go:256] remote version is much newer: v1.31.0; falling back to: stable-1.25
[upgrade/versions] Target version: v1.25.16
[upgrade/versions] Latest version in the v1.24 series: v1.24.17
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT CURRENT TARGET
kubelet 3 x v1.24.17 v1.25.16
Upgrade to the latest stable version:
COMPONENT CURRENT TARGET
kube-apiserver v1.24.17 v1.25.16
kube-controller-manager v1.24.17 v1.25.16
kube-scheduler v1.24.17 v1.25.16
kube-proxy v1.24.17 v1.25.16
CoreDNS v1.8.6 v1.9.3
etcd 3.5.6-0 3.5.6-0
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.25.16
Note: Before you can perform this upgrade, you have to update kubeadm to v1.25.16.
_____________________________________________________________________
The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.
API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io v1alpha1 v1alpha1 no
kubelet.config.k8s.io v1beta1 v1beta1 no
_____________________________________________________________________根据以上结果可以看到最高可以升级到 v1.25.16 版本，我们需要升级到v1.25.14版本是允许的；只要可允许升级的最高版本高于你的目标版本，就可以升级。
4.2将master节点升级到v1.25.14版本
注意：kubeadm upgrade命令也会自动对kubeadm在节点上所管理的证书执行续约操作。
如果需要略过证书续约操作，可以使用标志--certificate-renewal=false。
# 将 master升级到目标版本 kubeadm upgrade apply v1.25.14 #出现如下提示表示master节点升级完成了 [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.25.14". Enjoy!
4.3升级kubectl和kubelet
1.将当前节点标记为不可调度，并驱逐节点上的Pod kubectl drain <节点名称> --ignore-daemonsets ##说明： ## --ignore-daemonsets 无视DaemonSet管理下的Pod。即--ignore-daemonsets往往需要指定的,这是 #因为deamonset会忽略unschedulable标签(使用kubectl drain时会自动给节点打上不可调度标签), #由于deamonset控制器控制的pod被删除后可能马上又在此节点上启动起来,这样就会成为死循环。因此 #这里忽略daemonset 2.升级kubelet和kubectl组件 yum install -y kubelet-1.25.14-0 kubectl-1.25.14-0 --disableexcludes=kubernetes 3.重启kubelet systemctl daemon-reload systemctl restart kubelet 4.恢复当前节点上的Pod调度，使其上线 kubectl uncordon <节点名称>
此时查看版本发现master节点的版本已经升级成功了
5.升级node节点组件
工作节点上的升级过程应该一次执行一个节点，或者一次执行几个节点，以不影响运行工作负载所需的最小容量。
# 升级kubeadm yum install -y kubeadm-1.25.14-0 --disableexcludes=kubernetes # 查看版本 kubeadm version # 设置节点不可调度并排空节点。只有1个worker节点时忽略此步，因为可能会报错 kubectl drain k8s-node1 --ignore-daemonsets # 升级kubelet和kubectl组件 yum install -y kubelet-1.25.14-0 kubectl-1.25.14-0 --disableexcludes=kubernetes # 重启kubelet systemctl daemon-reload systemctl restart kubelet # 恢复当前节点上的Pod调度。只有1个worker节点时忽略此步 kubectl uncordon k8s-node01 # k8s-node01 为worker节点名称
6.验证集群
查看各个节点版本是否正常升级。
kubectl get nodes
7.查看证书时间
kubeadm alpha certs check-expiration
二进制方式升级
升级顺序如下步骤：
- etcd数据库备份升级。
- 升级master节点组件。
- 升级node节点组件。
- 升级calico和CoreDNS组件。
- 验证集群。
1.集群概况
节点名称 | 节点IP | 版本 | 升级版本 | 备注 |
k8s-master | 10.3.248.136 | v1.24.4 | v1.25.14 | master节点(kube-apiserver、controller、scheduler、kubelet、kube-proxy) |
k8s-node02 | 10.3.248.144 | v1.24.4 | v1.25.14 | node节点(kubelet、kube-proxy) |
mongodb | 10.3.248.143 | v1.24.4 | v1.25.14 | node节点(kubelet、kube-proxy) |
ocr | 10.3.248.139 | v1.24.4 | v1.25.14 | node节点(kubelet、kube-proxy) |
2.备份etcd数据并升级
2.1查看etcd集群的状态
[root@k8s-master][16:31:33] ~# /usr/local/bin/etcdctl --endpoints="https://10.3.248.136:2379,https://10.3.248.137:2379,https://10.3.248.144:2379" --cacert="/etc/etcd/ssl/ca.pem" --cert="/etc/etcd/ssl/etcd.pem" --key="/etc/etcd/ssl/etcd-key.pem" endpoint status --write-out="table" +---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | https://10.3.248.136:2379 | 2ea852d4423ced96 | 3.5.4 | 12 MB | false | false | 8 | 20804876 | 20804876 | | | https://10.3.248.137:2379 | 738eb00a101e8cf9 | 3.5.4 | 12 MB | true | false | 8 | 20804876 | 20804876 | | | https://10.3.248.144:2379 | dde4624daa86dd2e | 3.5.4 | 12 MB | false | false | 8 | 20804876 | 20804876 | | +---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
2.2.etcd数据备份
备份时选择主节点进行备份数据
[root@k8s-master][16:38:13] ~# /usr/local/bin/etcdctl --endpoints="https://10.3.248.137:2379" --cacert="/etc/etcd/ssl/ca.pem" --cert="/etc/etcd/ssl/etcd.pem" --key="/etc/etcd/ssl/etcd-key.pem" snapshot save etcd_bak.db
{"level":"info","ts":"2024-09-08T16:39:25.880+0800","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"etcd_bak.db.part"}
{"level":"info","ts":"2024-09-08T16:39:25.888+0800","logger":"client","caller":"v3/maintenance.go:211","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2024-09-08T16:39:25.888+0800","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://10.3.248.137:2379"}
{"level":"info","ts":"2024-09-08T16:39:25.993+0800","logger":"client","caller":"v3/maintenance.go:219","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2024-09-08T16:39:26.021+0800","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://10.3.248.137:2379","size":"12 MB","took":"now"}
{"level":"info","ts":"2024-09-08T16:39:26.021+0800","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"etcd_bak.db"}
Snapshot saved at etcd_bak.db2.3etcd数据库升级时先升级从节点然后再升级主节点。
可通过k8s官网升级changelog查看k8s版本对应的etcd版本。
官网地址https://github.com/kubernetes/kubernetes/tree/master/CHANGELOG
可以查看到升级k8sv1.25.14版本需要的etcd版本是3.5.9
2.4升级etcd版本
#下载
https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz
#备份旧版本
[root@k8s-master][16:57:05] /usr/local/bin# mv etcdctl etcdctlbak
[root@k8s-master][16:57:20] /usr/local/bin# mv etcd etcdbak
#解压新版本
[root@k8s-master][16:57:58] ~# tar -zxvf etcd-v3.5.9-linux-amd64.tar.gz --strip-components=1 -C /usr/local/bin etcd-v3.5.9-linux-amd64/etcd{,ctl}
etcd-v3.5.9-linux-amd64/etcdctl
etcd-v3.5.9-linux-amd64/etcd
#重启etcd
[root@k8s-master][16:58:18] ~# systemctl daemon-reload
[root@k8s-master][16:59:09] ~# systemctl restart etcd
#验证升级结果
[root@k8s-master][16:59:46] ~# /usr/local/bin/etcdctl --endpoints="https://10.3.248.136:2379,https://10.3.248.137:2379,https://10.3.248.144:2379" --cacert="/etc/etcd/ssl/ca.pem" --cert="/etc/etcd/ssl/etcd.pem" --key="/etc/etcd/ssl/etcd-key.pem" endpoint status --write-out="table"
+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://10.3.248.136:2379 | 2ea852d4423ced96 | 3.5.9 | 12 MB | false | false | 8 | 20808752 | 20808752 | |
| https://10.3.248.137:2379 | 738eb00a101e8cf9 | 3.5.4 | 12 MB | true | false | 8 | 20808752 | 20808752 | |
| https://10.3.248.144:2379 | dde4624daa86dd2e | 3.5.4 | 12 MB | false | false | 8 | 20808752 | 20808752 | |
+---------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
可以看到etcd的一个节点版本已经升级完成。其余节点升级方式同上，升级完成后进行版本查看正常升级即可。
升级完成后如下：
3.升级master节点组件
二进制包下载地址https://cdn.dl.k8s.io/release/v1.25.14/kubernetes-server-linux-amd64.tar.gz
3.1升级kube-apiserver组件
#备份 cp /usr/bin/kube-apiserver /usr/bin/kube-apiserverbak tar xf kubernetes-server-linux-amd64.tar.gz #停止apiserver服务 systemctl stop kube-apiserver.service #替换二进制文件 cp -a kubernetes/server/bin/kube-apiserver /usr/bin/ #启动新版的apiserver systemctl start kube-apiserver.service #查看版本 kube-apiserver --version Kubernetes v1.25.14
3.2升级kube-controller-manager组件
#备份 cp /usr/bin/kube-controller-manager /usr/bin/kube-controller-managerbak #停止kube-controller-manager服务 systemctl stop kube-controller-manager.service #替换二进制文件 cp -a kubernetes/server/bin/kube-controller-manager /usr/bin/ #启动新版controller-manager systemctl start kube-controller-manager.service #查看版本 # kube-controller-manager --version Kubernetes v1.25.14
3.3升级kube-scheduler组件
#备份 cp /usr/bin/kube-scheduler /usr/bin/kube-schedulerbak #停止kube-controller-manager服务 systemctl stop kube-scheduler.service #替换二进制文件 cp -a kubernetes/server/bin/kube-scheduler /usr/bin/ #启动新版controller-manager systemctl start kube-scheduler.service #查看版本 # kube-scheduler --version Kubernetes v1.25.14
3.4升级kubectl
#备份 cp /usr/bin/kubectl /usr/bin/kubectlbak #替换二进制文件 cp -a kubernetes/server/bin/kubectl /usr/bin/ #查看版本 # kubectl version --short Flag --short has been deprecated, and will be removed in the future. The --short output will become the default. Client Version: v1.25.14 Kustomize Version: v4.5.7 Server Version: v1.25.14
4.升级node节点组件
# 设置节点不可调度并排空节点。 [root@k8s-master][12:02:56] ~# kubectl drain ocr --ignore-daemonsets node/ocr already cordoned WARNING: ignoring DaemonSet-managed Pods: ingress-nginx/ingress-nginx-controller-thtp9, kube-system/calico-node-v8ddj evicting pod default/uesopconsole-5d78dcfb47-f5c8x pod/uesopconsole-5d78dcfb47-f5c8x evicted node/ocr drained
注意：如果使用了本地存储的情况下，可能无法排空当前节点，可能会出现如下报错
[root@k8s-master][16:56:31] ~# kubectl drain mongodb --ignore-daemonsets node/mongodb cordoned error: unable to drain node "mongodb" due to error:cannot delete Pods with local storage (use --delete-emptydir-data to override): kube-system/metrics-server-d49478bd-7wkbf, continuing command... There are pending nodes to be drained: mongodb cannot delete Pods with local storage (use --delete-emptydir-data to override): kube-system/metrics-server-d49478bd-7wkbf
此时我们选择就地升级即不用排空节点，直接升级node节点组件，可能会影响本地pod重启一次，如果升级时间较短的话，pod并不会重启。
4.1升级kubelet组件
#备份 cp /usr/bin/kubelet /usr/bin/kubeletbak tar xf kubernetes-server-linux-amd64.tar.gz #停止kubelet服务 systemctl stop kubelet.service #替换二进制文件 \cp kubernetes/server/bin/kubelet /usr/bin/ #启动新版kubelet systemctl start kubelet.service
4.2升级kube-proxy组件
#备份 cp /usr/bin/kube-proxy /usr/bin/kube-proxybak #停止kube-proxy服务 systemctl stop kube-proxy #替换二进制文件 \cp kubernetes/server/bin/kube-proxy /usr/bin/kube-proxy #启动新版kubelet systemctl start kube-proxy
恢复当前节点上的Pod调度。
kubectl uncordon ocr
其它node节点升级操作同上，升级完成kubelet和kube-proxy组件后，通过master节点查看版本已经升级
5.升级calico和CoreDNS组件
calico和coreDNS组件版本可根据实际需求进行版本升级，也可以不升级。
升级时根据部署方决定，如果是yaml方式进行部署的，下载最新的yaml文件和镜像修改对应参数重启pod即可。
calico官网链接：
coreDNS：
kubernetes/cluster/addons/dns/coredns/coredns.yaml.base at master · kubernetes/kubernetes · GitHub
总结
以上为个人经验，希望能给大家一个参考，也希望大家多多支持脚本之家。