OracleLinux 9.5 安装k8s – 个人笔记分享

环境准备
安装containerd
安装runc
切换containerd源
安装kubelet kubeadm kubectl
安装Calico网络插件
- 切换网络
部署Kubernetes Dashboard
Synology CSI
NFS CSI安装
Note

环境准备

关闭防火墙
systemctl stop firewalld
systemctl disable firewalld

修改主机名(每台需要不同)
hostnamectl set-hostname cnnode1.synohyperv.local

修改iscsi iqn名称(每台需要不同)
/etc/iscsi/initiatorname.iscsi

修改固定IP:
vi /etc/NetworkManager/system-connections/ens192.nmconnection

[ipv4]
method=manual
address1=10.61.22.15/22,10.61.23.254
dns=10.61.21.204;
dns-priority=0

修改解析
vi /etc/hosts

10.61.22.15 cnnode1.synohyperv.local cnnode1
10.61.22.16 cnnode2.synohyperv.local cnnode2
10.61.22.17 cnnode3.synohyperv.local cnnode3
10.61.22.18 cnnode4.synohyperv.local cnnode4

关闭SELINUX
sed -i '/^SELINUX=/c SELINUX=disabled' /etc/selinux/config && setenforce 0

关闭虚拟内存:

swapoff -a
sed -ri 's/.*swap.*/#&/' /etc/fstab

配置内核参数

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

手动启用

modprobe overlay
modprobe br_netfilter

检查是否有启用
lsmod | grep -E "overlay|br_netfilter"

配置系统配置

cat >> /etc/sysctl.conf <<EOF
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward=1
EOF

生效配置:
sysctl -p

安装containerd

项目地址:
https://github.com/containerd/containerd

安装v1.7.29

wget "https://github.com/containerd/containerd/releases/download/v1.7.29/containerd-1.7.29-linux-amd64.tar.gz"
tar Cxzvf /usr/local containerd-1.7.29-linux-amd64.tar.gz

配置服务:

wget "https://github.com/containerd/containerd/blob/main/containerd.service"
cp containerd.service /etc/systemd/system/containerd.service
systemctl daemon-reload
systemctl enable --now containerd

安装runc

项目地址:
https://github.com/opencontainers/runc

安装v1.2.9

wget "https://github.com/opencontainers/runc/releases/download/v1.2.9/runc.amd64"
install -m 755 runc.amd64 /usr/local/sbin/runc

切换containerd源

mkdir /etc/containerd
containerd config default > /etc/containerd/config.toml
sed -i 's/registry.k8s.io/registry.aliyuncs.com\/google_containers/' /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml

重启服务生效

systemctl daemon-reload
systemctl restart containerd

安装kubelet kubeadm kubectl

调整下载源，这边使用的是清华源:
https://mirrors.tuna.tsinghua.edu.cn/help/kubernetes/

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=kubernetes
baseurl=https://mirrors.tuna.tsinghua.edu.cn/kubernetes/yum/repos/kubernetes-el7-$basearch
name=Kubernetes
baseurl=https://mirrors.tuna.tsinghua.edu.cn/kubernetes/core:/stable:/v1.34/rpm/
enabled=1
gpgcheck=1
gpgkey=https://mirrors.tuna.tsinghua.edu.cn/kubernetes/core:/stable:/v1.34/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

安装:

yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
systemctl enable --now kubelet
kubeadm config images pull --image-repository registry.aliyuncs.com/google_containers

初始化主节点(init-default.yaml中的IP需要按实际配置):

kubeadm config print init-defaults > /etc/kubernetes/init-default.yaml
sed -i 's/registry.k8s.io/registry.aliyuncs.com\/google_containers/' /etc/kubernetes/init-default.yaml
sed -i 's/1.2.3.4/10.61.22.15/' /etc/kubernetes/init-default.yaml

准备初始化配置文件:

cat <<EOF | sudo tee kubeadm-init.yaml
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.34.2
controlPlaneEndpoint: "cnnode1.synohyperv.local:6443"
imageRepository: registry.aliyuncs.com/google_containers

networking:
  podSubnet: "192.168.215.0/21"
  serviceSubnet: "192.168.207.0/21"
EOF

执行初始化
kubeadm init --config kubeadm-init.yaml

初始化完成后结果如下:

完成初始化

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

其他节点使用初始化完成后的kubeadm join命令进行集群加入。
如果超时则使用以下命令生成:
kubeadm token create --print-join-command

加入后通过以下命令查看当前集群状态:
kubectl get nodes -A

修改NodePort范围:
vi /etc/kubernetes/manifests/kube-apiserver.yaml

添加NodePort限制到kube-apiserver配置中:
- --service-node-port-range=5000-30000

安装命令自动补全

yum install -y bash-completion
echo 'source /etc/profile.d/bash_completion.sh' >> ~/.bashrc
source ~/.bashrc

安装Calico网络插件

项目地址:
https://github.com/projectcalico/calico/blob/master/manifests/calico.yaml

这边使用的国内镜像源:
https://docker.aityp.com/

安装v3.30.3:
sed -i 's/v3\.31\.2/v3.30.3/g' calico.yaml

修改下载源为镜像源:
sed -i 's@quay.io/calico/@swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/calico/@g' calico.yaml

开始部署:
kubectl apply -f calico.yaml

确认服务状态:
kubectl get po -A -o wide

确认网络状态:
calicoctl get ippools -o wide

切换网络

修改/etc/kubernetes/manifests/kube-controller-manager.yaml中的--cluster-cidr与--service-cluster-ip-range

修改/etc/kubernetes/manifests/kube-apiserver.yaml中的--service-cluster-ip-range

修改kubectl edit cm -n kube-system kubeadm-config中的podSubnet与serviceSubnet

修改kubectl edit cm -n kube-system kube-proxy中的clusterCIDR

修改kubectl edit ippool default-ipv4-ippool中的cidr

删除所有的node:

kubectl delete node cnnode1.synohyperv.local
kubectl delete node cnnode2.synohyperv.local
kubectl delete node cnnode3.synohyperv.local
kubectl delete node cnnode4.synohyperv.local

在所有节点重启服务以及重建所有容器，以重新分配新的地址:

systemctl restart containerd
systemctl restart kubelet

部署Kubernetes Dashboard

项目地址:
https://github.com/kubernetes/dashboard

下载kubernetes-dashboard压缩包，通过以下命令部署:

helm upgrade --install kubernetes-dashboard ./kubernetes-dashboard-7.14.0.tgz \
  --namespace kubernetes-dashboard --create-namespace \
  --set=web.image.repository=swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/kubernetesui/dashboard-web \
  --set=web.image.tag=1.7.0 \
  --set=auth.image.repository=swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/kubernetesui/dashboard-auth \
  --set=auth.image.tag=1.4.0 \
  --set=metricsScraper.image.repository=swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/kubernetesui/dashboard-metrics-scraper \
  --set=metricsScraper.image.tag=1.2.2 \
  --set=api.image.repository=swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/kubernetesui/dashboard-api \
  --set=api.image.tag=1.14.0 \
  --set=kong.image.repository=swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/library/kong \
  --set=kong.image.tag=3.9

确认服务状态:
kubectl get po -A -o wide

修改服务端口:
kubectl patch svc kubernetes-dashboard-kong-proxy -n kubernetes-dashboard -p '{"spec":{"type":"NodePort","ports":[{"port":443,"nodePort":5001}]}}'

确认网络状态:
kubectl get svc -n kubernetes-dashboard

创建管理用户配置档:

cat > dashboard-adminuser.yaml << EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard
EOF

创建管理员用户:
kubectl apply -f dashboard-adminuser.yaml

生成长期登录许可:
kubectl -n kubernetes-dashboard create token admin-user --duration=87600h

Synology CSI

安装Synology CSI

安装external-snapshotter
https://github.com/kubernetes-csi/external-snapshotter

snapshot.storage.k8s.io_volumesnapshotclasses.yaml
snapshot.storage.k8s.io_volumesnapshotcontents.yaml
snapshot.storage.k8s.io_volumesnapshots.yaml

kubectl apply -f snapshot.storage.k8s.io_volumesnapshots.yaml
kubectl apply -f snapshot.storage.k8s.io_volumesnapshotclasses.yaml
kubectl apply -f snapshot.storage.k8s.io_volumesnapshotcontents.yaml

确认Snapshot CRD有安装:
kubectl get crd | grep snapshot

安装synology-csi
https://github.com/SynologyOpenSource/synology-csi

在所有节点预导入镜像:
ctr -n=k8s.io images import synology-csi-v1.2.1.tar

确认镜像:
ctr -n=k8s.io images ls | grep synology

创建秘钥
vi config/client-info.yml

内容如下:

clients:
- host: 192.168.1.1
  port: 5000
  https: false
  username: <username>
  password: <password>
- host: 192.168.1.2
  port: 5001
  https: true
  username: <username>
  password: <password>

安装:
./scripts/deploy.sh install --all
检查容器状态
kubectl get po -n synology-csi -o wide
确认组件是否有安装:

kubectl get storageclass
kubectl get csidriver

确认秘钥:
kubectl get secret -n synology-csi client-info-secret -o jsonpath="{.data['client-info\.yml']}" | base64 -d

添加NAS存储节点

修改秘钥文件:
vi config/client-info.yml

重建synology-csi(重建过程中不能执行 create/clone/delete/resize/snapshot PV 等操作):
./scripts/uninstall.sh; ./scripts/deploy.sh install --all

创建新的StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
  name: synology-iscsi-201-storage
provisioner: csi.san.synology.com
parameters:
  fsType: 'btrfs'
  dsm: '10.61.21.201'
  location: '/volume2'
  enableSpaceReclamation: 'true'
  formatOptions: '--nodiscard'
reclaimPolicy: Retain
allowVolumeExpansion: true

配置连接ISCSI存储

注意:如果使用该CSI会导致node节点端其他的iscsiadm配置被清除

创建StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
  name: synology-iscsi-202-storage
provisioner: csi.san.synology.com
parameters:
  fsType: 'btrfs'
  dsm: '10.61.21.202'
  location: '/volume1'
  formatOptions: '--nodiscard'
reclaimPolicy: Retain
allowVolumeExpansion: true

创建测试PVC synology-test-pvc-iscsi.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: synology-test-pvc-iscsi
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: synology-iscsi-202-storage

查看synology-test-pvc-iscsi.yaml配置:
kubectl get pvc

测试创建支持飘移的Pod test-iscsi-pod.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: iscsi-test
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: iscsi-test
  template:
    metadata:
      labels:
        app: iscsi-test
    spec:
      restartPolicy: Always
      containers:
      - name: app
        image: swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/library/busybox:latest
        command: ["sh", "-c", "sleep 36000"]
        volumeMounts:
        - mountPath: /iscsi1
          name: iscsi1
      volumes:
      - name: iscsi1
        persistentVolumeClaim:
          claimName: synology-test-pvc-iscsi

创建后查看NAS中target状态:

模拟故障转移从节点2飘移至节点4:

飘移后NAS中target状态:

如果希望删除pvs后一并将LUN删除，需要修改persistentVolumeReclaimPolicy为Delete。

查询pv名称:
kubectl get pv

修改pv配置:
kubectl edit pv pvc-d41a9426-fd06-4063-b246-86e7d72a0a7a

配置连接NFS存储

创建StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: synology-nfs-202-storage
provisioner: csi.san.synology.com
parameters:
  protocol: "nfs"
  dsm: "10.61.21.202"
  location: '/volume1'
  mountPermissions: '0755'
mountOptions:
  - nfsvers=4.1
reclaimPolicy: Retain
allowVolumeExpansion: true

创建测试PVC synology-test-pvc-nfs.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: synology-test-pvc-nfs
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: synology-nfs-202-storage

NAS端会自动创建共享文件夹:

测试创建支持飘移的Pod test-iscsi-pod.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: synonfs-test
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: synonfs-test
  template:
    metadata:
      labels:
        app: synonfs-test
    spec:
      restartPolicy: Always
      containers:
      - name: app
        image: swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/library/busybox:latest
        command: ["sh", "-c", "sleep 36000"]
        volumeMounts:
        - mountPath: /nfs1
          name: nfs1
      volumes:
      - name: nfs1
        persistentVolumeClaim:
          claimName: synology-test-pvc-nfs

模拟故障转移从节点3飘移至节点2:

NAS端可以查看NFS中的文件:

NFS CSI安装

安装csi-driver-nfs
https://github.com/kubernetes-csi/csi-driver-nfs

下载以下配置文件:

rbac-csi-nfs.yaml
csi-nfs-driverinfo.yaml
csi-nfs-controller.yaml
csi-nfs-node.yaml

修改下载源为镜像源:
sed -i 's@registry.k8s.io/sig-storage/@swr.cn-north-4.myhuaweicloud.com/ddn-k8s/registry.k8s.io/sig-storage/@g' *

检查容器状态
kubectl get po -A -o wide

应用配置:

kubectl apply -f rbac-csi-nfs.yaml
kubectl apply -f csi-nfs-driverinfo.yaml
kubectl apply -f csi-nfs-controller.yaml
kubectl apply -f csi-nfs-node.yaml

定义StorageClass nfs-sc.yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-sc
provisioner: nfs.csi.k8s.io
parameters:
  server: 10.61.21.202
  share: /volume1/k8snfs
reclaimPolicy: Retain
volumeBindingMode: Immediate
mountOptions:
  - nfsvers=4.1

应用nfs配置:
kubectl apply -f nfs-sc.yaml

查看StorageClass:
kubectl get sc

查看StorageClass的详细内容:
kubectl describe sc nfs-sc

创建nfs PVC nfs-test-pvc.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-test-pvc
spec:
  storageClassName: nfs-sc
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 20Gi

应用nfs配置:
kubectl apply -f nfs-test-pvc.yaml

查看nfs-test-pvc配置:
kubectl get pvc

测试创建支持飘移的Pod test-nfs-pod.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-test
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nfs-test
  template:
    metadata:
      labels:
        app: nfs-test
    spec:
      restartPolicy: Always
      containers:
      - name: app
        image: swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/library/busybox:latest
        command: ["sh", "-c", "sleep 36000"]
        volumeMounts:
        - mountPath: /nfstestdata
          name: nfs-test-storage
      volumes:
      - name: nfs-test-storage
        persistentVolumeClaim:
          claimName: nfs-test-pvc

创建测试容器:
kubectl apply -f test-nfs-pod.yaml

查看容器状态:
kubectl get pod -o wide | grep nfs-test

测试写入:
kubectl exec -it nfs-test-59984f489b-b6wql -- sh

模拟节点宕机:
kubectl drain cnnode3.synohyperv.local --ignore-daemonsets --delete-emptydir-data

查看容器状态:
kubectl get pod -o wide | grep nfs-test

恢复节点可调度:
kubectl uncordon cnnode3.synohyperv.local

Note

检查镜像
kubeadm config images list --kubernetes-version v1.33.0 --image-repository mirrors.tuna.tsinghua.edu.cn/kubernetes/google_containers

查看集群状态
kubectl get nodes -A

检查容器状态
kubectl get po -A -o wide

查看容器网络状态
kubectl get svc -n kubernetes-dashboard

部署项目
kubectl apply -f kube-flannel.yml

取消部署
kubectl delete -f kube-flannel.yml

配置容器网络
kubectl patch svc kubernetes-dashboard -n kubernetes-dashboard -p '{"spec":{"type":"NodePort","ports":[{"port":443,"nodePort":5001}]}}'

创建自动部署路径:
mkdir -p /etc/kubernetes/manifests

查看存储状态
kubectl get pv

编辑容器模板
kubectl get deploy
kubectl edit deploy synonfs-test

删除容器
kubectl delete -n ubernetes-dashboard all --all
kubectl delete namespace ubernetes-dashboard

kubectl delete pod iscsi-test -n default

查看有哪些镜像:
ctr -n=k8s.io images ls

导出导入镜像
ctr -n=k8s.io images export synology-csi-v1.2.1.tar docker.io/synology/synology-csi:v1.2.1

ctr -n=k8s.io images import synology-csi-v1.2.1.tar

查看命名空间容器
kubectl get all -n synology-csi

查看存储插件
kubectl get storageclass
kubectl get csidriver

检查存储状态
kubectl describe pvc synology-test-pvc-iscsi

检查容器状态
kubectl describe pod synonfs-test-66c9ddff7c-s7nqq -n default

检查容器日志
kubectl logs -f nfs-test-59984f489b-jfv9p -n default

查看节点内部IP分配情况
kubectl get node -o json | jq '.items[].spec.podCIDR'

查看集群证书有效期
kubeadm certs check-expiration

更新证书
先备份 /var/lib/etcd/ /etc/kubernetes/
kubeadm certs renew all

限制资源

resources:
    requests:
    cpu: "200m"
    memory: "256Mi"
limits:
    cpu: "500m"
    memory: "512Mi"