离线安装 Kubernetes (K8s)

简单记录一下离线环境使用Kubekey安装Kubernetes集群的步骤。

目录

1. 前提条件
2. 可选步骤
3. 步骤 11：导入容器镜像
4. 步骤 12：重装操作
5. 总结

---

前提条件

• 操作系统：本文以 麒麟V10 系统为例，需要保证可以从外部传输文件进来。
• 权限：需要具有 root 权限或使用 sudo 执行命令。
• 文件准备：确保所有必要的离线安装包已下载并解压到指定目录。如果安装过程有下载提示，请检查是否有文件缺失，正常情况是不会下载的。如果不存在请从已安装好的服务器上拷贝对应的文件夹。

---

可选步骤

---

步骤 1：解压所有文件

首先，解压所有下载的离线安装文件。并且在文件夹下有对应的二进制文件。

[root@master kubekey]# ls
cni  config-sample  containerd  crictl  etcd  helm  kube  logs  master  pki  runc  slave1  slave2  docker

---

步骤 ：安装 containerd

1 进入 containerd 目录并解压

[root@master kubekey]# cd containerd/
[root@master containerd]# ls
1.6.4
[root@master containerd]# cd 1.6.4/amd64/
[root@master amd64]# ls
containerd-1.6.4-linux-amd64.tar.gz
[root@master amd64]# tar -zxvf containerd-1.6.4-linux-amd64.tar.gz 

解压后，目录结构如下：

bin/
bin/ctr
bin/containerd-shim
bin/containerd
bin/containerd-shim-runc-v1
bin/containerd-shim-runc-v2
bin/containerd-stress

2 复制二进制文件到 /usr/local/bin/ 并赋予执行权限

[root@master amd64]# cp bin/* /usr/local/bin/
[root@master amd64]# chmod +x /usr/local/bin/ctr
[root@master amd64]# chmod +x /usr/local/bin/containerd*

---

步骤 ：配置并启动 containerd

1 创建配置目录并生成默认配置

[root@master amd64]# sudo mkdir -p /etc/containerd
[root@master amd64]# containerd config default | sudo tee /etc/containerd/config.toml

2 创建 systemd 服务文件

编辑 /etc/systemd/system/containerd.service 文件：

[root@master amd64]# sudo vim /etc/systemd/system/containerd.service

添加以下内容：

[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target

[Service]
ExecStart=/usr/local/bin/containerd
Restart=always
RestartSec=5
Delegate=yes
KillMode=process
OOMScoreAdjust=-999
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity

[Install]
WantedBy=multi-user.target

3 启动并启用 containerd 服务

[root@master amd64]# sudo systemctl daemon-reload
[root@master amd64]# sudo systemctl start containerd
[root@master amd64]# sudo systemctl enable containerd

---

步骤 ：安装 crictl

1 进入 crictl 目录并解压

[root@master kubekey]# cd crictl/
[root@master crictl]# ls
v1.24.0
[root@master crictl]# cd v1.24.0/amd64/
[root@master amd64]# ls
crictl-v1.24.0-linux-amd64.tar.gz
[root@master amd64]# tar -zxvf crictl-v1.24.0-linux-amd64.tar.gz 
crictl

2 复制并赋予执行权限

[root@master amd64]# chmod +x crictl
[root@master amd64]# cp crictl /usr/local/bin/

---

步骤 ：安装 etcd

1 进入 etcd 目录并解压

[root@master kubekey]# cd etcd/
[root@master etcd]# ls
v3.4.13
[root@master etcd]# cd v3.4.13/amd64/
[root@master amd64]# ls
etcd-v3.4.13-linux-amd64.tar.gz
[root@master amd64]# tar -zxvf etcd-v3.4.13-linux-amd64.tar.gz 
[root@master amd64]# cd etcd-v3.4.13-linux-amd64

2 复制并赋予执行权限

[root@master amd64]# cp etcd etcdctl /usr/local/bin/
[root@master amd64]# chmod +x /usr/local/bin/etcd*

---

步骤 ：安装 Helm

1 进入 Helm 目录并复制二进制文件

[root@master kubekey]# cd helm/v3.9.0/amd64/
[root@master amd64]# ls
helm
[root@master amd64]# chmod +x helm 
[root@master amd64]# cp helm /usr/local/bin/

---

步骤 ：安装 Kubernetes 二进制文件

1 进入 kube 目录并解压

[root@master kubekey]# cd kube/
[root@master kube]# ls
v1.23.10
[root@master kube]# cd v1.23.10/
[root@master v1.23.10]# ls
amd64/
[root@master v1.23.10]# cd amd64/
[root@master amd64]# ls
kubeadm  kubectl  kubelet

2 赋予执行权限并复制

[root@master amd64]# chmod +x *
[root@master amd64]# cp * /usr/local/bin/

---

步骤 ：安装 runc

1 进入 runc 目录并解压

[root@master kubekey]# cd runc/
[root@master runc]# cd v1.1.1/amd64/
[root@master amd64]# ls
runc.amd64
[root@master amd64]# chmod +x runc.amd64 

2 复制并重命名为 runc

[root@master amd64]# cp runc.amd64 /usr/local/bin/runc

3 验证安装

[root@master amd64]# runc --version
runc version 1.1.1
commit: v1.1.0-20-g52de29d7
spec: 1.0.2-dev

---

步骤 ：安装 CNI 插件

1 创建 CNI 插件目录

[root@master kubekey]# mkdir -p /opt/cni/bin/

2 解压并复制 CNI 插件

[root@master kubekey]# tar -zxvf cni/v1.2.0/amd64/cni-plugins-linux-amd64-v1.2.0.tgz -C /opt/cni/bin/

3 赋予执行权限

[root@master kubekey]# sudo chmod +x /opt/cni/bin/*

---

步骤 ：安装 calicoctl

1 复制 calicoctl 并赋予执行权限

[root@master kubekey]# cp cni/v3.26.1/amd64/calicoctl /usr/local/bin/
[root@master kubekey]# sudo chmod +x /usr/local/bin/calicoctl

2 验证安装

[root@master kubekey]# calicoctl version
Client Version:    v3.26.1
Git commit:        b1d192c95
Unable to detect installed Calico version

---

步骤 ：安装 docker

1 进入docker目录并解压

[root@km1 kubekey]# cd docker/
[root@km1 docker]# ls
24.0.6
[root@km1 docker]# cd 24.0.6/
[root@km1 24.0.6]# ls
amd64
[root@km1 24.0.6]# cd amd64/
[root@km1 amd64]# ls
docker-24.0.6.tgz
[root@km1 amd64]# tar -xvf docker-24.0.6.tgz 
docker/
docker/docker
docker/docker-init
docker/dockerd
docker/runc
docker/ctr
docker/containerd-shim-runc-v2
docker/containerd
docker/docker-proxy

2 移动文件到/usr/bin/下

mv docker/* /usr/bin/

3 编辑docker.service文件

vi /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target

4 添加docker.service文件的权限

chmod +x /usr/lib/systemd/system/docker.service

5 创建daemon.json文件

mkdir -p /etc/docker
vim /etc/docker/daemon.json
{}

6 验证安装

systemctl daemon-reload
systemctl start docker
systemctl enable docker
docker -v

7 可选-安装docker-compose

docker-compose下载地址：https://github.com/docker/compose/releases 选择对应的架构，如果可以联网，也可以直接执行命令下载

cd /home
wget https://github.com/docker/compose/releases/download/v2.25.0/docker-compose-linux-x86_64

安装并分配权限

mv docker-compose-linux-x86_64 /usr/bin/docker-compose
chmod +x /usr/bin/docker-compose

验证

docker-compose -v

---

步骤 11：导入容器镜像

11.1 创建镜像仓库

导入镜像docker.io/registry

docker load -i registry.tar

创建一个本地镜像存储路径

mkdir -p /root/registry-data 

启动registry容器

docker run 
--restart=always
-itd --name=registry  \
-v /root/registry-data:/var/lib/registry  \
-p 5000:5000 registry:latest

修改daemon.json

vim /etc/docker/daemon.json
{ 
    "insecure-registries":["127.0.0.1:5000"]
}

重启docker

systemctl daemon-reload
systemctl restart docker

11.2 导入镜像

进入存放镜像文件的目录并导入所有 .tar 文件：

[root@master kubekey]# cd image/image_file/
[root@master image_file]# ls
registry.cn-beijing.aliyuncs.com_kubesphereio_cni:v3.26.1.tar
registry.cn-beijing.aliyuncs.com_kubesphereio_coredns:1.8.6.tar
registry.cn-beijing.aliyuncs.com_kubesphereio_k8s-dns-node-cache:1.15.12.tar
registry.cn-beijing.aliyuncs.com_kubesphereio_kube-apiserver:v1.23.10.tar
registry.cn-beijing.aliyuncs.com_kubesphereio_kube-controller-manager:v1.23.10.tar
registry.cn-beijing.aliyuncs.com_kubesphereio_kube-controllers:v3.26.1.tar
registry.cn-beijing.aliyuncs.com_kubesphereio_kube-proxy:v1.23.10.tar
registry.cn-beijing.aliyuncs.com_kubesphereio_kube-scheduler:v1.23.10.tar
registry.cn-beijing.aliyuncs.com/kubesphereio_node:v3.26.1.tar

[root@master image_file]# for image in *.tar; do docker load -i "$image"; done
[root@master image_file]# docker images

导入完成后，您可以看到已导入的镜像列表。将镜像重新命名并push到镜像仓库中。

---

步骤 12：重装操作

在需要重新安装 Kubernetes 集群时，请按照以下步骤操作：

12.1 删除现有 etcd 数据

[root@master kubekey]# rm -rf ~/kube/kubekey/pki/etcd/*

12.2 删除现有集群

[root@master kubekey]# ./kk delete cluster -y -f config.yaml

---

12.3 安装Kubernetes

首先需要设置一下kubekey的镜像源

  registry:
    privateRegistry: "127.0.0.1:5000"
    namespaceOverride: ""
    registryMirrors: []
    insecureRegistries: []

之后执行即可安装

./kk create cluster -f config.yaml

---

12.4 安装kubesphere

首先执行下面命令确保命名空间

./kk create cluster --with-local-storage -f config.yaml

之后需要准备安装镜像，需要注意版本

curl -L -O https://github.com/kubesphere/ks-installer/releases/download/v3.4.1/images-list.txt
curl -L -O https://github.com/kubesphere/ks-installer/releases/download/v3.4.1/offline-installation-tool.sh
chmod +x offline-installation-tool.sh
./offline-installation-tool.sh -s -l images-list.txt -d ./kubesphere-images
./offline-installation-tool.sh -l images-list.txt -d ./kubesphere-images -r 127.0.0.1:5000

推送到自己搭建的镜像仓库中，如127.0.0.1:5000 根据需要选择拉取的镜像。例如，如果已经有一个 Kubernetes 集群了，可以在 images-list.text 中删除 ##k8s-images 和在它下面的相关镜像。 之后下载部署文件

curl -L -O https://github.com/kubesphere/ks-installer/releases/download/v3.4.1/cluster-configuration.yaml
curl -L -O https://github.com/kubesphere/ks-installer/releases/download/v3.4.1/kubesphere-installer.yaml

编辑cluster-configuration.yaml，在local_registry添加上本地镜像仓库的地址

spec:
  persistence:
    storageClass: ""        # If there is no default StorageClass in your cluster, you need to specify an existing StorageClass here.
  authentication:
    # adminPassword: ""     # Custom password of the admin user. If the parameter exists but the value is empty, a random password is generated. If the parameter does not exist, P@88w0rd is used.
    jwtSecret: ""           # Keep the jwtSecret consistent with the Host Cluster. Retrieve the jwtSecret by executing "kubectl -n kubesphere-system get cm kubesphere-config -o yaml | grep -v "apiVersion" | grep jwtSecret" on the Host Cluster.
  local_registry: "127.0.0.1:5000"        # Add your private registry address if it is needed.
  etcd:
    monitoring: false       # Enable or disable etcd monitoring dashboard installation. You have to create a Secret for etcd before you enable it.
    endpointIps: localhost  # etcd cluster EndpointIps. It can be a bunch of IPs here.
    port: 2379              # etcd port.
    tlsEnable: true
  common:
    core:
      console:
        enableMultiLogin: true  # Enable or disable simultaneous logins. It allows different users to log in with the same account at the same time.
        port: 30880
        type: NodePort

然后替换kubesphere-installer.yaml的仓库地址，也可以用脚本替换sed -i "s#^\s*image: kubesphere.*/ks-installer:.*# image: 127.0.0.1:5000/kubesphere/ks-installer:v3.4.0#" kubesphere-installer.yaml

spec:
  replicas: 1
  selector:
    matchLabels:
      app: ks-installer
  template:
    metadata:
      labels:
        app: ks-installer
    spec:
      serviceAccountName: ks-installer
      containers:
      - name: installer
        image: 127.0.0.1:5000/kubesphere/ks-installer:v3.4.1
        imagePullPolicy: "Always"

之后进行安装即可

kubectl apply -f kubesphere-installer.yaml
kubectl apply -f cluster-configuration.yaml

等待安装即可，然后查看日志会有地址用户名密码等信息

kubectl get pods -A
kubectl logs -n kubesphere-system ks-installer-xxxxxxxxxx

总结

保证有一个已经成功安装好的Kubernetes集群很重要，缺什么直接从里面取即可。 有很大概率会出现镜像缺失的问题，建议仔细检查一下，如果缺失随时补充。

---