GCDW技术栈- docker运行环境搭建

GCDW运行在k8s上,本文介绍docker运行环境的安装,特别是containerd服务的配置,与harbor镜像集群的交互证书的配置等。

上一步

GCDW技术栈-kubernets操作系统环境准备

安装docker

如果没有docker yum源,可以参考上一步的操作系统环境准备文章

yum install docker-ce docker-ce-cli containerd.io docker-compose-plugin

[root@vm246 172.16.3.246]# yum install docker-ce docker-ce-cli containerd.io docker-compose-plugin
Loaded plugins: fastestmirror, langpacks
Repository base is listed more than once in the configuration
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
Loading mirror speeds from cached hostfile
 * base: mirrors.bfsu.edu.cn
 * extras: mirrors.bfsu.edu.cn
 * updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package containerd.io.x86_64 0:1.6.21-3.1.el7 will be installed
--> Processing Dependency: container-selinux >= 2:2.74 for package: containerd.io-1.6.21-3.1.el7.x86_64
---> Package docker-ce.x86_64 3:24.0.2-1.el7 will be installed
--> Processing Dependency: docker-ce-rootless-extras for package: 3:docker-ce-24.0.2-1.el7.x86_64
---> Package docker-ce-cli.x86_64 1:24.0.2-1.el7 will be installed
--> Processing Dependency: docker-buildx-plugin for package: 1:docker-ce-cli-24.0.2-1.el7.x86_64
---> Package docker-compose-plugin.x86_64 0:2.18.1-1.el7 will be installed
--> Running transaction check
---> Package container-selinux.noarch 2:2.119.2-1.911c772.el7_8 will be installed
---> Package docker-buildx-plugin.x86_64 0:0.10.5-1.el7 will be installed
---> Package docker-ce-rootless-extras.x86_64 0:24.0.2-1.el7 will be installed
--> Processing Dependency: fuse-overlayfs >= 0.7 for package: docker-ce-rootless-extras-24.0.2-1.el7.x86_64
--> Processing Dependency: slirp4netns >= 0.4 for package: docker-ce-rootless-extras-24.0.2-1.el7.x86_64
--> Running transaction check
---> Package fuse-overlayfs.x86_64 0:0.7.2-6.el7_8 will be installed
--> Processing Dependency: libfuse3.so.3(FUSE_3.2)(64bit) for package: fuse-overlayfs-0.7.2-6.el7_8.x86_64
--> Processing Dependency: libfuse3.so.3(FUSE_3.0)(64bit) for package: fuse-overlayfs-0.7.2-6.el7_8.x86_64
--> Processing Dependency: libfuse3.so.3()(64bit) for package: fuse-overlayfs-0.7.2-6.el7_8.x86_64
---> Package slirp4netns.x86_64 0:0.4.3-4.el7_8 will be installed
--> Running transaction check
---> Package fuse3-libs.x86_64 0:3.6.1-4.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=====================================================================================================================================================================
 Package                                       Arch                       Version                                         Repository                            Size
=====================================================================================================================================================================
Installing:
 containerd.io                                 x86_64                     1.6.21-3.1.el7                                  docker-ce-stable                      34 M
 docker-ce                                     x86_64                     3:24.0.2-1.el7                                  docker-ce-stable                      24 M
 docker-ce-cli                                 x86_64                     1:24.0.2-1.el7                                  docker-ce-stable                      13 M
 docker-compose-plugin                         x86_64                     2.18.1-1.el7                                    docker-ce-stable                      12 M
Installing for dependencies:
 container-selinux                             noarch                     2:2.119.2-1.911c772.el7_8                       extras                                40 k
 docker-buildx-plugin                          x86_64                     0.10.5-1.el7                                    docker-ce-stable                      12 M
 docker-ce-rootless-extras                     x86_64                     24.0.2-1.el7                                    docker-ce-stable                     9.1 M
 fuse-overlayfs                                x86_64                     0.7.2-6.el7_8                                   extras                                54 k
 fuse3-libs                                    x86_64                     3.6.1-4.el7                                     extras                                82 k
 slirp4netns                                   x86_64                     0.4.3-4.el7_8                                   extras                                81 k

Transaction Summary
=====================================================================================================================================================================
Install  4 Packages (+6 Dependent packages)

Total download size: 105 M
Installed size: 372 M
Is this ok [y/d/N]: y
Downloading packages:
(1/10): container-selinux-2.119.2-1.911c772.el7_8.noarch.rpm                                                                                  |  40 kB  00:00:00
warning: /var/cache/yum/x86_64/7/docker-ce-stable/packages/docker-buildx-plugin-0.10.5-1.el7.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 621e9f35: NOKEY6 ETA
Public key for docker-buildx-plugin-0.10.5-1.el7.x86_64.rpm is not installed
(2/10): docker-buildx-plugin-0.10.5-1.el7.x86_64.rpm                                                                                          |  12 MB  00:00:07
(3/10): containerd.io-1.6.21-3.1.el7.x86_64.rpm                                                                                               |  34 MB  00:00:18
(4/10): docker-ce-24.0.2-1.el7.x86_64.rpm                                                                                                     |  24 MB  00:00:13
(5/10): docker-ce-rootless-extras-24.0.2-1.el7.x86_64.rpm                                                                                     | 9.1 MB  00:00:04
(6/10): fuse3-libs-3.6.1-4.el7.x86_64.rpm                                                                                                     |  82 kB  00:00:00
(7/10): fuse-overlayfs-0.7.2-6.el7_8.x86_64.rpm                                                                                               |  54 kB  00:00:00
(8/10): slirp4netns-0.4.3-4.el7_8.x86_64.rpm                                                                                                  |  81 kB  00:00:00
(9/10): docker-ce-cli-24.0.2-1.el7.x86_64.rpm                                                                                                 |  13 MB  00:00:07
(10/10): docker-compose-plugin-2.18.1-1.el7.x86_64.rpm                                                                                        |  12 MB  00:00:05
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                3.4 MB/s | 105 MB  00:00:31
Retrieving key from https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
Importing GPG key 0x621E9F35:
 Userid     : "Docker Release (CE rpm) <docker@docker.com>"
 Fingerprint: 060a 61c5 1b55 8a7f 742b 77aa c52f eb6b 621e 9f35
 From       : https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
Is this ok [y/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : 2:container-selinux-2.119.2-1.911c772.el7_8.noarch                                                                                               1/10
  Installing : containerd.io-1.6.21-3.1.el7.x86_64                                                                                                              2/10
  Installing : docker-buildx-plugin-0.10.5-1.el7.x86_64                                                                                                         3/10
  Installing : slirp4netns-0.4.3-4.el7_8.x86_64                                                                                                                 4/10
  Installing : fuse3-libs-3.6.1-4.el7.x86_64                                                                                                                    5/10
  Installing : fuse-overlayfs-0.7.2-6.el7_8.x86_64                                                                                                              6/10
  Installing : docker-compose-plugin-2.18.1-1.el7.x86_64                                                                                                        7/10
  Installing : 1:docker-ce-cli-24.0.2-1.el7.x86_64                                                                                                              8/10
  Installing : docker-ce-rootless-extras-24.0.2-1.el7.x86_64                                                                                                    9/10
  Installing : 3:docker-ce-24.0.2-1.el7.x86_64                                                                                                                 10/10
  Verifying  : 3:docker-ce-24.0.2-1.el7.x86_64                                                                                                                  1/10
  Verifying  : docker-compose-plugin-2.18.1-1.el7.x86_64                                                                                                        2/10
  Verifying  : fuse3-libs-3.6.1-4.el7.x86_64                                                                                                                    3/10
  Verifying  : fuse-overlayfs-0.7.2-6.el7_8.x86_64                                                                                                              4/10
  Verifying  : containerd.io-1.6.21-3.1.el7.x86_64                                                                                                              5/10
  Verifying  : slirp4netns-0.4.3-4.el7_8.x86_64                                                                                                                 6/10
  Verifying  : 2:container-selinux-2.119.2-1.911c772.el7_8.noarch                                                                                               7/10
  Verifying  : 1:docker-ce-cli-24.0.2-1.el7.x86_64                                                                                                              8/10
  Verifying  : docker-ce-rootless-extras-24.0.2-1.el7.x86_64                                                                                                    9/10
  Verifying  : docker-buildx-plugin-0.10.5-1.el7.x86_64                                                                                                        10/10

Installed:
  containerd.io.x86_64 0:1.6.21-3.1.el7    docker-ce.x86_64 3:24.0.2-1.el7    docker-ce-cli.x86_64 1:24.0.2-1.el7    docker-compose-plugin.x86_64 0:2.18.1-1.el7

Dependency Installed:
  container-selinux.noarch 2:2.119.2-1.911c772.el7_8        docker-buildx-plugin.x86_64 0:0.10.5-1.el7        docker-ce-rootless-extras.x86_64 0:24.0.2-1.el7
  fuse-overlayfs.x86_64 0:0.7.2-6.el7_8                     fuse3-libs.x86_64 0:3.6.1-4.el7                   slirp4netns.x86_64 0:0.4.3-4.el7_8

Complete!
[root@vm246 172.16.3.246]#

安装docker-compose

下载docker-compose-linux-x86_64 并改名mv到/usr/local/bin/docker-compose

注意下版本。我这里都是v2.18.1,和docker的版本一致

文件50M,网速不行的,可以只在1个节点下载,然后分发到别的。

[root@vm246 172.16.3.246]# wget https://github.com/docker/compose/releases/download/v2.18.1/docker-compose-linux-x86_64
--2023-06-15 18:26:54--  https://github.com/docker/compose/releases/download/v2.18.1/docker-compose-linux-x86_64
Resolving github.com (github.com)... 20.205.243.166
Connecting to github.com (github.com)|20.205.243.166|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/15045751/ebe621cd-2d6b-4306-b81c-eedc1b74e4da?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230616%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230616T012655Z&X-Amz-Expires=300&X-Amz-Signature=bbb81932fba33ad38be588df03770e70da5aff4ea444931750dd8320c591717d&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=15045751&response-content-disposition=attachment%3B%20filename%3Ddocker-compose-linux-x86_64&response-content-type=application%2Foctet-stream [following]
--2023-06-15 18:26:55--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/15045751/ebe621cd-2d6b-4306-b81c-eedc1b74e4da?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230616%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230616T012655Z&X-Amz-Expires=300&X-Amz-Signature=bbb81932fba33ad38be588df03770e70da5aff4ea444931750dd8320c591717d&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=15045751&response-content-disposition=attachment%3B%20filename%3Ddocker-compose-linux-x86_64&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.110.133, 185.199.109.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 54537935 (52M) [application/octet-stream]
Saving to: ‘docker-compose-linux-x86_64’

100%[===========================================================================================================================>] 54,537,935  13.0KB/s   in 35m 22s

2023-06-15 19:02:18 (25.1 KB/s) - ‘docker-compose-linux-x86_64’ saved [54537935/54537935]

[root@vm246 172.16.3.246]#

[root@vm246 172.16.3.246]# chmod a+x docker-compose-linux-x86_64
[root@vm246 172.16.3.246]# ./docker-compose-linux-x86_64  -v
Docker Compose version v2.18.1
[root@vm246 172.16.3.246]#

[root@vm246 172.16.3.246]# mv docker-compose-linux-x86_64  /usr/local/bin/docker-compose
[root@vm246 172.16.3.246]# docker-compose  -v
Docker Compose version v2.18.1
[root@vm246 172.16.3.246]#

查看版本

[root@vm246 ~]# docker compose version
Docker Compose version v2.18.1
[root@vm246 ~]#

修改docker的镜像服务器配置

其中https://172.16.3.249:8443是harbor镜像服务器的地址。

[root@k8s-81 ~]# cat /etc/docker/daemon.json
{
 "registry-mirrors": [
    "https://registry.docker-cn.com",
    "http://hub-mirror.c.163.com",
    "https://docker.mirrors.ustc.edu.cn",
    "https://172.16.3.249:8443"
  ],
  "insecure-registries": [
  ],

  "log-opts": {
              "max-size": "10m"
            }
}

将harbor的证书复制过来

mkdir -p /etc/docker/certs.d
scp -r 172.16.3.249:/etc/docker/certs.d/172.16.3.249\:8443 /etc/docker/certs.d/



[root@vm246 172.16.3.246]# vi /etc/docker/daemon.json
[root@vm246 172.16.3.246]# mkdir -p /etc/docker/certs.d
[root@vm246 172.16.3.246]# scp -r 172.16.3.249:/etc/docker/certs.d/172.16.3.249\:8443 /etc/docker/certs.d/
The authenticity of host '172.16.3.249 (172.16.3.249)' can't be established.
ECDSA key fingerprint is SHA256:Xs1gi6NKPEsAxLRIL2NHIv7jG1vt68oBlWZ0YUe/Swk.
ECDSA key fingerprint is MD5:b4:9c:dd:e1:3c:42:28:8d:db:c5:a0:73:30:2f:60:78.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.16.3.249' (ECDSA) to the list of known hosts.
root@172.16.3.249's password:
172.16.3.249.cert                                                                                                                  100% 2053     1.4MB/s   00:00
172.16.3.249.crt                                                                                                                   100% 2053     2.1MB/s   00:00
172.16.3.249.key                                                                                                                   100% 3247     3.8MB/s   00:00
ca.crt                                                                                                                             100% 2029     2.2MB/s   00:00
[root@vm246 172.16.3.246]

[root@vm246 ~]# ll /etc/docker/
total 4
drwxr-xr-x. 3 root root  31 Jun 15 19:09 certs.d
-rw-r--r--. 1 root root 274 Jun 15 19:07 daemon.json
[root@vm246 ~]# ll /etc/docker/certs.d/
total 0
drwxr-xr-x. 2 root root 93 Jun 15 19:09 172.16.3.249:8443
[root@vm246 ~]# ll /etc/docker/certs.d/172.16.3.249\:8443/
total 16
-rw-r--r--. 1 root root 2053 Jun 15 19:09 172.16.3.249.cert
-rw-r--r--. 1 root root 2053 Jun 15 19:09 172.16.3.249.crt
-rw-r--r--. 1 root root 3247 Jun 15 19:09 172.16.3.249.key
-rw-r--r--. 1 root root 2029 Jun 15 19:09 ca.crt
[root@vm246 ~]#


修改containerd的配置

将docker访问harbor的证书复制一份

cp -r /etc/docker/certs.d/172.16.3.249\:8443/ /etc/containerd/


[root@vm246 ~]# ll /etc/containerd/
total 4
-rw-r--r--. 1 root root 886 May  5 13:20 config.toml
[root@vm246 ~]# cp -r /etc/docker/certs.d/172.16.3.249\:8443/ /etc/containerd/
[root@vm246 ~]# ll /etc/containerd/
total 4
drwxr-xr-x. 2 root root  93 Jun 15 19:11 172.16.3.249:8443
-rw-r--r--. 1 root root 886 May  5 13:20 config.toml
[root@vm246 ~]# ll /etc/containerd/172.16.3.249\:8443/
total 16
-rw-r--r--. 1 root root 2053 Jun 15 19:11 172.16.3.249.cert
-rw-r--r--. 1 root root 2053 Jun 15 19:11 172.16.3.249.crt
-rw-r--r--. 1 root root 3247 Jun 15 19:11 172.16.3.249.key
-rw-r--r--. 1 root root 2029 Jun 15 19:11 ca.crt
[root@vm246 ~]#

生成默认配置文件

containerd config default > /etc/containerd/config.toml

修改配置文件

将containerd配置文件里面的pause:2.6的镜像,改成能用的,比如. 注意项目名称(如下例子是library)。镜像服务器证书配置看后面。

以前的 
sandbox_image = "registry.k8s.io/pause:3.6"
替换成
sandbox_image = "172.16.3.249:8443/library/pause:3.6"

如上的pause:3.6如果不想自己上传到harbor,也可以从其它仓库下载,比如

[root@vm248 ~]# ctr -n k8s.io i pull registry.aliyuncs.com/google_containers/pause:3.6
registry.aliyuncs.com/google_containers/pause:3.6:                                resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:c2280d2f5f56cf9c9a01bb64b2db4651e35efd6d62a54dcfc12049fe6449c5e4: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:fbe1a72f5dcd08ba4ca3ce3468c742786c1f6578c1f6bb401be1c4620d6ff705:    done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:6270bb605e12e581514ada5fd5b3216f727db55dc87d5889c790e4c760683fee:   done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 1.6 s                                                                    total:  5.1 Ki (3.2 KiB/s)
unpacking linux/amd64 sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db...
done: 61.92735ms
[root@vm248 ~]# ctr -n k8s.io i tag registry.aliyuncs.com/google_containers/pause:3.6 registry.k8s.io/pause:3.6
registry.k8s.io/pause:3.6

修改服务IP

以前的
stream_server_address = "127.0.0.1"
改成, 具体IP已每个节点的实际IP为准
stream_server_address = "10.0.2.81"

下图中为修改后示意,具体以实际IP为准

配置镜像服务器证书,包括如下2个部分,分别是镜像地址和证书文件位置

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = ""

      [plugins."io.containerd.grpc.v1.cri".registry.auths]

      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."172.16.3.249:8443".tls]
            ca_file = "/etc/containerd/172.16.3.249:8443/ca.crt"

      [plugins."io.containerd.grpc.v1.cri".registry.headers]

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."172.16.3.249:8443"]
          endpoint = ["https://172.16.3.249:8443"]

systemd配置,从false改成true

plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options
...
 SystemdCgroup = true

完整的配置文件例子,红色是我改动过的部分

[root@k8s-82 containerd]# cat config.toml
disabled_plugins = []
imports = []
oom_score = 0
plugin_dir = ""
required_plugins = []
root = "/var/lib/containerd"
state = "/run/containerd"
temp = ""
version = 2

[cgroup]
  path = ""

[debug]
  address = ""
  format = ""
  gid = 0
  level = ""
  uid = 0

[grpc]
  address = "/run/containerd/containerd.sock"
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216
  tcp_address = ""
  tcp_tls_ca = ""
  tcp_tls_cert = ""
  tcp_tls_key = ""
  uid = 0

[metrics]
  address = ""
  grpc_histogram = false

[plugins]

  [plugins."io.containerd.gc.v1.scheduler"]
    deletion_threshold = 0
    mutation_threshold = 100
    pause_threshold = 0.02
    schedule_delay = "0s"
    startup_delay = "100ms"

  [plugins."io.containerd.grpc.v1.cri"]
    device_ownership_from_security_context = false
    disable_apparmor = false
    disable_cgroup = false
    disable_hugetlb_controller = true
    disable_proc_mount = false
    disable_tcp_service = true
    enable_selinux = false
    enable_tls_streaming = false
    enable_unprivileged_icmp = false
    enable_unprivileged_ports = false
    ignore_image_defined_volumes = false
    max_concurrent_downloads = 3
    max_container_log_line_size = 16384
    netns_mounts_under_state_dir = false
    restrict_oom_score_adj = false
    sandbox_image = "172.16.3.249:8443/library/pause:3.6"
    selinux_category_range = 1024
    stats_collect_period = 10
    stream_idle_timeout = "4h0m0s"
    stream_server_address = "10.0.2.82"
    stream_server_port = "0"
    systemd_cgroup = false
    tolerate_missing_hugetlb_controller = true
    unset_seccomp_profile = ""

    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      conf_template = ""
      ip_pref = ""
      max_conf_num = 1

    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runc"
      disable_snapshot_annotations = true
      discard_unpacked_layers = false
      ignore_rdt_not_enabled_errors = false
      no_pivot = false
      snapshotter = "overlayfs"

      [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
        base_runtime_spec = ""
        cni_conf_dir = ""
        cni_max_conf_num = 0
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_path = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          base_runtime_spec = ""
          cni_conf_dir = ""
          cni_max_conf_num = 0
          container_annotations = []
          pod_annotations = []
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_path = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            BinaryName = ""
            CriuImagePath = ""
            CriuPath = ""
            CriuWorkPath = ""
            IoGid = 0
            IoUid = 0
            NoNewKeyring = false
            NoPivotRoot = false
            Root = ""
            ShimCgroup = ""
            SystemdCgroup = true

      [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
        base_runtime_spec = ""
        cni_conf_dir = ""
        cni_max_conf_num = 0
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_path = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options]

    [plugins."io.containerd.grpc.v1.cri".image_decryption]
      key_model = "node"

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = ""

      [plugins."io.containerd.grpc.v1.cri".registry.auths]

      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."172.16.3.249:8443".tls]
            ca_file = "/etc/containerd/172.16.3.249:8443/ca.crt"

      [plugins."io.containerd.grpc.v1.cri".registry.headers]

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."172.16.3.249:8443"]
          endpoint = ["https://172.16.3.249:8443"]

    [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""

  [plugins."io.containerd.internal.v1.opt"]
    path = "/opt/containerd"

  [plugins."io.containerd.internal.v1.restart"]
    interval = "10s"

  [plugins."io.containerd.internal.v1.tracing"]
    sampling_ratio = 1.0
    service_name = "containerd"

  [plugins."io.containerd.metadata.v1.bolt"]
    content_sharing_policy = "shared"

  [plugins."io.containerd.monitor.v1.cgroups"]
    no_prometheus = false

  [plugins."io.containerd.runtime.v1.linux"]
    no_shim = false
    runtime = "runc"
    runtime_root = ""
    shim = "containerd-shim"
    shim_debug = false

  [plugins."io.containerd.runtime.v2.task"]
    platforms = ["linux/amd64"]
    sched_core = false

  [plugins."io.containerd.service.v1.diff-service"]
    default = ["walking"]

  [plugins."io.containerd.service.v1.tasks-service"]
    rdt_config_file = ""

  [plugins."io.containerd.snapshotter.v1.aufs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.btrfs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.devmapper"]
    async_remove = false
    base_image_size = ""
    discard_blocks = false
    fs_options = ""
    fs_type = ""
    pool_name = ""
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.native"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.overlayfs"]
    root_path = ""
    upperdir_label = false

  [plugins."io.containerd.snapshotter.v1.zfs"]
    root_path = ""

  [plugins."io.containerd.tracing.processor.v1.otlp"]
    endpoint = ""
    insecure = false
    protocol = ""

[proxy_plugins]

[stream_processors]

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar"

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar+gzip"

[timeouts]
  "io.containerd.timeout.bolt.open" = "0s"
  "io.containerd.timeout.shim.cleanup" = "5s"
  "io.containerd.timeout.shim.load" = "5s"
  "io.containerd.timeout.shim.shutdown" = "3s"
  "io.containerd.timeout.task.state" = "2s"

[ttrpc]
  address = ""
  gid = 0
  uid = 0
[root@k8s-82 containerd]#

生成 CNI配置文件/etc/cni/net.d/10-containerd-net.conflist

cat << EOF | tee /etc/cni/net.d/10-containerd-net.conflist
{
 "cniVersion": "1.0.0",
 "name": "containerd-net",
 "plugins": [
   {
     "type": "bridge",
     "bridge": "cni0",
     "isGateway": true,
     "ipMasq": true,
     "promiscMode": true,
     "ipam": {
       "type": "host-local",
       "ranges": [
         [{
           "subnet": "10.88.0.0/16"
         }],
         [{
           "subnet": "2001:db8:4860::/64"
         }]
       ],
       "routes": [
         { "dst": "0.0.0.0/0" },
         { "dst": "::/0" }
       ]
     }
   },
   {
     "type": "portmap",
     "capabilities": {"portMappings": true},
     "externalSetMarkChain": "KUBE-MARK-MASQ"
   }
 ]
}
EOF

配置开机自启动

systemctl enable containerd
systemctl enable docker

[root@vm246 ~]# systemctl enable containerd
Created symlink from /etc/systemd/system/multi-user.target.wants/containerd.service to /usr/lib/systemd/system/containerd.service.
[root@vm246 ~]# systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
[root@vm246 ~]#

重启服务

systemctl daemon-reload
systemctl restart containerd
systemctl restart docker

检查各个服务的status,能正常启动,且没有报错为准。 特别是镜像拉取部分。一般包含关键字images

重点是containerd服务。

systemctl status containerd -l
systemctl status docker -l

成功标记

[root@vm246 ~]# systemctl status containerd -l
● containerd.service - containerd container runtime
   Loaded: loaded (/usr/lib/systemd/system/containerd.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2023-06-15 19:23:29 PDT; 22s ago
     Docs: https://containerd.io
  Process: 122442 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
 Main PID: 122447 (containerd)
    Tasks: 9
   Memory: 12.5M
   CGroup: /system.slice/containerd.service
           └─122447 /usr/bin/containerd

Jun 15 19:23:29 vm246 containerd[122447]: time="2023-06-15T19:23:29.875312841-07:00" level=error msg="failed to load cni during init, please check CRI plugin status before setting up network for pods" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
Jun 15 19:23:29 vm246 containerd[122447]: time="2023-06-15T19:23:29.876437032-07:00" level=info msg="Start subscribing containerd event"
Jun 15 19:23:29 vm246 containerd[122447]: time="2023-06-15T19:23:29.876571680-07:00" level=info msg="Start recovering state"
Jun 15 19:23:29 vm246 containerd[122447]: time="2023-06-15T19:23:29.876731048-07:00" level=info msg="Start event monitor"
Jun 15 19:23:29 vm246 containerd[122447]: time="2023-06-15T19:23:29.876776079-07:00" level=info msg="Start snapshots syncer"
Jun 15 19:23:29 vm246 containerd[122447]: time="2023-06-15T19:23:29.876804634-07:00" level=info msg="Start cni network conf syncer for default"
Jun 15 19:23:29 vm246 containerd[122447]: time="2023-06-15T19:23:29.876824365-07:00" level=info msg="Start streaming server"
Jun 15 19:23:29 vm246 containerd[122447]: time="2023-06-15T19:23:29.880223372-07:00" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
Jun 15 19:23:29 vm246 containerd[122447]: time="2023-06-15T19:23:29.880340135-07:00" level=info msg=serving... address=/run/containerd/containerd.sock
Jun 15 19:23:29 vm246 containerd[122447]: time="2023-06-15T19:23:29.880507655-07:00" level=info msg="containerd successfully booted in 0.045285s"
[root@vm246 ~]# systemctl status docker -l
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2023-06-15 19:23:31 PDT; 47s ago
     Docs: https://docs.docker.com
 Main PID: 122475 (dockerd)
    Tasks: 8
   Memory: 23.9M
   CGroup: /system.slice/docker.service
           └─122475 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Jun 15 19:23:31 vm246 systemd[1]: Starting Docker Application Container Engine...
Jun 15 19:23:31 vm246 dockerd[122475]: time="2023-06-15T19:23:31.090256047-07:00" level=info msg="Starting up"
Jun 15 19:23:31 vm246 dockerd[122475]: time="2023-06-15T19:23:31.128884422-07:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Jun 15 19:23:31 vm246 dockerd[122475]: time="2023-06-15T19:23:31.129218516-07:00" level=info msg="Loading containers: start."
Jun 15 19:23:31 vm246 dockerd[122475]: time="2023-06-15T19:23:31.294765086-07:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Jun 15 19:23:31 vm246 dockerd[122475]: time="2023-06-15T19:23:31.358672366-07:00" level=info msg="Loading containers: done."
Jun 15 19:23:31 vm246 dockerd[122475]: time="2023-06-15T19:23:31.381460228-07:00" level=info msg="Docker daemon" commit=659604f graphdriver=overlay2 version=24.0.2
Jun 15 19:23:31 vm246 dockerd[122475]: time="2023-06-15T19:23:31.381616108-07:00" level=info msg="Daemon has completed initialization"
Jun 15 19:23:31 vm246 dockerd[122475]: time="2023-06-15T19:23:31.418241862-07:00" level=info msg="API listen on /run/docker.sock"
Jun 15 19:23:31 vm246 systemd[1]: Started Docker Application Container Engine.
[root@vm246 ~]#

查看服务日志

journalctl -u docker -f -n 200
-- Logs begin at Fri 2023-06-16 15:32:21 CST. --
Jun 16 16:23:29 vm246 systemd[1]: Starting Docker Application Container Engine...
Jun 16 16:23:29 vm246 dockerd[14222]: time="2023-06-16T01:23:29.903461113-07:00" level=info msg="Starting up"
Jun 16 16:23:29 vm246 dockerd[14222]: time="2023-06-16T01:23:29.975099107-07:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Jun 16 16:23:29 vm246 dockerd[14222]: time="2023-06-16T01:23:29.976284285-07:00" level=info msg="Loading containers: start."
Jun 16 16:23:30 vm246 dockerd[14222]: time="2023-06-16T01:23:30.269691401-07:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Jun 16 16:23:30 vm246 dockerd[14222]: time="2023-06-16T01:23:30.337563745-07:00" level=info msg="Loading containers: done."
Jun 16 16:23:30 vm246 dockerd[14222]: time="2023-06-16T01:23:30.362873451-07:00" level=info msg="Docker daemon" commit=659604f graphdriver=overlay2 version=24.0.2
Jun 16 16:23:30 vm246 dockerd[14222]: time="2023-06-16T01:23:30.363081732-07:00" level=info msg="Daemon has completed initialization"
Jun 16 16:23:30 vm246 dockerd[14222]: time="2023-06-16T01:23:30.403721175-07:00" level=info msg="API listen on /run/docker.sock"
Jun 16 16:23:30 vm246 systemd[1]: Started Docker Application Container Engine.
Jun 16 16:34:48 vm246 systemd[1]: Stopping Docker Application Container Engine...
Jun 16 16:34:48 vm246 dockerd[14222]: time="2023-06-16T01:34:48.779962200-07:00" level=info msg="Processing signal 'terminated'"
Jun 16 16:34:48 vm246 dockerd[14222]: time="2023-06-16T01:34:48.784279762-07:00" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
Jun 16 16:34:48 vm246 dockerd[14222]: time="2023-06-16T01:34:48.785022718-07:00" level=info msg="Daemon shutdown complete"
Jun 16 16:34:48 vm246 systemd[1]: Stopped Docker Application Container Engine.
Jun 16 16:42:28 vm246 systemd[1]: Starting Docker Application Container Engine...
Jun 16 16:42:28 vm246 dockerd[16056]: time="2023-06-16T01:42:28.776106585-07:00" level=info msg="Starting up"
Jun 16 16:42:28 vm246 dockerd[16056]: time="2023-06-16T01:42:28.807766565-07:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Jun 16 16:42:28 vm246 dockerd[16056]: time="2023-06-16T01:42:28.808052262-07:00" level=info msg="Loading containers: start."
Jun 16 16:42:28 vm246 dockerd[16056]: time="2023-06-16T01:42:28.982890894-07:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Jun 16 16:42:29 vm246 dockerd[16056]: time="2023-06-16T01:42:29.043206290-07:00" level=info msg="Loading containers: done."
Jun 16 16:42:29 vm246 dockerd[16056]: time="2023-06-16T01:42:29.065665867-07:00" level=info msg="Docker daemon" commit=659604f graphdriver=overlay2 version=24.0.2
Jun 16 16:42:29 vm246 dockerd[16056]: time="2023-06-16T01:42:29.065775807-07:00" level=info msg="Daemon has completed initialization"
Jun 16 16:42:29 vm246 dockerd[16056]: time="2023-06-16T01:42:29.109587449-07:00" level=info msg="API listen on /run/docker.sock"
Jun 16 16:42:29 vm246 systemd[1]: Started Docker Application Container Engine.

下一步

GCDW技术栈- kubernets运行环境搭建