GCDW技术栈-kubernets操作系统环境准备

发表于 作者 laozizhu

GBase 8a云数仓gcdw运行在k8s上，本文是k8s安装前的准备工作，主要是操作系统层面的修改配置。所有变动均参考了官网的文档。

参考

GCDW 需要foundationdb作为元数据服务，用s3作为存储服务，镜像服务器(harbor)。这些服务器，建议单独部署，不要可k8s环境部署在同一台服务器上。最终能运行gcdw的单节点k8s,最少需要8核CPU、8G内存，只用于体验环境，比如笔记本虚拟机上。

请分别参考

GCDW元数据服务FoundationDB的集群模式配置和高可用测试

MinIO S3分布式集群搭建

GCDW技术栈，镜像仓库harbor安装，包括https 服务CA证书的生成(openssl)

样例说明

本文在10.0.2.81，主机名为 k8s-81的centos 7.9上安装k8s环境。同时为另外2台，共3台（）做准备。本文介绍以k8s-81为参考，另外2台的就不写了。

如下操作全部在操作系统root用户下进行。

主机名

配置文件为/etc/hosts，最终为3台服务器，其中第一台做master, 三台全部做worker。本文现在81上搭建单节点的k8s，后续再扩容。 

[root@k8s-81 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.2.81   k8s-81
10.0.2.82   k8s-82
10.0.2.83   k8s-83

[root@k8s-81 ~]#

设置主机名

hostnamectl set-hostname k8s-81

关闭SWAP

swapoff -a

注释/etc/fstab里面swap部分

[root@k8s-81 ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Fri Feb 10 10:18:48 2023
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=5cf16afe-8cd2-480b-81a3-a282871e69ca /                       xfs     defaults        0 0
UUID=83bfb85c-2769-442a-9bd4-960020d27172 /boot                   xfs     defaults        0 0
#UUID=a99cdd73-9785-41d1-9d15-6789186aa18b swap                    swap    defaults        0 0
[root@k8s-81 ~]#

检查MAC地址

主机的MAC地址不能一样，常见于虚拟机复制的场景。 可以在虚拟机网卡管理里，设置或重新生成MAC来解决。

[root@k8s-1 ~]# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:b1:15:fe brd ff:ff:ff:ff:ff:ff
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:63:df:29 brd ff:ff:ff:ff:ff:ff
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:63:df:29 brd ff:ff:ff:ff:ff:ff
[root@k8s-1 ~]#
[root@k8s-2 ~]# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:60:36:23 brd ff:ff:ff:ff:ff:ff
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:63:df:29 brd ff:ff:ff:ff:ff:ff
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:63:df:29 brd ff:ff:ff:ff:ff:ff
[root@k8s-2 ~]#
[root@k8s-3 ~]# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:2d:a3:56 brd ff:ff:ff:ff:ff:ff
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:63:df:29 brd ff:ff:ff:ff:ff:ff
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:63:df:29 brd ff:ff:ff:ff:ff:ff
[root@k8s-3 ~]#

检查UUID

不能出现重复的

[root@k8s-1 ~]# cat /sys/class/dmi/id/product_uuid
0E53E13A-1D5F-774D-99FC-16E6C0B79B97
[root@k8s-1 ~]#

检查端口

默认的6443端口不能被占用。

[root@k8s-1 ~]# nc 127.0.0.1 6443
Ncat: Connection refused.
[root@k8s-1 ~]#

关闭selinux

setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

关闭防火墙

systemctl stop firewalld
systemctl disable firewalld

docker运行时环境

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

检查

lsmod | grep br_netfilter
lsmod | grep overlay

设置 sysctl 参数

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# 应用 sysctl 参数而不重新启动
sudo sysctl --system

检查

sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward

yum源设置

请根据实际情况设置，如下只是一种源。包括基础，docker和kubernets的3个。

curl -o /etc/yum.repos.d/Centos-7.repo http://mirrors.aliyun.com/repo/Centos-7.repo
curl -o /etc/yum.repos.d/docker-ce.repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
        http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

执行刷新

yum clean all && yum makecache

安装systemd-resolved

yum install systemd-resolved

[root@vm246 172.16.3.246]# yum install systemd-resolved
Loaded plugins: fastestmirror, langpacks
Repository base is listed more than once in the configuration
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
Loading mirror speeds from cached hostfile
 * base: mirrors.bfsu.edu.cn
 * extras: mirrors.bfsu.edu.cn
 * updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package systemd-resolved.x86_64 0:219-78.el7_9.7 will be installed
--> Processing Dependency: systemd = 219-78.el7_9.7 for package: systemd-resolved-219-78.el7_9.7.x86_64
--> Running transaction check
---> Package systemd.x86_64 0:219-78.el7 will be updated
--> Processing Dependency: systemd = 219-78.el7 for package: systemd-python-219-78.el7.x86_64
--> Processing Dependency: systemd = 219-78.el7 for package: systemd-sysv-219-78.el7.x86_64
---> Package systemd.x86_64 0:219-78.el7_9.7 will be an update
--> Processing Dependency: systemd-libs = 219-78.el7_9.7 for package: systemd-219-78.el7_9.7.x86_64
--> Running transaction check
---> Package systemd-libs.x86_64 0:219-78.el7 will be updated
--> Processing Dependency: systemd-libs = 219-78.el7 for package: libgudev1-219-78.el7.x86_64
---> Package systemd-libs.x86_64 0:219-78.el7_9.7 will be an update
---> Package systemd-python.x86_64 0:219-78.el7 will be updated
---> Package systemd-python.x86_64 0:219-78.el7_9.7 will be an update
---> Package systemd-sysv.x86_64 0:219-78.el7 will be updated
---> Package systemd-sysv.x86_64 0:219-78.el7_9.7 will be an update
--> Running transaction check
---> Package libgudev1.x86_64 0:219-78.el7 will be updated
---> Package libgudev1.x86_64 0:219-78.el7_9.7 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

=====================================================================================================================================================================
 Package                                     Arch                              Version                                      Repository                          Size
=====================================================================================================================================================================
Installing:
 systemd-resolved                            x86_64                            219-78.el7_9.7                               updates                            422 k
Updating for dependencies:
 libgudev1                                   x86_64                            219-78.el7_9.7                               updates                            110 k
 systemd                                     x86_64                            219-78.el7_9.7                               updates                            5.1 M
 systemd-libs                                x86_64                            219-78.el7_9.7                               updates                            419 k
 systemd-python                              x86_64                            219-78.el7_9.7                               updates                            146 k
 systemd-sysv                                x86_64                            219-78.el7_9.7                               updates                             97 k

Transaction Summary
=====================================================================================================================================================================
Install  1 Package
Upgrade             ( 5 Dependent packages)

Total download size: 6.2 M
Is this ok [y/d/N]: y
Downloading packages:
No Presto metadata available for updates
(1/6): libgudev1-219-78.el7_9.7.x86_64.rpm                                                                                                    | 110 kB  00:00:00
(2/6): systemd-sysv-219-78.el7_9.7.x86_64.rpm                                                                                                 |  97 kB  00:00:00
(3/6): systemd-resolved-219-78.el7_9.7.x86_64.rpm                                                                                             | 422 kB  00:00:00
(4/6): systemd-python-219-78.el7_9.7.x86_64.rpm                                                                                               | 146 kB  00:00:00
(5/6): systemd-219-78.el7_9.7.x86_64.rpm                                                                                                      | 5.1 MB  00:00:02
(6/6): systemd-libs-219-78.el7_9.7.x86_64.rpm                                                                                                 | 419 kB  00:00:17
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                365 kB/s | 6.2 MB  00:00:17
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : systemd-libs-219-78.el7_9.7.x86_64                                                                                                               1/11
  Updating   : systemd-219-78.el7_9.7.x86_64                                                                                                                    2/11
  Updating   : systemd-sysv-219-78.el7_9.7.x86_64                                                                                                               3/11
  Installing : systemd-resolved-219-78.el7_9.7.x86_64                                                                                                           4/11
  Updating   : systemd-python-219-78.el7_9.7.x86_64                                                                                                             5/11
  Updating   : libgudev1-219-78.el7_9.7.x86_64                                                                                                                  6/11
  Cleanup    : systemd-sysv-219-78.el7.x86_64                                                                                                                   7/11
  Cleanup    : systemd-python-219-78.el7.x86_64                                                                                                                 8/11
  Cleanup    : systemd-219-78.el7.x86_64                                                                                                                        9/11
  Cleanup    : libgudev1-219-78.el7.x86_64                                                                                                                     10/11
  Cleanup    : systemd-libs-219-78.el7.x86_64                                                                                                                  11/11
  Verifying  : systemd-libs-219-78.el7_9.7.x86_64                                                                                                               1/11
  Verifying  : systemd-219-78.el7_9.7.x86_64                                                                                                                    2/11
  Verifying  : systemd-sysv-219-78.el7_9.7.x86_64                                                                                                               3/11
  Verifying  : systemd-resolved-219-78.el7_9.7.x86_64                                                                                                           4/11
  Verifying  : systemd-python-219-78.el7_9.7.x86_64                                                                                                             5/11
  Verifying  : libgudev1-219-78.el7_9.7.x86_64                                                                                                                  6/11
  Verifying  : systemd-python-219-78.el7.x86_64                                                                                                                 7/11
  Verifying  : systemd-sysv-219-78.el7.x86_64                                                                                                                   8/11
  Verifying  : systemd-libs-219-78.el7.x86_64                                                                                                                   9/11
  Verifying  : libgudev1-219-78.el7.x86_64                                                                                                                     10/11
  Verifying  : systemd-219-78.el7.x86_64                                                                                                                       11/11

Installed:
  systemd-resolved.x86_64 0:219-78.el7_9.7

Dependency Updated:
  libgudev1.x86_64 0:219-78.el7_9.7        systemd.x86_64 0:219-78.el7_9.7     systemd-libs.x86_64 0:219-78.el7_9.7     systemd-python.x86_64 0:219-78.el7_9.7
  systemd-sysv.x86_64 0:219-78.el7_9.7

Complete!
[root@vm246 172.16.3.246]#

FoundationDB的注意事项

当前gcdw的版本，要求fdb服务器的版本，必须和镜像内的fdbcli版本完全一致(6.3.13)，否则，会导致pod内无法连接foundationdb的现象。后续版本如果做了修正，我会更新本文档。

下一步

GCDW技术栈- docker运行环境安装

Post Views: 102

相关文章:

  1. GCDW技术栈，镜像仓库harbor安装，包括https 服务CA证书的生成(openssl)
  2. GCDW技术栈- docker运行环境搭建
  3. GCDW技术栈 - kubernets 1.26.0运行环境搭建
  4. GBase 8a GCDW存算分离主机版安装使用预览
  5. GCDW在k8s上的部署
  6. GCDW k8s版使用环境搭建ppt-内部用
  7. 国产CPU的技术参数，鲲鹏，飞腾等
  8. GBase 8a通过python UDF从客户端执行SQL语句
  9. GBase 8a透明存储加密encrypt功能使用
  10. GBase 8a实现用户磁盘空间配额限制