GCDW技术栈-kubernets操作系统环境准备

发表于 作者 laozizhu

GBase 8a云数仓gcdw运行在k8s上，本文是k8s安装前的准备工作，主要是操作系统层面的修改配置。所有变动均参考了官网的文档。

参考

GCDW 需要foundationdb作为元数据服务，用s3作为存储服务，镜像服务器(harbor)。这些服务器，建议单独部署，不要可k8s环境部署在同一台服务器上。最终能运行gcdw的单节点k8s,最少需要8核CPU、8G内存，只用于体验环境，比如笔记本虚拟机上。

请分别参考

GCDW元数据服务FoundationDB的集群模式配置和高可用测试

MinIO S3分布式集群搭建

GCDW技术栈，镜像仓库harbor安装，包括https 服务CA证书的生成(openssl)

样例说明

本文在10.0.2.81，主机名为 k8s-81的centos 7.9上安装k8s环境。同时为另外2台，共3台（）做准备。本文介绍以k8s-81为参考，另外2台的就不写了。

如下操作全部在操作系统root用户下进行。

主机名

配置文件为/etc/hosts，最终为3台服务器，其中第一台做master, 三台全部做worker。本文现在81上搭建单节点的k8s，后续再扩容。 

[root@k8s-81 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.2.81   k8s-81
10.0.2.82   k8s-82
10.0.2.83   k8s-83

[root@k8s-81 ~]#

设置主机名

hostnamectl set-hostname k8s-81

关闭SWAP

swapoff -a

注释/etc/fstab里面swap部分

[root@k8s-81 ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Fri Feb 10 10:18:48 2023
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=5cf16afe-8cd2-480b-81a3-a282871e69ca /                       xfs     defaults        0 0
UUID=83bfb85c-2769-442a-9bd4-960020d27172 /boot                   xfs     defaults        0 0
#UUID=a99cdd73-9785-41d1-9d15-6789186aa18b swap                    swap    defaults        0 0
[root@k8s-81 ~]#

检查MAC地址

主机的MAC地址不能一样，常见于虚拟机复制的场景。 可以在虚拟机网卡管理里，设置或重新生成MAC来解决。

[root@k8s-1 ~]# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:b1:15:fe brd ff:ff:ff:ff:ff:ff
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:63:df:29 brd ff:ff:ff:ff:ff:ff
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:63:df:29 brd ff:ff:ff:ff:ff:ff
[root@k8s-1 ~]#
[root@k8s-2 ~]# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:60:36:23 brd ff:ff:ff:ff:ff:ff
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:63:df:29 brd ff:ff:ff:ff:ff:ff
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:63:df:29 brd ff:ff:ff:ff:ff:ff
[root@k8s-2 ~]#
[root@k8s-3 ~]# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:2d:a3:56 brd ff:ff:ff:ff:ff:ff
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:63:df:29 brd ff:ff:ff:ff:ff:ff
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:63:df:29 brd ff:ff:ff:ff:ff:ff
[root@k8s-3 ~]#

检查UUID

不能出现重复的

[root@k8s-1 ~]# cat /sys/class/dmi/id/product_uuid
0E53E13A-1D5F-774D-99FC-16E6C0B79B97
[root@k8s-1 ~]#

检查端口

默认的6443端口不能被占用。

[root@k8s-1 ~]# nc 127.0.0.1 6443
Ncat: Connection refused.
[root@k8s-1 ~]#

关闭selinux

setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

关闭防火墙

systemctl stop firewalld
systemctl disable firewalld

docker运行时环境

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

检查

lsmod | grep br_netfilter
lsmod | grep overlay

设置 sysctl 参数

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# 应用 sysctl 参数而不重新启动
sudo sysctl --system

检查

sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward

yum源设置

请根据实际情况设置，如下只是一种源。包括基础，docker和kubernets的3个。

curl -o /etc/yum.repos.d/Centos-7.repo http://mirrors.aliyun.com/repo/Centos-7.repo
curl -o /etc/yum.repos.d/docker-ce.repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
        http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

执行刷新

yum clean all && yum makecache

安装systemd-resolved

yum install systemd-resolved

FoundationDB的注意事项

当前gcdwde版本，要求fdb服务器的版本，必须和镜像内的fdbcli版本完全一致(6.3.13)，否则，会导致pod内无法连接foundationdb的现象。后续版本如果做了修正，我会更新本文档。

下一步

GCDW技术栈- docker运行环境安装

Post Views: 28

相关文章:

  1. GBase 8a GCDW存算分离主机版安装使用预览
  2. GCDW在k8s上的部署
  3. GCDW技术栈，镜像仓库harbor安装，包括https 服务CA证书的生成(openssl)
  4. GCDW技术栈- docker运行环境搭建
  5. GCDW技术栈 – kubernets 1.26.0运行环境搭建
  6. GBase8a 服务的启动和停止方法
  7. GBase 8a集群杀死某个SQL进程的方法
  8. GBase8a 集群查看数据库参数的当前值
  9. GBase 8a和周week有关的函数和文章汇总
  10. sshd连接数配置