GBase 8a云数仓gcdw运行在k8s上,本文是k8s安装前的准备工作,主要是操作系统层面的修改配置。所有变动均参考了官网的文档。
参考
GCDW 需要foundationdb作为元数据服务,用s3作为存储服务,镜像服务器(harbor)。这些服务器,建议单独部署,不要可k8s环境部署在同一台服务器上。最终能运行gcdw的单节点k8s,最少需要8核CPU、8G内存,只用于体验环境,比如笔记本虚拟机上。
请分别参考
GCDW元数据服务FoundationDB的集群模式配置和高可用测试
GCDW技术栈,镜像仓库harbor安装,包括https 服务CA证书的生成(openssl)
样例说明
本文在10.0.2.81,主机名为 k8s-81的centos 7.9上安装k8s环境。同时为另外2台,共3台()做准备。本文介绍以k8s-81为参考,另外2台的就不写了。
如下操作全部在操作系统root用户下进行。
主机名
配置文件为/etc/hosts,最终为3台服务器,其中第一台做master, 三台全部做worker。本文现在81上搭建单节点的k8s,后续再扩容。
[root@k8s-81 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.2.81 k8s-81
10.0.2.82 k8s-82
10.0.2.83 k8s-83
[root@k8s-81 ~]#
设置主机名
hostnamectl set-hostname k8s-81
关闭SWAP
swapoff -a
注释/etc/fstab里面swap部分
[root@k8s-81 ~]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Fri Feb 10 10:18:48 2023
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=5cf16afe-8cd2-480b-81a3-a282871e69ca / xfs defaults 0 0
UUID=83bfb85c-2769-442a-9bd4-960020d27172 /boot xfs defaults 0 0
#UUID=a99cdd73-9785-41d1-9d15-6789186aa18b swap swap defaults 0 0
[root@k8s-81 ~]#
检查MAC地址
主机的MAC地址不能一样,常见于虚拟机复制的场景。 可以在虚拟机网卡管理里,设置或重新生成MAC来解决。
[root@k8s-1 ~]# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 08:00:27:b1:15:fe brd ff:ff:ff:ff:ff:ff
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
link/ether 52:54:00:63:df:29 brd ff:ff:ff:ff:ff:ff
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN mode DEFAULT group default qlen 1000
link/ether 52:54:00:63:df:29 brd ff:ff:ff:ff:ff:ff
[root@k8s-1 ~]#
[root@k8s-2 ~]# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 08:00:27:60:36:23 brd ff:ff:ff:ff:ff:ff
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
link/ether 52:54:00:63:df:29 brd ff:ff:ff:ff:ff:ff
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN mode DEFAULT group default qlen 1000
link/ether 52:54:00:63:df:29 brd ff:ff:ff:ff:ff:ff
[root@k8s-2 ~]#
[root@k8s-3 ~]# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 08:00:27:2d:a3:56 brd ff:ff:ff:ff:ff:ff
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
link/ether 52:54:00:63:df:29 brd ff:ff:ff:ff:ff:ff
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN mode DEFAULT group default qlen 1000
link/ether 52:54:00:63:df:29 brd ff:ff:ff:ff:ff:ff
[root@k8s-3 ~]#
检查UUID
不能出现重复的
[root@k8s-1 ~]# cat /sys/class/dmi/id/product_uuid
0E53E13A-1D5F-774D-99FC-16E6C0B79B97
[root@k8s-1 ~]#
检查端口
默认的6443端口不能被占用。
[root@k8s-1 ~]# nc 127.0.0.1 6443
Ncat: Connection refused.
[root@k8s-1 ~]#
关闭selinux
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
docker运行时环境
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter
检查
lsmod | grep br_netfilter
lsmod | grep overlay
设置 sysctl 参数
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
# 应用 sysctl 参数而不重新启动
sudo sysctl --system
检查
sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward
yum源设置
请根据实际情况设置,如下只是一种源。包括基础,docker和kubernets的3个。
curl -o /etc/yum.repos.d/Centos-7.repo http://mirrors.aliyun.com/repo/Centos-7.repo
curl -o /etc/yum.repos.d/docker-ce.repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
执行刷新
yum clean all && yum makecache
安装systemd-resolved
yum install systemd-resolved
FoundationDB的注意事项
当前gcdwde版本,要求fdb服务器的版本,必须和镜像内的fdbcli版本完全一致(6.3.13),否则,会导致pod内无法连接foundationdb的现象。后续版本如果做了修正,我会更新本文档。