使用Racnehr搭建跨数据中心集群

使用Racnehr搭建跨数据中心集群

Posted by Ksd on May 5, 2020

使用Racnehr搭建跨数据中心集群

背景

待补充

环境信息

数据中心 主机名 角色 公网IP 私有IP 备注
AWS ip-172-31-8-93 Rancher Server 35.183.208.191 172.31.8.93  
AWS ip-172-31-2-52 Etcd&Control 35.183.104.184 172.31.2.52  
AWS ip-172-31-8-33 worker 52.60.79.181 172.31.8.33  
GCE hailong-demo-gce-worker worker 35.188.150.213 35.188.150.213  

搭建集群

在Rancher Server 机器上安装单节点Rancher

# docker run -itd -p 80:80 -p 443:443 --restart=unless-stopped rancher/rancher:v2.4.3

在ETCD&Control节点想集群添加Etcd&Control角色

注意:需要将Node Public AddressNode Public Address都设置为公网IP地址或者只设置Node Public Address,否则添加worker节点后,worker节点的nginx-proxy将会报错:

2020/05/09 02:49:05 [error] 11#11: *711 upstream timed out (110: Operation timed out) while connecting to upstream, client: 127.0.0.1, server: 0.0.0.0:6443, upstream: "172.31.2.52:6443", bytes from/to client:0/0, bytes from/to upstream:0/0

然后将自动生成的命令copy到对应的Etcd&Control主机,执行

几分钟后,添加成功,节点状态变为Active

随后可以点击主机 右侧的 省略号 > Edit, 可以查看到rke.cattle.io/internal-ip已经设置为公网IP地址

添加worker节点

随后我们就可以在AWS和GCE虚拟机上添加worker节点,添加节点是需要分别指定Node Public AddressNode Internal Address的地址,这次可以按照虚拟机的公网IP和内网IP分别指定,不需要像Etcd节点那样同时都设置成公网IP。

下图为添加节点成功后的截图:

然后在确认下pod是否都启动了:

root@ip-172-31-8-93:~# kubectl get pods --all-namespaces
NAMESPACE       NAME                                      READY   STATUS      RESTARTS   AGE
cattle-system   cattle-cluster-agent-54f5f97854-qc4bm     1/1     Running     0          41m
cattle-system   cattle-node-agent-fl8jz                   1/1     Running     0          39m
cattle-system   cattle-node-agent-kplpq                   1/1     Running     0          18m
cattle-system   cattle-node-agent-lfjts                   1/1     Running     0          41m
cattle-system   kube-api-auth-jl4lw                       1/1     Running     0          41m
ingress-nginx   default-http-backend-67cf578fc4-x2j7w     1/1     Running     0          41m
ingress-nginx   nginx-ingress-controller-5sr8b            1/1     Running     0          39m
ingress-nginx   nginx-ingress-controller-v7bq2            1/1     Running     0          18m
kube-system     canal-j44zb                               2/2     Running     0          42m
kube-system     canal-k89nq                               2/2     Running     0          39m
kube-system     canal-qpxlt                               2/2     Running     0          18m
kube-system     coredns-7c5566588d-fwmjj                  1/1     Running     0          18m
kube-system     coredns-7c5566588d-kspws                  1/1     Running     0          42m
kube-system     coredns-autoscaler-65bfc8d47d-9g9qp       1/1     Running     0          42m
kube-system     metrics-server-6b55c64f86-vpl99           1/1     Running     0          42m
kube-system     rke-coredns-addon-deploy-job-qs49q        0/1     Completed   0          42m
kube-system     rke-ingress-controller-deploy-job-tqgkm   0/1     Completed   0          42m
kube-system     rke-metrics-addon-deploy-job-48px5        0/1     Completed   0          42m
kube-system     rke-network-plugin-deploy-job-45slw       0/1     Completed   0          42m

解决网络问题

在以上步骤完成后,跨数据中心的集群就已经搭建完成了,但跨数据中心的pod是不通的,可以登录到gce-worker节点和aws-worker节点flannel对应的dst都为内网地址,这会导致pod间网络不通。

AWS worker:

root@ip-172-31-8-33:~# bridge fdb show | grep flannel
72:b9:96:ab:69:60 dev flannel.1 dst 172.31.2.52 self permanent
2a:c8:84:70:13:90 dev flannel.1 dst 10.240.0.25 self permanent

GCE worker:

root@hailong-demo-gce-worker:~# bridge fdb show | grep flannel
d6:6f:c8:77:de:6c dev flannel.1 dst 172.31.8.33 self permanent
72:b9:96:ab:69:60 dev flannel.1 dst 172.31.2.52 self permanent

要解决这个问题,需要手动分别编辑worker节点和etcd&control的Annotations,添加flannel.alpha.coreos.com/public-ip-overwrite来解决,参考:https://coreos.com/flannel/docs/latest/kubernetes.html

参考:

随后导航到system project下,redeploy kube-system namespace下的canal workload,为了使配置生效。

接下来,再次在两台worker节点上验证flannel的dst地址 AWS worker:

root@ip-172-31-8-33:~# bridge fdb show | grep flannel
72:b9:96:ab:69:60 dev flannel.1 dst 35.183.104.184 self permanent
2a:c8:84:70:13:90 dev flannel.1 dst 35.188.150.213 self permanent

GCE worker:

root@hailong-demo-gce-worker:~# bridge fdb show | grep flannel
d6:6f:c8:77:de:6c dev flannel.1 dst 52.60.79.181 self permanent
72:b9:96:ab:69:60 dev flannel.1 dst 35.183.104.184 self permanent

可以看到,dst都已经变成了对端节点的公网IP。

验证

接下来就可以验证两个pod间通讯了。

root@ip-172-31-8-93:~# kubectl get pods -o wide
NAME                   READY   STATUS    RESTARTS   AGE   IP          NODE                      NOMINATED NODE   READINESS GATES
aws-76644dbb44-428wd   1/1     Running   0          18m   10.42.2.3   ip-172-31-8-33            <none>           <none>
gce-577c94b676-px8cf   1/1     Running   0          18m   10.42.1.6   hailong-demo-gce-worker   <none>           <none>
root@ip-172-31-8-93:~#
root@ip-172-31-8-93:~#
root@ip-172-31-8-93:~# kubectl exec -it aws-76644dbb44-428wd -- ping -c 4 10.42.1.6
PING 10.42.1.6 (10.42.1.6) 56(84) bytes of data.
64 bytes from 10.42.1.6: icmp_seq=1 ttl=62 time=28.9 ms
64 bytes from 10.42.1.6: icmp_seq=2 ttl=62 time=28.9 ms
64 bytes from 10.42.1.6: icmp_seq=3 ttl=62 time=28.9 ms
64 bytes from 10.42.1.6: icmp_seq=4 ttl=62 time=28.9 ms

遗留问题

  1. 使用kubectl exec gce 主机上的pod 会报错:
    root@ip-172-31-8-93:~# kubectl exec -it gce-577c94b676-bsmt7 bash
    Error from server: error dialing backend: dial tcp 10.240.0.25:10250: i/o timeout
    

问题的原因应该是 执行命令时,kube-api通过内网ip去连接worker节点的kubelet,所以才会导致timeout的情况 可能在worker节点的kubelet加上–node-ip参数就可以了,但目前没找到怎么在rancher中加这个参数。