K3S+Rancher2.x跨Region搭建方案

K3S+Rancher2.x跨Region搭建方案

Posted by Ksd on May 14, 2020

K3S+Rancher2.x跨Region搭建方案

背景

昨天看见群里有个用户提了一个问题,是说跨region搭建k3s集群后,虽然node的external ip已经设置成了公网IP,但创建了pod无法实现跨region访问。

然后按照正规的k3s搭建方法,加了--advertise-address--node-external-ip参数确实有一些问题,再后来给node加了一个 metadata.annotations.flannel.alpha.coreos.com/public-ip-overwrite: x.x.x.x解决

架构图

测试环境

实例名称 Region 公网IP 内外IP 备注
ip-172-31-3-18 ca-central-1 35.183.24.223 172.31.3.18 k3s server
ip-172-31-3-197 us-east-2 3.17.76.186 172.31.3.197 k3s server
ip-172-31-2-229 ca-central-1 35.183.182.237 172.31.2.229 db

搭建Rancher集群

参考:https://rancher2.docs.rancher.cn/docs/installation/k8s-install/_index

启动数据库

ip-172-31-2-229:

docker run --name mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=changeme -e MYSQL_DATABASE=k3sdb -d mysql:5.7

启动K3s集群:

ip-172-31-3-18:

curl -sfL https://get.k3s.io | sh -s - server   --datastore-endpoint="mysql://root:changeme@tcp(35.183.182.237:3306)/k3sdb"  --docker --advertise-address=35.183.24.223  --node-external-ip=35.183.24.223

ip-172-31-3-197:

curl -sfL https://get.k3s.io | sh -s - server   --datastore-endpoint="mysql://root:changeme@tcp(35.183.182.237:3306)/k3sdb"  --docker --advertise-address=3.17.76.186  --node-external-ip=3.17.76.186

修改node annotations

默认情况下,flannel将对端k3s服务器的内网IP作为下一跳的网关,但我们的环境是跨region集群,使用内网IP是无法连接的,所以这地方需要将flannel的下一跳网关改成公网IP,可以修改node的 annotations来实现,参考:https://coreos.com/flannel/docs/latest/kubernetes.html

kubectl edit node ip-172-31-3-18
apiVersion: v1
kind: Node
metadata:
  annotations:
    flannel.alpha.coreos.com/public-ip-overwrite: 35.183.24.223

另一个k3s节点也需要执行相同的操作,分别重启两次 k3s集群。 接下来就可以验证下上一步的操作是否成功了,我们可以看到 dst对应的IP已经变成了公网IP

root@ip-172-31-3-18:~# bridge fdb show | grep flannel
ea:da:e5:02:cd:af dev flannel.1 dst 3.17.76.186 self permanent

root@ip-172-31-3-197:~# bridge fdb show | grep flannel
4a:ab:06:55:dc:58 dev flannel.1 dst 35.183.24.223 self permanent

这时候我们可以部署一个副本为2个或更多的deployment,然后exec到pod里去ping跨主机的pid ip

root@ip-172-31-3-18:~# kubectl get pods -o wide
NAME                               READY   STATUS    RESTARTS   AGE     IP          NODE              NOMINATED NODE   READINESS GATES
nginx-deployment-bccf67fdd-6hmqq   1/1     Running   0          4h15m   10.42.0.8   ip-172-31-3-18    <none>           <none>
nginx-deployment-bccf67fdd-m9zb8   1/1     Running   0          4h15m   10.42.1.3   ip-172-31-3-197   <none>           <none>
root@ip-172-31-3-18:~# kubectl exec -it nginx-deployment-bccf67fdd-6hmqq ping 10.42.1.3
PING 10.42.1.3 (10.42.1.3) 56(84) bytes of data.
64 bytes from 10.42.1.3: icmp_seq=1 ttl=62 time=24.6 ms
64 bytes from 10.42.1.3: icmp_seq=2 ttl=62 time=24.5 ms
64 bytes from 10.42.1.3: icmp_seq=3 ttl=62 time=24.5 ms
64 bytes from 10.42.1.3: icmp_seq=4 ttl=62 time=24.5 ms
64 bytes from 10.42.1.3: icmp_seq=5 ttl=62 time=24.5 ms

安装Rancher

k3s集群安装成功后,就可以按照官网安装3个副本的rancher了,和其他集群无区别,这里就不过多介绍了,我已经测试过,rancher集群可以成功创建。