K8s集群 实现集群业务是否对外暴露的控制 （多LB实施方案）
架构图：
简述实施步骤：
1.新建集群会自动创建一个公网的LB，需要新创建一个内网LB绑定在此nginx controller 上，实现内网访问pod的目的。
2.新建专属内网的nginx controller 绑定内网LB只对内网提供服务，并且修改默认的ingress.class标签
1.基础Nginx Controller绑定内网LB
参考网址：https://help.aliyun.com/document_detail/151506.html
查看集群现存lb
# kubectl get svc -A |grep LoadBalancer kube-system nginx-ingress-lb LoadBalancer 192.168.223.150 47.108.153.86 80:30299/TCP,443:31348/TCP 86m
新建集群LB服务
（推荐）方式一：
集群服务 > 网络 > 服务 > 选择nginx-controller所在命名空间 > 创建 > 负载均衡.内网.新建slb.关联nginx
这种方式是最快捷的，或者也能选择手动创建
方式二：
1.创建负载均衡器 2.在此页面创建时，绑定已有负载均衡即可
或者，当您成功创建一个私网SLB实例后，您可以通过以下示例注解来配置Nginx Ingress Controller使用该SLB实例，
# nginx ingress slb service apiVersion: v1 kind: Service metadata: name: nginx-ingress-lb namespace: kube-system labels: app: nginx-ingress-lb annotations: # 指明SLB实例地址类型为私网类型。 service.beta.kubernetes.io/alicloud-loadbalancer-address-type: intranet # 修改为您的私网SLB实例ID。 service.beta.kubernetes.io/alicloud-loadbalancer-id: <YOUR_INTRANET_SLB_ID> # 是否自动创建SLB端口监听（会覆写已有端口监听），也可手动创建端口监听。 #service.beta.kubernetes.io/alicloud-loadbalancer-force-override-listeners: 'true' spec: type: LoadBalancer # route traffic to other nodes externalTrafficPolicy: "Cluster" ports: - port: 80 name: http targetPort: 80 - port: 443 name: https targetPort: 443 selector: # select app=ingress-nginx pods app: ingress-nginx
查询是否创建成功
# kubectl get svc -A |grep LoadBalancer kube-system nginx-ingress-lb LoadBalancer 192.168.223.150 47.108.153.86 80:30299/TCP,443:31348/TCP 100m kube-system vpc-lb LoadBalancer 192.168.37.202 172.24.43.76 80:32532/TCP,443:31330/TCP 3m19s
部署服务测试连通性
apiVersion: apps/v1 kind: Deployment metadata: name: hellok8s-dep namespace: default spec: replicas: 1 selector: matchLabels: app: hello-kubernetes template: metadata: labels: app: hello-kubernetes spec: containers: - name: hello-kubernetes image: paulbouwer/hello-kubernetes:1.4 #image: centos:7 #args: #- sleep #- "1000000" ports: - containerPort: 8080 --- apiVersion: v1 kind: Service metadata: name: hellok8s-svc namespace: default spec: ports: - port: 80 targetPort: 8080 selector: app: hello-kubernetes --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: hellok8s-ingress annotations: ##用来绑定特定的ingress-nginx #kubernetes.io/ingress.class: ack-nginx-vpc ##默认class kubernetes.io/ingress.class: nginx ##配置了ssl证书即打开此配置 #nginx.ingress.kubernetes.io/ssl-redirect: 'true' nginx.ingress.kubernetes.io/rewrite-target: / nginx.ingress.kubernetes.io/enable-cors: 'true' nginx.ingress.kubernetes.io/cors-allow-headers: DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization,Access-Control-Allow-Origin nginx.ingress.kubernetes.io/cors-allow-methods: PUT, GET, POST, OPTIONS, DELETE nginx.ingress.kubernetes.io/cors-allow-origin: '*' nginx.ingress.kubernetes.io/cors-allow-credentials: 'true' spec: ##按需配置ssl证书 #tls: #- secretName: da-e.top-tls # hosts: # - "*.test.com" rules: ##按需配置域名 #- host: www.test.com - http: paths: - path: /hellok8s pathType: Prefix backend: service: name: hellok8s-svc port: number: 80 - path: / pathType: Prefix backend: service: name: hellok8s-svc port: number: 80
# kubectl get pods NAME READY STATUS RESTARTS AGE hellok8s-dep-6588f6bd76-wlh4h 1/1 Running 0 60s 1 2 3 # kubectl get svc -A |grep LoadBalancer kube-system nginx-ingress-lb LoadBalancer 192.168.223.150 47.108.153.86 80:30299/TCP,443:31348/TCP 100m kube-system vpc-lb LoadBalancer 192.168.37.202 172.24.43.76 80:32532/TCP,443:31330/TCP 3m19s
验证内外网访问
浏览器访问公网LB地址
http://47.108.153.86/hellok8s
同一VPC机器访问内网LB地址
# curl 172.24.43.76 <!DOCTYPE html> <html> <head> <title>Hello Kubernetes!</title> <link rel="stylesheet" type="text/css" href="/css/main.css"> <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Ubuntu:300" > </head> <body> <div class="main"> <img src="/images/kubernetes.png"/> <div class="content"> <div id="message"> Hello world! </div> <div id="info"> <table> <tr> <th>pod:</th> <td>hellok8s-dep-6588f6bd76-wlh4h</td> </tr> <tr> <th>node:</th> <td>Linux (4.19.91-26.al7.x86_64)</td> </tr> </table> </div> </div> </div> </body> </html>
总结：这样做的好处：同一VPC访问服务时，可以走内网，节省流量
2.集群新建Nginx Controller内网访问LB
参考网址：https://help.aliyun.com/document_detail/151524.html
# kubectl create ns vpc-nginx namespace/vpc-nginx created
web页面创建应用nginx controller应用
容器服务 》应用市场 》ack-ingress-nginx-v1 》选择对应集群和命名空间（找不到命名空间时，刷新网页）
修改ingressClassResource的name字段
必须配置为专属标识，以区别两个nginx controller
默认为ack-nginx 修改为 ack-nginx-vpc
关闭公网访问 controller.service.external.enabled = false
打开内网controller.service.internal.enabled = true
（注意参考文档配置是 ack-ingress-nginx-v1版本）
等待部署完成
# kubectl -n vpc-nginx get svc |grep LoadBalancer ack-ingress-nginx-v1-default-controller-internal LoadBalancer 192.168.183.85 172.24.43.77 80:30389/TCP,443:31903/TCP 2m47s
验证阶段
##删除刚刚的测试pod # kubectl delete -f hello.yaml deployment.apps "hellok8s-dep" deleted service "hellok8s-svc" deleted ingress.networking.k8s.io "hellok8s-ingress" deleted
修改ingress.class配置 绑定内网Nginx Controller
##只修改ingress.annotations片段 ...... apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: hellok8s-ingress annotations: ##用来绑定特定的ingress-nginx kubernetes.io/ingress.class: ack-nginx-vpc ##默认class #kubernetes.io/ingress.class: nginx ##配置了ssl证书即打开此配置 #nginx.ingress.kubernetes.io/ssl-redirect: 'true' nginx.ingress.kubernetes.io/rewrite-target: / nginx.ingress.kubernetes.io/enable-cors: 'true' nginx.ingress.kubernetes.io/cors-allow-headers: DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization,Access-Control-Allow-Origin nginx.ingress.kubernetes.io/cors-allow-methods: PUT, GET, POST, OPTIONS, DELETE nginx.ingress.kubernetes.io/cors-allow-origin: '*' nginx.ingress.kubernetes.io/cors-allow-credentials: 'true' spec: ......
部署查看效果
# kubectl get ing NAME CLASS HOSTS ADDRESS PORTS AGE hellok8s-ingress <none> * 172.24.43.77 80 55s [root@iZ2vc69mi81oajsfk8bjefZ .kube]# kubectl get svc -A |grep LoadBalancer kube-system nginx-ingress-lb LoadBalancer 192.168.223.150 47.108.153.86 80:30299/TCP,443:31348/TCP 162m kube-system vpc-lb LoadBalancer 192.168.37.202 172.24.43.76 80:32532/TCP,443:31330/TCP 65m vpc-nginx ack-ingress-nginx-v1-default-controller-internal LoadBalancer 192.168.183.85 172.24.43.77 80:30389/TCP,443:31903/TCP 7m48s
发现ingress 已经成功绑定到内网的LB上
测试访问
## 公网私网LB失败，说明ingress未添加 [root@iZ2vc69mi81oajsfk8bjefZ .kube]# curl 47.108.153.86 <html> <head><title>404 Not Found</title></head> <body> <center><h1>404 Not Found</h1></center> <hr><center>nginx</center> </body> </html> [root@iZ2vc69mi81oajsfk8bjefZ .kube]# curl 172.24.43.76 <html> <head><title>404 Not Found</title></head> <body> <center><h1>404 Not Found</h1></center> <hr><center>nginx</center> </body> </html> ## 私网LB访问成功 说明ingress添加成功 [root@iZ2vc69mi81oajsfk8bjefZ .kube]# curl 172.24.43.77 <!DOCTYPE html> <html> <head> <title>Hello Kubernetes!</title> <link rel="stylesheet" type="text/css" href="/css/main.css"> <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Ubuntu:300" > </head> <body> <div class="main"> <img src="/images/kubernetes.png"/> <div class="content"> <div id="message"> Hello world! </div> <div id="info"> <table> <tr> <th>pod:</th> <td>hellok8s-dep-6588f6bd76-xwj7q</td> </tr> <tr> <th>node:</th> <td>Linux (4.19.91-26.al7.x86_64)</td> </tr> </table> </div> </div> </div> </body> </html>
总结：部署阶段稍麻烦，但是部署完成后，仅通过修改ingress配置绑定对应的Nginx Controller即可实现业务的对外控制
