前言:
ingress直译:进口;入口;初切;进入;进入资格;进入权。在kubernetes中,它指的是网络入口。
ingress概述:
通俗来讲,Ingress和之前提到的Service、Deployment等类似,也是一 个Kubernetes的资源对象,Deployment是用来部署应用的,Ingress就是实现 用域名的方式访问应用。Ingress实现的方式有很多,比如Nginx、HAProxy、 Treafik等(也就是控制器),就Nginx而言,和上述提到的传统服务架构用Nginx类似。 Ingress控制器在每个符合条件的宿主机上部署一个Pod,这个Pod里面运行的 就是Nginx进程,里面的实现逻辑和宿主机部署Nginx的方式并无太大区别, 关键区别是宿主机部署的Nginx需要更改Nginx的配置文件配置域名,而 Ingress则和其他Kubernetes资源文件一样,使用YAML文件进行配置,之后 Ingress控制器根据YAML文件定义的内容自动生成对应的配置文件。 在Kubernetes v1.1版中正式引用Ingress class的概念,用于从集群外部到集 群内部Service的HTTP和HTTPS路由,可以配置提供服务外部访问的URL、负载 均衡和终止SSL,并提供基于域名的虚拟主机。流量从Internet到Ingress再 到Services最后到Pod上,通常情况下,Ingress部署在所有的Node节点上, 暴露443和80端口(一般通过hostNetwork的方式部署Ingress),之后再通过 F5或公有云LB代理到对应的Ingress节点上,之后将域名解析到F5或公有云LB 即可实现基于域名的服务发布。
OK,为什么要用ingress呢?这个问题我先给打个10分。
确实,我们希望任何东西都是简单可靠可用的,但一台服务器上的端口是有限的,总共是65535好像,并且如果一个服务器如果像一面墙一样打满了孔洞,光管理这些端口就是一个非常大的工作量了,何况这些孔洞还有安全方面的风险(所以喽,有的安全性要求高的服务器会安装防火墙,就是堵这些孔洞的)。
回到k8s集群内,我们知道service的网络模式可以是NodePort或者EXTERNAL-IP或者是CLUSTER-IP或者是什么都没有的无头SERVICE(CLUSTER-IP 没有值)。
而想要把服务发布到集群外无外乎就是NodePort和EXTERNAL-IP两种方式了,CLUSTER-IP只能在集群内部使用,毕竟虚拟出来的IP地址嘛。
NodePort会在集群内的所有节点开辟一个随机的唯一的不重复的端口,并且这个端口还是由kube-apiserver限定的2700多个端口,关键的地方是随机和唯一不重复。
kube-apiserver的端口限定通常是它的配置文件(30000到32767):
KUBE_APISERVER_OPTS="--v=2 \ --logtostderr=false \ --service-node-port-range=30000-32767 \ --log-dir=/opt/kubernetes/logs \ --etcd-servers=https://192.168.217.16:2379,https://192.168.217.17:2379,https://192.168.217.18:2379 \ 。。。。省略 --service-cluster-ip-range=10.0.0.0/16 \ 。。。。省略
那么,如果有几十个service需要发布到集群外,使用NodePort就会有几十个上百个随机的端口,是不是十分的不安全和不方便呢(要查询出随机的端口,并记住的哦)。
EXTERNAL-IP 是使用互联网的IP,而互联网的IP资源是比较少的,这一条就把服务发布的路堵的差不多了。
而ingress 是在集群内创建一个service,此service可以绑定N个自定义的域名,并且它使用的端口是一个固定的端口,除非你重新生成这个service,而这就给了我们的无数可能,也因此,借由这个service,我们可以实现服务发布的特定流程,比如红蓝发布。
例如我这个ingress controller的service:
[root@k8s-master ~]# k get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-controller NodePort 10.0.191.241 <none> 80:32296/TCP,443:30766/TCP 9h
只要利用ingress资源清单文件绑定一个域名+32296这个端口就可以在集群外访问到了,如果是tls类型的域名,那么就是域名+30766端口在集群外就可以访问到。可以看到,端口是固定了的两个,那么孔洞也就十分少了,安全性自然是提高了嘛,剩下的管理工作就是管理好这些ingress清单文件就可以了,这就自然得达到了服务治理的目的。
总结一哈,ingress可以实现服务治理(管理集群内的每一个service),服务发布(比如红蓝,金丝雀等等都是基于ingress),网络黑白名单,https支持等等功能(主要的还是服务service)。
简单的流程说一哈:
(1)某一个web应用(例如,门户网站,WordPress这样的cms等等,此时的表现形式是一个pod实例)---->基于web应用创建一个需要发布到集群外的service(这个service一般不需要使用NodePort或者EXTERNAL-IP,因为如果是这两个类型已经可以发布service到集群外了,在使用ingress就是脱裤子放P了)
(2)ingress资源清单文件(此文件绑定上面的web应用的service和一个自定义域名)---->ingress 控制器绑定一个ingress class(指定使用哪个控制器,如果有多个controller的话)---->ingress service(ingress 控制器发出一个service(通常这个service是NodePort或者EXTERNAL-IP的网络类型,这个服务非常重要,是总入口,是在部署ingress控制器的时候生成)---->ingress service通过控制器处理web应用衍生的那个service,也就是(1)的service
一,
ingress-nginx的部署:
ingress-nginx是一种网络插件,部署比较简单,说实话,部署简单,用好ingress真的非常不容易
版本问题:
由于kubernetes的版本众多,因此,ingress作为其扩展功能插件是也随着kubernetes的版本有更新,并且ingress资源定义也随着版本变迁有一定的改变。
在本例中,使用的kubernetes版本是1.18,经测试ingress是和其匹配的,但如果用在高版本的kubernetes会有问题,在此提醒一哈(具体表现是在高版本kubernetes内部署不会成功,高版本指的是1.22及以后的版本)。
Support Versions table
Ingress-nginx version | k8s supported version | Alpine Version | Nginx Version |
v1.1.1 | 1.23, 1.22, 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v1.1.0 | 1.22, 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v1.0.5 | 1.22, 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v1.0.4 | 1.22, 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v1.0.3 | 1.22, 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v1.0.2 | 1.22, 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v1.0.1 | 1.22, 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v1.0.0 | 1.22, 1.21, 1.20, 1.19 | 3.13.5 | 1.20.1 |
v0.50.0 | 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v0.49.3 | 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v0.49.2 | 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v0.49.1 | 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v0.49.0 | 1.21, 1.20, 1.19 | 3.13.5 | 1.20.1 |
v0.48.1 | 1.21, 1.20, 1.19 | 3.13.5 | 1.20.1 |
v0.47.0 | 1.21, 1.20, 1.19 | 3.13.5 | 1.20.1 |
v0.46.0 | 1.21, 1.20, 1.19 | 3.13.2 | 1.19.6 |
#######附:
在Kubernetes 1.19之前,仅支持v1beta1 Ingress资源-从Kubernetes 1.19到1.21,同时支持v1beta1和v1 Ingress资源-在Kubernetes 1.22及更高版本中,仅支持v1 Ingress资源
yaml文件的下载:
版本号:GitHub - kubernetes/ingress-nginx: Ingress-NGINX Controller for Kubernetes
例如,下载1.1.1版本的yaml文件,如果是其它版本替换版本号就可以了
https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml
vim deploy-ingress.yaml
这么长的文件不用说了,很多同学看完必定会说:我的头要裂开了。没错,我也是这样的。
此文件主要有几个地方需要注意,第一,修改部署方式为daemonsets,第二,镜像部分我做了处理都是国内可以直接pull的,也就是本地化,第三,如果要部署的话,建议master节点打effect为NoScheduler的污点,毕竟ingress是一个重要服务,放到master和apiserver抢资源不太合适。
apiVersion: v1 kind: Namespace metadata: name: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx --- apiVersion: v1 kind: ServiceAccount metadata: labels: helm.sh/chart: ingress-nginx-3.33.0 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/version: 0.47.0 app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: controller name: ingress-nginx namespace: ingress-nginx automountServiceAccountToken: true --- apiVersion: v1 kind: ConfigMap metadata: labels: helm.sh/chart: ingress-nginx-3.33.0 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/version: 0.47.0 app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: controller name: ingress-nginx-controller namespace: ingress-nginx data: --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: helm.sh/chart: ingress-nginx-3.33.0 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/version: 0.47.0 app.kubernetes.io/managed-by: Helm name: ingress-nginx rules: - apiGroups: - '' resources: - configmaps - endpoints - nodes - pods - secrets verbs: - list - watch - apiGroups: - '' resources: - nodes verbs: - get - apiGroups: - '' resources: - services verbs: - get - list - watch - apiGroups: - extensions - networking.k8s.io # k8s 1.14+ resources: - ingresses verbs: - get - list - watch - apiGroups: - '' resources: - events verbs: - create - patch - apiGroups: - extensions - networking.k8s.io # k8s 1.14+ resources: - ingresses/status verbs: - update - apiGroups: - networking.k8s.io # k8s 1.14+ resources: - ingressclasses verbs: - get - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: labels: helm.sh/chart: ingress-nginx-3.33.0 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/version: 0.47.0 app.kubernetes.io/managed-by: Helm name: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: ingress-nginx subjects: - kind: ServiceAccount name: ingress-nginx namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: labels: helm.sh/chart: ingress-nginx-3.33.0 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/version: 0.47.0 app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: controller name: ingress-nginx namespace: ingress-nginx rules: - apiGroups: - '' resources: - namespaces verbs: - get - apiGroups: - '' resources: - configmaps - pods - secrets - endpoints verbs: - get - list - watch - apiGroups: - '' resources: - services verbs: - get - list - watch - apiGroups: - extensions - networking.k8s.io # k8s 1.14+ resources: - ingresses verbs: - get - list - watch - apiGroups: - extensions - networking.k8s.io # k8s 1.14+ resources: - ingresses/status verbs: - update - apiGroups: - networking.k8s.io # k8s 1.14+ resources: - ingressclasses verbs: - get - list - watch - apiGroups: - '' resources: - configmaps resourceNames: - ingress-controller-leader-nginx verbs: - get - update - apiGroups: - '' resources: - configmaps verbs: - create - apiGroups: - '' resources: - events verbs: - create - patch --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: labels: helm.sh/chart: ingress-nginx-3.33.0 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/version: 0.47.0 app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: controller name: ingress-nginx namespace: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: ingress-nginx subjects: - kind: ServiceAccount name: ingress-nginx namespace: ingress-nginx --- apiVersion: v1 kind: Service metadata: labels: helm.sh/chart: ingress-nginx-3.33.0 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/version: 0.47.0 app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: controller name: ingress-nginx-controller-admission namespace: ingress-nginx spec: type: ClusterIP ports: - name: https-webhook port: 443 targetPort: webhook selector: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/component: controller --- apiVersion: v1 kind: Service metadata: annotations: labels: helm.sh/chart: ingress-nginx-3.33.0 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/version: 0.47.0 app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: controller name: ingress-nginx-controller namespace: ingress-nginx spec: type: NodePort ports: - name: http port: 80 protocol: TCP targetPort: http - name: https port: 443 protocol: TCP targetPort: https selector: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/component: controller --- apiVersion: apps/v1 kind: DaemonSet metadata: labels: helm.sh/chart: ingress-nginx-3.33.0 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/version: 0.47.0 app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: controller name: ingress-nginx-controller namespace: ingress-nginx spec: selector: matchLabels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/component: controller revisionHistoryLimit: 10 minReadySeconds: 0 template: metadata: labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/component: controller spec: dnsPolicy: ClusterFirstWithHostNet containers: - name: controller image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v0.50.0 imagePullPolicy: IfNotPresent lifecycle: preStop: exec: command: - /wait-shutdown args: - /nginx-ingress-controller - --election-id=ingress-controller-leader - --ingress-class=nginx - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller - --validating-webhook=:8443 - --validating-webhook-certificate=/usr/local/certificates/cert - --validating-webhook-key=/usr/local/certificates/key securityContext: capabilities: drop: - ALL add: - NET_BIND_SERVICE runAsUser: 101 allowPrivilegeEscalation: true env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: LD_PRELOAD value: /usr/local/lib/libmimalloc.so livenessProbe: failureThreshold: 5 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 ports: - name: http containerPort: 80 protocol: TCP - name: https containerPort: 443 protocol: TCP - name: webhook containerPort: 8443 protocol: TCP volumeMounts: - name: webhook-cert mountPath: /usr/local/certificates/ readOnly: true resources: requests: cpu: 100m memory: 90Mi nodeSelector: kubernetes.io/os: linux serviceAccountName: ingress-nginx terminationGracePeriodSeconds: 300 volumes: - name: webhook-cert secret: secretName: ingress-nginx-admission --- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration metadata: labels: helm.sh/chart: ingress-nginx-3.33.0 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/version: 0.47.0 app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: admission-webhook name: ingress-nginx-admission webhooks: - name: validate.nginx.ingress.kubernetes.io matchPolicy: Equivalent rules: - apiGroups: - networking.k8s.io apiVersions: - v1beta1 operations: - CREATE - UPDATE resources: - ingresses failurePolicy: Fail sideEffects: None admissionReviewVersions: - v1 - v1beta1 clientConfig: service: namespace: ingress-nginx name: ingress-nginx-controller-admission path: /networking/v1beta1/ingresses --- apiVersion: v1 kind: ServiceAccount metadata: name: ingress-nginx-admission annotations: helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded labels: helm.sh/chart: ingress-nginx-3.33.0 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/version: 0.47.0 app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: admission-webhook namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: ingress-nginx-admission annotations: helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded labels: helm.sh/chart: ingress-nginx-3.33.0 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/version: 0.47.0 app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: admission-webhook rules: - apiGroups: - admissionregistration.k8s.io resources: - validatingwebhookconfigurations verbs: - get - update --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: ingress-nginx-admission annotations: helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded labels: helm.sh/chart: ingress-nginx-3.33.0 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/version: 0.47.0 app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: admission-webhook roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: ingress-nginx-admission subjects: - kind: ServiceAccount name: ingress-nginx-admission namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: ingress-nginx-admission annotations: helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded labels: helm.sh/chart: ingress-nginx-3.33.0 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/version: 0.47.0 app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: admission-webhook namespace: ingress-nginx rules: - apiGroups: - '' resources: - secrets verbs: - get - create --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: ingress-nginx-admission annotations: helm.sh/hook: pre-install,pre-upgrade,post-install,post-upgrade helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded labels: helm.sh/chart: ingress-nginx-3.33.0 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/version: 0.47.0 app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: admission-webhook namespace: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: ingress-nginx-admission subjects: - kind: ServiceAccount name: ingress-nginx-admission namespace: ingress-nginx --- apiVersion: batch/v1 kind: Job metadata: name: ingress-nginx-admission-create annotations: helm.sh/hook: pre-install,pre-upgrade helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded labels: helm.sh/chart: ingress-nginx-3.33.0 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/version: 0.47.0 app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: admission-webhook namespace: ingress-nginx spec: template: metadata: name: ingress-nginx-admission-create labels: helm.sh/chart: ingress-nginx-3.33.0 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/version: 0.47.0 app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: admission-webhook spec: containers: - name: create image: jettech/kube-webhook-certgen:v1.5.1 imagePullPolicy: IfNotPresent args: - create - --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc - --namespace=$(POD_NAMESPACE) - --secret-name=ingress-nginx-admission env: - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace restartPolicy: OnFailure serviceAccountName: ingress-nginx-admission securityContext: runAsNonRoot: true runAsUser: 2000 --- apiVersion: batch/v1 kind: Job metadata: name: ingress-nginx-admission-patch annotations: helm.sh/hook: post-install,post-upgrade helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded labels: helm.sh/chart: ingress-nginx-3.33.0 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/version: 0.47.0 app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: admission-webhook namespace: ingress-nginx spec: template: metadata: name: ingress-nginx-admission-patch labels: helm.sh/chart: ingress-nginx-3.33.0 app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/version: 0.47.0 app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: admission-webhook spec: containers: - name: patch image: docker.io/jettech/kube-webhook-certgen:v1.5.1 imagePullPolicy: IfNotPresent args: - patch - --webhook-name=ingress-nginx-admission - --namespace=$(POD_NAMESPACE) - --patch-mutating=false - --secret-name=ingress-nginx-admission - --patch-failure-policy=Fail env: - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace restartPolicy: OnFailure serviceAccountName: ingress-nginx-admission securityContext: runAsNonRoot: true runAsUser: 2000
apply上述文件后,将会出现这么些pod:
[root@master ~]# k get po -n ingress-nginx -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES ingress-nginx-admission-create-xc2z4 0/1 Completed 0 4d2h 192.168.169.133 k8s-node2 <none> <none> ingress-nginx-admission-patch-7xgst 0/1 Completed 3 4d2h 192.168.235.197 k8s-master <none> <none> ingress-nginx-controller-987b747f-xrzn8 1/1 Running 17 4d2h 10.244.169.139 k8s-node2 <none> <none> ingress-nginx-controller-ph46w 1/1 Running 0 163m 10.244.36.93 k8s-node1 <none> <none> ingress-nginx-controller-t2nxd 1/1 Running 0 163m 10.244.169.163 k8s-node2 <none> <none>
将会有这么两个service:
注意31702这个端口,这个端口以后会常用的,这两个service是部署文件里产生的。
[root@master ~]# k get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-controller NodePort 10.0.0.102 <none> 80:31702/TCP,443:31675/TCP 4d2h ingress-nginx-controller-admission ClusterIP 10.0.0.12 <none> 443/TCP 4d2h
节点情况如下:
[root@master ~]# k get no -owide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME k8s-master Ready <none> 36d v1.18.3 192.168.217.16 <none> CentOS Linux 7 (Core) 5.16.9-1.el7.elrepo.x86_64 docker://20.10.7 k8s-node1 Ready <none> 36d v1.18.3 192.168.217.17 <none> CentOS Linux 7 (Core) 5.16.9-1.el7.elrepo.x86_64 docker://20.10.7 k8s-node2 Ready <none> 36d v1.18.3 192.168.217.18 <none> CentOS Linux 7 (Core) 5.16.9-1.el7.elrepo.x86_64 docker://20.10.7
二,
编写ingress资源清单文件
vim ingress-http.yaml
(两个自定义域名nginx.test.com和tomcat.test.com 绑定到了nginx-service和tomcat-service 这两服务上了)
这里要注意了,为什么两个servicePort都是80了,因为上面的ingress-nginx-controller是80端口嘛,由于我的kubernetes版本是1.18,因此还是使用注解方式。
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress-http namespace: dev annotations: nginx.ingress.kubernetes.io/rewrite-target: / spec: rules: - host: nginx.test.com http: paths: - path: / backend: serviceName: nginx-service servicePort: 80 - host: tomcat.test.com http: paths: - path: / backend: serviceName: tomcat-service servicePort: 80
此文件执行过后,ingress的情况如下:
可以看到此ingress是绑定到了192.168.217.18也就是node2节点了,绑定了两个域名,ADDRESS是node2节点的IP。
[root@master ~]# k get ing -A NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE dev ingress-http <none> nginx.test.com,tomcat.test.com 192.168.217.18 80 3h23m
建立namespace:
[root@master ~]# cat tomcat-nginx-ns.yaml apiVersion: v1 kind: Namespace metadata: name: dev ---
vim tomcat-nginx.yaml
建立两个deployment的pod,可提供web功能的,一个nginx 一个tomcat,两个都做了node选择,和ingress-nginx-controller处于同一个节点。
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment namespace: dev spec: replicas: 1 selector: matchLabels: app: nginx-pod template: metadata: labels: app: nginx-pod spec: containers: - name: nginx image: nginx:1.17.1 ports: - containerPort: 80 nodeName: k8s-node2 --- apiVersion: apps/v1 kind: Deployment metadata: name: tomcat-deployment namespace: dev spec: replicas: 1 selector: matchLabels: app: tomcat-pod template: metadata: labels: app: tomcat-pod spec: containers: - name: tomcat image: tomcat:8.5-jre10-slim ports: - containerPort: 8080 nodeName: k8s-node2
vim tomcat-nginx-svc.yaml
这里又需要注意了,两个service一个无头service,一个普通的clusterip,一会使用了ingress清单文件就可以将这两个服务发布到集群外了。
--- apiVersion: v1 kind: Service metadata: name: nginx-service namespace: dev spec: ports: - port: 80 name: nginx clusterIP: None selector: app: nginx-pod --- apiVersion: v1 kind: Service metadata: name: tomcat-service namespace: dev spec: selector: app: tomcat-pod type: ClusterIP ports: - protocol: TCP port: 80 targetPort: 8080
三,
OK,以上文件都apply后,就可以看结果了:
宿主机做hosts域名解析:
OK,如果nginx部署到node1节点会怎么样呢?
报错504
这就说明一个问题,kubectl get ingress -A 查询出来的那个IP地址也就是ingress的节点和要发布的service对应的pod要在同一个节点下,和service的类型没有关系,即使service是无头的也是OK的,ingress-nginx-controller会帮我们自己处理好的。并且多个service都是通过同一个端口发不出来的,只是域名不同而已。
四,
改造成https也就是使用ssl的域名(实验性质,当然还是使用自签的证书,实际生产环境肯定是使用备案过的证书哦)
a,
生成自签证书
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=tomcat.test.com/ST=BeiJing/L=BeiJing/O=devops/OU=unicorn"
可以看到,生成了这么两个玩意
[root@master ~]# ls tls* tls.crt tls.key
b,
生成secret,证书存放到secret里
注意,这里要指定namespace,要不ingress controller找不到证书
kubectl create secret tls tls-secret --key=tls.key --cert tls.crt -n dev
c,
编写ingress文件
vim ingress-https.yaml
主要就是添加了tls相关,域名还是不变的以及一个注解,并且引用了前面打入的证书
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: test-ingress3 namespace: dev annotations: nginx.ingress.kubernetes.io/rewrite-target: / kubernetes.io/ingress.class: nginx # nginx.ingress.kubernetes.io/backend-protocol: HTTPS nginx.ingress.kubernetes.io/ssl-redirect: 'true' # # nginx.ingress.kubernetes.io/use-regex: 'true' spec: tls: - hosts: - tomcat.test.com secretName: tls-secret rules: - host: tomcat.test.com http: paths: - path: / backend: serviceName: tomcat-service servicePort: 80
查看ingress,可以看到多了一个443,还是绑定的node2节点(ingress 控制器前面部署的时候搞错了,就部署在了一个节点,daemonset没有使用的。)
[root@master ~]# k get ing -A NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE dev ingress-http <none> nginx.test.com,tomcat.test.com 192.168.217.18 80 11h dev test-ingress3 <none> tomcat.test.com 192.168.217.18 80, 443 4m16s
d,
验证;
需要先查询一哈ingress的service提供的端口,查询出端口是31675
[root@master ~]# vim ingress-https.yaml [root@master ~]# k get svc -A NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 37d dev nginx-service ClusterIP None <none> 80/TCP 16h dev tomcat-service ClusterIP 10.0.92.37 <none> 80/TCP 16h ingress-nginx ingress-nginx-controller NodePort 10.0.0.102 <none> 80:31702/TCP,443:31675/TCP 4d14h ingress-nginx ingress-nginx-controller-admission ClusterIP 10.0.0.12 <none> 443/TCP 4d14h kube-system coredns ClusterIP 10.0.0.2 <none> 53/UDP,53/TCP 36d
OK,https证书启用成功,此网站的证书只是没有注册的自产证书,但功能是完好的。
总结:
那么现在这个ingress controller插件是可以使用的,ingress统一了要发布服务的端口,可以看到即使多个门户,也可以简单的以域名来区分,端口是统一的31702(http)或者31675(https),从而达到了服务治理的目的(其它功能,比如黑白名单,重定向,二级域名跳转等等留待以后研究哈):
[root@master ~]# k get ing -A NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE dev ingress-http <none> nginx.test.com,tomcat.test.com 192.168.217.18 80 11h dev test-ingress3 <none> tomcat.test.com 192.168.217.18 80, 443 4m16s
ingress-nginx-controller截取的部分日志:
192.168.217.18 - - [03/Oct/2022:09:03:03 +0000] "GET /favicon.ico HTTP/1.1" 404 555 "http://nginx.test.com:31702/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36 Core/1.94.175.400 QQBrowser/11.1.5155.400" 423 0.001 [dev-nginx-service-80] [] 10.244.36.97:80 555 0.001 404 3b851fa39d2268e5fbd230fc5f8d1d59 192.168.217.18 - - [03/Oct/2022:09:03:05 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36 Core/1.94.175.400 QQBrowser/11.1.5155.400" 581 0.001 [dev-nginx-service-80] [] 10.244.36.97:80 0 0.001 304 21366d34e3103a50236738d6b1dd00e7 192.168.217.18 - - [03/Oct/2022:09:03:07 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36 Core/1.94.175.400 QQBrowser/11.1.5155.400" 581 0.002 [dev-nginx-service-80] [] 10.244.36.97:80 0 0.002 304 69bcb2182681ea3be696fe3b449e286c 192.168.217.18 - - [03/Oct/2022:09:03:09 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36 Core/1.94.175.400 QQBrowser/11.1.5155.400" 581 0.001 [dev-nginx-service-80] [] 10.244.36.97:80 0 0.000 304 3622a7bf35abbdf08c43084c89fd0110 192.168.217.18 - - [03/Oct/2022:09:06:15 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36 Core/1.94.175.400 QQBrowser/11.1.5155.400" 500 0.003 [dev-nginx-service-80] [] 10.244.36.97:80 612 0.003 200 5f6ba8d7070c9b7984acf2012fa57a5b 192.168.217.18 - - [03/Oct/2022:09:06:17 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36 Core/1.94.175.400 QQBrowser/11.1.5155.400" 581 0.002 [dev-nginx-service-80] [] 10.244.36.97:80 0 0.001 304 12d114732628f18df5988661bf79fc83 192.168.217.18 - - [03/Oct/2022:09:16:42 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" 460 0.001 [dev-nginx-service-80] [] 10.244.36.97:80 612 0.001 200 298738d5c747741af42d8f13fb4c4566
可以看到我是使用的QQ浏览器(也用了谷歌105版本),192.168.217.18:31702 代理了10.244.36.97:80
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES dev nginx-deployment-b785b4498-5r8jn 1/1 Running 0 16m 10.244.36.97 k8s-node1 <none> <none>