前言:
ingress是kubernetes内的一个重要功能插件,这个使得服务治理成为一个可能,当然,结合微服务更为妥当了。
不管是什么插件,还是服务,第一步当然是要能顺利的部署到系统当中,这样,才会有后续的使用可能,否则一切无从谈起,对吧。
而kubernetes的版本众多,因此,它的插件版本也是非常多得,而本文将就kubernetes-1.23集群内部署ingress-nginx做一个详细的描述,并指出它(ingress-nginx)和旧版本的不同
一,
部署方法简介
按照部署手法来说,插件一般是使用helm安装并管理的,其次是使用yaml清单文件,但很明显,yaml清单文件的方式更为灵活,能够做一些简单的定制,从可定制性来说,无疑yaml清单是更好的。
本文将使用yaml清单文件的形式部署
按照官方的说法,yaml清单文件的部署效果是和使用helm部署基本一致的(好像说是根据helm生成的)。
二,
ingress-nginx和kubernetes的版本对应关系:
GitHub - kubernetes/ingress-nginx at controller-v1.6.4
Supported Versions table
Supported versions for the ingress-nginx project mean that we have completed E2E tests, and they are passing for the versions listed. Ingress-Nginx versions may work on older versions but the project does not make that guarantee.
(只是一个大概的版本对应表,以下内容是确定的)
Ingress-NGINX version | k8s supported version | Alpine Version | Nginx Version |
v1.6.4 | 1.26, 1.25, 1.24, 1.23 | 3.17.0 | 1.21.6 |
v1.5.1 | 1.25, 1.24, 1.23 | 3.16.2 | 1.21.6 |
v1.4.0 | 1.25, 1.24, 1.23, 1.22 | 3.16.2 | 1.19.10† |
v1.3.1 | 1.24, 1.23, 1.22, 1.21, 1.20 | 3.16.2 | 1.19.10† |
v1.3.0 | 1.24, 1.23, 1.22, 1.21, 1.20 | 3.16.0 | 1.19.10† |
v1.2.1 | 1.23, 1.22, 1.21, 1.20, 1.19 | 3.14.6 | 1.19.10† |
v1.1.3 | 1.23, 1.22, 1.21, 1.20, 1.19 | 3.14.4 | 1.19.10† |
v1.1.2 | 1.23, 1.22, 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v1.1.1 | 1.23, 1.22, 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v1.1.0 | 1.22, 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v1.0.5 | 1.22, 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v1.0.4 | 1.22, 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v1.0.3 | 1.22, 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v1.0.2 | 1.22, 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v1.0.1 | 1.22, 1.21, 1.20, 1.19 | 3.14.2 | 1.19.9† |
v1.0.0 | 1.22, 1.21, 1.20, 1.19 | 3.13.5 | 1.20.1 |
根据以上表,可以确定 kubernetes-1.23-1.26都可以使用ingress-nginx-1.6.4
三,
ingress-nginx的yaml清单文件下载地址(其它版本只需要修改v后面的数字即可):
OK,不谈是安装哪个版本的ingress-nginx,所有的版本的yaml文件都需要一个本土化修改
这里说的本土化和Java开发的时候说的国际化其实挺类似的,主要原因是国内的墙的问题,造成很多镜像无法下载,需要修改成国内可以下载的同一镜像,
例如(上面下载的文件里包含的镜像),
registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20220916-gd32f8c343@sha256:39c5b2e3310dc4264d638ad28d9d1d96c4cbb2b2dcfb52368fe4e3c63f61e10f
修改为:
dyrnq/kube-webhook-certgen:v20220916-gd32f8c343
registry.k8s.io/ingress-nginx/controller:v1.6.4@sha256:15be4666c53052484dd2992efacf2f50ea77a78ae8aa21ccd91af6baaa7ea22f
修改为:
registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.6.4
ingress-nginx只涉及以上两个镜像
其次是网络的修改,这里要说明一下,ingress-nginx和网络模型是解耦的,也就是说,不需要关心网络模型是IPVS,还是iptables,也不需要关心是flannel还是calico或者是其它的网络插件,主要是修改ingress的对外服务(service)为NodePort,如果有必要,将port固定即可。
apiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.6.4 name: ingress-nginx-controller namespace: ingress-nginx spec: externalTrafficPolicy: Local ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - appProtocol: http name: http port: 80 protocol: TCP targetPort: http - appProtocol: https name: https port: 443 protocol: TCP targetPort: https selector: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx type: NodePort
下面附上完整的修改后的部署yaml清单文件(以上所说修改已经完成,port固定为31080和31443):
cat deploy-ingress.yaml apiVersion: v1 kind: Namespace metadata: labels: app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx name: ingress-nginx --- apiVersion: v1 automountServiceAccountToken: true kind: ServiceAccount metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.6.4 name: ingress-nginx namespace: ingress-nginx --- apiVersion: v1 kind: ServiceAccount metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.6.4 name: ingress-nginx-admission namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.6.4 name: ingress-nginx namespace: ingress-nginx rules: - apiGroups: - "" resources: - namespaces verbs: - get - apiGroups: - "" resources: - configmaps - pods - secrets - endpoints verbs: - get - list - watch - apiGroups: - "" resources: - services verbs: - get - list - watch - apiGroups: - networking.k8s.io resources: - ingresses verbs: - get - list - watch - apiGroups: - networking.k8s.io resources: - ingresses/status verbs: - update - apiGroups: - networking.k8s.io resources: - ingressclasses verbs: - get - list - watch - apiGroups: - coordination.k8s.io resourceNames: - ingress-nginx-leader resources: - leases verbs: - get - update - apiGroups: - coordination.k8s.io resources: - leases verbs: - create - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - discovery.k8s.io resources: - endpointslices verbs: - list - watch - get --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.6.4 name: ingress-nginx-admission namespace: ingress-nginx rules: - apiGroups: - "" resources: - secrets verbs: - get - create --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.6.4 name: ingress-nginx rules: - apiGroups: - "" resources: - configmaps - endpoints - nodes - pods - secrets - namespaces verbs: - list - watch - apiGroups: - coordination.k8s.io resources: - leases verbs: - list - watch - apiGroups: - "" resources: - nodes verbs: - get - apiGroups: - "" resources: - services verbs: - get - list - watch - apiGroups: - networking.k8s.io resources: - ingresses verbs: - get - list - watch - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - networking.k8s.io resources: - ingresses/status verbs: - update - apiGroups: - networking.k8s.io resources: - ingressclasses verbs: - get - list - watch - apiGroups: - discovery.k8s.io resources: - endpointslices verbs: - list - watch - get --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.6.4 name: ingress-nginx-admission rules: - apiGroups: - admissionregistration.k8s.io resources: - validatingwebhookconfigurations verbs: - get - update --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.6.4 name: ingress-nginx namespace: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: ingress-nginx subjects: - kind: ServiceAccount name: ingress-nginx namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.6.4 name: ingress-nginx-admission namespace: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: ingress-nginx-admission subjects: - kind: ServiceAccount name: ingress-nginx-admission namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: labels: app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.6.4 name: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: ingress-nginx subjects: - kind: ServiceAccount name: ingress-nginx namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.6.4 name: ingress-nginx-admission roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: ingress-nginx-admission subjects: - kind: ServiceAccount name: ingress-nginx-admission namespace: ingress-nginx --- apiVersion: v1 data: allow-snippet-annotations: "true" kind: ConfigMap metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.6.4 name: ingress-nginx-controller namespace: ingress-nginx --- apiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.6.4 name: ingress-nginx-controller namespace: ingress-nginx spec: externalTrafficPolicy: Local ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - appProtocol: http name: http port: 80 protocol: TCP targetPort: http nodePort: 31080 - appProtocol: https name: https port: 443 protocol: TCP targetPort: https nodePort: 31443 selector: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx type: NodePort --- apiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.6.4 name: ingress-nginx-controller-admission namespace: ingress-nginx spec: ports: - appProtocol: https name: https-webhook port: 443 targetPort: webhook selector: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx type: ClusterIP --- apiVersion: apps/v1 kind: DaemonSet metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.6.4 name: ingress-nginx-controller namespace: ingress-nginx spec: minReadySeconds: 0 revisionHistoryLimit: 10 selector: matchLabels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx template: metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx spec: containers: - args: - /nginx-ingress-controller - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller - --election-id=ingress-nginx-leader - --controller-class=k8s.io/ingress-nginx - --ingress-class=nginx - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller - --validating-webhook=:8443 - --validating-webhook-certificate=/usr/local/certificates/cert - --validating-webhook-key=/usr/local/certificates/key env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: LD_PRELOAD value: /usr/local/lib/libmimalloc.so image: registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:v1.6.4 imagePullPolicy: IfNotPresent lifecycle: preStop: exec: command: - /wait-shutdown livenessProbe: failureThreshold: 5 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 name: controller ports: - containerPort: 80 name: http protocol: TCP - containerPort: 443 name: https protocol: TCP - containerPort: 8443 name: webhook protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 resources: requests: cpu: 100m memory: 90Mi securityContext: allowPrivilegeEscalation: true capabilities: add: - NET_BIND_SERVICE drop: - ALL runAsUser: 101 volumeMounts: - mountPath: /usr/local/certificates/ name: webhook-cert readOnly: true dnsPolicy: ClusterFirst nodeSelector: kubernetes.io/os: linux serviceAccountName: ingress-nginx terminationGracePeriodSeconds: 300 volumes: - name: webhook-cert secret: secretName: ingress-nginx-admission --- apiVersion: batch/v1 kind: Job metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.6.4 name: ingress-nginx-admission-create namespace: ingress-nginx spec: template: metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.6.4 name: ingress-nginx-admission-create spec: containers: - args: - create - --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc - --namespace=$(POD_NAMESPACE) - --secret-name=ingress-nginx-admission env: - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: dyrnq/kube-webhook-certgen:v20220916-gd32f8c343 imagePullPolicy: IfNotPresent name: create securityContext: allowPrivilegeEscalation: false nodeSelector: kubernetes.io/os: linux restartPolicy: OnFailure securityContext: fsGroup: 2000 runAsNonRoot: true runAsUser: 2000 serviceAccountName: ingress-nginx-admission --- apiVersion: batch/v1 kind: Job metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.6.4 name: ingress-nginx-admission-patch namespace: ingress-nginx spec: template: metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.6.4 name: ingress-nginx-admission-patch spec: containers: - args: - patch - --webhook-name=ingress-nginx-admission - --namespace=$(POD_NAMESPACE) - --patch-mutating=false - --secret-name=ingress-nginx-admission - --patch-failure-policy=Fail env: - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace image: dyrnq/kube-webhook-certgen:v20220916-gd32f8c343 imagePullPolicy: IfNotPresent name: patch securityContext: allowPrivilegeEscalation: false nodeSelector: kubernetes.io/os: linux restartPolicy: OnFailure securityContext: fsGroup: 2000 runAsNonRoot: true runAsUser: 2000 serviceAccountName: ingress-nginx-admission --- apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.6.4 name: nginx spec: controller: k8s.io/ingress-nginx --- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration metadata: labels: app.kubernetes.io/component: admission-webhook app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.6.4 name: ingress-nginx-admission webhooks: - admissionReviewVersions: - v1 clientConfig: service: name: ingress-nginx-controller-admission namespace: ingress-nginx path: /networking/v1/ingresses failurePolicy: Fail matchPolicy: Equivalent name: validate.nginx.ingress.kubernetes.io rules: - apiGroups: - networking.k8s.io apiVersions: - v1 operations: - CREATE - UPDATE resources: - ingresses sideEffects: None
四,
与旧版的ingress-nginx的一些不同点
云原生|kubernetes|ingress-nginx插件部署以及简单的应用(修订版---适用于kubernetes-1.18-1.21)_安装ingress插件_晚风_END的博客-CSDN博客
这个是1.18集群使用的,主要的不同点集中在ingress资源的api version和一些定义方面的细小差异
1,apiVersion: networking.k8s.io/v1(新版)
apiVersion: extensions/v1beta1(旧版)
2,新版取消了注解,也就是annotations(元数据这不用注解),统一使用ingressClassName(在spec模块使用)
3,ingress直接绑定的是ingress的服务,而旧版绑定的是节点ip:
[root@k8s-master ~]# kubectl get ing -n dev NAME CLASS HOSTS ADDRESS PORTS AGE ingress-http nginx nginx.test.com,tomcat.test.com 10.109.115.193 80 16h [root@k8s-master ~]# kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-controller NodePort 10.109.115.193 <none> 80:31680/TCP,443:31111/TCP 4h47m ingress-nginx-controller-admission ClusterIP 10.110.7.6 <none> 443/TCP 4h47m
4,
新版新增pathType,旧版是使用注解
附注:
有3种支持的path类型:
ImplementationSpecific:对于这种path类型,匹配取决于IngressClass。可以将其视为一个单独的pathType或者将其认为和Prefix或者Exact路径类型一样。
Exact:精确匹配URL路径,并且区分大小写
Prefix: 根据URL中的,被/分割的前缀进行匹配。匹配区分大小写并且按照元素对路径进行匹配。path元素指的是路径中由/分隔符分隔的标签列表。
下面是ingress资源清单文件:
cat >ingress-http.yaml <<EOF apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-http namespace: dev spec: ingressClassName: nginx rules: - host: nginx.test.com http: paths: - backend: service: name: nginx-service port: number: 80 path: / pathType: ImplementationSpecific - host: tomcat.test.com http: paths: - backend: service: name: tomcat-service port: number: 80 path: / pathType: ImplementationSpecific EOF
使用方面和原来的旧版剩下基本相同。
创建测试pod:
cat tomcat-nginx.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment namespace: dev spec: replicas: 1 selector: matchLabels: app: nginx-pod template: metadata: labels: app: nginx-pod spec: containers: - name: nginx image: nginx:1.18 ports: - containerPort: 80 nodeName: k8s-node2 --- apiVersion: apps/v1 kind: Deployment metadata: name: tomcat-deployment namespace: dev spec: replicas: 1 selector: matchLabels: app: tomcat-pod template: metadata: labels: app: tomcat-pod spec: containers: - name: tomcat image: tomcat:8.5-jre10-slim ports: - containerPort: 8080 nodeName: k8s-node2
测试pod的service:
[root@k8s-master ~]# cat tomcat-nginx-svc.yaml --- apiVersion: v1 kind: Service metadata: name: nginx-service namespace: dev spec: ports: - port: 80 name: nginx clusterIP: None selector: app: nginx-pod --- apiVersion: v1 kind: Service metadata: name: tomcat-service namespace: dev spec: selector: app: tomcat-pod type: ClusterIP ports: - protocol: TCP port: 80 targetPort: 8080
此时,应用上面的ingress资源yaml文件,可以看到绑定了10.109.115.193这个IP地址,但,实际访问的时候仍然是用节点IP+31080
[root@k8s-master ~]# kubectl get svc -n dev NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx-service ClusterIP None <none> 80/TCP 17h tomcat-service ClusterIP 10.104.157.194 <none> 80/TCP 17h [root@k8s-master ~]# kubectl get ing -n dev NAME CLASS HOSTS ADDRESS PORTS AGE ingress-http nginx nginx.test.com,tomcat.test.com 10.109.115.193 80 17h [root@k8s-master ~]# kubectl get svc -n ingress-nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-controller NodePort 10.109.115.193 <none> 80:31080/TCP,443:31111/TCP 5h21m ingress-nginx-controller-admission ClusterIP 10.110.7.6 <none> 443/TCP 5h21m