使用Kmesh作为阿里云服务网格ASM Sidecarless模式数据面-阿里云开发者社区

概述

阿里云服务网格 ASM 支持 Sidecar 模式和 Sidecarless 模式。Sidecar 模式，即在每个服务实例旁运行一个代理，是当前最主流、稳定的方案，但这种架构引入了显著的延迟和资源开销。为了解决 Sidecar 模式所带来的延迟和资源消耗，近些年出现了不同的 Sidecarless 模式方案，例如 Istio Ambient，Istio Ambient 通过在每个节点部署 ztunnel，对节点上运行的 Pod 进行四层流量代理，并引入了 waypoint 对七层流量进行代理。虽然 Sidecarless 模式能够降低延迟和资源消耗，但其稳定性以及功能的完善程度仍有待提升。

阿里云服务网格 ASM 当前支持不同的 Sidecarless 模式，例如 Istio Ambient 模式，ACMG 模式以及 Kmesh 等。Kmesh （具体参考https://kmesh.net/）是一款基于 ebpf + 可编程内核实现的高性能服务网格数据面软件。通过将流量治理下沉到内核，实现网格内服务通信无需经过代理软件，大大缩减了流量转发路径，有效提升了服务访问的转发性能。

Kmesh 简介

Kmesh 的双引擎模式（dual-engine）模式使用 eBPF 在内核空间拦截流量，部署 Waypoint Proxy 来处理复杂的 L7 流量管理，从而将 L4 和 L7 的治理在内核空间（eBPF）和用户空间（Waypoint）之间分离。相比 Istio 的 Ambient Mesh，延迟减少了 30%。相较于内核原生模式，双引擎模式无需对内核进行增强，具有更广泛的适用性。

当前阿里云服务网格ASM支持使用 Kmesh 双引擎模式作为服务网格的数据面之一，从而实现更加高效的服务治理。具体来说, 可以使用 ASM 作为控制面，并在 ACK Kubernetes集群中部署 Kmesh 作为数据面。

在阿里云 ACK 集群中部署 Kmesh 并连接 ASM 控制面

使用前提

参考阿里云服务网格ASM的官方文档, 创建一个ASM集群实例与ACK Kubernetes 集群, 并将ACK Kubernetes集群添加到ASM集群实例中进行管理。具体操作步骤, 可以参考文档： https://help.aliyun.com/zh/asm/getting-started/add-a-cluster-to-an-asm-instance-1?spm=a2c4g.11186623.help-menu-147365.d_1_2.436356ccEXYKBU

安装 Kmesh

执行以下命令，将 Kmesh 项目下载到本地。

git clone https://github.com/kmesh-net/kmesh.git && cd kmesh

查看 ASM 控制面 Service 信息

下载完成后，您首先需要执行以下命令，查看当前 ASM 控制面在集群中的 Service 名称，以配置 Kmesh 与 ASM 控制面的连接。

kubectl get svc -n istio-system | grep istiod
# istiod-1-22-6   ClusterIP   None   <none>   15012/TCP   2d

使用 kubectl 安装 Kmesh

您可以使用 kubectl 在ACK Kubernetes集群中安装 Kmesh，但在安装前，请为 Kmesh DaemonSet 添加ClusterId和xdsAddress环境变量，用于 Kmesh 和 ASM 控制面认证与连接。ClusterId为您部署 Kmesh 的 ACK Kubernetes集群 Id，xdsAddress为 ASM 控制面的 Service 信息。

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kmesh
  labels:
    app: kmesh
  namespace: kmesh-system
spec:
    spec:
      containers:
        - env:
            # 修改 xdsAddress 为 ASM 控制面 Service
            - name: XDS_ADDRESS
              value: "istiod-1-22-6.istio-system.svc:15012"
            # 添加阿里云 ACK 集群 id
            - name: CLUSTER_ID
              value: "cluster-id"
    ...

修改以下命令中的 Kmesh Daemonset 环境变量后，执行以部署 Kmesh。

kubectl apply -f -<<EOF
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kmesh
  labels:
    app: kmesh
  namespace: kmesh-system
spec:
  selector:
    matchLabels:
      app: kmesh
  template:
    metadata:
      labels:
        app: kmesh
      annotations:
        prometheus.io/path: "status/metric"
        prometheus.io/port: "15020"
        prometheus.io/scrape: "true"
    spec:
      tolerations:
        - effect: NoSchedule
          operator: Exists
        - key: CriticalAddonsOnly
          operator: Exists
        - effect: NoExecute
          operator: Exists
      volumes:
        # use cgroup requires
        - name: mnt
          hostPath:
            path: /mnt
        # for eBPF program into the host machine
        - name: sys-fs-bpf
          hostPath:
            path: /sys/fs/bpf
        # required for compiling and building ko
        - name: lib-modules
          hostPath:
            path: /lib/modules
        # k8s default cni conflist path
        - name: cni
          hostPath:
            path: /etc/cni/net.d
        # k8s default cni path
        - name: kmesh-cni-install-path
          hostPath:
            path: /opt/cni/bin
        - name: host-procfs
          hostPath:
            path: /proc
            type: Directory
        - name: istiod-ca-cert
          configMap:
            defaultMode: 420
            name: istio-ca-root-cert
        - name: istio-token
          projected:
            defaultMode: 420
            sources:
              - serviceAccountToken:
                  audience: istio-ca
                  expirationSeconds: 43200
                  path: istio-token
        # 修改 xdsAddress 为 ASM 控制面 Service
        - name: XDS_ADDRESS
          value: "istiod-1-22-6.istio-system.svc:15012"
        # 添加阿里云 ACK 集群 id
        - name: CLUSTER_ID
          value: "cluster-id"
      containers:
        - name: kmesh
          image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/kmesh:latest
          imagePullPolicy: IfNotPresent
          command: ["/bin/sh", "-c"]
          args:
            [
              "./start_kmesh.sh --mode=dual-engine --enable-bypass=false --enable-bpf-log=true",
            ]
          securityContext:
            privileged: true
            capabilities:
              add: ["all"]
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: INSTANCE_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: XDS_ADDRESS
              value: "istiod.istio-system.svc:15012"
            - name: SERVICE_ACCOUNT
              valueFrom:
                fieldRef:
                  fieldPath: spec.serviceAccountName
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          volumeMounts:
            - name: mnt
              mountPath: /mnt
              readOnly: false
            - name: sys-fs-bpf
              mountPath: /sys/fs/bpf
              readOnly: false
            - name: lib-modules
              mountPath: /lib/modules
              readOnly: false
            # k8s default cni conflist path
            - name: cni
              mountPath: /etc/cni/net.d
              readOnly: false
            # k8s default cni path
            - name: kmesh-cni-install-path
              mountPath: /opt/cni/bin
              readOnly: false
            - mountPath: /host/proc
              name: host-procfs
              readOnly: true
            - name: istiod-ca-cert
              mountPath: /var/run/secrets/istio
            - name: istio-token
              mountPath: /var/run/secrets/tokens
          resources:
            limits:
              # image online-compile needs 800Mi, or only 200Mi
              memory: "800Mi"
              cpu: "1"
      priorityClassName: system-node-critical
      serviceAccountName: kmesh
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kmesh
  labels:
    app: kmesh
rules:
- apiGroups: [""]
  resources: ["pods","services","namespaces"]
  verbs: ["get", "update", "patch", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["daemonsets"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kmesh
  labels:
    app: kmesh
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kmesh
subjects:
- kind: ServiceAccount
  name: kmesh
  namespace: kmesh-system
---
apiVersion: v1
kind: Namespace
metadata:
  name: kmesh-system
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: add-listener-filter
  namespace: istio-system
  labels:
    asm-system: 'true'
    provider: asm
spec:
  workloadSelector:
    labels:
      gateway.istio.io/managed: istio.io-mesh-controller
  configPatches:
  - applyTo: LISTENER
    match:
      proxy:
        proxyVersion: .*
    patch:
      operation: ADD
      value:
        name: kmesh-listener
        address:
          socket_address:
            protocol: TCP
            address: 0.0.0.0
            port_value: 15019
        additional_addresses:
        - address:
            socket_address:
              protocol: TCP
              address: "::"
              port_value: 15019
        default_filter_chain:
          filters:
          - name: envoy.filters.network.tcp_proxy
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
              stat_prefix: kmesh
              cluster: main_internal
        filter_chains:
        - filter_chain_match:
            application_protocols:
            - "http/1.1"
            - "h2c"
          filters:
          - name: envoy.filters.network.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              stat_prefix: kmesh
              route_config:
                name: default
                virtual_hosts:
                - name: default
                  domains:
                  - '*'
                  routes:
                  - match:
                      prefix: "/"
                    route:
                      cluster: main_internal
              http_filters:
              - name: waypoint_downstream_peer_metadata
                typed_config:
                  "@type": type.googleapis.com/udpa.type.v1.TypedStruct
                  type_url: type.googleapis.com/io.istio.http.peer_metadata.Config
                  value:
                    downstream_discovery:
                    - workload_discovery: {}
                    shared_with_upstream: true
              - name: envoy.filters.http.router
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
        listener_filters:
        - name: "envoy.listener.kmesh_tlv"
          typed_config:
            "@type": "type.googleapis.com/udpa.type.v1.TypedStruct"
            "type_url": "type.googleapis.com/envoy.listener.kmesh_tlv.config.KmeshTlv"
        - name: "envoy.filters.listener.http_inspector"
          typed_config:
            "@type": "type.googleapis.com/envoy.extensions.filters.listener.http_inspector.v3.HttpInspector"
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: skip-tunneling
  namespace: istio-system
  labels:
    asm-system: 'true'
    provider: asm
spec:
  workloadSelector:
    labels:
      gateway.istio.io/managed: istio.io-mesh-controller
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      proxy:
        proxyVersion: .*
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.tcp_proxy
    patch:
      operation: REPLACE
      value:
        name: envoy.filters.network.tcp_proxy
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
          stat_prefix: kmesh_original_dst_cluster
          cluster: kmesh_original_dst_cluster
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: add-original-dst-cluster
  namespace: istio-system
  labels:
    asm-system: 'true'
    provider: asm
spec:
  workloadSelector:
    labels:
      gateway.istio.io/managed: istio.io-mesh-controller
  configPatches:
  - applyTo: CLUSTER
    match:
      proxy:
        proxyVersion: .*
      context: SIDECAR_INBOUND
    patch:
      operation: ADD
      value:
        name: "kmesh_original_dst_cluster"
        type: ORIGINAL_DST
        connect_timeout: 2s
        lb_policy: CLUSTER_PROVIDED
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kmesh
  namespace: kmesh-system
EOF

检查 Kmesh 服务状态

安装完毕后，执行以下命令查看 Kmesh 服务启动状态。

kubectl get pods -A | grep kmesh
# kmesh-system   kmesh-l5z2j   1/1   Running   0    117m

执行以下命令查看 Kmesh 服务运行状态。

kubectl logs -f -n kmesh-system kmesh-l5z2j
# time="2024-02-19T10:16:52Z" level=info msg="service node sidecar~192.168.11.53~kmesh-system.kmesh-system~kmesh-system.svc.cluster.local connect to discovery address istiod.istio-system.svc:15012" subsys=controller/envoy
# time="2024-02-19T10:16:52Z" level=info msg="options InitDaemonConfig successful" subsys=manager
# time="2024-02-19T10:16:53Z" level=info msg="bpf Start successful" subsys=manager
# time="2024-02-19T10:16:53Z" level=info msg="controller Start successful" subsys=manager
# time="2024-02-19T10:16:53Z" level=info msg="command StartServer successful" subsys=manager
# time="2024-02-19T10:16:53Z" level=info msg="start write CNI config\n" subsys="cni installer"
# time="2024-02-19T10:16:53Z" level=info msg="kmesh cni use chained\n" subsys="cni installer"
# time="2024-02-19T10:16:54Z" level=info msg="Copied /usr/bin/kmesh-cni to /opt/cni/bin." subsys="cni installer"
# time="2024-02-19T10:16:54Z" level=info msg="kubeconfig either does not exist or is out of date, writing a new one" subsys="cni installer"
# time="2024-02-19T10:16:54Z" level=info msg="wrote kubeconfig file /etc/cni/net.d/kmesh-cni-kubeconfig" subsys="cni installer"
# time="2024-02-19T10:16:54Z" level=info msg="command Start cni successful" subsys=manager

您可以通过执行以下命令来对指定命名空间启用 Kmesh。

kubectl label namespace default istio.io/dataplane-mode=Kmesh

流量调度示例

部署示例应用和流量调度规则

在为 default 命名空间启用 Kmesh 后，执行以下命令安装示例应用。

kubectl apply -f samples/fortio/fortio-route.yaml
kubectl apply -f samples/fortio/netutils.yaml

执行以下命令查看实例应用运行状态。

kubectl get pod 
# NAME                         READY   STATUS    RESTARTS   AGE
# fortio-v1-596b55cb8b-sfktr   1/1     Running   0          57m
# fortio-v2-76997f99f4-qjsmd   1/1     Running   0          57m
# netutils-575f5c569-lr98z     1/1     Running   0          67m
kubectl describe pod netutils-575f5c569-lr98z | grep Annotations
# Annotations:      kmesh.net/redirection: enabled

当您看到应用 Pod 具有kmesh.net/redirection: enabled时，代表 Kmesh 转发已经对该 Pod 启用。

执行以下命令查看当前定义的流量调度规则，可以看出，此时定义了 90% 的流量流向 v1 版本的 fortio，10% 的流量流向 v2 版本的 fortio。

kubectl get virtualservices -o yaml
# apiVersion: v1
# items:
# - apiVersion: networking.istio.io/v1beta1
#   kind: VirtualService
#   metadata:
#     annotations:
#       kubectl.kubernetes.io/last-applied-configuration: |
#         {"apiVersion":"networking.istio.io/v1alpha3","kind":"VirtualService","metadata":{"annotations":{},"name":"fortio","namespace":"default"},"spec":{"hosts":["fortio"],"http":[{"route":[{"destination":{"host":"fortio","subset":"v1"},"weight":90},{"destination":{"host":"fortio","subset":"v2"},"weight":10}]}]}}
#     creationTimestamp: "2024-07-09T09:00:36Z"
#     generation: 1
#     name: fortio
#     namespace: default
#     resourceVersion: "11166"
#     uid: 0a07f283-ac26-4d86-b3bd-ce6aa07dc628
#   spec:
#     hosts:
#     - fortio
#     http:
#     - route:
#       - destination:
#           host: fortio
#           subset: v1
#         weight: 90
#       - destination:
#           host: fortio
#           subset: v2
#         weight: 10
# kind: List
# metadata:
#   resourceVersion: ""

发起测试流量

您可以通过执行以下命令发起测试流量，您应该可以看到，此时只有大约 10% 的流量流向 v2 版本的 fortio。

for i in {1..20}; do kubectl exec -it $(kubectl get pod | grep netutils | awk '{print $1}') -- curl -v $(kubectl get svc -owide | grep fortio | awk '{print $3}'):80 | grep "Server:"; done
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 2
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 1
# < Server: 2
# < Server: 1
# < Server: 1

使用Kmesh作为阿里云服务网格ASM Sidecarless模式数据面

概述