环境介绍
在本例中一共部署了3个容器：
- front-envoy 容器作为 API 网关，所有的入向请求都通过 front-envoy 容器进行路由。front-envoy 容器暴露了 8080，8443 端口分别来接受 HTTP，HTTPS 请求，并根据路径分别将它们路由到对应的服务上，以及通过 8001 端口来接受 Envoy 自带的 admin 服务。
- 分别部署 service1 和 service2 两个Flask 应用程序，在该容器中启动 Envoy 服务， 通过 loopback 地址将请求路由到 Flask 应用程序。
service1 & service2 服务代码
service1 和 service2 都使用相同的代码启动 Flask 服务，通过 SERVICE_NAME 这个环境变量在访问的时候可以区分服务是 service1 还是 service2 。
# service.py from flask import Flask from flask import request import os import requests import socket import sys app = Flask(__name__) @app.route('/service/<service_number>') def hello(service_number): return ('Hello from behind Envoy (service {})! hostname: {} resolved' 'hostname: {}\n'.format(os.environ['SERVICE_NAME'], socket.gethostname(), socket.gethostbyname(socket.gethostname()))) if __name__ == "__main__": app.run(host='127.0.0.1', port=8080, debug=True)
service1 & service2 的 envoy 服务配置
在 service1 和 service2 容器中还启动了 envoy 服务，外部客户端（本例中是 front-envoy 容器）访问 service1 和 service2 时是去访问 envoy , 然后由 envoy 通过 loopback 地址 将请求路由到 Flask 应用程序。
# service-envoy.yaml static_resources: #定义静态资源 listeners: #监听器 - address: socket_address: address: 0.0.0.0 #envoy监听地址 port_value: 8000 #envoy监听端口号 filter_chains: - filters: - name: cr7_filters #自定义filters的名字 typed_config: #https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager codec_type: auto #默认配置，connection manager 使用的编解码器，自动适配HTTP/1.1和HTTP/2 stat_prefix: ingress_http #connection manager 发出统计信息时使用的前缀 route_config: #静态配置路由管理，可选参数有3个：rds（通过RDS API动态加载）,route_config（静态）,scoped_routes（根据请求参数匹配路） name: cr7_route #自定义路由配置名称 virtual_hosts: #定义一组虚拟主机 - name: cr7_service #自定义虚拟主机名称 domains: #匹配所有域名 - "*" routes: - match: prefix: "/service" #匹配的URI路径 route: cluster: cr7_cluster #上游集群名称 http_filters: - name: envoy.filters.http.router #实现HTTP转发 # - name: cr7-router # typed_config: # "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router clusters: - name: cr7_cluster #自定义上游集群名称 connect_timeout: 0.25s #新连接到上游集群主机的超时时间 #https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/cluster/v3/cluster.proto#envoy-v3-api-enum-config-cluster-v3-cluster-discoverytype type: strict_dns #服务发现机制：通过域名解析 lb_policy: round_robin #负载均衡策略 load_assignment: #设置负载均衡的成员， 取代了V2 API中的hosts字段 cluster_name: cr7_upstream endpoints: - lb_endpoints: - endpoint: address: socket_address: address: 127.0.0.1 #通过loopback转发给本地flask服务 port_value: 8080 #https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/bootstrap/v3/bootstrap.proto.html?highlight=access_log_path#envoy-v3-api-msg-config-bootstrap-v3-admin admin: #envoy管理接口 access_log_path: "/tmp/access.log" address: socket_address: address: 0.0.0.0 port_value: 8001
service1 & service2 Dockerfile 文件
service1 和 service2 容器启动脚本：首先启动Flask 服务，然后启动 envoy 服务。
# start_service.sh #!/bin/sh python3 /code/service.py & envoy -c /etc/service-envoy.yaml --service-cluster "service${SERVICE_NAME}"
#Dockerfile-service FROM envoyproxy/envoy-alpine-dev:latest RUN apk update && apk add py3-pip bash curl RUN pip3 install -q Flask==0.11.1 requests==2.18.4 RUN mkdir /code ADD ./service.py /code ADD ./start_service.sh /usr/local/bin/start_service.sh RUN chmod u+x /usr/local/bin/start_service.sh ENTRYPOINT ["/bin/sh", "/usr/local/bin/start_service.sh"]
front-envoy envoy 配置文件
front-envoy 容器中只有 envoy 服务， 负责接收所有入访的流量，并且根据URI请求路径分发给service1 或者 service2 。另外还配置了 HTTPS 加密，生成了证书和私钥。
# front-envoy.yaml static_resources: listeners: - address: socket_address: address: 0.0.0.0 port_value: 8080 filter_chains: - filters: - name: envoy.filters.network.http_connection_manager typed_config: "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager codec_type: auto stat_prefix: ingress_http route_config: name: local_route virtual_hosts: - name: backend domains: - "*" routes: - match: prefix: "/service/1" route: cluster: service1 - match: prefix: "/service/2" route: cluster: service2 http_filters: - name: envoy.filters.http.router typed_config: {} - address: socket_address: address: 0.0.0.0 port_value: 8443 filter_chains: - filters: - name: envoy.filters.network.http_connection_manager typed_config: "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager codec_type: auto stat_prefix: ingress_http route_config: name: local_route virtual_hosts: - name: backend domains: - "*" routes: - match: prefix: "/service/1" route: cluster: service1 - match: prefix: "/service/2" route: cluster: service2 http_filters: - name: envoy.filters.http.router typed_config: {} transport_socket: name: envoy.transport_sockets.tls typed_config: "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext common_tls_context: tls_certificates: # The following self-signed certificate pair is generated using: # $ openssl req -x509 -newkey rsa:2048 -keyout a/front-proxy-key.pem -out a/front-proxy-crt.pem -days 3650 -nodes -subj '/CN=front-envoy' # # Instead of feeding it as an inline_string, certificate pair can also be fed to Envoy # via filename. Reference: https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/base.proto#config-core-v3-datasource. # # Or in a dynamic configuration scenario, certificate pair can be fetched remotely via # Secret Discovery Service (SDS). Reference: https://www.envoyproxy.io/docs/envoy/latest/configuration/security/secret. certificate_chain: inline_string: | -----BEGIN CERTIFICATE----- MIICqDCCAZACCQCquzpHNpqBcDANBgkqhkiG9w0BAQsFADAWMRQwEgYDVQQDDAtm cm9udC1lbnZveTAeFw0yMDA3MDgwMTMxNDZaFw0zMDA3MDYwMTMxNDZaMBYxFDAS BgNVBAMMC2Zyb250LWVudm95MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC AQEAthnYkqVQBX+Wg7aQWyCCb87hBce1hAFhbRM8Y9dQTqxoMXZiA2n8G089hUou oQpEdJgitXVS6YMFPFUUWfwcqxYAynLK4X5im26Yfa1eO8La8sZUS+4Bjao1gF5/ VJxSEo2yZ7fFBo8M4E44ZehIIocipCRS+YZehFs6dmHoq/MGvh2eAHIa+O9xssPt ofFcQMR8rwBHVbKy484O10tNCouX4yUkyQXqCRy6HRu7kSjOjNKSGtjfG+h5M8bh 10W7ZrsJ1hWhzBulSaMZaUY3vh5ngpws1JATQVSK1Jm/dmMRciwlTK7KfzgxHlSX 58ENpS7yPTISkEICcLbXkkKGEQIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQCmj6Hg vwOxWz0xu+6fSfRL6PGJUGq6wghCfUvjfwZ7zppDUqU47fk+yqPIOzuGZMdAqi7N v1DXkeO4A3hnMD22Rlqt25vfogAaZVToBeQxCPd/ALBLFrvLUFYuSlS3zXSBpQqQ Ny2IKFYsMllz5RSROONHBjaJOn5OwqenJ91MPmTAG7ujXKN6INSBM0PjX9Jy4Xb9 zT+I85jRDQHnTFce1WICBDCYidTIvJtdSSokGSuy4/xyxAAc/BpZAfOjBQ4G1QRe 9XwOi790LyNUYFJVyeOvNJwveloWuPLHb9idmY5YABwikUY6QNcXwyHTbRCkPB2I m+/R4XnmL4cKQ+5Z -----END CERTIFICATE----- private_key: inline_string: | -----BEGIN PRIVATE KEY----- MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC2GdiSpVAFf5aD tpBbIIJvzuEFx7WEAWFtEzxj11BOrGgxdmIDafwbTz2FSi6hCkR0mCK1dVLpgwU8 VRRZ/ByrFgDKcsrhfmKbbph9rV47wtryxlRL7gGNqjWAXn9UnFISjbJnt8UGjwzg Tjhl6EgihyKkJFL5hl6EWzp2Yeir8wa+HZ4Achr473Gyw+2h8VxAxHyvAEdVsrLj zg7XS00Ki5fjJSTJBeoJHLodG7uRKM6M0pIa2N8b6HkzxuHXRbtmuwnWFaHMG6VJ oxlpRje+HmeCnCzUkBNBVIrUmb92YxFyLCVMrsp/ODEeVJfnwQ2lLvI9MhKQQgJw tteSQoYRAgMBAAECggEAeDGdEkYNCGQLe8pvg8Z0ccoSGpeTxpqGrNEKhjfi6NrB NwyVav10iq4FxEmPd3nobzDPkAftfvWc6hKaCT7vyTkPspCMOsQJ39/ixOk+jqFx lNa1YxyoZ9IV2DIHR1iaj2Z5gB367PZUoGTgstrbafbaNY9IOSyojCIO935ubbcx DWwL24XAf51ez6sXnI8V5tXmrFlNXhbhJdH8iIxNyM45HrnlUlOk0lCK4gmLJjy9 10IS2H2Wh3M5zsTpihH1JvM56oAH1ahrhMXs/rVFXXkg50yD1KV+HQiEbglYKUxO eMYtfaY9i2CuLwhDnWp3oxP3HfgQQhD09OEN3e0IlQKBgQDZ/3poG9TiMZSjfKqL xnCABMXGVQsfFWNC8THoW6RRx5Rqi8q08yJrmhCu32YKvccsOljDQJQQJdQO1g09 e/adJmCnTrqxNtjPkX9txV23Lp6Ak7emjiQ5ICu7iWxrcO3zf7hmKtj7z+av8sjO mDI7NkX5vnlE74nztBEjp3eC0wKBgQDV2GeJV028RW3b/QyP3Gwmax2+cKLR9PKR nJnmO5bxAT0nQ3xuJEAqMIss/Rfb/macWc2N/6CWJCRT6a2vgy6xBW+bqG6RdQMB xEZXFZl+sSKhXPkc5Wjb4lQ14YWyRPrTjMlwez3k4UolIJhJmwl+D7OkMRrOUERO EtUvc7odCwKBgBi+nhdZKWXveM7B5N3uzXBKmmRz3MpPdC/yDtcwJ8u8msUpTv4R JxQNrd0bsIqBli0YBmFLYEMg+BwjAee7vXeDFq+HCTv6XMva2RsNryCO4yD3I359 XfE6DJzB8ZOUgv4Dvluie3TB2Y6ZQV/p+LGt7G13yG4hvofyJYvlg3RPAoGAcjDg +OH5zLN2eqah8qBN0CYa9/rFt0AJ19+7/smLTJ7QvQq4g0gwS1couplcCEnNGWiK 72y1n/ckvvplmPeAE19HveMvR9UoCeV5ej86fACy8V/oVpnaaLBvL2aCMjPLjPP9 DWeCIZp8MV86cvOrGfngf6kJG2qZTueXl4NAuwkCgYEArKkhlZVXjwBoVvtHYmN2 o+F6cGMlRJTLhNc391WApsgDZfTZSdeJsBsvvzS/Nc0burrufJg0wYioTlpReSy4 ohhtprnQQAddfjHP7rh2LGt+irFzhdXXQ1ybGaGM9D764KUNCXLuwdly0vzXU4HU q5sGxGrC1RECGB5Zwx2S2ZY= -----END PRIVATE KEY----- clusters: - name: service1 connect_timeout: 0.25s type: strict_dns #服务发现机制：通过域名解析 lb_policy: round_robin http2_protocol_options: {} load_assignment: cluster_name: service1 endpoints: - lb_endpoints: - endpoint: address: socket_address: address: service1 # 通过 DNS 解析 service1 可以得到 service1 容器的 IP 地址 port_value: 8000 - name: service2 connect_timeout: 0.25s type: strict_dns lb_policy: round_robin http2_protocol_options: {} load_assignment: cluster_name: service2 endpoints: - lb_endpoints: - endpoint: address: socket_address: address: service2 port_value: 8000 admin: access_log_path: "/dev/null" address: socket_address: address: 0.0.0.0 port_value: 8001 layered_runtime: layers: - name: static_layer_0 static_layer: envoy: resource_limits: listener: example_listener_name: connection_limit: 10000
front-envoy Dockerfile 文件
# Dockerfile-frontenvoy FROM envoyproxy/envoy-dev:latest RUN apt-get update && apt-get -q install -y \ curl COPY ./front-envoy.yaml /etc/front-envoy.yaml RUN chmod go+r /etc/front-envoy.yaml CMD ["/usr/local/bin/envoy", "-c", "/etc/front-envoy.yaml", "--service-cluster", "front-proxy"]
docker-compose.yml 文件
# docker-compose.yaml version: "3.7" services: front-envoy: build: context: . dockerfile: Dockerfile-frontenvoy networks: - envoymesh expose: - "8080" - "8443" - "8001" ports: - "8080:8080" - "8443:8443" - "8001:8001" service1: build: context: . dockerfile: Dockerfile-service volumes: - ./service-envoy.yaml:/etc/service-envoy.yaml networks: envoymesh: aliases: - service1 environment: - SERVICE_NAME=1 #通过环境变量来区分服务 expose: - "8000" service2: build: context: . dockerfile: Dockerfile-service volumes: - ./service-envoy.yaml:/etc/service-envoy.yaml networks: envoymesh: aliases: - service2 environment: - SERVICE_NAME=2 expose: - "8000" networks: envoymesh: {}
运行验证
步骤一：安装 Docker
确保你已安装较新版本的 docker 和 docker-compose 。
步骤二：克隆仓库
git clone https://github.com/cr7258/envoy-lab.git
步骤三：启动所有容器
- up：启动容器
- -d：在后台运行
- --build：重新构建镜像
cd envoy-lab/front-proxy docker-compose up -d --build
查看容器：
[root@envoy ~]# docker-compose ps Name Command State Ports ----------------------------------------------------------------------------------------------------------------------------------------------- root_front-envoy_1 /docker-entrypoint.sh /usr ... Up 10000/tcp, 0.0.0.0:8001->8001/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:8443->8443/tcp root_service1_1 /bin/sh /usr/local/bin/sta ... Up 10000/tcp, 8000/tcp root_service2_1 /bin/sh /usr/local/bin/sta ... Up 10000/tcp, 8000/tcp
步骤四：测试 Envoy 的路由能力
你现在可以通过 front-envoy 向两个服务发送请求。
向 service1 发请求：
[root@envoy ~]# curl -v localhost:8080/service/1 * About to connect() to localhost port 8080 (#0) * Trying ::1... * Connected to localhost (::1) port 8080 (#0) > GET /service/1 HTTP/1.1 > User-Agent: curl/7.29.0 > Host: localhost:8080 > Accept: */* > < HTTP/1.1 200 OK < content-type: text/html; charset=utf-8 < content-length: 89 < server: envoy < date: Mon, 15 Mar 2021 15:29:28 GMT < x-envoy-upstream-service-time: 2 < Hello from behind Envoy (service 1)! hostname: e60ba6d0671c resolvedhostname: 172.18.0.2
向 service2 发请求：
[root@envoy ~]# curl -v localhost:8080/service/2 * About to connect() to localhost port 8080 (#0) * Trying ::1... * Connected to localhost (::1) port 8080 (#0) > GET /service/2 HTTP/1.1 > User-Agent: curl/7.29.0 > Host: localhost:8080 > Accept: */* > < HTTP/1.1 200 OK < content-type: text/html; charset=utf-8 < content-length: 89 < server: envoy < date: Mon, 15 Mar 2021 15:29:32 GMT < x-envoy-upstream-service-time: 2 < Hello from behind Envoy (service 2)! hostname: 9727cf7b9303 resolvedhostname: 172.18.0.4 * Connection #0 to host localhost left intact
能看到，每个请求在发送给前端 Envoy 后被正确路由到相应的应用程序。
我们也可以通过 HTTPS 请求前端 Envoy 后的服务。例如，向 service1：
[root@envoy ~]# curl https://localhost:8443/service/1 -k -v * About to connect() to localhost port 8443 (#0) * Trying ::1... * Connected to localhost (::1) port 8443 (#0) * Initializing NSS with certpath: sql:/etc/pki/nssdb * skipping SSL peer certificate verification * SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 * Server certificate: * subject: CN=front-envoy * start date: 7月 08 01:31:46 2020 GMT * expire date: 7月 06 01:31:46 2030 GMT * common name: front-envoy * issuer: CN=front-envoy > GET /service/1 HTTP/1.1 > User-Agent: curl/7.29.0 > Host: localhost:8443 > Accept: */* > < HTTP/1.1 200 OK < content-type: text/html; charset=utf-8 < content-length: 89 < server: envoy < date: Mon, 15 Mar 2021 15:30:10 GMT < x-envoy-upstream-service-time: 2 < Hello from behind Envoy (service 1)! hostname: e60ba6d0671c resolvedhostname: 172.18.0.2 * Connection #0 to host localhost left intact
步骤五：测试 Envoy 的负载均衡能力
现在增加 service1 的节点数量来演示 Envoy 的负载均衡能力：
[root@envoy ~]# docker-compose scale service1=3 WARNING: The scale command is deprecated. Use the up command with the --scale flag instead. Starting root_service1_1 ... done Creating root_service1_2 ... done Creating root_service1_3 ... done
现在，如果我们多次向 service1 发送请求，前端 Envoy 将通过 round-robin 轮询三台 service1 机器来实现负载均衡：
docker-compose exec -T front-envoy bash -c "\ curl -s http://localhost:8080/service/1 \ && curl -s http://localhost:8080/service/1 \ && curl -s http://localhost:8080/service/1" \ | grep Hello | grep "service 1" # 返回结果 Hello from behind Envoy (service 1)! hostname: 707d6f830af2 resolvedhostname: 172.18.0.5 Hello from behind Envoy (service 1)! hostname: 64eebf06b9db resolvedhostname: 172.18.0.6 Hello from behind Envoy (service 1)! hostname: e60ba6d0671c resolvedhostname: 172.18.0.2
步骤六：进入容器并 curl admin
当 Envoy 启动时，也会同时启动一个 admin 服务并绑定指定的端口。在示例配置中 admin 绑定到了 8001 端口。
我们可以通过 curl 它获得有用的信息：
[root@envoy ~]# curl localhost:8001/stats cluster.service1.external.upstream_rq_200: 7 ... cluster.service1.membership_change: 2 cluster.service1.membership_total: 3 ... cluster.service1.upstream_cx_http2_total: 3 ... cluster.service1.upstream_rq_total: 7 ... cluster.service2.external.upstream_rq_200: 2 ... cluster.service2.membership_change: 1 cluster.service2.membership_total: 1 ... cluster.service2.upstream_cx_http2_total: 1 ... cluster.service2.upstream_rq_total: 2 ...
能看到，我们可以获取上游集群的成员数量，它们完成的请求数量，有关 http 入口的信息以及大量其他有用的统计数据。
