Kubernetes的Sidecar模式

实践

apiVersion: v1
kind: Pod
metadata:
  name: test-sidecar
spec:
  volumes:
    - name: share-path
      emptyDir: {}
  containers:
  - name: sidecar
    image: busybox
    command: ["/bin/sh", "-c", "sleep 3600"]
    volumeMounts:
    - name: share-path
      mountPath: /var/share/sidecar
    lifecycle:
      postStart:
        exec:
          command: ["/bin/sh", "-c", "sleep 20"]
  - name: main
    image: busybox
    command: ["/bin/sh", "-c", "sleep 6666"]
    volumeMounts:
    - name: share-path
      mountPath: /var/share/main

创建日志

在多个容器的Pod中，通常业务容器需要依赖sidecar。启动时sidecar需要先启动，退出时sidecar需要在业务容器退出后再退出。k8s目前对于sidecar的生命周期比较有争议，见issue、sidecarcontainers。

Kubernetes Pod 内有两种容器: 初始化容器(init container)和应用容器(app container)。

kubectl describe pod/test-sidecar -n default

Events:
  Type    Reason     Age                From               Message
  ----    ------     ----               ----               -------
  Normal  Scheduled  11m                default-scheduler  Successfully assigned default/test-sidecar to rds005-worker-1
  Normal  Pulled     11m                kubelet            Successfully pulled image "busybox" in 405.005566ms
  Normal  Created    11m                kubelet            Created container main
  Normal  Pulling    11m                kubelet            Pulling image "busybox"
  Normal  Pulled     11m                kubelet            Successfully pulled image "busybox" in 140.242714ms
  Normal  Started    10m                kubelet            Started container main
  Normal  Created    74s (x2 over 11m)  kubelet            Created container sidecar
  Normal  Pulling    74s (x2 over 11m)  kubelet            Pulling image "busybox"
  Normal  Pulled     74s                kubelet            Successfully pulled image "busybox" in 385.214008ms
  Normal  Started    73s (x2 over 11m)  kubelet            Started container sidecar

此方案可能存在的缺点：

如果sidecar启动失败或者hook失败，其他容器会立即启动

退出顺序

容器启动顺序比较好解决，退出顺序则是按照相反的顺序，业务容器先退出，之后sidecar再退出。

目前，在kubelet删除pod步骤如下;

• 遍历容器，每个容器起一个goroutine删除
• 删除时，先执行pre stop hook，得到gracePeriod=DeletionGracePeriodSeconds-period(stophook)
• 再调用cri删除接口m.runtimeService.StopContainer(containerID.ID, gracePeriod)

如果在sidecar的pre stop hook检测业务容器状态，那么可以延迟退出。

apiVersion: v1
kind: Pod
metadata:
  name: test-stop
spec:
  containers:
  - name: sidecar
    image: busybox
    command: 
    - "/bin/sh"
    - "-c"
    - |
      trap "touch /lifecycle/sidecar-terminated" 15
      until [ -f "/lifecycle/sidecar-terminated" ];do
        date
        sleep 1
      done
      sleep 5
      cat /lifecycle/main-terminated
      t=$(date)
      echo "sidecar exit at $t"
    lifecycle:
      preStop:
        exec:
          command:
          - "/bin/sh"
          - "-c"
          - |
            until [ -f "/lifecycle/main-terminated" ];do
              sleep 1
            done
            t=$(date)
            echo "main exit at $t" > /lifecycle/main-terminated
    volumeMounts:
    - name: lifecycle
      mountPath: /lifecycle
  - name: main
    image: busybox
    command: 
    - "/bin/sh"
    - "-c"
    - |
      trap "touch /lifecycle/main-terminated" 15
      until [ -f "/lifecycle/main-terminated" ];do
        date
        sleep 1
      done
    volumeMounts:
    - name: lifecycle
      mountPath: /lifecycle
  volumes:
  - name: lifecycle
    emptyDir: {}

通过测试，使用postStopHook可以达到sidecar延迟退出的目的，但这种方式也有一些缺点

• 配置复杂，多个sidecar都需要配置postStop监听业务容器状态
• 业务容器需要有可观察性（提供特定形式的健康检测）
• poststop执行异常，会等到最大优雅退出时间（默认30s）后才终止

总结

相关命令

# 进入指定的pod容器中
kubectl exec -it test-sidecar -c sidecar -- sh