一.系统环境

本文主要基于Kubernetes1.22.2和Linux操作系统Ubuntu 18.04。

服务器版本	docker软件版本	Kubernetes(k8s)集群版本	CPU架构
Ubuntu 18.04.5 LTS	Docker version 20.10.14	v1.22.2	x86_64

Kubernetes集群架构：k8scludes1作为master节点，k8scludes2，k8scludes3作为worker节点。

服务器	操作系统版本	CPU架构	进程	功能描述
k8scludes1/192.168.110.128	Ubuntu 18.04.5 LTS	x86_64	docker，kube-apiserver，etcd，kube-scheduler，kube-controller-manager，kubelet，kube-proxy，coredns，calico	k8s master节点
k8scludes2/192.168.110.129	Ubuntu 18.04.5 LTS	x86_64	docker，kubelet，kube-proxy，calico	k8s worker节点
k8scludes3/192.168.110.130	Ubuntu 18.04.5 LTS	x86_64	docker，kubelet，kube-proxy，calico	k8s worker节点

二.前言

在当今的云计算时代，容器编排技术已成为企业部署和管理应用程序的关键组成部分。Kubernetes 作为最流行的容器编排平台，提供了强大的功能来部署、管理和扩展容器化应用程序。然而，随着应用程序的日益复杂和部署环境的扩大，确保 Kubernetes 集群的安全性变得越来越重要。在这篇博客中，我们将深入了解 Kubernetes 审计的概念，并探讨如何在 Kubernetes 环境中实施审计策略。

使用Kubernetes 审计（Auditing）的前提是已经有一套可以正常运行的Kubernetes集群，关于Kubernetes(k8s)集群的安装部署，可以查看博客《Ubuntu 安装部署Kubernetes(k8s)集群》https://www.cnblogs.com/renshengdezheli/p/17632858.html。

三.Kubernetes 审计简介

Kubernetes 审计（Auditing）功能提供了与安全相关的、按时间顺序排列的记录集，记录每个用户、使用 Kubernetes API 的应用以及控制面自身引发的活动。

审计功能使得集群管理员能够回答以下问题：

发生了什么？
什么时候发生的？
谁触发的？
活动发生在哪个（些）对象上？
在哪观察到的？
它从哪触发的？
活动的后续处理行为是什么？

审计记录最初产生于 kube-apiserver 内部。每个请求在不同执行阶段都会生成审计事件；这些审计事件会根据特定策略被预处理并写入后端。策略确定要记录的内容和用来存储记录的后端，当前的后端支持日志文件和 webhook。

每个请求都可被记录其相关的阶段（stage），阶段（stage）可以理解为什么时候记录。。已定义的阶段有：

RequestReceived - 此阶段对应审计处理器接收到请求后，并且在委托给其余处理器之前生成的事件。
ResponseStarted - 在响应消息的头部发送后，响应消息体发送前生成的事件。只有长时间运行的请求（例如 watch）才会生成这个阶段。
ResponseComplete - 当响应消息体完成并且没有更多数据需要传输的时候。
Panic- 当 panic 发生时生成。

审计日志记录功能会增加 API server 的内存消耗，因为需要为每个请求存储审计所需的某些上下文。内存消耗取决于审计日志记录的配置。

Kubernetes 审计是一种监控和记录 Kubernetes 集群中资源操作的方法，用于确保集群的安全性和符合性。通过审计，管理员可以跟踪对集群资源的访问和修改，以便在发生安全事件时进行调查和响应。Kubernetes 提供了审计日志记录的框架，允许管理员自定义审计策略，以确定哪些资源操作应该被记录。

四.审计策略简介

审计策略定义了关于应记录哪些事件以及应包含哪些数据的规则。审计策略对象结构定义在 audit.k8s.io API 组。处理事件时，将按顺序与规则列表进行比较。第一个匹配规则设置事件的审计级别（Audit Level），审计级别（Audit Level）可以理解为记录什么？ 已定义的审计级别有：

None - 符合这条规则的日志将不会记录。
Metadata - 记录请求的元数据（请求的用户、时间戳、资源、动词等等），但是不记录请求或者响应的消息体。
Request - 记录事件的元数据和请求的消息体，但是不记录响应的消息体。这不适用于非资源类型的请求。
RequestResponse - 记录事件的元数据，请求和响应的消息体。这不适用于非资源类型的请求。

你可以使用 --audit-policy-file 标志将包含策略的文件传递给 kube-apiserver。如果不设置该标志，则不记录事件。注意 rules 字段必须在审计策略文件中提供。没有（0）规则的策略将被视为非法配置。

审计策略定义了哪些资源操作应该被审计以及审计记录的格式。在 Kubernetes 中，审计策略通过 Admission Controllers 实现，可以通过 Webhook 的方式进行集成。审计策略可以根据资源的类型、操作的类型和用户身份等信息进行过滤，以满足不同场景下的审计需求。

五.启用审计

5.1 引入审计

本次使用etcd2机器作为kubernetes集群的客户端，kctom是kubeconfig文件，etcd2机器使用kctom连接kubernetes集群，现在用户tom没有权限，需要先授权。kubeconfig文件在博客《Kubernetes(k8s)访问控制：身份认证》里已经详细介绍过了，这里不再赘述。

[root@etcd2 ~]# ls kc* -lh
-rw------- 1 root root 5.5K 5月   5 17:46 kctom
[root@etcd2 ~]# kubectl get node --kubeconfig=kctom
Error from server (Forbidden): nodes is forbidden: User "tom" cannot list resource "nodes" in API group "" at the cluster scope

去kubernetes 集群给 tom用户授予 cluster-admin 权限，也可以授予部分权限。关于授权的详细信息，请查看博客《Kubernetes(k8s)访问控制：权限管理之RBAC授权/鉴权》。

root@k8scludes1:~# kubectl create clusterrolebinding crbindtom --clusterrole=cluster-admin --user=tom
clusterrolebinding.rbac.authorization.k8s.io/crbindtom created

授予cluster-admin权限之后，客户端就具有权限了。

[root@etcd2 ~]# kubectl get node --kubeconfig=kctom
NAME         STATUS   ROLES                  AGE   VERSION
k8scludes1   Ready    control-plane,master   66d   v1.22.2
k8scludes2   Ready    <none>                 65d   v1.22.2
k8scludes3   Ready    <none>                 65d   v1.22.2

编辑pod配置文件，表示使用nginx镜像创建pod。

root@k8scludes1:~# vim pod.yaml 
root@k8scludes1:~# cat pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: podtest
  name: podtest
spec:
  #当需要关闭容器时，立即杀死容器而不等待默认的30秒优雅停机时长。
  terminationGracePeriodSeconds: 0
  containers:
  - image: hub.c.163.com/library/nginx:latest
    #imagePullPolicy: IfNotPresent：表示如果本地已经存在该镜像，则不重新下载；否则从远程 Docker Hub 下载该镜像
    imagePullPolicy: IfNotPresent
    name: podtest
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

在admissioncontr命名空间创建一个pod。

root@k8scludes1:~# kubectl apply -f pod.yaml 
pod/podtest created

在客户端etcd2机器可以看到这个pod。

[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom -n admissioncontr
NAME      READY   STATUS    RESTARTS   AGE
podtest   1/1     Running   0          34s

在客户端删除pod。

[root@etcd2 ~]# kubectl delete pod podtest --kubeconfig=kctom -n admissioncontr
pod "podtest" deleted

kubectl get ev （查看事件），可以看到pod被删除掉了，但是看不到pod被谁删除掉了！

[root@etcd2 ~]# kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide
LAST SEEN   TYPE     REASON    OBJECT        SUBOBJECT                  SOURCE                MESSAGE                                                                            FIRST SEEN   COUNT   NAME
3m37s       Normal   Pulled    pod/podtest   spec.containers{podtest}   kubelet, k8scludes3   Container image "hub.c.163.com/library/nginx:latest" already present on machine   3m37s        1       podtest.16fab43e14342a3f
3m37s       Normal   Created   pod/podtest   spec.containers{podtest}   kubelet, k8scludes3   Created container podtest                                                          3m37s        1       podtest.16fab43e1db484f4
3m36s       Normal   Started   pod/podtest   spec.containers{podtest}   kubelet, k8scludes3   Started container podtest                                                          3m36s        1       podtest.16fab43e3315ada7
2m47s       Normal   Killing   pod/podtest   spec.containers{podtest}   kubelet, k8scludes3   Stopping container podtest                                                         2m47s        1       podtest.16fab449c468f9af

我们的k8s集群会被各种用户连接使用，我们想知道连接k8s集群的各个用户做了什么操作？应该怎么办？通过审计解决！

5.2 启用审计

创建audit目录存放yaml文件。

root@k8scludes1:~# mkdir audit
root@k8scludes1:~# cd audit/
root@k8scludes1:~/audit# pwd
/root/audit

创建命名空间audit。

root@k8scludes1:~/audit# kubectl create ns audit
namespace/audit created

切换到命名空间audit。

root@k8scludes1:~/audit# kubens audit
Context "kubernetes-admin@kubernetes" modified.
Active namespace is "audit".
root@k8scludes1:~/audit# kubectl get pod 
No resources found in audit namespace.

kubernetes默认并没有启用审计，需要启用审计功能。指定审计策略文件的路径：--audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml

root@k8scludes1:~# vim /etc/kubernetes/manifests/kube-apiserver.yaml 
root@k8scludes1:~# grep audit-policy-file /etc/kubernetes/manifests/kube-apiserver.yaml
    - --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml

创建审计策略文件，文件内容可以先为空。

root@k8scludes1:~# mkdir /etc/kubernetes/audit/
root@k8scludes1:~# cd /etc/kubernetes/audit/
root@k8scludes1:/etc/kubernetes/audit# touch audit-policy.yaml
root@k8scludes1:/etc/kubernetes/audit# ls /etc/kubernetes/audit/audit-policy.yaml
/etc/kubernetes/audit/audit-policy.yaml

指定审计日志存放的路径：--audit-log-path=/var/log/kubernetes/audit/audit.log 。审计日志格式为JSONlines 格式。

root@k8scludes1:/etc/kubernetes/audit# vim /etc/kubernetes/manifests/kube-apiserver.yaml
root@k8scludes1:/etc/kubernetes/audit# grep audit-log-path /etc/kubernetes/manifests/kube-apiserver.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log

创建审计日志文件。

root@k8scludes1:/etc/kubernetes/audit# mkdir -p /var/log/kubernetes/audit/
root@k8scludes1:/etc/kubernetes/audit# touch /var/log/kubernetes/audit/audit.log
root@k8scludes1:/etc/kubernetes/audit# ls /var/log/kubernetes/audit/audit.log
/var/log/kubernetes/audit/audit.log

审计日志参数如下：

--audit-log-path 指定用来写入审计事件的日志文件路径。不指定此标志会禁用日志后端。- 意味着标准化
--audit-log-maxage 定义保留旧审计日志文件的最大天数
--audit-log-maxbackup 定义要保留的审计日志文件的最大数量
--audit-log-maxsize 定义审计日志文件轮转之前的最大大小（兆字节）

配置审计日志的参数。

root@k8scludes1:/etc/kubernetes/audit# vim /etc/kubernetes/manifests/kube-apiserver.yaml
root@k8scludes1:/etc/kubernetes/audit# grep audit /etc/kubernetes/manifests/kube-apiserver.yaml
    - --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log
    - --audit-log-maxage=365
    - --audit-log-maxbackup=730
    - --audit-log-maxsize=250

如果你的集群控制面以 Pod 的形式运行 kube-apiserver，需要通过 hostPath 数据卷来访问策略文件和日志文件所在的目录，这样审计记录才会持久保存下来。

现在/etc/kubernetes/audit/audit-policy.yaml和/var/log/kubernetes/audit/audit.log在宿主机里是存在的，但是kube-apiserver是以pod的方式运行的，pod里并不存在这两个文件，需要通过hostPath 数据卷进行映射。

/etc/kubernetes/manifests/kube-apiserver.yaml所有的修改如下截图：

注意：/var/log/kubernetes/audit/的readOnly为false，否则不能写入日志！

自此，成功启用审计了，下面开始配置审计策略。

六.审计策略

6.1 记录审计阶段为：ResponseStarted，审计级别为Metadata，apiVersion为group: "" 的日志

先不用重启kubelet，因为审计策略文件还没写好。

现在编写审计策略文件，该审计策略表示什么都不记录。

root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml
root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "ResponseStarted"
rules:
  - level: None
    resources:
    - group: ""

重启kubelet使配置生效。

root@k8scludes1:~# systemctl restart kubelet
root@k8scludes1:~# kubectl get node
NAME         STATUS   ROLES                  AGE   VERSION
k8scludes1   Ready    control-plane,master   66d   v1.22.2
k8scludes2   Ready    <none>                 66d   v1.22.2
k8scludes3   Ready    <none>                 66d   v1.22.2

在客户端etcd2机器执行命令：kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide。

[root@etcd2 ~]# kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide
No resources found in admissioncontr namespace.

查看审计日志，没有任何日志记录。

root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log

审计级别修改为Metadata，表示记录请求的元数据（请求的用户、时间戳、资源、动词等等），但是不记录请求或者响应的消息体。

root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml
root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "ResponseStarted"
rules:
  - level: Metadata
    resources:
    - group: ""

重启kubelet使配置生效。

root@k8scludes1:~# systemctl daemon-reload ; systemctl restart kubelet

在客户端etcd2机器执行命令：kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide 。

[root@etcd2 ~]# kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide
No resources found in admissioncontr namespace.

查看审计日志，发现还是没有日志产生，看来重启kubelet还不行，现在重启机器。

root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log

现在直接重启k8scludes1机器，k8s worker节点不用重启。

root@k8scludes1:~# reboot

重启k8scludes1机器之后，现在/var/log/kubernetes/audit/audit.log里有很多数据，先删除数据，然后客户端执行命令，查看审计日志。

root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log | wc -l
3317

清空日志。

root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log

客户端执行kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide命令。

[root@etcd2 ~]# kubectl get ev --kubeconfig=kctom -n admissioncontr -o wide
No resources found in admissioncontr namespace.

现在不停的产生日志。

root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log | wc -l
32

这是最新的日志。

root@k8scludes1:~# tail -1 /var/log/kubernetes/audit/audit.log
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"72736d33-508f-4279-89a5-d95aea7b98c5","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/ingress-nginx/configmaps/ingress-controller-leader","verb":"update","user":{"username":"system:serviceaccount:ingress-nginx:ingress-nginx","uid":"70a36f2c-225f-450d-849e-2432db224f40","groups":["system:serviceaccounts","system:serviceaccounts:ingress-nginx","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["ingress-nginx-controller-684bbc4b45-jmpxk"],"authentication.kubernetes.io/pod-uid":["308fdf30-854b-49f7-b041-4e5d770c0667"]}},"sourceIPs":["192.168.110.130"],"userAgent":"nginx-ingress-controller/v1.0.0 (linux/amd64) ingress-nginx/041eb167c7bfccb1d1653f194924b0c5fd885e10","objectRef":{"resource":"configmaps","namespace":"ingress-nginx","name":"ingress-controller-leader","uid":"db625974-8e33-4bd6-a2e2-1816301af942","apiVersion":"v1","resourceVersion":"2739119"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-06-22T13:03:36.751232Z","stageTimestamp":"2022-06-22T13:03:36.759442Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by RoleBinding \"ingress-nginx/ingress-nginx\" of Role \"ingress-nginx\" to ServiceAccount \"ingress-nginx/ingress-nginx\"","mutation.webhook.admission.k8s.io/round_0_index_1":"{\"configuration\":\"gatekeeper-mutating-webhook-configuration\",\"webhook\":\"mutation.gatekeeper.sh\",\"mutated\":false}"}}

客户端不执行命令就产生了太多审计日志了，需要修改审计策略。

root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log | wc -l
1174

6.2 只记录audit命名空间里的日志

修改审计策略，现在配置只记录某个命名空间里的审计日志，namespaces: ["audit"]表示只记录audit命名空间里的日志。

root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml 
root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "ResponseStarted"
rules:
  - level: Metadata
    resources:
    - group: ""
    namespaces: ["audit"]

重启kubelet使其生效，但是审计策略没有生效，重启机器。

root@k8scludes1:~# systemctl restart kubelet

重启k8s master节点。

root@k8scludes1:~# reboot
root@k8scludes1:~# kubectl get node
NAME         STATUS   ROLES                  AGE   VERSION
k8scludes1   Ready    control-plane,master   66d   v1.22.2
k8scludes2   Ready    <none>                 66d   v1.22.2
k8scludes3   Ready    <none>                 66d   v1.22.2

重启k8s master节点之后，清空日志。

root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log

客户端执行kubectl get pod --kubeconfig=kctom -n default -o wide。

[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom  -n default -o wide
No resources found in default namespace.

客户端执行 kubectl get pod --kubeconfig=kctom -n audit -o wide。

[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom  -n audit -o wide
No resources found in audit namespace.

查看审计日志，只记录了audit命名空间的操作，default命名空间的操作没有记录。

root@k8scludes1:~# tail -2 /var/log/kubernetes/audit/audit.log
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"08b8fca7-8390-4f59-8a21-6ff3541fd2a2","stage":"RequestReceived","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"requestReceivedTimestamp":"2022-06-22T13:26:08.732751Z","stageTimestamp":"2022-06-22T13:26:08.732751Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"08b8fca7-8390-4f59-8a21-6ff3541fd2a2","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-06-22T13:26:08.732751Z","stageTimestamp":"2022-06-22T13:26:08.741665Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"crbindtom\" of ClusterRole \"cluster-admin\" to User \"tom\""}}

6.3 只记录audit命名空间的pods操作日志

修改审计策略，该审计策略表示只记录audit命名空间的pods操作。

root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml 
root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "ResponseStarted"
rules:
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods"]
    namespaces: ["audit"]

重启机器使审计策略生效。

root@k8scludes1:~# reboot

重启之后，清空日志。

root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log

在客户端执行kubectl get pod --kubeconfig=kctom -n audit -o wide。

[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom  -n audit -o wide
No resources found in audit namespace.

在客户端执行 kubectl get svc --kubeconfig=kctom -n audit -o wide 。

[root@etcd2 ~]# kubectl get svc --kubeconfig=kctom  -n audit -o wide
No resources found in audit namespace.

查看日志，发现只记录了audit命名空间的pod操作，svc操作没有记录。

root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"4748e76d-e9cb-46de-af28-d467171ca105","stage":"RequestReceived","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"requestReceivedTimestamp":"2022-06-23T00:24:45.351036Z","stageTimestamp":"2022-06-23T00:24:45.351036Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"4748e76d-e9cb-46de-af28-d467171ca105","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-06-23T00:24:45.351036Z","stageTimestamp":"2022-06-23T00:24:45.434147Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"clusterrolebinding1\" of ClusterRole \"clusterole1\" to User \"tom\""}}

6.4 只记录audit命名空间的pods,services,deployments操作日志

编辑审计策略文件，表示只记录audit命名空间的pods,services,deployments操作，因为deployments的apiVersion的父级为apps，所以需要group: "apps" 。

root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml 
root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "ResponseStarted"
rules:
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods","services"]
    - group: "apps"
      resources: ["deployments"]
    namespaces: ["audit"]

6.5 只记录audit命名空间的pods操作,审计级别为RequestResponse

编辑审计策略文件，表示只记录audit命名空间的pods操作,审计级别为RequestResponse，记录事件的元数据，请求和响应的消息体。

root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml
root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "ResponseStarted"
rules:
  - level: RequestResponse
    resources:
    - group: ""
      resources: ["pods"]
    namespaces: ["audit"]

重启机器使审计策略生效。

root@k8scludes1:~# reboot

创建一个pod。

root@k8scludes1:~# kubectl apply -f pod.yaml 
pod/podtest created
root@k8scludes1:~# kubectl get pod
NAME      READY   STATUS    RESTARTS   AGE
podtest   1/1     Running   0          5s

清空审计日志。

root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log

客户端执行kubectl get pod --kubeconfig=kctom -n audit -o wide 。

[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom  -n audit -o wide
NAME      READY   STATUS    RESTARTS   AGE   IP            NODE         NOMINATED NODE   READINESS GATES
podtest   1/1     Running   0          48s   10.244.1.97   k8scludes3   <none>           <none>

查看审计日志，记录的内容很丰富，元数据，请求和响应的消息体都显示出来了。

root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","auditID":"d6836f9b-4918-49f2-80dc-ad02e246949b","stage":"RequestReceived","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"requestReceivedTimestamp":"2022-06-23T00:43:30.782241Z","stageTimestamp":"2022-06-23T00:43:30.782241Z"}
......
23T00:42:46Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{".":{},"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"PodScheduled\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.97\"}":{".":{},"f:ip":{}}},"f:startTime":{}}},"subresource":"status"}]}}}]},"requestReceivedTimestamp":"2022-06-23T00:43:30.782241Z","stageTimestamp":"2022-06-23T00:43:30.785883Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"clusterrolebinding1\" of ClusterRole \"clusterole1\" to User \"tom\""}}

6.6 只记录audit命名空间下的tom用户的pods操作，其他用户操作不记录

编辑审计策略文件，设置审计策略：只记录audit命名空间下的tom用户的pods操作，其他用户的操作不记录。

root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml 
root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "ResponseStarted"
rules:
  - level: Metadata
    users: ["tom"]
    resources:
    - group: ""
      resources: ["pods"]
    namespaces: ["audit"]

重启机器使审计策略生效。

root@k8scludes1:~# reboot

清空日志。

root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log

使用管理员用户查看pod。

root@k8scludes1:~# kubectl get pod -n audit
NAME      READY   STATUS    RESTARTS   AGE
podtest   1/1     Running   0          29m

使用tom用户在客户端查看pod 。

[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom  -n audit -o wide
NAME      READY   STATUS    RESTARTS   AGE   IP            NODE         NOMINATED NODE   READINESS GATES
podtest   1/1     Running   0          29m   10.244.1.97   k8scludes3   <none>           <none>

查看日志，可以发现只记录了tom用户的pod查询操作。

root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"a70f1b39-3c54-4ed8-ab45-9cbb7b4688ef","stage":"RequestReceived","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"requestReceivedTimestamp":"2022-06-23T01:12:24.103036Z","stageTimestamp":"2022-06-23T01:12:24.103036Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"a70f1b39-3c54-4ed8-ab45-9cbb7b4688ef","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/audit/pods?limit=500","verb":"list","user":{"username":"tom","groups":["normalusertom","system:authenticated"]},"sourceIPs":["192.168.110.131"],"userAgent":"kubectl/v1.23.1 (linux/amd64) kubernetes/86ec240","objectRef":{"resource":"pods","namespace":"audit","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2022-06-23T01:12:24.103036Z","stageTimestamp":"2022-06-23T01:12:24.107872Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"crbindtom\" of ClusterRole \"cluster-admin\" to User \"tom\""}}

6.7 rules规则是从上往下匹配的，第一条规则已经匹配了，第二条就不匹配了

编辑审计策略文件，有两条策略，一条是：apiVersion为group: ""的操作不记录日志，另外一条是：只记录audit命名空间下tom用户的pod操作。

root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml
root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "ResponseStarted"
rules:
  #apiVersion为group: ""的操作不记录日志
  - level: None
    resources:
    - group: ""
  #只记录audit命名空间下tom用户的pod操作
  - level: Metadata
    users: ["tom"]
    resources:
    - group: ""
      resources: ["pods"]
    namespaces: ["audit"]

重启机器使配置生效。

root@k8scludes1:~# reboot

清空日志。

root@k8scludes1:~# >/var/log/kubernetes/audit/audit.log

使用管理员用户查询audit命名空间的pod。

root@k8scludes1:~# kubectl get pod -n audit
NAME      READY   STATUS    RESTARTS   AGE
podtest   1/1     Running   0          37m

使用tom用户在客户端机器etcd2上查询audit命名空间的pod。

[root@etcd2 ~]# kubectl get pod --kubeconfig=kctom  -n audit -o wide
NAME      READY   STATUS    RESTARTS   AGE   IP            NODE         NOMINATED NODE   READINESS GATES
podtest   1/1     Running   0          37m   10.244.1.97   k8scludes3   <none>           <none>

查看审计日志，可以发现在客户端使用tom用户查询pod和使用管理员用户查看pod都没有生成审计日志，rules规则是从上往下匹配的，第一条规则已经匹配了，第二条就不匹配了。

root@k8scludes1:~# cat /var/log/kubernetes/audit/audit.log

6.8 在 Metadata 级别为所有请求生成日志

编辑审计策略文件，在 Metadata 级别为所有请求生成日志。

root@k8scludes1:~# vim /etc/kubernetes/audit/audit-policy.yaml 
root@k8scludes1:~# cat /etc/kubernetes/audit/audit-policy.yaml
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
- level: Metadata
SHELL 复制 全屏

七.总结

在 Kubernetes 1.22.2 环境中实施审计策略可以帮助管理员监控和记录集群中的资源操作，确保集群的安全性和符合性。通过启用审计 Admission Controller 和配置相应的审计策略，我们可以灵活地控制审计记录的格式和范围。

致力于一条龙式的为您解决问题

Kubernetes 审计（Auditing）

一.系统环境

二.前言

三.Kubernetes 审计简介

四.审计策略简介

五.启用审计

5.1 引入审计

5.2 启用审计

六.审计策略

6.1 记录审计阶段为：ResponseStarted，审计级别为Metadata，apiVersion为group: "" 的日志

6.2 只记录audit命名空间里的日志

6.3 只记录audit命名空间的pods操作日志

6.4 只记录audit命名空间的pods,services,deployments操作日志

6.5 只记录audit命名空间的pods操作,审计级别为RequestResponse

6.6 只记录audit命名空间下的tom用户的pods操作，其他用户操作不记录

6.7 rules规则是从上往下匹配的，第一条规则已经匹配了，第二条就不匹配了

6.8 在 Metadata 级别为所有请求生成日志

七.总结

热门文章

最新文章

相关课程

相关电子书

相关实验场景

推荐镜像

热门

活动广场

任务中心

开发者评测

高校计划

乘风者计划

训练营

阿里云MVP

话题

直播

下载

镜像站

技术资料

插件

Kubernetes 审计（Auditing）

一.系统环境

二.前言

三.Kubernetes 审计简介

四.审计策略简介

五.启用审计

5.1 引入审计

5.2 启用审计

六.审计策略

6.1 记录审计阶段为：ResponseStarted，审计级别为Metadata，apiVersion为group: "" 的日志

6.2 只记录audit命名空间里的日志

6.3 只记录audit命名空间的pods操作日志

6.4 只记录audit命名空间的pods,services,deployments操作日志

6.5 只记录audit命名空间的pods操作,审计级别为RequestResponse

6.6 只记录audit命名空间下的tom用户的pods操作，其他用户操作不记录

6.7 rules规则是从上往下匹配的，第一条规则已经匹配了，第二条就不匹配了

6.8 在 Metadata 级别为所有请求生成日志

七.总结

热门文章

最新文章

相关课程

相关电子书

相关实验场景

推荐镜像