四、单机版Crane部署流程🍑
4.1. Crane系统一键化安装⚒️
🎆 Plan A：执行此命令直接一键化安装部署
# 执行此命令，可以一键部署，但是需要访问外网。 curl -sf https://raw.githubusercontent.com/gocrane/crane/main/hack/local-env-setup.sh | sh -
安装过程演示：
[root@Crane ~]# curl -sf https://raw.githubusercontent.com/gocrane/crane/main/hack/local-env-setup.sh | sh - Step1: Create local cluster: /root/.kube/config_crane Deleting cluster "crane" ... Creating cluster "crane" ... ✓ Ensuring node image (kindest/node:v1.21.1) 🖼 ✓ Preparing nodes 📦 ✓ Writing configuration 📜 ✓ Starting control-plane 🕹️ ✓ Installing CNI 🔌 ✓ Installing StorageClass 💾 Set kubectl context to "kind-crane" You can now use your cluster with: kubectl cluster-info --context kind-crane --kubeconfig /root/.kube/config_crane Thanks for using kind! 😊 Step1: Create local cluster finished. Step2: Installing Prometheus "prometheus-community" has been added to your repositories NAME: prometheus LAST DEPLOYED: Wed May 10 12:12:54 2023 NAMESPACE: crane-system STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: The Prometheus server can be accessed via port 8080 on the following DNS name from within your cluster: prometheus-server.crane-system.svc.cluster.local Get the Prometheus server URL by running these commands in the same shell: export POD_NAME=$(kubectl get pods --namespace crane-system -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}") kubectl --namespace crane-system port-forward $POD_NAME 9090 ################################################################################# ###### WARNING: Persistence is disabled!!! You will lose your data when ##### ###### the Server pod is terminated. ##### ################################################################################# ################################################################################# ###### WARNING: Pod Security Policy has been disabled by default since ##### ###### it deprecated after k8s 1.25+. use ##### ###### (index .Values "prometheus-node-exporter" "rbac" ##### ###### . "pspEnabled") with (index .Values ##### ###### "prometheus-node-exporter" "rbac" "pspAnnotations") ##### ###### in case you still need it. ##### ################################################################################# For more information on running Prometheus, visit: https://prometheus.io/ Step2: Installing Prometheus finished. Step3: Installing Grafana NAME: grafana LAST DEPLOYED: Wed May 10 12:13:00 2023 NAMESPACE: crane-system STATUS: deployed REVISION: 1 NOTES: 1. Get your 'admin' user password by running: kubectl get secret --namespace crane-system grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo 2. The Grafana server can be accessed via port 8082 on the following DNS name from within your cluster: grafana.crane-system.svc.cluster.local Get the Grafana URL to visit by running these commands in the same shell: export POD_NAME=$(kubectl get pods --namespace crane-system -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}") kubectl --namespace crane-system port-forward $POD_NAME 3000 3. Login with the password from step 1 and the username: admin ################################################################################# ###### WARNING: Persistence is disabled!!! You will lose your data when ##### ###### the Grafana pod is terminated. ##### ################################################################################# Step3: Installing Grafana finished. Step4: Installing Crane "crane" has been added to your repositories Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "crane" chart repository ...Successfully got an update from the "grafana" chart repository ...Successfully got an update from the "prometheus-community" chart repository Update Complete. ⎈Happy Helming!⎈ NAME: crane LAST DEPLOYED: Wed May 10 12:13:06 2023 NAMESPACE: crane-system STATUS: deployed REVISION: 1 TEST SUITE: None NAME: fadvisor LAST DEPLOYED: Wed May 10 12:13:10 2023 NAMESPACE: crane-system STATUS: deployed REVISION: 1 TEST SUITE: None Step4: Installing Crane finished. NAME READY UP-TO-DATE AVAILABLE AGE craned 0/1 1 0 3s fadvisor 0/1 1 0 0s grafana 0/1 1 0 10s metric-adapter 0/1 1 0 3s prometheus-kube-state-metrics 0/1 1 0 16s prometheus-server 0/1 1 0 16s Please wait for all pods ready After all pods ready, Get the Crane Dashboard URL to visit by running these commands in the same shell: export KUBECONFIG=/root/.kube/config_crane kubectl -n crane-system port-forward service/craned 9090:9090
⏭️Plan B：如果访问网络发生错误，可以使用本地安装包执行安装操作，具体执行命令如下：
# 上传Crane安装包到系统中 [root@Crane training]# pwd /root/training [root@Crane training]# ll total 4 drwxr-xr-x. 7 root root 4096 May 10 13:50 installation # 进入training目录，查看文件内容 [root@Crane installation]# ll total 228 -rw-r--r--. 1 root root 4206 May 10 13:50 components.yaml drwxr-xr-x. 5 root root 120 May 10 13:50 crane -rw-r--r--. 1 root root 1232 May 10 13:50 effective-hpa.yaml drwxr-xr-x. 3 root root 77 May 10 13:50 fadvisor drwxr-xr-x. 5 root root 124 May 10 13:50 grafana -rw-r--r--. 1 root root 199848 May 10 13:50 grafana_override_values.yaml drwxr-xr-x. 3 root root 96 May 10 13:50 kube-state-metrics -rw-r--r--. 1 root root 3777 May 10 13:50 local-env-setup.sh -rw-r--r--. 1 root root 522 May 10 13:50 nginx-deployment.yaml -rw-r--r--. 1 root root 615 May 10 13:50 php-apache.yaml drwxr-xr-x. 3 root root 140 May 10 13:50 prometheus -rw-r--r--. 1 root root 4915 May 10 13:50 prometheus_override_values.yaml # 💥必须在 installation 的上级目录执行下面这一操作（即不能修改这条命令🚫），否则安装失败。💢 bash installation/local-env-setup.sh
等待片刻时间， 查看所有的Pod 是否都正常启动运行，如下所示。再进行下一步相关操作。
[root@Crane ~]# kubectl get pods -n crane-system NAME READY STATUS RESTARTS AGE craned-75d5fcff49-2ppnn 2/2 Running 0 121m fadvisor-6c6867dcb9-tscxm 1/1 Running 0 121m grafana-8fb6974cc-kzgzf 1/1 Running 0 121m metric-adapter-789b5b8bc5-hnt9g 1/1 Running 0 121m prometheus-kube-state-metrics-69c44479cb-jlzmh 1/1 Running 0 121m prometheus-prometheus-node-exporter-4xmrg 1/1 Running 0 121m prometheus-server-6cb8bc86c4-wxdsz 2/2 Running 0 121m
4.2. 访问Crane Dashboard💨
重新打开一个新的终端，执行如下命令：
# 🌈每打开一个终端进行操作时，都需要执行配置环境变量这一条命令(不然会出现8080端口被拒绝的提示)，如下图所示。 export KUBECONFIG=/root/.kube/config_crane
# 执行此命令，访问Crane Dashboard。如下图所示。 kubectl -n crane-system port-forward service/craned 9090:9090
💢重点💢：本实验使用虚拟机进行安装部署，直接执行127.0.0.1:9090或者192.168.200.60:9090，均无法访问到DashBoard。此时，系统需要将本地的端口做下反向代理，将9090端口转发给80，在浏览器中直接输入主机IP地址192.168.200.60即可访问。具体执行操作如下所示。
# 安装nginx服务 yum install -y nginx # 修改nginx.conf配置文件，修改内容如下： server_name 192.168.200.60; location / { proxy_pass http://127.0.0.1:9090; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection keep-alive; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } # 按:wq报错退出。 # 检查nginx配置文件是否正确，执行结果如下所示。 [root@Crane ~]# nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful # 启动nginx服务并设置开机自启动 systemctl start && systemctl enable nginx && systemctl status nginx
如果还访问不了Dashboard，请自行检查pods是否都已经正常启动；系统环境设置是否做了限制（如防火墙是否关闭或开启对应端口）。
4.3. 添加集群🛠️
添加集群。
添加完成。
五、集群功能演示🛠️
5.1. 使用智能弹性EffectiveHPA🥒
Crane提供了一种名为EffectiveHorizontalPodAutoscaler（EHPA）的弹性伸缩产品，它基于社区HPA技术实现了弹性控制功能，并支持更多的弹性触发策略，包括预测、观测和周期等，从而实现更高效的弹性控制，并确保了服务质量。简而言之，EHPA是一种高效的弹性伸缩方案，可以为服务提供更好的保障。
5.1.1 安装Metrics Server
# 执行命令安装Metrics Server。需要在installation上一级目录下执行此命令。💥 [root@Crane training]# kubectl apply -f installation/components.yaml serviceaccount/metrics-server created clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created clusterrole.rbac.authorization.k8s.io/system:metrics-server created rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created service/metrics-server created deployment.apps/metrics-server created apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
查看该pod是否正常启动。
[root@Crane ~]# kubectl get pod -n kube-system | grep metrics-server metrics-server-79c88ff4f-mz96g 1/1 Running 0 16m
5.1.2 创建测试应用
使用以下命令启动一个 Deployment 用 hpa-example 镜像运行一个容器， 然后将其暴露为一个 服务（Service）
[root@Crane training]# kubectl apply -f installation/php-apache.yaml deployment.apps/php-apache created service/php-apache created [root@Crane training]# kubectl apply -f installation/nginx-deployment.yaml deployment.apps/nginx-deployment created [root@Crane training]# kubectl get pods NAME READY STATUS RESTARTS AGE nginx-deployment-758fd5cc9f-27bpz 1/1 Running 0 108s nginx-deployment-758fd5cc9f-mxxx9 1/1 Running 0 108s nginx-deployment-758fd5cc9f-np46c 1/1 Running 0 108s nginx-deployment-758fd5cc9f-p8s9q 1/1 Running 0 108s nginx-deployment-758fd5cc9f-tc2mj 1/1 Running 0 108s php-apache-7d59cc57d4-8tnph 1/1 Running 0 2m5s
5.1.3 创建 EffectiveHPA
[root@Crane training]# kubectl apply -f installation/effective-hpa.yaml effectivehorizontalpodautoscaler.autoscaling.crane.io/php-apache created
# 查看 EffectiveHPA 的状态信息 [root@Crane training]# kubectl get ehpa NAME STRATEGY MINPODS MAXPODS SPECIFICPODS REPLICAS AGE php-apache Auto 1 10 0 10s
5.1.4 增加负载测试
# 打开新的终端窗口，配置环境变量 export KUBECONFIG=${HOME}/.kube/config_crane kubectl run -i --tty load-generator --rm --image=busybox:1.28 --restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://php-apache; done"
可以按CTRL+C终止上述请求。随着请求增多，CPU利用率会不断提升，可以看到 EffectiveHPA 会自动扩容实例。如下所示。
增加负载后，相关信息参数如下所示。
[root@crane training]# kubectl get hpa ehpa-php-apache --watch NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE ehpa-php-apache Deployment/php-apache 29%/50% 1 10 1 107s ehpa-php-apache Deployment/php-apache 250%/50% 1 10 1 2m1s ehpa-php-apache Deployment/php-apache 240%/50% 1 10 4 2m16s ehpa-php-apache Deployment/php-apache 109%/50% 1 10 5 2m31s ehpa-php-apache Deployment/php-apache 70%/50% 1 10 5 2m46s
[root@Crane training]# kubectl get ehpa NAME STRATEGY MINPODS MAXPODS SPECIFICPODS REPLICAS AGE php-apache Auto 1 10 6 7m1s [root@Crane training]# kubectl get pods NAME READY STATUS RESTARTS AGE nginx-deployment-758fd5cc9f-27bpz 1/1 Running 0 5m22s nginx-deployment-758fd5cc9f-mxxx9 1/1 Running 0 5m22s nginx-deployment-758fd5cc9f-np46c 1/1 Running 0 5m22s nginx-deployment-758fd5cc9f-p8s9q 1/1 Running 0 5m22s nginx-deployment-758fd5cc9f-tc2mj 1/1 Running 0 5m22s php-apache-7d59cc57d4-4zf6j 1/1 Running 0 57s php-apache-7d59cc57d4-6b5h8 1/1 Running 0 72s php-apache-7d59cc57d4-8tnph 1/1 Running 0 5m39s php-apache-7d59cc57d4-kjkff 1/1 Running 0 27s php-apache-7d59cc57d4-lwnpp 1/1 Running 0 72s php-apache-7d59cc57d4-zhdfk 1/1 Running 0 57s
通过下面截图对比可知，创建的应用在增加负载压力测试过程中，发生了自动扩缩容
5.2. 成本展示 🍆
集群总
通过登录Grafana数据可视化展示平台进行查验。本实验的输入的地址是：http://192.168.200.60/grafana。
默认账号/密码：admin/admin
export KUBECONFIG=${HOME}/.kube/config_crane kubectl -n crane-system port-forward service/grafana 8082:8082
5.3. 应用资源优化🥕
在 dashboard 中看到相关的成本数据，是因为在添加集群的时候安装了推荐的规则。
推荐框架会自动分析集群的各种资源的运行情况并给出优化建议。
Crane 的推荐模块会定期检测发现集群资源配置的问题，并给出优化建议。
智能推荐提供了多种 Recommender 来实现面向不同资源的优化推荐。
在成本分析>推荐规则页面可以看到安装的两个推荐规则。
这些推荐规则实际上在将 K8s 集群接入Dashboard时安装上的 RecommendationRule CRD 对象：
[root@Crane ~]# kubectl get RecommendationRule NAME RUNINTERVAL AGE idlenodes-rule 24h 5h8m workloads-rule 24h 5h8m
RecommendationRule 是一个集群维度的对象，该推荐规则会对所有命名空间中的 Deployments 和 StatefulSets 做资源推荐和副本数推荐。
需要注意的是资源类型和 recommenders 需要可以匹配，比如 Resource 推荐默认只支持 Deployments 和 StatefulSets。
👀 查看闲置节点推荐规则的资源对象👀
[root@Crane ~]# kubectl get recommendationrule idlenodes-rule -oyaml apiVersion: analysis.crane.io/v1alpha1 kind: RecommendationRule metadata: creationTimestamp: "2023-05-10T04:27:24Z" generation: 2 labels: analysis.crane.io/recommendation-rule-preinstall: "true" name: idlenodes-rule resourceVersion: "3494" uid: 034152a2-e4ae-4d3b-8223-624f2315e067 spec: namespaceSelector: any: true recommenders: - name: IdleNode resourceSelectors: - apiVersion: v1 kind: Node runInterval: 24h status: lastUpdateTime: "2023-05-10T04:27:24Z" recommendations: - lastStartTime: "2023-05-10T04:27:24Z" message: 'Failed to run recommendation flow in recommender IdleNode: Node crane-control-plane is not a idle node ' recommenderRef: name: IdleNode targetRef: apiVersion: v1 kind: Node name: crane-control-plane runNumber: 1
👀 查看集群生成的多个优化建议 Recommendation 对象👀
[root@Crane ~]# kubectl get recommendations -A NAMESPACE NAME TYPE TARGETKIND TARGETNAMESPACE TARGETNAME STRATEGY PERIODSECONDS ADOPTIONTYPE AGE crane-system workloads-rule-resource-254v6 Resource Deployment crane-system metric-adapter Once StatusAndAnnotation 5h12m crane-system workloads-rule-resource-7c4jg Resource Deployment crane-system prometheus-kube-state-metrics Once StatusAndAnnotation 5h12m crane-system workloads-rule-resource-hwr7p Resource Deployment crane-system prometheus-server Once StatusAndAnnotation 5h12m crane-system workloads-rule-resource-m5ws6 Resource Deployment crane-system grafana Once StatusAndAnnotation 5h12m
👀查看任意优化建议对象👀
kubectl get recommend workloads-rule-resource-254v6 -n crane-system -oyaml
apiVersion: analysis.crane.io/v1alpha1 kind: Recommendation metadata: annotations: analysis.crane.io/run-number: "1" creationTimestamp: "2023-05-10T04:27:24Z" generateName: workloads-rule-resource- generation: 2 labels: analysis.crane.io/recommendation-rule-name: workloads-rule analysis.crane.io/recommendation-rule-recommender: Resource analysis.crane.io/recommendation-rule-uid: ae95350e-5bfb-4fa7-955c-a69907d17b70 analysis.crane.io/recommendation-target-kind: Deployment analysis.crane.io/recommendation-target-name: metric-adapter analysis.crane.io/recommendation-target-version: v1 app: metric-adapter app.kubernetes.io/instance: crane app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: crane app.kubernetes.io/version: v0.10.0 helm.sh/chart: crane-0.10.0 name: workloads-rule-resource-254v6 namespace: crane-system ownerReferences: - apiVersion: analysis.crane.io/v1alpha1 blockOwnerDeletion: false controller: false kind: RecommendationRule name: workloads-rule uid: ae95350e-5bfb-4fa7-955c-a69907d17b70 resourceVersion: "3515" uid: b00b3e5d-400d-455d-a338-7526c5d7d6c1 spec: adoptionType: StatusAndAnnotation completionStrategy: completionStrategyType: Once targetRef: apiVersion: apps/v1 kind: Deployment name: metric-adapter namespace: crane-system type: Resource status: action: Patch conditions: - lastTransitionTime: "2023-05-10T04:27:24Z" message: Recommendation is ready reason: RecommendationReady status: "True" type: Ready currentInfo: '{"spec":{"template":{"spec":{"containers":[{"name":"metric-adapter","resources":{"requests":{"cpu":"0","memory":"0"}}}]}}}}' lastUpdateTime: "2023-05-10T04:27:24Z" recommendedInfo: '{"spec":{"template":{"spec":{"containers":[{"name":"metric-adapter","resources":{"requests":{"cpu":"114m","memory":"120586239"}}}]}}}}' recommendedValue: | resourceRequest: containers: - containerName: metric-adapter target: cpu: 114m memory: "120586239" targetRef: {} [root@Crane ~]#
通过Web控制面板也可以查看上述信息
对于闲置节点推荐，由于节点的下线在不同平台上的步骤不同，用户可以根据自身需求进行节点的下线或者缩容。
应用在监控系统（比如 Prometheus）中的历史数据越久，推荐结果就越准确，建议生产上超过两周时间。
对新建应用的预测往往不准。
