helm是一款管理kubernetes yaml文件的强大工具，能实现整个项目所需yaml一次性编译，包括service account、rbac、configmap、storageClass、controller manager等等资源。
背景：
mysql作为业务常用的后端库，广泛用于数据存储和分析，但作为有状态服务，部署起来挑战不小，各种资源都需要apply yaml文件，容易出现报错失败且不好排查，helm就可以很好的解决，并且很高效。
一、准备工作
1、资源清单
helm 采用开源bitnami库的mysql进行操作
primary为主库、secondary为从库
二、部署helm
安装地址 PATH
下载包：helm-v3.11.1-linux-amd64.tar.gz
解压：
[root@localhost ~]# tar -xf helm-v3.11.1-linux-amd64.tar.gz [root@localhost ~]# cd linux-amd64/ [root@localhost linux-amd64]# ll 总用量 45788 -rwxr-xr-x 1 1001 1002 46870528 2月 9 01:29 helm -rw-r--r-- 1 1001 1002 11373 2月 9 01:39 LICENSE -rw-r--r-- 1 1001 1002 3367 2月 9 01:39 README.md [root@localhost linux-amd64]# cp helm /usr/local/bin/ [root@localhost linux-amd64]# helm version version.BuildInfo{Version:"v3.11.1", GitCommit:"293b50c65d4d56187cd4e2f390f0ada46b4c4737", GitTreeState:"clean", GoVersion:"go1.18.10"}
1、生成locat-path
bitnami本身不带创建storageclass，但有状态服务需要可持续化存储，所以本地需要自建一个storageclass供mysql生成pv和pvc使用。
[root@localhost ~]# mkdir mysql [root@localhost mysql]# pwd /root/mysql
[root@localhost mysql]# cat > local-path-storage.yaml << EOF apiVersion: v1 kind: Namespace metadata: name: local-path-storage --- apiVersion: v1 kind: ServiceAccount metadata: name: local-path-provisioner-service-account namespace: local-path-storage --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: local-path-provisioner-role rules: - apiGroups: [ "" ] resources: [ "nodes", "persistentvolumeclaims", "configmaps" ] verbs: [ "get", "list", "watch" ] - apiGroups: [ "" ] resources: [ "endpoints", "persistentvolumes", "pods" ] verbs: [ "*" ] - apiGroups: [ "" ] resources: [ "events" ] verbs: [ "create", "patch" ] - apiGroups: [ "storage.k8s.io" ] resources: [ "storageclasses" ] verbs: [ "get", "list", "watch" ] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: local-path-provisioner-bind roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: local-path-provisioner-role subjects: - kind: ServiceAccount name: local-path-provisioner-service-account namespace: local-path-storage --- apiVersion: apps/v1 kind: Deployment metadata: name: local-path-provisioner namespace: local-path-storage spec: replicas: 1 selector: matchLabels: app: local-path-provisioner template: metadata: labels: app: local-path-provisioner spec: serviceAccountName: local-path-provisioner-service-account containers: - name: local-path-provisioner image: rancher/local-path-provisioner:v0.0.22 imagePullPolicy: IfNotPresent command: - local-path-provisioner - --debug - start - --config - /etc/config/config.json volumeMounts: - name: config-volume mountPath: /etc/config/ env: - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace volumes: - name: config-volume configMap: name: local-path-config --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: local-path provisioner: rancher.io/local-path volumeBindingMode: WaitForFirstConsumer reclaimPolicy: Delete --- kind: ConfigMap apiVersion: v1 metadata: name: local-path-config namespace: local-path-storage data: config.json: |- { "nodePathMap":[ { "node":"DEFAULT_PATH_FOR_NON_LISTED_NODES", "paths":["/opt/local-path-provisioner"] } ] } setup: |- #!/bin/sh set -eu mkdir -m 0777 -p "$VOL_DIR" teardown: |- #!/bin/sh set -eu rm -rf "$VOL_DIR" helperPod.yaml: |- apiVersion: v1 kind: Pod metadata: name: helper-pod spec: containers: - name: helper-pod image: busybox imagePullPolicy: IfNotPresent EOF
#若报错：local-path-storage not found [root@localhost mysql]# kubectl create ns local-path-storage [root@localhost mysql]# kubectl apply -f local-path-storage.yaml #设置为默认存储类 [root@localhost mysql]# kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
2、部署MySQL主从
helm是依靠模板内yaml文件获取values.yaml中的值实现freestyle风格部署。
如下是简单部署：
[root@localhost mysql]# cat > values.yaml << EOF global: storageClass: "local-path" auth: rootPassword: test123456 architecture: replication metrics: enabled: true primary: service: type: NodePort nodePorts: mysql: "30336" EOF
storageClass：使用NFS存储
auth.rootPassword：nysql 的密码
architecture： 架构，这里是集群部署，默认是主从
values与模板内定义对应关系，有的涉及到if/else end的语法可以自己去看看资料。
spec: type: {{ .Values.primary.service.type }} 解释： .为顶域 primary.service.type取的值如下 primary: service: type: NodePort
运行chart为release，运行后副本默认为1，可以在values.yaml文件内定义，如下：
replicas: {{ .Values.secondary.replicaCount }} secondary: replicaCount: 2
[root@localhost mysql]# kubectl create ns component [root@localhost mysql]# helm install mysql bitnami/mysql -f values.yaml -n component [root@localhost mysql]# helm list -A NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION mysql component 1 2023-03-24 10:41:28.897163202 +0800 CST deployed mysql-9.7.0 8.0.32 [root@localhost mysql]# helm status mysql -n component NAME: mysql LAST DEPLOYED: Fri Mar 24 10:41:28 2023 NAMESPACE: component STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: CHART NAME: mysql CHART VERSION: 9.7.0 APP VERSION: 8.0.32 ** Please be patient while the chart is being deployed ** Tip: Watch the deployment status using the command: kubectl get pods -w --namespace component Services: echo Primary: mysql-primary.component.svc.cluster.local:3306 echo Secondary: mysql-secondary.component.svc.cluster.local:3306 Execute the following to get the administrator credentials: echo Username: root MYSQL_ROOT_PASSWORD=$(kubectl get secret --namespace component mysql -o jsonpath="{.data.mysql-root-password}" | base64 -d) To connect to your database: 1. Run a pod that you can use as a client: kubectl run mysql-client --rm --tty -i --restart='Never' --image docker.io/bitnami/mysql:8.0.32-debian-11-r14 --namespace component --env MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD --command -- bash 2. To connect to primary service (read/write): mysql -h mysql-primary.component.svc.cluster.local -uroot -p"$MYSQL_ROOT_PASSWORD" 3. To connect to secondary service (read-only): mysql -h mysql-secondary.component.svc.cluster.local -uroot -p"$MYSQL_ROOT_PASSWORD" To access the MySQL Prometheus metrics from outside the cluster execute the following commands: kubectl port-forward --namespace component svc/mysql-metrics 9104:9104 & curl http://127.0.0.1:9104/metrics
#master [root@localhost mysql]# kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE component mysql-primary-0 2/2 Running 0 5h32m component mysql-secondary-0 2/2 Running 0 5h32m kube-system coredns-6c5f44f974-bmnk8 1/1 Running 2 27d kube-system coredns-6c5f44f974-cn4mb 1/1 Running 2 27d kube-system kube-flannel-ds-7q4ws 1/1 Running 2 27d kube-system kube-flannel-ds-bpkkc 1/1 Running 2 27d local-path-storage local-path-provisioner-7c795b5576-mt69b 1/1 Running 0 5h38m #node1 [root@localhost ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c7dff0a4c87b bitnami/mysqld-exporter "/bin/bash -ec 'pass…" 4 hours ago Up 4 hours k8s_metrics_mysql-primary-0_component_a585312d-4f91-46e2-a06a-8d8c5bcba493_0 30832a206c86 bitnami/mysql "/opt/bitnami/script…" 4 hours ago Up 4 hours k8s_mysql_mysql-primary-0_component_a585312d-4f91-46e2-a06a-8d8c5bcba493_0 972771a0c0ab registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2 "/pause" 4 hours ago Up 4 hours k8s_POD_mysql-primary-0_component_a585312d-4f91-46e2-a06a-8d8c5bcba493_0 [root@localhost mysql]# kubectl get svc -A NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE component mysql-metrics ClusterIP 10.254.250.224 <none> 9104/TCP 5h47m component mysql-primary NodePort 10.254.216.254 <none> 3306:30336/TCP 5h47m component mysql-primary-headless ClusterIP None <none> 3306/TCP 5h47m component mysql-secondary ClusterIP 10.254.93.194 <none> 3306/TCP 5h47m component mysql-secondary-headless ClusterIP None <none> 3306/TCP 5h47m default kubernetes ClusterIP 10.254.0.1 <none> 443/TCP 28d kube-system kube-dns ClusterIP 10.254.0.2 <none> 53/UDP,53/TCP,9153/TCP 28d
[root@localhost mysql]# kubectl run mysql-client --rm --tty -i --restart='Never' --image docker.io/bitnami/mysql:8.0.32-debian-11-r14 --namespace component --env MYSQL_ROOT_PASSWORD=test123456 --command -- bash If you don't see a command prompt, try pressing enter. I have no name!@mysql-client:/$ mysql -h mysql-primary.component.svc.cluster.local -uroot -p"test123456" mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 8083 Server version: 8.0.32 Source distribution Copyright (c) 2000, 2023, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | my_database | | mysql | | performance_schema | | sys | +--------------------+ 5 rows in set (0.00 sec) mysql> show master status\G; *************************** 1. row *************************** File: mysql-bin.000003 Position: 157 Binlog_Do_DB: Binlog_Ignore_DB: Executed_Gtid_Set: 1 row in set (0.01 sec)
查看sql、io线程是否running ###slave库 操作 I have no name!@mysql-client:/$ mysql -h mysql-secondary.component.svc.cluster.local -uroot -p"test123456" mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 8173 Server version: 8.0.32 Source distribution Copyright (c) 2000, 2023, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> show slave status\G; *************************** 1. row *************************** Slave_IO_State: Waiting for source to send event Master_Host: mysql-primary Master_User: replicator Master_Port: 3306 Connect_Retry: 10 Master_Log_File: mysql-bin.000003 Read_Master_Log_Pos: 157 Relay_Log_File: mysql-relay-bin.000006 Relay_Log_Pos: 373 Relay_Master_Log_File: mysql-bin.000003 Slave_IO_Running: Yes Slave_SQL_Running: Yes
三、分析bitnami
bitnami官网地址
拉取bitnami mysql chart
[root@localhost mysql]# helm repo add bitnami https://charts.bitnami.com/bitnami [root@localhost mysql]# helm repo list NAME URL bitnami https://charts.bitnami.com/bitnami [root@localhost mysql]# helm search repo mysql NAME CHART VERSION APP VERSION DESCRIPTION bitnami/mysql 9.7.0 8.0.32 MySQL is a fast, reliable, scalable, and easy t... bitnami/phpmyadmin 10.4.5 5.2.1 phpMyAdmin is a free software tool written in P... bitnami/mariadb 11.5.5 10.6.12 MariaDB is an open source, community-developed ... bitnami/mariadb-galera 7.5.4 10.6.12 MariaDB Galera is a multi-primary database clus... [root@localhost mysql]# helm pull bitnami/mysql [root@localhost mysql]# ll 总用量 56 -rw-r--r-- 1 root root 2933 3月 24 10:35 local-path-storage.yaml -rw-r--r-- 1 root root 46303 3月 24 15:16 mysql-9.7.0.tgz [root@localhost mysql]# tar xf mysql-9.7.0.tgz [root@localhost mysql]# cd mysql [root@localhost mysql]# ll 总用量 140 -rw-r--r-- 1 root root 219 3月 23 01:37 Chart.lock drwxr-xr-x 3 root root 20 3月 24 14:26 charts #子chart -rw-r--r-- 1 root root 787 3月 23 01:37 Chart.yaml -rw-r--r-- 1 root root 68526 3月 23 01:37 README.md drwxr-xr-x 4 root root 290 3月 24 14:26 templates #总的模板目录 -rw-r--r-- 1 root root 5495 3月 23 01:37 values.schema.json -rw-r--r-- 1 root root 53803 3月 23 01:37 values.yaml #变量修改地方 [root@localhost mysql]# tree templates templates ├── extra-list.yaml ├── _helpers.tpl ├── metrics-svc.yaml #metrics svc ├── networkpolicy.yaml ├── NOTES.txt ├── primary #主库 │ ├── configmap.yaml │ ├── initialization-configmap.yaml #初始化配置 │ ├── pdb.yaml │ ├── statefulset.yaml #默认副本为1，metrics │ ├── svc-headless.yaml │ └── svc.yaml ├── prometheusrule.yaml ├── rolebinding.yaml ├── role.yaml ├── secondary #从库 │ ├── configmap.yaml │ ├── pdb.yaml │ ├── statefulset.yaml │ ├── svc-headless.yaml │ └── svc.yaml ├── secrets.yaml ├── serviceaccount.yaml └── servicemonitor.yaml
四、拓展
1、更新版本和查看历史版本
[root@localhost mysql]# kubectl get svc -A NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE component mysql-metrics ClusterIP 10.254.250.224 <none> 9104/TCP 5h47m component mysql-primary NodePort 10.254.216.254 <none> 3306:30336/TCP 5h47m component mysql-primary-headless ClusterIP None <none> 3306/TCP 5h47m component mysql-secondary ClusterIP 10.254.93.194 <none> 3306/TCP 5h47m component mysql-secondary-headless ClusterIP None <none> 3306/TCP 5h47m default kubernetes ClusterIP 10.254.0.1 <none> 443/TCP 28d kube-system kube-dns ClusterIP 10.254.0.2 <none> 53/UDP,53/TCP,9153/TCP 28d ##若需要从库被navicat链接到参与查询，需要把该release进行更新 # helm upgrade -f values.yaml chartname 远程chart -n namespace # helm history chartname -n namespace [root@localhost mysql]# cat values.yaml global: storageClass: "local-path" auth: rootPassword: test123456 architecture: replication metrics: enabled: true primary: service: type: NodePort nodePorts: mysql: "30336" secondary: service: type: NodePort nodePorts: mysql: "30366" [root@localhost mysql]# helm history mysql -n component REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 1 Fri Mar 24 10:41:28 2023 deployed mysql-9.7.0 8.0.32 Install complete [root@localhost mysql]# helm upgrade -f values.yaml mysql bitnami/mysql -n component Release "mysql" has been upgraded. Happy Helming! NAME: mysql LAST DEPLOYED: Fri Mar 24 16:36:53 2023 NAMESPACE: component STATUS: deployed REVISION: 2 TEST SUITE: None NOTES: CHART NAME: mysql CHART VERSION: 9.7.0 APP VERSION: 8.0.32 [root@localhost mysql]# helm history mysql -n component REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 1 Fri Mar 24 10:41:28 2023 superseded mysql-9.7.0 8.0.32 Install complete 2 Fri Mar 24 16:36:53 2023 deployed mysql-9.7.0 8.0.32 Upgrade complete ###从库service类型变化为nodeport [root@localhost mysql]# kubectl get svc -A NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE component mysql-metrics ClusterIP 10.254.250.224 <none> 9104/TCP 5h55m component mysql-primary NodePort 10.254.216.254 <none> 3306:30336/TCP 5h55m component mysql-primary-headless ClusterIP None <none> 3306/TCP 5h55m component mysql-secondary NodePort 10.254.93.194 <none> 3306:30366/TCP 5h55m component mysql-secondary-headless ClusterIP None <none> 3306/TCP 5h55m default kubernetes ClusterIP 10.254.0.1 <none> 443/TCP 28d kube-system kube-dns ClusterIP 10.254.0.2 <none> 53/UDP,53/TCP,9153/TCP 28d
若service type为clusterip临时给图形化连接
# kubectl port-forward --address=ip pods/cluster-mysql-primary-0 port
2、版本回滚
#helm rollback chartname 1 回滚到第一个版本 [root@localhost mysql]# helm rollback mysql 1 -n component Rollback was a success! Happy Helming! [root@localhost mysql]# helm history mysql -n component REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 1 Fri Mar 24 10:41:28 2023 superseded mysql-9.7.0 8.0.32 Install complete 2 Fri Mar 24 16:36:53 2023 superseded mysql-9.7.0 8.0.32 Upgrade complete 3 Fri Mar 24 16:43:21 2023 deployed mysql-9.7.0 8.0.32 Rollback to 1
3、替换值安装时只是测试不安装
# helm install --debug --dry-run chartname chart文件 -n names
