devops-k8s部署jenkins和动态创建slave节点

2022-12-13 1056

版权

本文内容由阿里云实名注册用户自发贡献，版权归原作者所有，阿里云开发者社区不拥有其著作权，亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容，填写侵权投诉表单进行举报，一经查实，本社区将立刻删除涉嫌侵权内容。

本文涉及的产品

容器服务 Serverless 版 ACK Serverless，952元额度多规格

容器服务 Serverless 版 ACK Serverless，317元额度多规格

简介： 1. K8S部署jenkins2. 动态创建slave集群节点3. 使用PodTemplate构建流水线

环境准备

k8s环境：node1 和 node2

创建命名空间

╰─# kubectl create ns devopsnamespace/devops created

创建jenkins-master

deployment文件：

kind: Deployment
apiVersion: apps/v1
metadata:  labels:    k8s-app: jenkins
  name: jenkins
  namespace: devops
spec:  replicas: 1  revisionHistoryLimit: 10  selector:    matchLabels:      k8s-app: jenkins
  template:    metadata:      labels:        k8s-app: jenkins
      namespace: devops
      name: jenkins
    spec:      containers:        - name: jenkins
          image: jenkins/jenkins:2.381          imagePullPolicy: IfNotPresent
          ports:            - containerPort: 30080              name: web
              protocol: TCP
            - containerPort: 30081              name: agent
              protocol: TCP
          resources:            limits:              cpu: 1000m
              memory: 2Gi
            requests:              cpu: 500m
              memory: 512Mi
          livenessProbe:            httpGet:              path: /login
              port: 30080            initialDelaySeconds: 60            timeoutSeconds: 5            failureThreshold: 12          readinessProbe:            httpGet:              path: /login
              port: 30080            initialDelaySeconds: 60            timeoutSeconds: 5            failureThreshold: 12          volumeMounts:            - name: jenkins-home
              mountPath: /var/lib/jenkins
          env:            - name: JENKINS_HOME
              value: /var/lib/jenkins
            - name: JENKINS_OPTS 
              value: --httpPort=30080
            - name: JENKINS_SLAVE_AGENT_PORT
              value: "30081"      volumes:        - name: jenkins-home
          hostPath:             path: /data/devops/jenkins
            type: Directory
      serviceAccountName: jenkins
---apiVersion: v1
kind: ServiceAccount
metadata:  labels:    k8s-app: jenkins
  name: jenkins
  namespace: devops
---kind: Service
apiVersion: v1
metadata:  labels:    k8s-app: jenkins
  name: jenkins
  namespace: devops
spec:  type: NodePort
  ports:    - name: web
      port: 30080      targetPort: 30080      nodePort: 30080    - name: slave
      port: 30081      targetPort: 30081      nodePort: 30081  selector:    k8s-app: jenkins
---kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata: name: jenkins
 namespace: devops
rules: - apiGroups: [""]   resources: ["pods","configmaps","namespaces"]   verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""]   resources: ["pods/exec"]   verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""]   resources: ["pods/log"]   verbs: ["get","list","watch"] - apiGroups: [""]   resources: ["secrets"]   verbs: ["get"]---apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: name: jenkins
 namespace: devops
roleRef: apiGroup: rbac.authorization.k8s.io
 kind: Role
 name: jenkins
subjects: - kind: ServiceAccount
   name: jenkins
   namespace: devops

通过一下命令启动jenkins-master

kubectl create -f deployment.yml

在启动jenkins过程中，容器状态为 CrashLoopBackOff ，查看日志出现 java.nio.file.AccessDeniedException: /var/lib/jenkins/war

错误信息时，容器挂载的宿主机目录，无权限访问，需要执行 chmod 777 /data/devops/jenkins , 然后在重启一下pod

╰─# kubectl get pods -n devops                                                                                                                                                                                                1 ↵
NAME                       READY   STATUS             RESTARTS   AGE
jenkins-65dbc65fbd-kkxbc   0/1     CrashLoopBackOff   5          5m26s
╭─root@k201 ~/jenkins-k8s 
╰─# kubectl logs pods/jenkins-65dbc65fbd-kkxbc -n devops                                                               
Running from: /usr/share/jenkins/jenkins.war
webroot: /var/lib/jenkins/war
2022-12-12 09:47:16.120+0000 [id=1] INFO  winstone.Logger#logInternal: Beginning extraction from war file
2022-12-12 09:47:16.139+0000 [id=1] WARNING winstone.Logger#logInternal: Failed to recreate dirs /var/lib/jenkins/war
java.nio.file.AccessDeniedException: /var/lib/jenkins/war
  at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
  at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
  at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
  at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389)
  at java.base/java.nio.file.Files.createDirectory(Files.java:690)
  at java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:797)
  at java.base/java.nio.file.Files.createDirectories(Files.java:783)
  at winstone.HostConfiguration.getWebRoot(HostConfiguration.java:265)
  at winstone.HostConfiguration.<init>(HostConfiguration.java:87)
  at winstone.HostGroup.initHost(HostGroup.java:65)
  at winstone.HostGroup.<init>(HostGroup.java:44)
  at winstone.Launcher.<init>(Launcher.java:150)
  at winstone.Launcher.main(Launcher.java:389)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.base/java.lang.reflect.Method.invoke(Method.java:566)
  at executable.Main.main(Main.java:356)
2022-12-12 09:47:16.143+0000 [id=1] WARNING winstone.Logger#logInternal: Failed to create dirs /var/lib/jenkins/war/META-INF
2022-12-12 09:47:16.229+0000 [id=1] INFO  winstone.Logger#logInternal: Jetty shutdown successfully
java.io.FileNotFoundException: /var/lib/jenkins/war/META-INF/MANIFEST.MF (No such file or directory)
  at java.base/java.io.FileOutputStream.open0(Native Method)
  at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)
  at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:237)
  at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:187)
  at winstone.HostConfiguration.getWebRoot(HostConfiguration.java:309)
  at winstone.HostConfiguration.<init>(HostConfiguration.java:87)
  at winstone.HostGroup.initHost(HostGroup.java:65)
  at winstone.HostGroup.<init>(HostGroup.java:44)
  at winstone.Launcher.<init>(Launcher.java:150)
  at winstone.Launcher.main(Launcher.java:389)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.base/java.lang.reflect.Method.invoke(Method.java:566)
  at executable.Main.main(Main.java:356)
2022-12-12 09:47:16.232+0000 [id=1] SEVERE  winstone.Logger#logInternal: Container startup failed
java.io.FileNotFoundException: /var/lib/jenkins/war/META-INF/MANIFEST.MF (No such file or directory)
  at java.base/java.io.FileOutputStream.open0(Native Method)
  at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)
  at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:237)
  at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:187)
  at winstone.HostConfiguration.getWebRoot(HostConfiguration.java:309)
  at winstone.HostConfiguration.<init>(HostConfiguration.java:87)
  at winstone.HostGroup.initHost(HostGroup.java:65)
  at winstone.HostGroup.<init>(HostGroup.java:44)
  at winstone.Launcher.<init>(Launcher.java:150)
  at winstone.Launcher.main(Launcher.java:389)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.base/java.lang.reflect.Method.invoke(Method.java:566)
  at executable.Main.main(Main.java:356)
╭─root@k201 ~/jenkins-k8s 
╰─#

动态配置jenkins Slave节点

安装kubernetes插件（安装完成后最好重启一下）。配置插件信息系统设置 -> 节点管理 -> Configure Clouds ->增加一个云。

制作Kubernetes证书

进入集群服务器 .kube/config
复制config文件中的certificate-authority-data内容，生成base64文件 ca.crt
复制config文件中的client-certificate-data内容，生成base64文件 client.crt
复制config文件中的client-key-data内容，生成base64文件 client.key

echo zzzzzzzzz | base64 –d > client.key

生成证书（会输入密码需要记住后面jenkins需要配置）

openssl pkcs12 -export -out cert.pfx -inkey client.key -in client.crt -certfile ca.crt

下载证书 cert.pfx

Jenkins新建凭据

注意：这里上传证书文件，通过 openssl 文件生成的 cert.pfx 的文件

配置Salve集群节点

将ca.crt 内容复制到服务证书key 选择证书凭据测试连接

注意：

获取kubenetes地址方式:

╰─# kubectl config view  | grep server 
    server: https://192.168.10.201:6443╭─root@k201 ~/jenkins-k8s 
╰─#

kubernetes 服务证书key为 ca.crt 文件中的内容

创建pipeline

流水线示例

以PodTemplate模板创建脚本

def label = "slave-${UUID.randomUUID().toString()}"podTemplate(cloud: 'kubernetes', namespace:'devops',label: label, serviceAccount: 'jenkins',containers: [  containerTemplate(name: 'golang', image: 'golang:1.14.2-alpine3.11', command: 'cat', ttyEnabled: true),  containerTemplate(name: 'docker', image: 'docker:latest', command: 'cat', ttyEnabled: true),  containerTemplate(name: 'kubectl', image: 'cnych/kubectl', command: 'cat', ttyEnabled: true)
],volumes: [  hostPathVolume(mountPath: '/home/jenkins/.kube', hostPath: '/root/.kube'),  hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock')
]) {  node(label) {    stage('单元测试') {      echo "测试阶段"}    stage('代码编译打包') {      container('golang') {        echo "代码编译打包阶段"}}    stage('构建 Docker 镜像') {      container('docker') {        echo "构建 Docker 镜像阶段"}}    stage('运行 Kubectl') {      container('kubectl') {        echo "查看 K8S 集群 Pod 列表"        sh "kubectl get pods"}}}}

devops-k8s部署jenkins和动态创建slave节点

环境准备

创建命名空间

创建jenkins-master

动态配置jenkins Slave节点

Jenkins新建凭据

配置Salve集群节点

创建pipeline

流水线示例

热门文章

最新文章

相关课程

相关电子书

相关实验场景

推荐镜像