备注:
- sonarqube默认用户名密码: admin/admin
- 卸载命令：docker-compose -f jenkins-compose.yml down -v
六、Jenkins自动打包部署配置
项目部署有多种方式，从最原始的可运行jar包直接部署到JDK环境下运行，到将可运行的jar包放到docker容器中运行，再到现在比较流行的把可运行的jar包和docker放到k8s的pod环境中运行。每一种新的部署方式都是对原有部署方式的改进和优化，这里不着重介绍每种方式的优缺点，只简单说明一下使用Kubernetes 的原因：Kubernetes 主要提供弹性伸缩、服务发现、自我修复，版本回退、负载均衡、存储编排等功能。
日常开发部署过程中的基本步骤如下：
- 提交代码到gitlab代码仓库
- gitlab通过webhook触发Jenkins构建代码质量检查
- Jenkins需通过手动触发，来拉取代码、编译、打包、构建Docker镜像、发布到私有镜像仓库Harbor、执行kubectl命令从Harbor拉取Docker镜像部署至k8s
1、安装Kubernetes plugin插件、Git Parameter插件（用于流水线参数化构建）、
插件（用于多个微服务时，选择需要构建的微服务）、 Pipeline Utility Steps插件（用于读取maven工程的.yaml、pom.xml等）和 Kubernetes Continuous Deploy（一定要使用1.0版本，从官网下载然后上传） ，Jenkins --> 系统管理 --> 插件管理 --> 可选插件 --> Kubernetes plugin /Git Parameter/Extended Choice Parameter ,选中后点击Install without restart按钮进行安装
Kubernetes plugin
Extended Choice Parameter
image.png
Git Parameter
Blueocean目前还不支持Git Parameter插件和Extended Choice Parameter插件，Git Parameter是通过Git Plugin读取分支信息，我们这里使用Pipeline script而不是使用Pipeline script from SCM，是因为我们不希望把构建信息放到代码里，这样做可以开发和部署分离。
2、配置Kubernetes plugin插件，Jenkins --> 系统管理 --> 节点管理 --> Configure Clouds --> Add a new cloud -> Kubernetes
2dbd8ea1886ae30659926345724bb1b.png
3、增加kubernetes证书
cat ~/.kube/config # 以下步骤暂不使用，将certificate-authority-data、client-certificate-data、client-key-data替换为~/.kube/config里面具体的值 #echo certificate-authority-data | base64 -d > ca.crt #echo client-certificate-data | base64 -d > client.crt #echo client-key-data | base64 -d > client.key # 执行以下命令，自己设置密码 #openssl pkcs12 -export -out cert.pfx -inkey client.key -in client.crt -certfile ca.crt
系统管理-->凭据-->系统-->全局凭据
image.png
4、添加访问Kubernetes的凭据信息，这里填入上面登录Kubernetes Dashboard所创建的token即可，添加完成之后选择刚刚添加的凭据，然后点击连接测试，如果提示连接成功，那么说明我们的Jenkins可以连接Kubernetes了
设置token
连接测试
5、jenkins全局配置jdk、git和maven
jenkinsci/blueocean镜像默认安装了jdk和git，这里需要登录容器找到路径，然后配置进去。
通过命令进入jenkins容器，并查看JAVA_HOEM和git路径
[root@localhost ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 0520ebb9cc5d jenkinsci/blueocean "/sbin/tini -- /usr/…" 2 days ago Up 30 hours 50000/tcp, 0.0.0.0:18080->8080/tcp, :::18080->8080/tcp root-jenkins-1 [root@localhost ~]# docker exec -it 0520ebb9cc5d /bin/bash bash-5.1# echo $JAVA_HOME /opt/java/openjdk bash-5.1# which git /usr/bin/git
通过命令查询可知，JAVA_HOME=/opt/java/openjdk GIT= /usr/bin/git ， 在Jenkins全局工具配置中配置
image.png
Maven可以在宿主机映射的/data/docker/ci/jenkins/home中安装，然后配置时，配置容器路径为/var/jenkins_home下的Maven安装路径
image.png
在系统配置中设置MAVEN_HOME供Pipeline script调用，如果执行脚本时提示没有权限，那么在宿主Maven目录的bin目录下执行chmod 777 *
image.png
6、为k8s新建harbor-key，用于k8s拉取私服镜像，配置在代码的k8s-deployment.yml中使用。
kubectl create secret docker-registry harbor-key --docker-server=172.16.20.175 --docker-username='robot$gitegg' --docker-password='Jqazyv7vvZiL6TXuNcv7TrZeRdL8U9n3'
7、新建pipeline流水线任务
新建pipeline
8、配置流水线任务参数
172.16.20.175_18080_job_gitegg-cloud_param.png
9、配置pipeline发布脚本
在流水线下面选择Pipeline script
image.png
pipeline { agent any parameters { gitParameter branchFilter: 'origin/(.*)', defaultValue: 'master', name: 'Branch', type: 'PT_BRANCH', description:'请选择需要构建的代码分支' choice(name: 'BaseImage', choices: ['openjdk:8-jdk-alpine'], description: '请选择基础运行环境') choice(name: 'Environment', choices: ['dev','test','prod'],description: '请选择要发布的环境：dev开发环境、test测试环境、prod 生产环境') extendedChoice( defaultValue: 'gitegg-gateway,gitegg-oauth,gitegg-plugin/gitegg-code-generator,gitegg-service/gitegg-service-base,gitegg-service/gitegg-service-extension,gitegg-service/gitegg-service-system', description: '请选择需要构建的微服务', multiSelectDelimiter: ',', name: 'ServicesBuild', quoteValue: false, saveJSONParameterToFile: false, type: 'PT_CHECKBOX', value:'gitegg-gateway,gitegg-oauth,gitegg-plugin/gitegg-code-generator,gitegg-service/gitegg-service-base,gitegg-service/gitegg-service-extension,gitegg-service/gitegg-service-system', visibleItemCount: 6) string(name: 'BuildParameter', defaultValue: 'none', description: '请输入构建参数') } environment { PRO_NAME = "gitegg" BuildParameter="${params.BuildParameter}" ENV = "${params.Environment}" BRANCH = "${params.Branch}" ServicesBuild = "${params.ServicesBuild}" BaseImage="${params.BaseImage}" k8s_token = "7696144b-3b77-4588-beb0-db4d585f5c04" } stages { stage('Clean workspace') { steps { deleteDir() } } stage('Process parameters') { steps { script { if("${params.ServicesBuild}".trim() != "") { def ServicesBuildString = "${params.ServicesBuild}" ServicesBuild = ServicesBuildString.split(",") for (service in ServicesBuild) { println "now got ${service}" } } if("${params.BuildParameter}".trim() != "" && "${params.BuildParameter}".trim() != "none") { BuildParameter = "${params.BuildParameter}" } else { BuildParameter = "" } } } } stage('Pull SourceCode Platform') { steps { echo "${BRANCH}" git branch: "${Branch}", credentialsId: 'gitlabTest', url: 'http://172.16.20.188:2080/root/gitegg-platform.git' } } stage('Install Platform') { steps{ echo "==============Start Platform Build==========" sh "${MAVEN_HOME}/bin/mvn -DskipTests=true clean install ${BuildParameter}" echo "==============End Platform Build==========" } } stage('Pull SourceCode') { steps { echo "${BRANCH}" git branch: "${Branch}", credentialsId: 'gitlabTest', url: 'http://172.16.20.188:2080/root/gitegg-cloud.git' } } stage('Build') { steps{ script { echo "==============Start Cloud Parent Install==========" sh "${MAVEN_HOME}/bin/mvn -DskipTests=true clean install -P${params.Environment} ${BuildParameter}" echo "==============End Cloud Parent Install==========" def workspace = pwd() for (service in ServicesBuild) { stage ('buildCloud${service}') { echo "==============Start Cloud Build ${service}==========" sh "cd ${workspace}/${service} && ${MAVEN_HOME}/bin/mvn -DskipTests=true clean package -P${params.Environment} ${BuildParameter} jib:build -Djib.httpTimeout=200000 -DsendCredentialsOverHttp=true -f pom.xml" echo "==============End Cloud Build ${service}============" } } } } } stage('Sync to k8s') { steps { script { echo "==============Start Sync to k8s==========" def workspace = pwd() mainpom = readMavenPom file: 'pom.xml' profiles = mainpom.getProfiles() def version = mainpom.getVersion() def nacosAddr = "" def nacosConfigPrefix = "" def nacosConfigGroup = "" def dockerHarborAddr = "" def dockerHarborProject = "" def dockerHarborUsername = "" def dockerHarborPassword = "" def serverPort = "" def commonDeployment = "${workspace}/k8s-deployment.yaml" for(profile in profiles) { // 获取对应配置 if (profile.getId() == "${params.Environment}") { nacosAddr = profile.getProperties().getProperty("nacos.addr") nacosConfigPrefix = profile.getProperties().getProperty("nacos.config.prefix") nacosConfigGroup = profile.getProperties().getProperty("nacos.config.group") dockerHarborAddr = profile.getProperties().getProperty("docker.harbor.addr") dockerHarborProject = profile.getProperties().getProperty("docker.harbor.project") dockerHarborUsername = profile.getProperties().getProperty("docker.harbor.username") dockerHarborPassword = profile.getProperties().getProperty("docker.harbor.password") } } for (service in ServicesBuild) { stage ('Sync${service}ToK8s') { echo "==============Start Sync ${service} to k8s==========" dir("${workspace}/${service}") { pom = readMavenPom file: 'pom.xml' echo "group: artifactId: ${pom.artifactId}" def deployYaml = "k8s-deployment-${pom.artifactId}.yaml" yaml = readYaml file : './src/main/resources/bootstrap.yml' serverPort = "${yaml.server.port}" if(fileExists("${workspace}/${service}/k8s-deployment.yaml")){ commonDeployment = "${workspace}/${service}/k8s-deployment.yaml" } else { commonDeployment = "${workspace}/k8s-deployment.yaml" } script { sh "sed 's#{APP_NAME}#${pom.artifactId}#g;s#{IMAGE_URL}#${dockerHarborAddr}#g;s#{IMAGE_PROGECT}#${PRO_NAME}#g;s#{IMAGE_TAG}#${version}#g;s#{APP_PORT}#${serverPort}#g;s#{SPRING_PROFILE}#${params.Environment}#g' ${commonDeployment} > ${deployYaml}" kubernetesDeploy configs: "${deployYaml}", kubeconfigId: "${k8s_token}" } } echo "==============End Sync ${service} to k8s==========" } } echo "==============End Sync to k8s==========" } } } } }
常见问题：
1、Pipeline Utility Steps 第一次执行会报错Scripts not permitted to use method或者Scripts not permitted to use staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods getProperties java.lang.Object
解决：系统管理-->In-process Script Approval->点击 Approval
a5cd8534ee59d1b6018941947f4c2fc.png
a7060adfc34ff1a73e685d4436a3e81.png
2、通过NFS服务将所有容器的日志统一存放在NFS的服务端
3、Kubernetes Continuous Deploy,使用1.0.0版本，否则报错，不兼容
4、解决docker注册到内网问题
spring: cloud: inetutils: ignored-interfaces: docker0
5、配置ipvs模式，kube-proxy监控Pod的变化并创建相应的ipvs规则。ipvs相对iptables转发效率更高。除此以外，ipvs支持更多的LB算法。
kubectl edit cm kube-proxy -n kube-system
修改mode: "ipvs"
image.png
重新加载kube-proxy配置文件
kubectl delete pod -l k8s-app=kube-proxy -n kube-system
查看ipvs规则
ipvsadm -Ln
6、k8s集群内部访问外部服务，nacos，redis等
- a、内外互通模式，在部署的服务设置hostNetwork: true
spec: hostNetwork: true
- b、Endpoints模式
kind: Endpoints apiVersion: v1 metadata: name: nacos namespace: default subsets: - addresses: - ip: 172.16.20.188 ports: - port: 8848
apiVersion: v1 kind: Service metadata: name: nacos namespace: default spec: type: ClusterIP ports: - port: 8848 targetPort: 8848 protocol: TCP
- c、service的type: ExternalName模式,“ExternalName” 使用 CNAME 重定向，因此无法执行端口重映射，域名使用
EndPoints和type: ExternalName
以上外部新建yaml，不要用内部的，这些需要在环境设置时配置好。
7、k8s常用命令：
查看pod: kubectl get pods
查看service: kubectl get svc
查看endpoints: kubectl get endpoints
安装： kubectl apply -f XXX.yaml
删除：kubectl delete -f xxx.yaml
删除pod: kubectl delete pod podName
删除service: kubectl delete service serviceName
进入容器： kubectl exec -it podsNamexxxxxx -n default -- /bin/sh
