三年前刚刚接触 kubernetes 不久，搭建的线上交易会平台装的 kubernetes 集群使用的是 kubeadm。每年都会做一次集群升级、刷新许可证，今天再做一次升级。
一、升级 kubeadm
1. 查看 kubeadm 版本
[root@xxx install]# kubeadm version kubeadm version: &version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.16", GitCommit:"7a98bb2b7c9112935387825f2fce1b7d40b76236", GitTreeState:"clean", BuildDate:"2021-02-17T11:58:49Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
2. 列出 kubeadm 版本
[root@xxxxx install]# yum list --showduplicates kubeadm --disableexcludes=kubernetes Loaded plugins: fastestmirror, langpacks Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast Determining fastest mirrors Installed Packages kubeadm.x86_64 1.18.16-0 @kubernetes Available Packages kubeadm.x86_64 1.17.0-0 kubernetes kubeadm.x86_64 1.17.1-0 kubernetes kubeadm.x86_64 1.17.2-0 kubernetes kubeadm.x86_64 1.17.3-0 kubernetes kubeadm.x86_64 1.17.4-0 kubernetes kubeadm.x86_64 1.17.5-0 kubernetes kubeadm.x86_64 1.17.6-0 kubernetes kubeadm.x86_64 1.17.7-0 kubernetes kubeadm.x86_64 1.17.7-1 kubernetes kubeadm.x86_64 1.17.8-0 kubernetes kubeadm.x86_64 1.17.9-0 kubernetes kubeadm.x86_64 1.17.11-0 kubernetes kubeadm.x86_64 1.17.12-0 kubernetes kubeadm.x86_64 1.17.13-0 kubernetes kubeadm.x86_64 1.17.14-0 kubernetes kubeadm.x86_64 1.17.15-0 kubernetes kubeadm.x86_64 1.17.16-0 kubernetes kubeadm.x86_64 1.17.17-0 kubernetes kubeadm.x86_64 1.18.0-0 kubernetes kubeadm.x86_64 1.18.1-0 kubernetes kubeadm.x86_64 1.18.2-0 kubernetes kubeadm.x86_64 1.18.3-0 kubernetes kubeadm.x86_64 1.18.4-0 kubernetes kubeadm.x86_64 1.18.4-1 kubernetes kubeadm.x86_64 1.18.5-0 kubernetes kubeadm.x86_64 1.18.6-0 kubernetes kubeadm.x86_64 1.18.8-0 kubernetes kubeadm.x86_64 1.18.9-0 kubernetes kubeadm.x86_64 1.18.10-0 kubernetes kubeadm.x86_64 1.18.12-0 kubernetes kubeadm.x86_64 1.18.13-0 kubernetes kubeadm.x86_64 1.18.14-0 kubernetes kubeadm.x86_64 1.18.15-0 kubernetes kubeadm.x86_64 1.18.16-0 kubernetes kubeadm.x86_64 1.18.17-0 kubernetes kubeadm.x86_64 1.18.18-0 kubernetes kubeadm.x86_64 1.18.19-0 kubernetes kubeadm.x86_64 1.18.20-0 kubernetes kubeadm.x86_64 1.19.0-0 kubernetes kubeadm.x86_64 1.19.1-0 kubernetes kubeadm.x86_64 1.19.2-0 kubernetes kubeadm.x86_64 1.19.3-0 kubernetes kubeadm.x86_64 1.19.4-0 kubernetes kubeadm.x86_64 1.19.5-0 kubernetes kubeadm.x86_64 1.19.6-0 kubernetes kubeadm.x86_64 1.19.7-0 kubernetes kubeadm.x86_64 1.19.8-0 kubernetes kubeadm.x86_64 1.19.9-0 kubernetes kubeadm.x86_64 1.19.10-0 kubernetes kubeadm.x86_64 1.19.11-0 kubernetes kubeadm.x86_64 1.19.12-0 kubernetes kubeadm.x86_64 1.19.13-0 kubernetes kubeadm.x86_64 1.19.14-0 kubernetes kubeadm.x86_64 1.19.15-0 kubernetes kubeadm.x86_64 1.19.16-0 kubernetes kubeadm.x86_64 1.20.0-0 kubernetes kubeadm.x86_64 1.20.1-0 kubernetes kubeadm.x86_64 1.20.2-0 kubernetes kubeadm.x86_64 1.20.4-0 kubernetes kubeadm.x86_64 1.20.5-0 kubernetes kubeadm.x86_64 1.20.6-0 kubernetes kubeadm.x86_64 1.20.7-0 kubernetes kubeadm.x86_64 1.20.8-0 kubernetes kubeadm.x86_64 1.20.9-0 kubernetes kubeadm.x86_64 1.20.10-0 kubernetes kubeadm.x86_64 1.20.11-0 kubernetes kubeadm.x86_64 1.20.12-0 kubernetes kubeadm.x86_64 1.20.13-0 kubernetes kubeadm.x86_64 1.20.14-0 kubernetes kubeadm.x86_64 1.20.15-0 kubernetes kubeadm.x86_64 1.21.0-0 kubernetes kubeadm.x86_64 1.21.1-0 kubernetes kubeadm.x86_64 1.21.2-0 kubernetes kubeadm.x86_64 1.21.3-0 kubernetes kubeadm.x86_64 1.21.4-0 kubernetes kubeadm.x86_64 1.21.5-0 kubernetes kubeadm.x86_64 1.21.6-0 kubernetes kubeadm.x86_64 1.21.7-0 kubernetes kubeadm.x86_64 1.21.8-0 kubernetes kubeadm.x86_64 1.21.9-0 kubernetes kubeadm.x86_64 1.21.10-0 kubernetes kubeadm.x86_64 1.21.11-0 kubernetes kubeadm.x86_64 1.21.12-0 kubernetes kubeadm.x86_64 1.21.13-0 kubernetes kubeadm.x86_64 1.21.14-0 kubernetes kubeadm.x86_64 1.22.0-0 kubernetes kubeadm.x86_64 1.22.1-0 kubernetes kubeadm.x86_64 1.22.2-0 kubernetes kubeadm.x86_64 1.22.3-0 kubernetes kubeadm.x86_64 1.22.4-0 kubernetes kubeadm.x86_64 1.22.5-0 kubernetes kubeadm.x86_64 1.22.6-0 kubernetes kubeadm.x86_64 1.22.7-0 kubernetes kubeadm.x86_64 1.22.8-0 kubernetes kubeadm.x86_64 1.22.9-0 kubernetes kubeadm.x86_64 1.22.10-0 kubernetes kubeadm.x86_64 1.22.11-0 kubernetes kubeadm.x86_64 1.22.12-0 kubernetes kubeadm.x86_64 1.22.13-0 kubernetes kubeadm.x86_64 1.22.14-0 kubernetes kubeadm.x86_64 1.22.15-0 kubernetes kubeadm.x86_64 1.23.0-0 kubernetes kubeadm.x86_64 1.23.1-0 kubernetes kubeadm.x86_64 1.23.2-0 kubernetes kubeadm.x86_64 1.23.3-0 kubernetes kubeadm.x86_64 1.23.4-0 kubernetes kubeadm.x86_64 1.23.5-0 kubernetes kubeadm.x86_64 1.23.6-0 kubernetes kubeadm.x86_64 1.23.7-0 kubernetes kubeadm.x86_64 1.23.8-0 kubernetes kubeadm.x86_64 1.23.9-0 kubernetes kubeadm.x86_64 1.23.10-0 kubernetes kubeadm.x86_64 1.23.11-0 kubernetes kubeadm.x86_64 1.23.12-0 kubernetes kubeadm.x86_64 1.23.13-0 kubernetes kubeadm.x86_64 1.24.0-0 kubernetes kubeadm.x86_64 1.24.1-0 kubernetes kubeadm.x86_64 1.24.2-0 kubernetes kubeadm.x86_64 1.24.3-0 kubernetes kubeadm.x86_64 1.24.4-0 kubernetes kubeadm.x86_64 1.24.5-0 kubernetes kubeadm.x86_64 1.24.6-0 kubernetes kubeadm.x86_64 1.24.7-0 kubernetes kubeadm.x86_64 1.25.0-0 kubernetes kubeadm.x86_64 1.25.1-0 kubernetes kubeadm.x86_64 1.25.2-0 kubernetes kubeadm.x86_64 1.25.3-0 kubernetes
3. 升级 kubeadm
注意：kubeadm 是不支持跨版本的升级的，所以我当前是1.18 版本，我升级时最高只能升级到1.19版本
[root@xxxxx install]# yum install -y kubeadm-1.19.16-0 --disableexcludes=kubernetes Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile Resolving Dependencies --> Running transaction check ---> Package kubeadm.x86_64 0:1.18.16-0 will be updated ---> Package kubeadm.x86_64 0:1.19.16-0 will be an update --> Processing Dependency: kubelet >= 1.19.0 for package: kubeadm-1.19.16-0.x86_64 --> Processing Dependency: kubectl >= 1.19.0 for package: kubeadm-1.19.16-0.x86_64 --> Processing Dependency: cri-tools >= 1.19.0 for package: kubeadm-1.19.16-0.x86_64 --> Running transaction check ---> Package cri-tools.x86_64 0:1.13.0-0 will be updated ---> Package cri-tools.x86_64 0:1.26.0-0 will be an update ---> Package kubectl.x86_64 0:1.17.1-0 will be updated ---> Package kubectl.x86_64 0:1.26.1-0 will be an update ---> Package kubelet.x86_64 0:1.17.1-0 will be updated ---> Package kubelet.x86_64 0:1.26.1-0 will be an update --> Processing Dependency: kubernetes-cni >= 1.2.0 for package: kubelet-1.26.1-0.x86_64 --> Running transaction check ---> Package kubernetes-cni.x86_64 0:0.8.7-0 will be updated ---> Package kubernetes-cni.x86_64 0:1.2.0-0 will be an update --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================================= Package Arch Version Repository Size ============================================================================================================================================================================================= Updating: kubeadm x86_64 1.19.16-0 kubernetes 8.3 M Updating for dependencies: cri-tools x86_64 1.26.0-0 kubernetes 8.6 M kubectl x86_64 1.26.1-0 kubernetes 11 M kubelet x86_64 1.26.1-0 kubernetes 22 M kubernetes-cni x86_64 1.2.0-0 kubernetes 17 M Transaction Summary ============================================================================================================================================================================================= Upgrade 1 Package (+4 Dependent packages) Total download size: 66 M Downloading packages: Delta RPMs disabled because /usr/bin/applydeltarpm not installed. (1/5): 6e8fc5b12b06a19517237776f07bf8f171fbfec1e0345232ea945264d84790c3-kubeadm-1.19.16-0.x86_64.rpm | 8.3 MB 00:01:13 (2/5): 3f5ba2b53701ac9102ea7c7ab2ca6616a8cd5966591a77577585fde1c434ef74-cri-tools-1.26.0-0.x86_64.rpm | 8.6 MB 00:01:15 (3/5): 7c5ee9df7097fe780a8fd2e87541d5c4dba86120a96aec5eb4c9517ee88148ee-kubectl-1.26.1-0.x86_64.rpm | 11 MB 00:01:33 (4/5): 2dcb121663166d78efad52d20fcbdc6f23fe67665d319930905a3e722e05ec30-kubelet-1.26.1-0.x86_64.rpm | 22 MB 00:03:11 (5/5): 0f2a2afd740d476ad77c508847bad1f559afc2425816c1f2ce4432a62dfe0b9d-kubernetes-cni-1.2.0-0.x86_64.rpm | 17 MB 00:02:27 --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 216 kB/s | 66 MB 00:05:13 Running transaction check Running transaction test Transaction test succeeded Running transaction Updating : kubelet-1.26.1-0.x86_64 1/10 Updating : kubernetes-cni-1.2.0-0.x86_64 2/10 Updating : cri-tools-1.26.0-0.x86_64 3/10 Updating : kubectl-1.26.1-0.x86_64 4/10 Updating : kubeadm-1.19.16-0.x86_64 5/10 Cleanup : kubeadm-1.18.16-0.x86_64 6/10 Cleanup : kubelet-1.17.1-0.x86_64 7/10 Cleanup : kubernetes-cni-0.8.7-0.x86_64 8/10 Cleanup : cri-tools-1.13.0-0.x86_64 9/10 Cleanup : kubectl-1.17.1-0.x86_64 10/10 Verifying : kubectl-1.26.1-0.x86_64 1/10 Verifying : cri-tools-1.26.0-0.x86_64 2/10 Verifying : kubernetes-cni-1.2.0-0.x86_64 3/10 Verifying : kubeadm-1.19.16-0.x86_64 4/10 Verifying : kubelet-1.26.1-0.x86_64 5/10 Verifying : kubernetes-cni-0.8.7-0.x86_64 6/10 Verifying : kubectl-1.17.1-0.x86_64 7/10 Verifying : cri-tools-1.13.0-0.x86_64 8/10 Verifying : kubeadm-1.18.16-0.x86_64 9/10 Verifying : kubelet-1.17.1-0.x86_64 10/10 Updated: kubeadm.x86_64 0:1.19.16-0 Dependency Updated: cri-tools.x86_64 0:1.26.0-0 kubectl.x86_64 0:1.26.1-0 kubelet.x86_64 0:1.26.1-0 kubernetes-cni.x86_64 0:1.2.0-0 Complete!
4. 验证升级
[root@xxxxx install]# kubeadm version kubeadm version: &version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.16", GitCommit:"e37e4ab4cc8dcda84f1344dda47a97bb1927d074", GitTreeState:"clean", BuildDate:"2021-10-27T16:24:44Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
- 可以看到，这里已经成功从1.18版本升级到了1.19版本
二、升级集群
1. 列出升级计划
[root@xxxxx install]# kubeadm upgrade plan [upgrade/config] Making sure the configuration is correct: [upgrade/config] Reading configuration from the cluster... [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [preflight] Running pre-flight checks. [upgrade] Running cluster health checks [upgrade] Fetching available versions to upgrade to [upgrade/versions] Cluster version: v1.18.16 [upgrade/versions] kubeadm version: v1.19.16 I0120 17:01:26.289546 19651 version.go:255] remote version is much newer: v1.26.1; falling back to: stable-1.19 [upgrade/versions] Latest stable version: v1.19.16 [upgrade/versions] Latest stable version: v1.19.16 [upgrade/versions] Latest version in the v1.18 series: v1.18.20 [upgrade/versions] Latest version in the v1.18 series: v1.18.20 Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': COMPONENT CURRENT AVAILABLE kubelet 2 x v1.17.1 v1.18.20 Upgrade to the latest version in the v1.18 series: COMPONENT CURRENT AVAILABLE kube-apiserver v1.18.16 v1.18.20 kube-controller-manager v1.18.16 v1.18.20 kube-scheduler v1.18.16 v1.18.20 kube-proxy v1.18.16 v1.18.20 CoreDNS 1.6.7 1.7.0 etcd 3.4.3-0 3.4.3-0 You can now apply the upgrade by executing the following command: kubeadm upgrade apply v1.18.20 _____________________________________________________________________ Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': COMPONENT CURRENT AVAILABLE kubelet 2 x v1.17.1 v1.19.16 Upgrade to the latest stable version: COMPONENT CURRENT AVAILABLE kube-apiserver v1.18.16 v1.19.16 kube-controller-manager v1.18.16 v1.19.16 kube-scheduler v1.18.16 v1.19.16 kube-proxy v1.18.16 v1.19.16 CoreDNS 1.6.7 1.7.0 etcd 3.4.3-0 3.4.13-0 You can now apply the upgrade by executing the following command: kubeadm upgrade apply v1.19.16 _____________________________________________________________________ The table below shows the current state of component configs as understood by this version of kubeadm. Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually upgrade to is denoted in the "PREFERRED VERSION" column. API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED kubeproxy.config.k8s.io v1alpha1 v1alpha1 no kubelet.config.k8s.io v1beta1 v1beta1 no _____________________________________________________________________
2. 升级kubernetes 集群
[root@xxxxx install]# kubeadm upgrade apply v1.19.16 --force [upgrade/config] Making sure the configuration is correct: [upgrade/config] Reading configuration from the cluster... [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [preflight] Running pre-flight checks. [upgrade] Running cluster health checks [upgrade/version] You have chosen to change the cluster version to "v1.19.16" [upgrade/versions] Cluster version: v1.18.16 [upgrade/versions] kubeadm version: v1.19.16 [upgrade/version] Found 1 potential version compatibility errors but skipping since the --force flag is set: - There are kubelets in this cluster that are too old that have these versions [v1.17.1] [upgrade/prepull] Pulling images required for setting up a Kubernetes cluster [upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection [upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull' [upgrade/apply] Upgrading your Static Pod-hosted control plane to version "v1.19.16"... Static pod: kube-apiserver-xxxxxx hash: 797734e8767dae171f91b69b7666c850 Static pod: kube-controller-manager-xxxxx hash: 883239f813b1769b1503caa752eee48c Static pod: kube-scheduler-xxxxx hash: 05c41878f5fac1c1d0cf49da86047e1d [upgrade/etcd] Upgrading to TLS for etcd Static pod: etcd-xxxx hash: 1ffc829a4d3541b26460c8d579426907 [upgrade/staticpods] Preparing for "etcd" upgrade [upgrade/staticpods] Renewing etcd-server certificate [upgrade/staticpods] Renewing etcd-peer certificate [upgrade/staticpods] Renewing etcd-healthcheck-client certificate [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-01-20-17-06-51/etcd.yaml" [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) Static pod: etcd-xxxxx hash: 1ffc829a4d3541b26460c8d579426907 Static pod: etcd-xxxx hash: f5f7eff9bafb27bc6001310fd32d88db [apiclient] Found 1 Pods for label selector component=etcd [upgrade/staticpods] Component "etcd" upgraded successfully! [upgrade/etcd] Waiting for etcd to become available [upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests086047809" [upgrade/staticpods] Preparing for "kube-apiserver" upgrade [upgrade/staticpods] Renewing apiserver certificate [upgrade/staticpods] Renewing apiserver-kubelet-client certificate [upgrade/staticpods] Renewing front-proxy-client certificate [upgrade/staticpods] Renewing apiserver-etcd-client certificate [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-01-20-17-06-51/kube-apiserver.yaml" [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) Static pod: kube-apiserver-xxxx hash: 797734e8767dae171f91b69b7666c850 Static pod: kube-apiserver-xxxxx hash: 797734e8767dae171f91b69b7666c850 Static pod: kube-apiserver-xxxx hash: 797734e8767dae171f91b69b7666c850 Static pod: kube-apiserver-xxxx hash: bb40b51157c0de652b015900b3f618bd [apiclient] Found 1 Pods for label selector component=kube-apiserver [upgrade/staticpods] Component "kube-apiserver" upgraded successfully! [upgrade/staticpods] Preparing for "kube-controller-manager" upgrade [upgrade/staticpods] Renewing controller-manager.conf certificate [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-01-20-17-06-51/kube-controller-manager.yaml" [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) Static pod: kube-controller-manager-xxxx hash: 883239f813b1769b1503caa752eee48c Static pod: kube-controller-manager-xxxx hash: 2b1b8618c20a834b73cde0592827bf54 [apiclient] Found 1 Pods for label selector component=kube-controller-manager [upgrade/staticpods] Component "kube-controller-manager" upgraded successfully! [upgrade/staticpods] Preparing for "kube-scheduler" upgrade [upgrade/staticpods] Renewing scheduler.conf certificate [upgrade/staticpods] Moved new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backed up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2023-01-20-17-06-51/kube-scheduler.yaml" [upgrade/staticpods] Waiting for the kubelet to restart the component [upgrade/staticpods] This might take a minute or longer depending on the component/version gap (timeout 5m0s) Static pod: kube-scheduler-xxxxx hash: 05c41878f5fac1c1d0cf49da86047e1d Static pod: kube-scheduler-xxxx hash: fb04d3c61e9503154c23197a2d7e8243 [apiclient] Found 1 Pods for label selector component=kube-scheduler [upgrade/staticpods] Component "kube-scheduler" upgraded successfully! [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.19" in namespace kube-system with the configuration for the kubelets in the cluster [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.19.16". Enjoy! [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
3. 验证升级
[root@xxxx install]# kubectl version --output=json { "clientVersion": { "major": "1", "minor": "26", "gitVersion": "v1.26.1", "gitCommit": "8f94681cd294aa8cfd3407b8191f6c70214973a4", "gitTreeState": "clean", "buildDate": "2023-01-18T15:58:16Z", "goVersion": "go1.19.5", "compiler": "gc", "platform": "linux/amd64" }, "kustomizeVersion": "v4.5.7", "serverVersion": { "major": "1", "minor": "19", "gitVersion": "v1.19.16", "gitCommit": "e37e4ab4cc8dcda84f1344dda47a97bb1927d074", "gitTreeState": "clean", "buildDate": "2021-10-27T16:20:18Z", "goVersion": "go1.15.15", "compiler": "gc", "platform": "linux/amd64" } }
- 可以看到kubernetes已经升级到1.19版本了。
三、命令概要
kubeadm version yum list --showduplicates kubeadm --disableexcludes=kubernetes yum install -y kubeadm-${yournextversion} --disableexcludes=kubernetes kubeadm version kubeadm upgrade plan kubeadm upgrade apply ${yournextversion} --force kubectl version --output=json
