etcdctl的v3版本与v2版本使用命令有所不同，本文介绍etcdctl v3版本的命令工具的使用方式。

1. etcdctl的安装

etcdctl的二进制文件可以在 github.com/coreos/etcd/releases 选择对应的版本下载，例如可以执行以下install_etcdctl.sh的脚本，修改其中的版本信息。

#!/bin/bash
ETCD_VER=v3.3.4
ETCD_DIR=etcd-download
DOWNLOAD_URL=https://github.com/coreos/etcd/releases/download
# Download
mkdir ${ETCD_DIR}
cd ${ETCD_DIR}
wget ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz 
tar -xzvf etcd-${ETCD_VER}-linux-amd64.tar.gz
# install
cd etcd-${ETCD_VER}-linux-amd64
cp etcdctl /usr/local/bin/

2. etcdctl V3

使用etcdctlv3的版本时，需设置环境变量ETCDCTL_API=3。

export ETCDCTL_API=3
或者在`/etc/profile`文件中添加环境变量
vi /etc/profile
...
export ETCDCTL_API=3
...
source /etc/profile

查看当前etcdctl的版本信息etcdctl version。

[root@k8s-dbg-master-1 etcd]# etcdctl version
etcdctl version: 3.3.4
API version: 3.3

更多命令帮助可以查询etcdctl —help。

[root@k8s-dbg-master-1 etcd]# etcdctl --help
NAME:
  etcdctl - A simple command line client for etcd3.
USAGE:
  etcdctl
VERSION:
  3.3.4
API VERSION:
  3.3
COMMANDS:
  get     Gets the key or a range of keys
  put     Puts the given key into the store
  del     Removes the specified key or range of keys [key, range_end)
  txn     Txn processes all the requests in one transaction
  compaction    Compacts the event history in etcd
  alarm disarm    Disarms all alarms
  alarm list    Lists all alarms
  defrag      Defragments the storage of the etcd members with given endpoints
  endpoint health   Checks the healthiness of endpoints specified in `--endpoints` flag
  endpoint status   Prints out the status of endpoints specified in `--endpoints` flag
  endpoint hashkv   Prints the KV history hash for each endpoint in --endpoints
  move-leader   Transfers leadership to another etcd cluster member.
  watch     Watches events stream on keys or prefixes
  version     Prints the version of etcdctl
  lease grant   Creates leases
  lease revoke    Revokes leases
  lease timetolive  Get lease information
  lease list    List all active leases
  lease keep-alive  Keeps leases alive (renew)
  member add    Adds a member into the cluster
  member remove   Removes a member from the cluster
  member update   Updates a member in the cluster
  member list   Lists all members in the cluster
  snapshot save   Stores an etcd node backend snapshot to a given file
  snapshot restore  Restores an etcd member snapshot to an etcd directory
  snapshot status   Gets backend snapshot status of a given file
  make-mirror   Makes a mirror at the destination etcd cluster
  migrate     Migrates keys in a v2 store to a mvcc store
  lock      Acquires a named lock
  elect     Observes and participates in leader election
  auth enable   Enables authentication
  auth disable    Disables authentication
  user add    Adds a new user
  user delete   Deletes a user
  user get    Gets detailed information of a user
  user list   Lists all users
  user passwd   Changes password of user
  user grant-role   Grants a role to a user
  user revoke-role  Revokes a role from a user
  role add    Adds a new role
  role delete   Deletes a role
  role get    Gets detailed information of a role
  role list   Lists all roles
  role grant-permission Grants a key to a role
  role revoke-permission  Revokes a key from a role
  check perf    Check the performance of the etcd cluster
  help      Help about any command
OPTIONS:
      --cacert=""       verify certificates of TLS-enabled secure servers using this CA bundle
      --cert=""         identify secure client using this TLS certificate file
      --command-timeout=5s      timeout for short running command (excluding dial timeout)
      --debug[=false]       enable client-side debug logging
      --dial-timeout=2s       dial timeout for client connections
  -d, --discovery-srv=""      domain name to query for SRV records describing cluster endpoints
      --endpoints=[127.0.0.1:2379]    gRPC endpoints
      --hex[=false]       print byte strings as hex encoded strings
      --insecure-discovery[=true]   accept insecure SRV records describing cluster endpoints
      --insecure-skip-tls-verify[=false]  skip server certificate verification
      --insecure-transport[=true]   disable transport security for client connections
      --keepalive-time=2s     keepalive time for client connections
      --keepalive-timeout=6s      keepalive timeout for client connections
      --key=""          identify secure client using this TLS key file
      --user=""         username[:password] for authentication (prompt if password is not supplied)
  -w, --write-out="simple"      set the output format (fields, json, protobuf, simple, table)

3. etcdctl 常用命令

3.1. 指定etcd集群

HOST_1=10.240.0.17
HOST_2=10.240.0.18
HOST_3=10.240.0.19
ENDPOINTS=$HOST_1:2379,$HOST_2:2379,$HOST_3:2379
etcdctl --endpoints=$ENDPOINTS member list

执行查询时前缀是固定的，如下所示，使用这个前缀再加上etcd的查找命令即可成功查询：

ETCDCTL_API=3 /usr/local/bin/etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
--key=/etc/kubernetes/pki/etcd/healthcheck-client.key

命令行太长了，不容易记住，使用alias命令用来设置指令的别名。

alias etcdctl="ETCDCTL_API=3 /usr/local/bin/etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key"

3.2. 增删改查

1、增

etcdctl --endpoints=$ENDPOINTS put foo "Hello World!"

2、查

etcdctl --endpoints=$ENDPOINTS get foo
etcdctl --endpoints=$ENDPOINTS --write-out="json" get foo
# 查看所有etcd的所有key，执行以下命令
etcdctl get / --prefix --keys-only
# 查看某个POD信息
# 输出信息中，有少量不可见字符，这是因为etcd中存储的并不是json的原文，而是protocol buffer序列化后的数据，不过还是有部分内容是可读的；
etcdctl get /registry/pods/default/nginx-deployment-6b474476c4-qsscn

基于相同前缀查找

etcdctl --endpoints=$ENDPOINTS put web1 value1
etcdctl --endpoints=$ENDPOINTS put web2 value2
etcdctl --endpoints=$ENDPOINTS put web3 value3
etcdctl --endpoints=$ENDPOINTS get web --prefix

列出所有的key

etcdctl --endpoints=$ENDPOINTS get / --prefix --keys-only

3、删

etcdctl --endpoints=$ENDPOINTS put key myvalue
etcdctl --endpoints=$ENDPOINTS del key
etcdctl --endpoints=$ENDPOINTS put k1 value1
etcdctl --endpoints=$ENDPOINTS put k2 value2
etcdctl --endpoints=$ENDPOINTS del k --prefix

3.3. 集群状态

集群状态主要是etcdctl endpoint status 和etcdctl endpoint health两条命令。

etcdctl --write-out=table --endpoints=$ENDPOINTS endpoint status
+------------------+------------------+---------+---------+-----------+-----------+------------+
|     ENDPOINT     |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+------------------+------------------+---------+---------+-----------+-----------+------------+
| 10.240.0.17:2379 | 4917a7ab173fabe7 | 3.0.0   | 45 kB   | true      |         4 |      16726 |
| 10.240.0.18:2379 | 59796ba9cd1bcd72 | 3.0.0   | 45 kB   | false     |         4 |      16726 |
| 10.240.0.19:2379 | 94df724b66343e6c | 3.0.0   | 45 kB   | false     |         4 |      16726 |
+------------------+------------------+---------+---------+-----------+-----------+------------+
etcdctl --endpoints=$ENDPOINTS endpoint health
10.240.0.17:2379 is healthy: successfully committed proposal: took = 3.345431ms
10.240.0.19:2379 is healthy: successfully committed proposal: took = 3.767967ms
10.240.0.18:2379 is healthy: successfully committed proposal: took = 4.025451ms

3.4. 集群成员

跟集群成员相关的命令如下：

  member add        Adds a member into the cluster
  member remove   Removes a member from the cluster
  member update   Updates a member in the cluster
  member list       Lists all members in the cluster

例如 etcdctl member list列出集群成员的命令。

etcdctl --endpoints=http://172.16.5.4:12379 member list -w table
+-----------------+---------+-------+------------------------+-----------------------------------------------+
|       ID        | STATUS  | NAME  |       PEER ADDRS       |                 CLIENT ADDRS                  |
+-----------------+---------+-------+------------------------+-----------------------------------------------+
| c856d92a82ba66a | started | etcd0 | http://172.16.5.4:2380 | http://172.16.5.4:2379,http://172.16.5.4:4001 |
+-----------------+---------+-------+------------------------+-----------------------------------------------+

4. etcdctl get

使用etcdctl {command} --help可以查看具体命令的帮助信息。

# etcdctl get --help
NAME:
  get - Gets the key or a range of keys
USAGE:
  etcdctl get [options] <key> [range_end]
OPTIONS:
      --consistency="l"     Linearizable(l) or Serializable(s)
      --from-key[=false]    Get keys that are greater than or equal to the given key using byte compare
      --keys-only[=false]   Get only the keys
      --limit=0       Maximum number of results
      --order=""      Order of results; ASCEND or DESCEND (ASCEND by default)
      --prefix[=false]      Get keys with matching prefix
      --print-value-only[=false]  Only write values when using the "simple" output format
      --rev=0       Specify the kv revision
      --sort-by=""      Sort target; CREATE, KEY, MODIFY, VALUE, or VERSION
GLOBAL OPTIONS:
      --cacert=""       verify certificates of TLS-enabled secure servers using this CA bundle
      --cert=""         identify secure client using this TLS certificate file
      --command-timeout=5s      timeout for short running command (excluding dial timeout)
      --debug[=false]       enable client-side debug logging
      --dial-timeout=2s       dial timeout for client connections
      --endpoints=[127.0.0.1:2379]    gRPC endpoints
      --hex[=false]       print byte strings as hex encoded strings
      --insecure-skip-tls-verify[=false]  skip server certificate verification
      --insecure-transport[=true]   disable transport security for client connections
      --key=""          identify secure client using this TLS key file
      --user=""         username[:password] for authentication (prompt if password is not supplied)
  -w, --write-out="simple"      set the output format (fields, json, protobuf, simple, table)

数据结构说明

/registry/apiregistration.k8s.io/apiservices/{版本}.{api名称}

包含 Kubernetes 中 API 服务的定义，因此我们可以在其中找到 Kubernetes 使用的所有现有核心 API，例如 /registry/apiregistration.k8s.io/apiservices/v1.batch或 /registry/apiregistration.k8s.io/apiservices/v1beta1.rbac。 authorization.k8s.io或自定义 API 定义（请参阅 https://github.com/kubernetes-incubator/apiserver-builder/blob/master/docs/concepts/aggregation.md）。您可以通过在 Etcd 中读取该键的值（您将获得人类可读的 json）或更友好的方式使用 kubectl get apiservice v1beta1.authorization.k8s.io -o json（与直接值相同）来获取有关 API 的信息等访问）

/registry/clusterroles/{角色名称}

包含 Kubernetes 中所有集群范围角色的定义，因此我们可以在那里找到 /registry/clusterroles/cluster-admin 或 /registry/clusterroles/system:kube-scheduler 之类的内容。Etcd 中的数据是人类可读但难以理解的——我们可以看到一些操作，如获取、修补、更新 API 的某些部分

/registry/clusterrolebindings/{实体名称}

包含集群范围内的角色和用户/组/服务帐户之间的绑定，因此我们可以在那里找到 /registry/clusterrolebindings/cluster-admin或 /registry/clusterrolebindings/kubeadm:kubelet-bootstrap 之类的东西。Etcd 中的数据是人类可读但难以理解的。

/registry/roles/{namespace}/{role name} 和 /registry/rolebindings/{namespace}/{entity name}

与集群角色/绑定中的故事相同，但受命名空间限制，例如 /registry/roles/kube-system/system:controller:token-cleaner

/registry/serviceaccounts/{namespace}/{name}

所有服务帐户的定义

/registry/configmaps/{命名空间}/{地图名称}

所有配置映射存储为 yamls

/registry/controllerrevisions/{namespace}/{pod}

我发现 ControllerRevision 资源用于在 DaemonSet 和 StatefulSet ( https://kubernetes.io/docs/tasks/manage-daemon/rollback-daemon-set/ ) 中提供回滚可能性。在 Etcd 中，我们可以找到 pod 规范的快照。

/registry/daemonsets/{namespace}/{name} 和 /registry/deployments/{namespace}/{name} 等。

在这些键下，Kubernetes 存储有关不同部署的信息，例如 DaemonSet、Deployment、ReplicaSet、Job 等。在部署的情况下有趣的是，我们看到那里描述了 last-applied-configuration https://kubernetes.io/docs/concepts/overview/对象管理-kubectl/declarative-config/#merge-patch-calculation

/registry/minions/{节点名称}

Kubernetes 节点以前被称为“minions”，所以在 Etcd 中的名称仍然没有改变。我们看到有大量数据描述节点，例如：

CPU内核

内存大小

kubelet 的状态：例如 kubelet 有足够的可用磁盘空间或 kubelet 有足够的 PID 可用

IP地址

主机名

Docker 版本

Docker image /registry/ranges/servicenodeportss 在节点上可用

/registry/namespaces/{namespace}

只是定义命名空间。还有特定命名空间的状态，如 Active 或 Terminating。

/registry/pods/{namespace}/{pod 名称}

集群中运行的每个 pod 的状态。包含很多信息，如 pod IP、挂载的卷、docker 映像等。

/registry/ranges/serviceips

服务的 CIDR

/registry/ranges/servicenodeports

暴露服务的端口范围

/registry/secrets/{namespace}/{pod}

集群中的所有秘密都以默认模式存储为纯文本。有关加密，请参阅 https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/

/registry/services/endpoints/{namespace}/{name}

服务定义。Kubernetes 计算特定服务选择了哪些 Pod，并将该信息存储在服务值中，以便我们可以在那里看到 Pod 的 IP 地址和名称。

文章参考：

https://coreos.com/etcd/docs/latest/demo.html

https://jakubbujny.wordpress.com/2018/09/02/what-stores-kubernetes-in-etcd/

Kubernetes：etcdctl3的常用命令