kubernetes 1.8 高可用安装（三）

3、master 组件安装（etcd/api-server/controller/scheduler）

3.1 etcd集群安装

确定你要安装的master机器， 上面安装rpm包，配置kubelet

注意：

所有的image，我都已经放到docker hub仓库，需要的可以去下载

安装rpm包

yum localinstall -y kubectl-1.8.0-1.x86_64.rpm kubelet-1.8.0-1.x86_64.rpm kubernetes-cni-0.5.1-1.x86_64.rpm

创建manitest目录

mkdir -p /etc/kubernetes/manifests

修改kubelet配置
/etc/systemd/system/kubelet.service.d/kubelet.conf

[Service]
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.12 --cluster-domain=cluster.local"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs"
Environment="KUBELET_EXTRA_ARGS=--v=2  --pod-infra-container-image=foxchan/google_containers/pause-amd64:3.0 --fail-swap-on=false"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS  $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS

注意：

--cluster-dns=10.96.0.12    这个IP自己规划，记得和创建证书时候的IP段保持一致

--fail-swap-on=false             1.8开始，如果机器开启了swap，kubulet会无法启动，默认参数是true

启动kubelet

systemctl daemon-reload
systemctl restart kubelet

3.2 安装etcd集群

创建etcd.yaml,并放到 /etc/kubernetes/manifests

注意：
提前创建日志文件，便于挂载
/var/log/kube-apiserver.log
/var/log/kube-etcd.log
/var/log/kube-controller-manager.log
/var/log/kube-scheduler.log

#根据挂载配置创建相关目录
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: etcd
    tier: control-plane
  name: etcd-server
  namespace: kube-system
spec:
  hostNetwork: true
  containers:
  - image: foxchan/google_containers/etcd-amd64:3.0.17
    name: etcd-container
    command:
    - /bin/sh
    - -c
    - /usr/local/bin/etcd
      --name=etcd0
      --initial-advertise-peer-urls=http://master_IP:2380
      --listen-peer-urls=http://master_IP:2380
      --advertise-client-urls=http://master_IP:2379
      --listen-client-urls=http://master_IP:2379,http://127.0.0.1:2379
      --data-dir=/var/etcd/data
      --initial-cluster-token=emar-etcd-cluster
      --initial-cluster=etcd0=http://master_IP1:2380,etcd1=http://master_IP2:2380,etcd2=http://master_IP3:2380
      --initial-cluster-state=new 1>>/var/log/kube-etcd.log 2>&1
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /health
        port: 2379
        scheme: HTTP
      initialDelaySeconds: 15
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /var/log/kube-etcd.log
      name: logfile
    - mountPath: /var/etcd
      name: varetcd
    - mountPath: /etc/ssl/certs
      name: certs
    - mountPath: /etc/kubernetes/
      name: k8s
      readOnly: true
  volumes:
  - hostPath:
      path: /var/log/kube-etcd.log
    name: logfile
  - hostPath:
      path: /var/etcd/data
    name: varetcd
  - hostPath:
      path: /etc/ssl/certs
    name: certs
  - hostPath:
      path: /etc/kubernetes/
    name: k8s
status: {}

3台master机器 重复操作3.1-3.2，
参数说明

• --name=etcd0 每个etcd name都是唯一

• client-urls 修改对应的机器ip

kubelet 会定时查看manifests目录，拉起 里面的配置文件

3.3 安装kube-apiserver

创建kube-apiserver.yaml,并放到 /etc/kubernetes/manifests

#根据挂载配置创建相关目录
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  hostNetwork: true
  containers:
  - command:
    - /bin/sh
    - -c
    - /usr/local/bin/kube-apiserver
      --kubelet-https=true
      --enable-bootstrap-token-auth=true
      --token-auth-file=/etc/kubernetes/token.csv
      --service-cluster-ip-range=10.96.0.0/12
      --tls-cert-file=/etc/kubernetes/pki/kubernetes.pem
      --tls-private-key-file=/etc/kubernetes/pki/kubernetes-key.pem
      --client-ca-file=/etc/kubernetes/pki/ca.pem
      --service-account-key-file=/etc/kubernetes/pki/ca-key.pem
      --insecure-port=9080
      --secure-port=6443
      --insecure-bind-address=0.0.0.0
      --bind-address=0.0.0.0
      --advertise-address=master_IP
      --storage-backend=etcd3
      --etcd-servers=http://master_IP1:2379,http://master_IP2:2379,http://master_IP3:2379
      --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds，NodeRestriction
      --allow-privileged=true
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
      --authorization-mode=Node,RBAC
      --v=2 1>>/var/log/kube-apiserver.log 2>&1
    image: foxchan/google_containers/kube-apiserver-amd64:v1.8.1
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: kube-apiserver
    resources:
      requests:
        cpu: 250m
    volumeMounts:
    - mountPath: /etc/kubernetes/
      name: k8s
      readOnly: true
    - mountPath: /etc/ssl/certs
      name: certs
    - mountPath: /etc/pki
      name: pki
    - mountPath: /var/log/kube-apiserver.log
      name: logfile
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/kubernetes
    name: k8s
  - hostPath:
      path: /etc/ssl/certs
    name: certs
  - hostPath:
      path: /etc/pki
    name: pki
  - hostPath:
      path: /var/log/kube-apiserver.log
    name: logfile
status: {}

参数说明：

• --advertise-address 修改对应机器的ip

• --enable-bootstrap-token-auth Bootstrap Token authenticator

• --authorization-mode 授权模型增加了 Node 参数，因为 1.8 后默认 system:node role 不会自动授予 system:nodes 组

• 由于以上原因，--admission-control 同时增加了 NodeRestriction 参数

  
  

检测：可以看到api已经正常

kubectl --server=https://master_IP:6443 \
--certificate-authority=/etc/kubernetes/pki/ca.pem  \
--client-certificate=/etc/kubernetes/pki/admin.pem \
--client-key=/etc/kubernetes/pki/admin-key.pem \
get componentstatuses
NAME                 STATUS      MESSAGE                                                                                        ERROR
controller-manager   Unhealthy   Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: getsockopt: connection refused   
scheduler            Healthy     ok                                                                                             
etcd-1               Healthy     {"health": "true"}                                                                             
etcd-0               Healthy     {"health": "true"}                                                                             
etcd-2               Healthy     {"health": "true"}

3.4 安装kube-controller-manager

创建kube-controller-manager.yaml,并放到 /etc/kubernetes/manifests

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-controller-manager
    tier: control-plane
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - command:
    - /bin/sh
    - -c
    - /usr/local/bin/kube-controller-manager 
      --master=127.0.0.1:9080
      --controllers=*,bootstrapsigner,tokencleaner
      --root-ca-file=/etc/kubernetes/pki/ca.pem
      --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem
      --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem
      --service-account-private-key-file=/etc/kubernetes/pki/ca-key.pem
      --leader-elect=true 
      --v=2 1>>/var/log/kube-controller-manager.log 2>&1
    image: foxchan/google_containers/kube-controller-manager-amd64:v1.8.1
    livenessProbe:
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10252
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: kube-controller-manager
    volumeMounts:
    - mountPath: /etc/kubernetes/
      name: k8s
      readOnly: true
    - mountPath: /var/log/kube-controller-manager.log
      name: logfile
    - mountPath: /etc/ssl/certs
      name: certs
    - mountPath: /etc/pki
      name: pki
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/kubernetes
    name: k8s
  - hostPath:
      path: /var/log/kube-controller-manager.log
    name: logfile
  - hostPath:
      path: /etc/ssl/certs
    name: certs
  - hostPath:
      path: /etc/pki
    name: pki
status: {}

参数说明

• --controllers=*,tokencleaner,bootstrapsigner 启用bootstrap token

3.5 安装kube-scheduler

3.5.1 配置scheduler.conf

cd /etc/kubernetes
export KUBE_APISERVER="https://master_VIP:6443"

# set-cluster
kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/pki/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=scheduler.conf

# set-credentials
kubectl config set-credentials system:kube-scheduler \
  --client-certificate=/etc/kubernetes/pki/scheduler.pem \
  --embed-certs=true \
  --client-key=/etc/kubernetes/pki/scheduler-key.pem \
  --kubeconfig=scheduler.conf

# set-context
kubectl config set-context system:kube-scheduler@kubernetes \
  --cluster=kubernetes \
  --user=system:kube-scheduler \
  --kubeconfig=scheduler.conf

# set default context
kubectl config use-context system:kube-scheduler@kubernetes --kubeconfig=scheduler.conf

scheduler.conf文件生成后将这个文件分发到各个Master节点的/etc/kubernetes目录下

3.5.2创建kube-scheduler.yaml,并放到 /etc/kubernetes/manifests

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-scheduler
    tier: control-plane
  name: kube-scheduler
  namespace: kube-system
spec:
  hostNetwork: true
  containers:
  - command:
    - /bin/sh
    - -c
    - /usr/local/bin/kube-scheduler 
      --address=127.0.0.1
      --leader-elect=true 
      --kubeconfig=/etc/kubernetes/scheduler.conf 
      --v=2 1>>/var/log/kube-scheduler.log 2>&1
    image: foxchan/google_containers/kube-scheduler-amd64:v1.8.1
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10251
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: kube-scheduler
    resources:
      requests:
        cpu: 100m
    volumeMounts:
    - mountPath: /var/log/kube-scheduler.log
      name: logfile
    - mountPath: /etc/kubernetes/scheduler.conf
      name: kubeconfig
      readOnly: true
  volumes:
  - hostPath:
      path: /var/log/kube-scheduler.log
    name: logfile
  - hostPath:
      path: /etc/kubernetes/scheduler.conf
    name: kubeconfig
status: {}

到这里三个Master节点上的kube-scheduler部署完成，通过选举出一个leader工作。
查看kube-scheduler日志

 tail -f kube-scheduler.log
I1024 05:20:44.704783       7 event.go:218] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"kube-scheduler", UID:"1201fc85-b7e1-11e7-9792-525400b406cc", APIVersion:"v1", ResourceVersion:"87114", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' kvm-sh002154 became leader

查看Kubernetes Master集群各个核心组件的状态全部正常

kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-2               Healthy   {"health": "true"}
etcd-0               Healthy   {"health": "true"}
etcd-1               Healthy   {"health": "true"}