备案控制台
探索云世界
开发者社区 开发与运维 文章 正文

CentOS 7.4搭建Kubernetes 1.8.5集群

2017-11-12 2525
版权
版权声明：
本文内容由阿里云实名注册用户自发贡献，版权归原作者所有，阿里云开发者社区不拥有其著作权，亦不承担相应法律责任。具体规则请查看《 阿里云开发者社区用户服务协议》和 《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容，填写 侵权投诉表单进行举报，一经查实，本社区将立刻删除涉嫌侵权内容。
简介：

环境介绍

角色 操作系统 IP 主机名 Docker版本
master,node CentOS 7.4 192.168.0.210 node210 17.11.0-ce
node CentOS 7.4 192.168.0.211 node211 17.11.0-ce
node CentOS 7.4 192.168.0.212 node212 17.11.0-ce

1.基础环境配置(所有服务器执行)
a.SELinux关闭

sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
setenforce 0

b.Docker安装

curl -sSL https://get.docker.com/ | sh

c.配置国内Docker镜像加速器

curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://e2a6d434.m.daocloud.io

d.开启Docker开机自动启动

systemctl enable docker.service
systemctl restart docker

2.kubernetes证书准备(master执行)
a.为将文件复制到Node节点，节省部署时间，我这里做ssh信任免密复制

ssh-genkey -t rsa
ssh-copy-id 192.168.0.211
ssh-copy-id 192.168.0.212

b.下载证书生成工具

yum -y install wget
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
chmod +x cfssl_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl

wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x cfssljson_linux-amd64
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl-certinfo_linux-amd64
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo

c.CA证书制作
#目录准备

mkdir /root/ssl
cd /root/ssl

#创建CA证书配置
vim ca-config.json

{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      }
    }
  }
}

#创建CA证书请求文件
vim ca-csr.json

{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "JIANGXI",
      "L": "NANCHANG",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

#生成CA证书和私钥
cfssl gencert -initca ca-csr.json | cfssljson -bare ca

#创建kubernetes证书签名请求
vim kubernetes-csr.json

{
    "CN": "kubernetes",
    "hosts": [
      "127.0.0.1",
      "192.168.0.210",      #修改成自己主机的IP
      "192.168.0.211",      #修改成自己主机的IP
      "192.168.0.212",      #修改成自己主机的IP
      "10.254.0.1",
      "kubernetes",
      "node210",      #修改成自己主机的主机名
      "node211",      #修改成自己主机的主机名
      "node212",      #修改成自己主机的主机名
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "JIANGXI",
            "L": "JIANGXI",
            "O": "k8s",
            "OU": "System"
        }
    ]
}

#生成kubernetes证书及私钥
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes

#创建admin证书签名请求
vim admin-csr.json

{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "JIANGXI",
      "L": "JIANGXI",
      "O": "system:masters",
      "OU": "System"
    }
  ]
}

#生成admin证书及私钥
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

#创建 kube-proxy 证书签名请求
vim kube-proxy-csr.json

{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "JIANGXI",
      "L": "JIANGXI",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

#生成证书及私钥
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

#分发证书

mkdir -p /etc/kubernetes/ssl
cp -r *.pem /etc/kubernetes/ssl

cd /etc
scp -r kubernetes/ 192.168.0.211:/etc/
scp -r kubernetes/ 192.168.0.212:/etc/

3.etcd集群安装及配置
a.下载etcd，并分发至节点

wget https://github.com/coreos/etcd/releases/download/v3.2.11/etcd-v3.2.11-linux-amd64.tar.gz
tar zxf etcd-v3.2.11-linux-amd64.tar.gz
mv etcd-v3.2.11-linux-amd64/etcd* /usr/local/bin
scp -r /usr/local/bin/etc* 192.168.0.211:/usr/local/bin/
scp -r /usr/local/bin/etc* 192.168.0.212:/usr/local/bin/

b.创建etcd服务启动文件
vim /usr/lib/systemd/system/etcd.service

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/local/bin/etcd \
  --name ${ETCD_NAME} \
  --cert-file=/etc/kubernetes/ssl/kubernetes.pem \
  --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
  --peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem \
  --peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
  --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --initial-advertise-peer-urls ${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
  --listen-peer-urls ${ETCD_LISTEN_PEER_URLS} \
  --listen-client-urls ${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
  --advertise-client-urls ${ETCD_ADVERTISE_CLIENT_URLS} \
  --initial-cluster-token ${ETCD_INITIAL_CLUSTER_TOKEN} \
  --initial-cluster infra1=https://192.168.0.210:2380,infra2=https://192.168.0.211:2380,infra3=https://192.168.0.212:2380 \
  --initial-cluster-state new \
  --data-dir=${ETCD_DATA_DIR}
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

c.创建必要的目录

mkdir -p /var/lib/etcd/
mkdir /etc/etcd

d.编辑etcd的配置文件
vim /etc/etcd/etcd.conf
node210的配置文件/etc/etcd/etcd.conf为

# [member]
ETCD_NAME=infra1
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.0.210:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.210:2379"

#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.210:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.210:2379"

node211的配置文件/etc/etcd/etcd.conf为

# [member]
ETCD_NAME=infra2
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.0.211:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.211:2379"

#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.211:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.211:2379"

node212的配置文件/etc/etcd/etcd.conf为

# [member]
ETCD_NAME=infra3
ETCD_DATA_DIR="/var/lib/etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.0.212:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.0.212:2379"

#[cluster]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.212:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.212:2379"

#在所有节点执行，启动etcd

systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
systemctl status etcd

如果报错，就需要查看/var/log/messages文件进行排错

e.测试集群是否正常

验证ETCD是否成功启动
etcdctl \
  --ca-file=/etc/kubernetes/ssl/ca.pem \
  --cert-file=/etc/kubernetes/ssl/kubernetes.pem \
  --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
  cluster-health

4.配置kubernetes参数
a.下载kubernetes编译好的二进制文件并进行分发

wget https://dl.k8s.io/v1.8.5/kubernetes-server-linux-amd64.tar.gz
tar zxf kubernetes-server-linux-amd64.tar.gz 
cp -rf kubernetes/server/bin/{kube-apiserver,kube-controller-manager,kubectl,kubefed,kubelet,kube-proxy,kube-scheduler} /usr/local/bin/
scp -r kubernetes/server/bin/{kubelet,kube-proxy} 192.168.0.211:/usr/local/bin/
scp -r kubernetes/server/bin/{kubelet,kube-proxy} 192.168.0.212:/usr/local/bin/

#查看kubernetes最新版，可到https://github.com/kubernetes/kubernetes/releases
然后进入 CHANGELOG-x.x.md就可限制二进制的下载地址

b.创建 TLS Bootstrapping Token

cd /etc/kubernetes

export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ')

cat > token.csv <<EOF
${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF

c.创建 kubelet bootstrapping kubeconfig 文件

cd /etc/kubernetes
export KUBE_APISERVER="https://192.168.0.210:6443"

#设置集群参数

kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap.kubeconfig

#设置客户端认证参数

kubectl config set-credentials kubelet-bootstrap \
  --token=${BOOTSTRAP_TOKEN} \
  --kubeconfig=bootstrap.kubeconfig

#设置上下文参数

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubelet-bootstrap \
  --kubeconfig=bootstrap.kubeconfig

#设置默认上下文
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

d.创建 kube-proxy kubeconfig 文件
export KUBE_APISERVER="https://192.168.0.210:6443"
#设置集群参数

kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig

#设置客户端认证参数

kubectl config set-credentials kube-proxy \
  --client-certificate=/etc/kubernetes/ssl/kube-proxy.pem \
  --client-key=/etc/kubernetes/ssl/kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig

#设置上下文参数

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig

#设置默认上下文
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

e.创建 kubectl kubeconfig 文件
export KUBE_APISERVER="https://192.168.0.210:6443"
#设置集群参数

kubectl config set-cluster kubernetes \
  --certificate-authority=/etc/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER}

#设置客户端认证参数

kubectl config set-credentials admin \
  --client-certificate=/etc/kubernetes/ssl/admin.pem \
  --embed-certs=true \
  --client-key=/etc/kubernetes/ssl/admin-key.pem

#设置上下文参数

kubectl config set-context kubernetes \
  --cluster=kubernetes \
  --user=admin

#设置默认上下文
kubectl config use-context kubernetes

f.将2个bootstrap.kubeconfig kube-proxy.kubeconfig文件分发至其余服务器

scp -r *.kubeconfig 192.168.0.211:/etc/kubernetes/
scp -r *.kubeconfig 192.168.0.212:/etc/kubernetes/

5.MASTER安装及配置
a.apiserver安装配置
#apiserver服务启动文件
vim /usr/lib/systemd/system/kube-apiserver.service

[Unit]
Description=Kubernetes API Service
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
After=etcd.service

[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/apiserver
ExecStart=/usr/local/bin/kube-apiserver \
        $KUBE_LOGTOSTDERR \
        $KUBE_LOG_LEVEL \
        $KUBE_ETCD_SERVERS \
        $KUBE_API_ADDRESS \
        $KUBE_API_PORT \
        $KUBELET_PORT \
        $KUBE_ALLOW_PRIV \
        $KUBE_SERVICE_ADDRESSES \
        $KUBE_ADMISSION_CONTROL \
        $KUBE_API_ARGS
Restart=on-failure
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

#配置kubernetes默认配置
vim /etc/kubernetes/config

###
# kubernetes system config
#
# The following values are used to configure various aspects of all
# kubernetes services, including
#
#   kube-apiserver.service
#   kube-controller-manager.service
#   kube-scheduler.service
#   kubelet.service
#   kube-proxy.service
# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR="--logtostderr=true"

# journal message level, 0 is debug
KUBE_LOG_LEVEL="--v=0"

# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow-privileged=true"

# How the controller-manager, scheduler, and proxy find the apiserver
#KUBE_MASTER="--master=http://sz-pg-oam-docker-test-001.tendcloud.com:8080"
KUBE_MASTER="--master=http://192.168.0.210:8080"

#配置apiserver参数
vim /etc/kubernetes/apiserver

###
## kubernetes system config
##
## The following values are used to configure the kube-apiserver
##
#
## The address on the local server to listen to.
#KUBE_API_ADDRESS="--insecure-bind-address=sz-pg-oam-docker-test-001.tendcloud.com"
KUBE_API_ADDRESS="--advertise-address=192.168.0.210 --bind-address=192.168.0.210 --insecure-bind-address=192.168.0.210"
#
## The port on the local server to listen on.
#KUBE_API_PORT="--port=8080"
#
## Port minions listen on
#KUBELET_PORT="--kubelet-port=10250"
#
## Comma separated list of nodes in the etcd cluster
KUBE_ETCD_SERVERS="--etcd-servers=https://192.168.0.210:2379,https://192.168.0.211:2379,https://192.168.0.212:2379"
#
## Address range to use for services
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"
#
## default admission control policies
KUBE_ADMISSION_CONTROL="--admission-control=ServiceAccount,NamespaceLifecycle,NamespaceExists,LimitRanger,ResourceQuota"
#
## Add your own!
KUBE_API_ARGS="--authorization-mode=RBAC --runtime-config=rbac.authorization.k8s.io/v1beta1 --kubelet-https=true --enable-bootstrap-token-auth --token-auth-file=/etc/kubernetes/token.csv --service-node-port-range=30000-32767 --tls-cert-file=/etc/kubernetes/ssl/kubernetes.pem --tls-private-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --client-ca-file=/etc/kubernetes/ssl/ca.pem --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem --etcd-cafile=/etc/kubernetes/ssl/ca.pem --etcd-certfile=/etc/kubernetes/ssl/kuberne
#如果出现错误，查看/var/log/messages
tes.pem --etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem --enable-swagger-ui=true --apiserver-count=3 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/lib/audit.log --
event-ttl=1h"

#启动apiserver

systemctl daemon-reload
systemctl enable kube-apiserver
systemctl start kube-apiserver
systemctl status kube-apiserver

#如果出现错误，查看/var/log/messages

b.controller-manager服务配置
#controller-manager服务启动文件
vim /usr/lib/systemd/system/kube-controller-manager.service

[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/controller-manager
ExecStart=/usr/local/bin/kube-controller-manager \
        $KUBE_LOGTOSTDERR \
        $KUBE_LOG_LEVEL \
        $KUBE_MASTER \
        $KUBE_CONTROLLER_MANAGER_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

#配置controller-manager服务配置文件
vim /etc/kubernetes/controller-manager

#The following values are used to configure the kubernetes controller-manager

#defaults from config and apiserver should be adequate

#Add your own!
KUBE_CONTROLLER_MANAGER_ARGS="--address=127.0.0.1 --service-cluster-ip-range=10.254.0.0/16 --cluster-name=kubernetes --cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem --cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem  --service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem --root-ca-file=/etc/kubernetes/ssl/ca.pem --leader-elect=true"

#启动controller-manager服务
systemctl daemon-reload
systemctl enable kube-controller-manager
systemctl start kube-controller-manager

c.scheduler服务安装及配置
#配置scheduler服务启动文件
vim /usr/lib/systemd/system/kube-scheduler.service

[Unit]
Description=Kubernetes Scheduler Plugin
Documentation=https://github.com/GoogleCloudPlatform/kubernetes

[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/scheduler
ExecStart=/usr/local/bin/kube-scheduler \
            $KUBE_LOGTOSTDERR \
            $KUBE_LOG_LEVEL \
            $KUBE_MASTER \
            $KUBE_SCHEDULER_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

#配置scheduler服务配置文件
vim /etc/kubernetes/scheduler

#kubernetes scheduler config
#default config should be adequate
#Add your own!
KUBE_SCHEDULER_ARGS="--leader-elect=true --address=127.0.0.1"

#启动scheduler服务

systemctl daemon-reload
systemctl enable kube-scheduler
systemctl start kube-scheduler

d.测试master是否正常
kubectl get componentstatuses
#结果如下说明正常

NAME                 STATUS    MESSAGE              ERROR
scheduler            Healthy   ok                   
controller-manager   Healthy   ok                   
etcd-0               Healthy   {"health": "true"}   
etcd-1               Healthy   {"health": "true"}   
etcd-2               Healthy   {"health": "true"}

6.node安装(所有节点)
a.flannel安装及配置(容器网络我们采用flannel)
#yum安装flannel
yum install -y flannel
#检查node节点证书情况
ls /etc/kubernetes/ssl
#修改flannel.service配置文件如下
vi /usr/lib/systemd/system/flanneld.service

[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service

[Service]
Type=notify
EnvironmentFile=/etc/sysconfig/flanneld
EnvironmentFile=-/etc/sysconfig/docker-network
ExecStart=/usr/bin/flanneld-start \
  -etcd-endpoints=${ETCD_ENDPOINTS} \
  -etcd-prefix=${ETCD_PREFIX} \
  $FLANNEL_OPTIONS
ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure

[Install]
WantedBy=multi-user.target
RequiredBy=docker.service

#修改flannel配置文件
vi /etc/sysconfig/flanneld

# Flanneld configuration options

# etcd url location.  Point this to the server where etcd runs
ETCD_ENDPOINTS="https://192.168.0.210:2379,https://192.168.0.211:2379,https://192.168.0.212:2379"

# etcd config key.  This is the configuration key that flannel queries
# For address range assignment
ETCD_PREFIX="/kube-centos/network"

# Any additional options that you want to pass
FLANNEL_OPTIONS="-etcd-cafile=/etc/kubernetes/ssl/ca.pem -etcd-certfile=/etc/kubernetes/ssl/kubernetes.pem -etcd-keyfile=/etc/kubernetes/ssl/kubernetes-key.pem"

#在etcd中创建网络配置

etcdctl --endpoints=https://192.168.0.210:2379,https://192.168.0.211:2379,https://192.168.0.212:2379 \
  --ca-file=/etc/kubernetes/ssl/ca.pem \
  --cert-file=/etc/kubernetes/ssl/kubernetes.pem \
  --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
  mkdir /kube-centos/network

etcdctl --endpoints=https://192.168.0.210:2379,https://192.168.0.211:2379,https://192.168.0.212:2379 \
  --ca-file=/etc/kubernetes/ssl/ca.pem \
  --cert-file=/etc/kubernetes/ssl/kubernetes.pem \
  --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
  mk /kube-centos/network/config '{"Network":"172.30.0.0/16","SubnetLen":24,"Backend":{"Type":"vxlan"}}'

#flannel服务启动
systemctl daemon-reload
systemctl enable flanneld
systemctl start flanneld
systemctl status flanneld

b.配置Docker服务启动文件,集成flannel
vim /usr/lib/systemd/system/docker.service

在ExecStart上增加
EnvironmentFile=-/run/flannel/docker
EnvironmentFile=-/run/docker_opts.env
EnvironmentFile=-/run/flannel/subnet.env
修改如下
ExecStart=/usr/bin/dockerd --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU}

效果如下:

[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
EnvironmentFile=-/run/flannel/docker
EnvironmentFile=-/run/docker_opts.env
EnvironmentFile=-/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU}
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

#重启启动Docker服务

systemctl daemon-reload
systemctl restart docker
systemctl status docker

切记要先启动flannel，再启动Docker

c.查询etcd是否分配网络

etcdctl --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/etc/kubernetes/ssl/ca.pem \
  --cert-file=/etc/kubernetes/ssl/kubernetes.pem \
  --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
  ls /kube-centos/network/subnets

结果大致如下

/kube-centos/network/subnets/172.30.1.0-24
/kube-centos/network/subnets/172.30.54.0-24
/kube-centos/network/subnets/172.30.99.0-24
etcdctl --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/etc/kubernetes/ssl/ca.pem \
  --cert-file=/etc/kubernetes/ssl/kubernetes.pem \
  --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
  get /kube-centos/network/config

结果大致如下

{"Network":"172.30.0.0/16","SubnetLen":24,"Backend":{"Type":"vxlan"}}

d.安装及配置kubelet
#创建kubelet服务启动文件
vim /usr/lib/systemd/system/kubelet.service

[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/var/lib/kubelet
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/kubelet
ExecStart=/usr/local/bin/kubelet \
            $KUBE_LOGTOSTDERR \
            $KUBE_LOG_LEVEL \
            $KUBELET_API_SERVER \
            $KUBELET_ADDRESS \
            $KUBELET_PORT \
            $KUBELET_HOSTNAME \
            $KUBE_ALLOW_PRIV \
            $KUBELET_POD_INFRA_CONTAINER \
            $KUBELET_ARGS
Restart=on-failure

[Install]
WantedBy=multi-user.target

#kubelet认证配置文件
vim /etc/kubernetes/kubelet.kubeconfig

apiVersion: v1
kind: Config
clusters:
  - cluster:
      server: http://192.168.0.210:8080
    name: local
contexts:
  - context:
      cluster: local
    name: local
current-context: local

#kubelet配置文件
vim /etc/kubernetes/kubelet

node210下/etc/kubernetes/kubelet内容如下

###
## kubernetes kubelet (minion) config
#
## The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=192.168.0.210"
#
## The port for the info server to serve on
#KUBELET_PORT="--port=10250"
#
## You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override=192.168.0.210"
#
## location of the api-server
#KUBELET_API_SERVER="--api-servers=http://192.168.0.210:8080"
KUBELET_API_SERVER=" "
#
## pod infrastructure container
#KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest"
KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=kubernetes/pause"
#
## Add your own!
#KUBELET_ARGS="--cgroup-driver=cgroupfs --cluster-dns=10.254.0.2 --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --require-kubeconfig --cert-dir=/etc/kubernetes/ssl --cluster-domain=cluster.local --hairpin-mode promiscuous-bridge --serialize-image-pulls=false"
KUBELET_ARGS="--cgroup-driver=cgroupfs --cluster-dns=10.254.0.2 --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --cert-dir=/etc/kubernetes/ssl --cluster-domain=cluster.local --hairpin-mode promiscuous-bridge --serialize-image-pulls=false --fail-swap-on=false"

node211里配置文件如下

###
## kubernetes kubelet (minion) config
#
## The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=192.168.0.211"
#
## The port for the info server to serve on
#KUBELET_PORT="--port=10250"
#
## You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override=192.168.0.211"
#
## location of the api-server
#KUBELET_API_SERVER="--api-servers=http://192.168.0.210:8080"
KUBELET_API_SERVER=" "
#
## pod infrastructure container
#KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest"
KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=kubernetes/pause"
#
## Add your own!
#KUBELET_ARGS="--cgroup-driver=cgroupfs --cluster-dns=10.254.0.2 --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --require-kubeconfig --cert-dir=/etc/kubernetes/ssl --cluster-domain=cluster.local --hairpin-mode promiscuous-bridge --serialize-image-pulls=false"
KUBELET_ARGS="--cgroup-driver=cgroupfs --cluster-dns=10.254.0.2 --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --cert-dir=/etc/kubernetes/ssl --cluster-domain=cluster.local --hairpin-mode promiscuous-bridge --serialize-image-pulls=false --fail-swap-on=false"

node212里配置文件如下

###
## kubernetes kubelet (minion) config
#
## The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=192.168.0.212"
#
## The port for the info server to serve on
#KUBELET_PORT="--port=10250"
#
## You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname-override=192.168.0.212"
#
## location of the api-server
#KUBELET_API_SERVER="--api-servers=http://192.168.0.210:8080"
KUBELET_API_SERVER=" "
#
## pod infrastructure container
#KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest"
KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=kubernetes/pause"
#
## Add your own!
#KUBELET_ARGS="--cgroup-driver=cgroupfs --cluster-dns=10.254.0.2 --experimental-bootstrap-kubeconfig=/etc/kubernetes/bootstrap.kubeconfig --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --require-kubeconfig --cert-dir=/etc/kubernetes/ssl --cluster-domain=cluster.local --hairpin-mode promiscuous-bridge --serialize-image-pulls=false"
KUBELET_ARGS="--cgroup-driver=cgroupfs --cluster-dns=10.254.0.2 --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --cert-dir=/etc/kubernetes/ssl --cluster-domain=cluster.local --hairpin-mode promiscuous-bridge --serialize-image-pulls=false --fail-swap-on=false"

#启动kubelet服务

mkdir -p /var/lib/kubelet

systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet
systemctl status kubelet

#这里很容易出错，出错时查看/var/log/messages看日志进行排错

#检查kubelet服务是否正常

kubectl get nodes
NAME            STATUS    ROLES     AGE       VERSION
192.168.0.210   Ready     <none>    14h       v1.8.5
192.168.0.211   Ready     <none>    14h       v1.8.5
192.168.0.212   Ready     <none>    14h       v1.8.5

c.安装及配置kube-proxy
#配置kube-proxy服务启动文件
vim /usr/lib/systemd/system/kube-proxy.service

[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/proxy
ExecStart=/usr/local/bin/kube-proxy \
        $KUBE_LOGTOSTDERR \
        $KUBE_LOG_LEVEL \
        $KUBE_MASTER \
        $KUBE_PROXY_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

#kube-proxy配置文件如下：
node210:
vim /etc/kubernetes/proxy

###
# kubernetes proxy config

# default config should be adequate

# Add your own!
KUBE_PROXY_ARGS="--bind-address=192.168.0.210 --hostname-override=192.168.0.210 --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig --cluster-cidr=10.254.0.0/16"

node211:
vim /etc/kubernetes/proxy

###
# kubernetes proxy config

# default config should be adequate

# Add your own!
KUBE_PROXY_ARGS="--bind-address=192.168.0.211 --hostname-override=192.168.0.211 --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig --cluster-cidr=10.254.0.0/16"

node212:
vim /etc/kubernetes/proxy

###
# kubernetes proxy config

# default config should be adequate

# Add your own!
KUBE_PROXY_ARGS="--bind-address=192.168.0.212--hostname-override=192.168.0.212 --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig --cluster-cidr=10.254.0.0/16"

#启动kube-proxy服务

systemctl daemon-reload
systemctl enable kube-proxy
systemctl start kube-proxy
systemctl status kube-proxy

d.在所有节点默认开启forward为accept
vim /usr/lib/systemd/system/forward.service

[Unit]
Description=iptables forward
Documentation=http://iptables.org/
After=network.target docker.service

[Service]
Type=forking
ExecStart=/usr/sbin/iptables -P FORWARD ACCEPT
ExecReload=/usr/sbin/iptables -P FORWARD ACCEPT
ExecStop=/usr/sbin/iptables -P FORWARD ACCEPT
PrivateTmp=true

[Install]
WantedBy=multi-user.target

#启动forward服务

systemctl daemon-reload
systemctl enable forward
systemctl start forward
systemctl status forward

7.测试集群是否工作正常
a.创建一个deploy
kubectl run nginx --replicas=2 --labels="run=nginx-service" --image=nginx --port=80
b.映射服务到外网可访问
kubectl expose deployment nginx --type=NodePort --name=nginx-service
c.查看服务状态

kubectl describe svc example-service
Name:                   nginx-service
Namespace:              default
Labels:                 run=nginx-service
Annotations:            <none>
Selector:               run=nginx-service
Type:                   NodePort
IP:                     10.254.84.99
Port:                   <unset> 80/TCP
NodePort:               <unset> 30881/TCP
Endpoints:              172.30.1.2:80,172.30.54.2:80
Session Affinity:       None
Events:                 <none>

d.查看pods启动情况

kubectl get pods
NAME                     READY     STATUS    RESTARTS   AGE
nginx-2317272628-nsfrr   1/1       Running   0          1m
nginx-2317272628-qbbgg   1/1       Running   0          1m

e.在外网通过
http://192.168.0.210:30881
http://192.168.0.211:30881
http://192.168.0.212:30881
都可以访问nginx页面

若无法访问，可通过iptables -nL查看forward链是否开启


本文转自 rong341233 51CTO博客，原文链接:http://blog.51cto.com/fengwan/2049124


文章标签：
容器服务Kubernetes版
云解析DNS
容器
Kubernetes
网络安全
开发工具
Docker
Perl
应用服务中间件
Linux
网络协议
nginx
关键词：
容器服务Kubernetes版集群
容器服务Kubernetes版 CentOS
CentOS集群
centos容器服务Kubernetes版
CentOS Kubernetes
相关实践学习
容器服务Serverless版ACK Serverless 快速入门：在线魔方应用部署和监控
通过本实验，您将了解到容器服务Serverless版ACK Serverless 的基本产品能力，即可以实现快速部署一个在线魔方应用，并借助阿里云容器服务成熟的产品生态，实现在线应用的企业级监控，提升应用稳定性。
云原生实践公开课
课程大纲 开篇：如何学习并实践云原生技术 基础篇： 5 步上手 Kubernetes 进阶篇：生产环境下的 K8s 实践 相关的阿里云产品：容器服务&nbsp;ACK 容器服务&nbsp;Kubernetes&nbsp;版（简称&nbsp;ACK）提供高性能可伸缩的容器应用管理能力，支持企业级容器化应用的全生命周期管理。整合阿里云虚拟化、存储、网络和安全能力，打造云端最佳容器化应用运行环境。 了解产品详情:&nbsp;https://www.aliyun.com/product/kubernetes
科技小能手
目录
相关文章
天下无贼001
|
27天前
|
Prometheus 监控 Kubernetes
Kubernetes 集群监控与日志管理实践
【2月更文挑战第29天】 在微服务架构日益普及的当下，Kubernetes 已成为容器编排的事实标准。然而，随着集群规模的扩大和业务复杂度的提升，有效的监控和日志管理变得至关重要。本文将探讨构建高效 Kubernetes 集群监控系统的策略，以及实施日志聚合和分析的最佳实践。通过引入如 Prometheus 和 Fluentd 等开源工具，我们旨在为运维专家提供一套完整的解决方案，以保障系统的稳定性和可靠性。
天下无贼001
40 0
jerrywangsap
|
13天前
|
数据库 存储 监控
什么是 SAP HANA 内存数据库 的 Delta Storage
什么是 SAP HANA 内存数据库 的 Delta Storage
jerrywangsap
16 0
什么是 SAP HANA 内存数据库 的 Delta Storage
jianz123
|
2天前
|
Kubernetes 搜索推荐 Docker
使用 kubeadm 部署 Kubernetes 集群（二）k8s环境安装
使用 kubeadm 部署 Kubernetes 集群（二）k8s环境安装
jianz123
34 17
风水道人
|
15天前
|
消息中间件 Kubernetes Kafka
Terraform阿里云创建资源1分钟创建集群一键发布应用Terraform 创建 Kubernetes 集群
Terraform阿里云创建资源1分钟创建集群一键发布应用Terraform 创建 Kubernetes 集群
风水道人
11 0
风水道人
|
15天前
|
Kubernetes 安全 网络安全
搭建k8s集群kubeadm搭建Kubernetes二进制搭建Kubernetes集群
搭建k8s集群kubeadm搭建Kubernetes二进制搭建Kubernetes集群
风水道人
101 0
李杰-@yd
|
22天前
|
Kubernetes Cloud Native Docker
【云原生】kubeadm快速搭建K8s集群Kubernetes1.19.0
Kubernetes 是一个开源平台，用于管理容器化工作负载和服务，提供声明式配置和自动化。源自 Google 的大规模运维经验，它拥有广泛的生态支持。本文档详细介绍了 Kubernetes 集群的搭建过程，包括服务器配置、Docker 和 Kubernetes 组件的安装，以及 Master 和 Node 的部署。此外，还提到了使用 Calico 作为 CNI 网络插件，并提供了集群功能的测试步骤。
李杰-@yd
181 0
mrq4nk6ni2neg
|
25天前
|
Prometheus 监控 Kubernetes
Kubernetes 集群的监控与日志管理实践
【2月更文挑战第31天】 在微服务架构日益普及的今天，容器编排工具如Kubernetes已成为部署、管理和扩展容器化应用的关键平台。然而，随着集群规模的扩大和业务复杂性的增加，如何有效监控集群状态、及时响应系统异常，以及管理海量日志信息成为了运维人员面临的重要挑战。本文将深入探讨 Kubernetes 集群监控的最佳实践和日志管理的高效策略，旨在为运维团队提供一套系统的解决思路和操作指南。
mrq4nk6ni2neg
21 0
三分钟热度的鱼
|
26天前
|
Kubernetes 监控 Linux
容器服务ACK常见问题之新增一台CentOS 5.4内核的节点失败如何解决
容器服务ACK（阿里云容器服务 Kubernetes 版）是阿里云提供的一种托管式Kubernetes服务，帮助用户轻松使用Kubernetes进行应用部署、管理和扩展。本汇总收集了容器服务ACK使用中的常见问题及答案，包括集群管理、应用部署、服务访问、网络配置、存储使用、安全保障等方面，旨在帮助用户快速解决使用过程中遇到的难题，提升容器管理和运维效率。
三分钟热度的鱼
18 1
三分钟热度的鱼
|
26天前
|
监控 Kubernetes 安全
容器服务ACK常见问题之centos系统配置失败如何解决
容器服务ACK（阿里云容器服务 Kubernetes 版）是阿里云提供的一种托管式Kubernetes服务，帮助用户轻松使用Kubernetes进行应用部署、管理和扩展。本汇总收集了容器服务ACK使用中的常见问题及答案，包括集群管理、应用部署、服务访问、网络配置、存储使用、安全保障等方面，旨在帮助用户快速解决使用过程中遇到的难题，提升容器管理和运维效率。
三分钟热度的鱼
24 0
zhishijike
|
26天前
|
Kubernetes Linux Docker
centos离线安装 k8s （实操可用）
centos离线安装 k8s （实操可用）
zhishijike
19 4

热门文章

最新文章

  • 1
    服务器Centos7 静默安装Oracle Database 12.2
  • 2
    【MYSQL】—— MySQL 在 Centos 7环境安装
  • 3
    Kubernetes的“厨房”：架构是菜谱，组件是厨具，资源对象是食材（上）
  • 4
    在Kubernetes上运行Flink应用程序时
  • 5
    CentOS7(6.5)升级docker到较新版
  • 6
    CentOS通过DNSpod实现动态域名
  • 7
    CentOS 6 新装服务器部署流程
  • 8
    centos7下配置tomcat
  • 9
    Centos下 Mysql同步备份
  • 10
    centos（7） 使用yum进行安装lamp环境
  • 1
    如何通过计算巢在ACK集群上使用Istio服务网格
    29
  • 2
    Docker容器管理
    35
  • 3
    云原生K8S场景自动化响应ECS系统事件
    118
  • 4
    云计算中的容器化技术：Docker与Kubernetes的实践
    61
  • 5
    934.【kubernetes】kubeadm版本更新证书
    46
  • 6
    TCP 中的 Delay ACK 和 Nagle 算法
    31
  • 7
    K8S基于NFS来动态创建PV【亲测可用】
    65
  • 8
    深入浅出：使用Docker容器化部署Python Web应用
    67
  • 9
    深入探讨 Prometheus 在 Kubernetes 上的部署和实战操作
    161
  • 10
    K8S客户端二 使用Rancher部署服务
    84

    • 相关课程

    更多
  • 体验-Kubernetes 对象及资源规范
  • 阿里云K8S微服务部署案例
  • Kubernetes极速入门
  • Serverless 容器从入门到精通: - Serverless Kubernetes
  • Kubernetes云原生管理实践
  • Kubernetes入门

    • 相关电子书

    更多
  • ACK 云原生弹性方案—云原生时代的加速器
  • ACK集群类型选择最佳实践
  • 企业运维之云原生和Kubernetes 实战

    • 相关实验场景

    更多
  • 在ACK Serverless中部署Stable Diffusion应用
  • 基于 ACK Serverless 解锁你家萌宠的 AI 形象
  • 基于ACK Serverless实现容器应用弹性伸缩
  • 使用ACK Serverless容器化部署大语言模型FastChat
  • 通过EDAS实现K8s微服务应用的金丝雀发布
  • 通过ACK Serverless搭建企业级网站应用

    • 推荐镜像

    更多
  • centos
  • centos-vault
  • kubernetes
    • 下一篇
    部署LAMP环境（Alibaba Cloud Linux 3）
    为什么选择阿里云什么是云计算全球基础设施技术领先稳定可靠安全合规分析师报告
    产品和定价全部产品免费试用产品动态产品定价价格计算器云上成本管理
    解决方案技术解决方案
    文档与社区文档开发者社区天池大赛培训与认证
    权益中心免费试用高校计划企业扶持计划推荐返现计划
    支持与服务基础服务企业增值服务迁云服务官网公告健康看板信任中心
    关注阿里云
    关注阿里云公众号或下载阿里云APP，关注云资讯，随时随地运维管控云服务
    阿里云APP阿里云微信
    联系我们：4008013260
    法律声明Cookies政策廉正举报安全举报联系我们加入我们
    阿里巴巴集团淘宝网天猫全球速卖通阿里巴巴国际交易市场1688阿里妈妈飞猪阿里云计算AliOS万网高德UC友盟优酷钉钉支付宝达摩院淘宝海外阿里云盘饿了么
    © 2009-2024 Aliyun.com 版权所有 增值电信业务经营许可证： 浙B2-20080101 域名注册服务机构许可： 浙D3-20210002 京D3-20220015
    浙公网安备 33010602009975号浙B2-20080101-4