k8s环境部署及使用方式-阿里云开发者社区

安装k8s+docker集群环境（基于centos7系统）

集群机器：
centos-master = 192.168.121.9
centos-minion-1 = 192.168.121.65
centos-minion-2 = 192.168.121.66
centos-minion-3 = 192.168.121.67

1.配置yum源
[centos-master]:cat /etc/yum.repos.d/virt7-docker-common-release.repo

[virt7-docker-common-release] name=virt7-docker-common-release baseurl=http://cbs.centos.org/repos/virt7-docker-common-release/x86_64/os/ gpgcheck=0

加载安装包：
[centos-master]:yum repolist

virt7-docker-common-release virt7-docker-common-release 15

2.安装集群必要软件—-etcd/flannel/kubernetes
Etcd服务在k8s集群中用于配置共享和服务发现。

Flannel是针对k8s设计一个网络规划服务，让集群中的不同节点主机创建的Docker容器都具有全集群唯一的虚拟IP地址。

[centos-master]:yum -y install –enablerepo=virt7-docker-common-release kubernetes etcd flannel

3.如果集群中没有使用DNS解析，那么需要在master节点的/etc/hosts中添加node的主机名信息，比如：

echo "192.168.121.9 centos-master
192.168.121.65 centos-minion-1
192.168.121.66 centos-minion-2
192.168.121.67 centos-minion-3" >> /etc/hosts

4.修改配置master节点的kubernetes配置文件/etc/kubernetes/config

#表示错误日志记录到文件还是输出到stderr
KUBE_LOGTOSTDERR="--logtostderr=true" #日志等级
KUBE_LOG_LEVEL="--v=0" #允许运行特权容器
KUBE_ALLOW_PRIV="--allow-privileged=false" #apiserver的服务地址，controller-manager、scheduler及kubelet都会用到这个配置，这里配置为192.168.121.9
KUBE_MASTER="--master=http://192.168.121.9:8080"

5.k8s集群中涉及的端口比较多，所以centos中的防火墙需要设置对应规则，并需关闭selinux。为确保k8s集群的正常运行，我们可以直接关闭iptables与seliinux服务。

setenforce 0
systemctl stop firewalld.service
systemctl stop firewalld.service

6.修改配置master节点的etcd配置文件/etc/etcd/etcd.conf
etcd服务的可调参数比较多，根据需求开启对应功能，此处我们大概调整如下几个功能：

# [member] #etcd名称
ETCD_NAME=default #etcd数据存储位置
ETCD_DATA_DIR="/var/lib/etcd/default.etcd" #监听的端口
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379" #[cluster] #集群监听的端口
ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"

7.修改配置master节点的apiserver配置文件/etc/kubernetes/apiserver

#监听的接口，如果配置为127.0.0.1则只监听localhost，配置为0.0.0.0会监听所有接口，这里配置为0.0.0.0
KUBE_API_ADDRESS="--address=0.0.0.0" #apiserver的监听端口，默认8080
KUBE_API_PORT="--port=8080" #minion上kubelet监听的端口，默认10250
KUBELET_PORT="--kubelet-port=10250" #etcd服务地址，前面已经启动了etcd服务,端口为2379
KUBE_ETCD_SERVERS="--etcd-servers=http://127.0.0.1:2379" #kubernetes可以分配的ip的范围，kubernetes启动的每一个pod以及serveice都会分配一个ip地址，将从这个范围分配
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16" #需要额外添加的配置项，简单地启用一个集群无需配置
KUBE_API_ARGS=""

8.启动并且配置etcd的网段，此网段一定是未被使用的

systemctl start etcd
etcdctl mkdir /kube-centos/network
etcdctl mk /kube-centos/network/config "{ \"Network\": \"172.30.0.0/16\", \"SubnetLen\": 24, \"Backend\": { \"Type\": \"vxlan\" } }"

9.修改配置master节点的flanneld配置文件/etc/sysconfig/flanneld

#etcd的访问地址及端口
FLANNEL_ETCD_ENDPOINTS="http://192.168.121.9:2379" #服务范围
FLANNEL_ETCD_PREFIX="/kube-centos/network" #其他
FLANNEL_OPTIONS=""

10.启动k8s集群

for SERVICES in etcd kube-apiserver kube-controller-manager kube-scheduler flanneld; do
 systemctl restart $SERVICES
 systemctl enable $SERVICES
 systemctl status $SERVICES
done

以上就是k8s，master节点的安装及配置
接下来，配置nodes节点

11.修改配置nodes节点kubelet的配置文件/etc/kubernetes/kubelet

#minion监听的地址，每个minion根据实际的ip配置,这里配置为0.0.0.0
KUBELET_ADDRESS="--address=0.0.0.0" #监听的端口
KUBELET_PORT="--port=10250" #apiserver的访问地址及端口
KUBELET_API_SERVER="--api-servers=http://192.168.121.9:8080" #额外增加的参数
KUBELET_ARGS="--logtostderr=false --v=0 --log-dir=/data/logs/kubernetes"

12.修改配置nodes节点flanneld的配置文件/etc/sysconfig/flanneld

#etcd的访问地址及端口
FLANNEL_ETCD="http://192.168.121.9:2379" #etcd服务范围
FLANNEL_ETCD_KEY="/kube-centos/network"

13.启动k8s集群服务

for SERVICES in kube-proxy kubelet flanneld docker; do
 systemctl restart $SERVICES
 systemctl enable $SERVICES
 systemctl status $SERVICES done

14.设置kubectl的配置文件

kubectl config set-cluster default-cluster --server=http://192.168.121.9:8080
kubectl config set-context default-context --cluster=default-cluster --user=default-admin
kubectl config use-context default-context

15.检查集群状态
[centos-master]:kubectl get nodes

NAME STATUS AGE VERSION
centos-minion-1 Ready 3d v1.5.0+fff5156
centos-minion-2 Ready 3d v1.5.0+fff5156
centos-minion-3 Ready 3d v1.5.0+fff5156

至此，集群构建完毕

搭建私有库

私有库用于系统内部存储成品镜像，能够快速进行下载及被k8s调度。

1.下载并启动私有库

[centos-master]:docker run --name registry -v /etc/localtime:/etc/localtime -v /opt/registry:/var/lib/registry -p 5000:5000 -itd docker.io/registry

#--name 表示启动的容器后名称，此处为registry #-v 表示挂载路径 格式为宿主机路径:容器内路径 #-p 表示映射端口 格式为宿主机端口:容器内端口 #-itd docker的内部参数，此处声明后台运行容器并分配一个伪终端并绑定到容器的标准输入上，后跟镜像名称此处为docker.io/registry

2.创建一个secret服务，用于k8s调度私有库容器时的“令牌”。简单来说，secret服务就是一个存储密码的服务

[centos-master]:kubectl create secret docker-registry registrykey --docker-server=registry.evehicle.cn --docker-username=docker --docker-password=docker --docker-email=lienhua@zhongchuangsanyou.com 
[centos-master]:kubectl get secret
NAME TYPE DATA AGE
registrykey kubernetes.io/dockercfg 1 6s

此时登录时会提示认证错误

[centos-master]:docker login -u docker -p docker -e lienhua@zhongchuangsanyou.com registry.evehicle.cn
Flag --email has been deprecated, will be removed in 1.13.
Error response from daemon: login attempt to https://registry.evehicle.cn/v2/ failed with status: 401 Unauthorized

这是因为Docker官方是推荐采用Secure Registry的工作模式的，即transport采用tls。这样我们就需要为Registry配置tls所需的key和crt文件了

3.配置nginx反向代理
[centos-master]: cat registry.evehicle.cn.conf

# For versions of nginx > 1.3.9 that include chunked transfer encoding support # Replace with appropriate values where necessary upstream docker-registry {
 server 192.168.121.9:5000;
 #server 10.44.170.95:5000;
}

# uncomment if you want a 301 redirect for users attempting to connect # on port 80 # NOTE: docker client will still fail. This is just for convenience # server { # listen *:80; # server_name my.docker.registry.com; # return 301 https://$server_name$request_uri; # } server {
 listen 443;
 server_name registry.evehicle.cn;

 ssl on;
 ssl_certificate ssl/registry.evehicle.cn.crt;
 ssl_certificate_key ssl/registry.evehicle.cn.key;

 client_max_body_size 0; # disable any limits to avoid HTTP 413 for large image uploads # required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486) chunked_transfer_encoding on;

 location / {
 auth_basic "Restricted";
 auth_basic_user_file passwd;
 add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;

 proxy_pass http://docker-registry;
 proxy_set_header Host $http_host; # required for docker client's sake proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Forwarded-Proto $scheme;
 proxy_read_timeout 900;
 }

 location /_ping {
 auth_basic off;
 include docker-registry.conf;
 }

 location /v1/_ping {
 auth_basic off;
 include docker-registry.conf;
 }

 location /v2/_ping {
 auth_basic off;
 include docker-registry.conf;
 }
}

将key及crt证书文件放到../ssl目录下。使用htpasswd生成密码放于./上一级目录

 htpasswd -bcm passwd docker docker
 #-c：创建一个加密文件 #-m：md5加密，默认可不填写 #-b：表示用户名密码在命令行中一并输入，不用分别填写

4.再次登录

[centos-master]:docker login -u docker -p docker -e lienhua@zhongchuangsanyou.com registry.evehicle.cn 
Login Succeeded
表示成功，此时再pull\push既在私有库中进行

构建服务

docker的本意是将代码包含在容器内制作成镜像形成“产品”。但出于公司的（频繁修改代码及服务器资源受限）的特殊性，我们将代码以“外挂”的形式运行在宿主机上。下面以部署官网（apache）服务为例：
1.从docker的公有库里下载centos7的原生镜像

[centos-master]:docker pull centos

Using default tag: latest
Trying to pull repository docker.io/library/centos ...
latest: Pulling from docker.io/library/centos
d9aaf4d82f24: Downloading [> ] 540 kB/73.39 MB
d9aaf4d82f24: Pulling fs layer
Digest: sha256:eba772bac22c86d7d6e72421b4700c3f894ab6e35475a34014ff8de74c10872e
Status: Downloaded newer image for centos:latest

2.编写Dockerfile制造apache基础镜像

######httpd####
FROM centos
MAINTAINER lienhua lienhua@zhongchuangsanyou.com
RUN yum -y install epel-release
RUN yum -y install httpd php php-mysql php-memcache* php-mbstring
ADD httpd.conf /etc/httpd/conf/httpd.conf

EXPOSE 80

CMD ["/usr/sbin/apachectl", "-D", "FOREGROUND"]

其中httpd.conf文件需要在当前目录下真实存在，此处其内容为

ServerRoot "/etc/httpd" Listen 80
Listen 8080
Include conf.modules.d/*.conf
Include zcsy/*.conf
User apache
Group apache
ServerAdmin root@localhost
<Directory /> AllowOverride none
 Require all denied
</Directory> DocumentRoot "/var/www/html" <Directory "/var/www"> AllowOverride None
 Require all granted
</Directory> <Directory "/var/www/html"> Options Indexes FollowSymLinks
 AllowOverride None
 Require all granted
</Directory> <IfModule dir_module> DirectoryIndex index.html
</IfModule> <Files ".ht*"> Require all denied
</Files> ErrorLog "logs/error_log" LogLevel warn
<IfModule log_config_module> LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
 LogFormat "%h %l %u %t \"%r\" %>s %b" common
 <IfModule logio_module> LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
 </IfModule> CustomLog "logs/access_log" combined
</IfModule> <IfModule alias_module> ScriptAlias /cgi-bin/ "/var/www/cgi-bin/" </IfModule> <Directory "/var/www/cgi-bin"> AllowOverride None
 Options None
 Require all granted
</Directory> <IfModule mime_module> TypesConfig /etc/mime.types
 AddType application/x-compress .Z
 AddType application/x-gzip .gz .tgz
 AddType application/x-httpd-php .php
 AddType text/html .shtml
 AddOutputFilter INCLUDES .shtml
</IfModule> AddDefaultCharset UTF-8
<IfModule mime_magic_module> MIMEMagicFile conf/magic
</IfModule> EnableSendfile off EnableMMAP off IncludeOptional conf.d/*.conf

执行[centos-master]:docker build -t registry.evehicle.cn/httpd . 命令制作名为”registry.evehicle.cn/httpd”的镜像（注意此处的点必须要有，并且其意义代表当前目录下的Dockerfile文件）

3.将制作好的镜像上传到私有库

docker push registry.evehicle.cn/httpd

4.编写启动apache服务的yaml文件

[centos-master]:cat 13-rc-httpd.yaml

apiVersion: v1
kind: ReplicationController
metadata:
 name: 13-rc-httpd
 labels:
 name: 13-rc-httpd
spec:
 replicas: 2
 selector:
 name: 13-rc-httpd
 template:
 metadata:
 labels:
 name: 13-rc-httpd
 spec:
 containers:
 - name: 13-rc-httpd
 image: registry.evehicle.cn/httpd
 env:
 - name: LANG  value: en_US.UTF-8
 ports:
 - containerPort: 80  hostPort: 80
 volumeMounts:
 - name: time
 mountPath: /etc/localtime
 - name: zcsy
 mountPath: /etc/httpd/zcsy
 - name: deploy
 mountPath: /docker/httpd/deploy
 - name: log
 mountPath: /var/log/httpd
 volumes:
 - name: time
 hostPath:
 path: /etc/localtime
 - name: zcsy
 hostPath:
 path: /docker/httpd/zcsy
 - name: deploy
 hostPath:
 path: /docker/httpd/deploy
 - name: log
 hostPath:
 path: /docker/httpd/log
 nodeSelector:
 slave: "13"
 imagePullSecrets:
 - name: registrykey

5.给其中一个node加上标签为“13”

kubectl label nodes centos-minion-1 slave=13

6.此时拥有标签“13”的nodes应具备的条件

/docker/httpd/zcsy下需要有官网的配置文件

<VirtualHost *:80> ServerName www.evehicle.cn
 DocumentRoot /var/deploy/wordpress/
 RewriteEngine on RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-d
 RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-f
 RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} !^.*\.(ico|pdf|flv|jpe?g|js|gif|png|html|shtml|zip|xml|gz|rar|swf|txt|apk|bmp|css|m4a|ogg|mp3|ipa|plist)$
 RewriteCond %{REQUEST_URI} !^/server-status$
 RewriteRule . /index.php [QSA,PT,L] </VirtualHost> <Directory /var/deploy/wordpress/> Options FollowSymLinks
 AllowOverride All Require all granted
</Directory>

以及/docker/httpd/deploy下需要有官网的代码

7.运行yaml文件启动容器

[centos-master]: kuberctl create -f 13-rc-httpd.yaml

8.查看服务

[centos-master]: kuberctl get rc

NAME DESIRED CURRENT AGE
13-rc-httpd 2 2 168d

9.程序中涉及的mysql\redis\memcache等服务也需使用容器运行起来

[centos-master]: docker pull redis
[centos-master]: docker tag registry.evehicle.cn/redis redis
[centos-master]: docker push registry.evehicle.cn/redis
[centos-master]: kubectl create -f rc-redis.yaml
[centos-master]: cat rc-redis.yaml

apiVersion: v1
kind: ReplicationController
metadata:
 name: redis
 labels:
 name: redis
spec:
 replicas: 2
 selector:
 name: redis
 template:
 metadata:
 labels:
 name: redis
 spec:
 containers:
 - name: redis
 image: registry.evehicle.cn/redis
 ports:
 - containerPort: 6379  hostPort: 6379
 volumeMounts:
 - name: data
 mountPath: /data
 - name: time
 mountPath: /etc/localtime
 volumes:
 - name: data
 hostPath:
 path: /docker/redis/6379
 - name: time
 hostPath:
 path: /etc/localtime
 nodeSelector:
 slave: "13"
 imagePullSecrets:
 - name: registrykey

启动memcache
[centos-master]: docker pull memcache
[centos-master]: docker tag registry.evehicle.cn/memcached memcache
[centos-master]: docker push registry.evehicle.cn/memcached
[centos-master]: kubectl create -f rc-memcached.yaml
[centos-master]: cat rc-memcached.yaml

apiVersion: v1
kind: ReplicationController
metadata:
 name: memcached
 labels:
 name: memcached
spec:
 replicas: 3
 selector:
 name: memcached
 template:
 metadata:
 labels:
 name: memcached
 spec:
 containers:
 - name: memcached
 image: registry.evehicle.cn/memcached
 ports:
 - containerPort: 11211
 hostPort: 11211 #nodeSelector: # slave: "13"
 imagePullSecrets:
 - name: registrykey

制造mysql镜像
[centos-master]: cat Dockerfile

FROM alpine


COPY startup.sh /startup.sh
RUN addgroup mysql && \
 adduser -H -D -s /bin/false -G mysql mysql && \
 apk add --update mysql mysql-client && rm -f /var/cache/apk/* && \
 mkdir /data && \
 chown -R mysql:mysql /data /etc/mysql && \
 chmod 755 /startup.sh \
 ;


WORKDIR /data VOLUME /data VOLUME /etc/mysql


EXPOSE 3306 CMD ["/startup.sh"]

启动mysql（建议mysql在宿主机启动）
[centos-master]: docker build -t registry.evehicle.cn/mysql
[centos-master]: docker push registry.evehicle.cn/mysql
[centos-master]: kubectl create -f rc-mysql.yaml
[centos-master]: cat rc-mysql.yaml

apiVersion: v1
kind: ReplicationController
metadata:
 name: 13-rc-mysql
 labels:
 name: 13-rc-mysql
spec:
 replicas: 2
 selector:
 name: 13-rc-mysql
 template:
 metadata:
 labels:
 name: 13-rc-mysql
 spec:
 containers:
 - name: 13-rc-mysql
 image: registry.evehicle.cn/mysql
 env:
 - name: MYSQL_DATABASE  value: admin
 - name: MYSQL_USER  value: tony
 - name: MYSQL_PASSWORD  value: 456
 - name: MYSQL_ROOT_PASSWORD  value: 123
 ports:
 - containerPort: 3306  hostPort: 3306
 volumeMounts:
 - name: time
 mountPath: /etc/localtime
 - name: data
 mountPath: /data
 - name: etc
 mountPath: /etc/mysql
 - name: run
 mountPath: /run/mysqld
 volumes:
 - name: time
 hostPath:
 path: /etc/localtime
 - name: data
 hostPath:
 path: /docker/mysql/data
 - name: etc
 hostPath:
 path: /docker/mysql/etc
 - name: run
 hostPath:
 path: /docker/mysql/run
 nodeSelector:
 slave: "13"
 imagePullSecrets:
 - name: registrykey

为方便代码编写及统一管理，应提前做好内部DNS解析。将所负责的应用规整到对应的机器上。

本文转自CSDN-k8s环境部署及使用方式

k8s环境部署及使用方式

搭建私有库

构建服务

热门文章

最新文章

相关课程

相关电子书

相关实验场景

推荐镜像

热门

活动广场

任务中心

开发者评测

高校计划

乘风者计划

训练营

阿里云MVP

话题

直播

下载

镜像站

技术资料

插件

k8s环境部署及使用方式

搭建私有库

构建服务

热门文章

最新文章

相关课程

相关电子书

相关实验场景

推荐镜像