一.devicemapper介绍
|
1
2
3
4
5
6
7
8
9
|
Device Mapper是Linux系统中基于内核的高级卷管理技术框架。Docker的devicemapper存储驱动就是基于该框架的精简置备和快照功能来实现镜像和容器的管理。
注:Device Mapper是Linux的一种技术框架，而devicemapper是Docker Engine基于Device Mapper提供的一种存储驱动。
早期的Docker运行在Ubuntu和Debian Linux上并使用AUFS作为后端存储。Docker流行之后，越来越多的的公司希望在Red Hat Enterprise Linux这类企业级的操作系统上面运行Docker，但可惜的是RHEL的内核并不支持AUFS。
这个时候红帽公司出手了，决定和Docker公司合作去开发一种基于Device Mapper技术的后端存储，也就是现在的devicemapper。
devicemapper驱动将每一个Docker镜像和容器存储在它自身的具有精简置备(thin-provisioned)、写时拷贝(copy-on-write)和快照功能(snapshotting)的虚拟设备上。由于Device Mapper技术是在块(block)层面而非文件层面，所以Docker Engine的devicemapper存储驱动使用的是块设备来存储数据而非文件系统。
|
二.devicemapper的模式
|
1
2
3
4
5
6
7
8
9
10
|
devicemapper是RHEL下Docker Engine的默认存储驱动，它有两种配置模式:loop-lvm和direct-lvm。
loop-lvm是默认的模式，它使用OS层面离散的文件来构建精简池(thin pool)。该模式主要是设计出来让Docker能够简单的被”开箱即用(out-of-the-box)”而无需额外的配置。但如果是在生产环境的部署Docker，官方明文不推荐使用该模式。我们使用docker info命令可以看到以下警告:
WARNING: Usage of loopback devices is strongly discouraged
for
production use. Either use `–storage-opt dm.thinpooldev` or use `–storage-opt dm.no_warn_on_loop_devices=
true
` to suppress this warning.
direct-lvm是Docker推荐的生产环境的推荐模式，他使用块设备来构建精简池来存放镜像和容器的数据。
前段时间有篇很不错的微信文章是关于老司机填devicemapper坑的血泪史，仔细研读之后发现老司机使用的是loop-lvm模式，那个坑有可能由此引起，最终老司机使用overlayfs的存储驱动解决了问题。
注:Linux内核在3.18以上才能支持overlayfs，但RHEL 7.2的内核版本为3.10，所以原生并不支持。但是的确有人在RHEL7.2上成功应用了overlayfs驱动，个人猜测可能是手动在内核里面加载了overlay的模块。
|
三.配置direct-lvm模式
1.停止Docker并备份
如果Docker服务已在运行且有需要保留的镜像和容器，停服务前把相关数据给备份。个人也强烈建议如果是在生产环境使用Docker的话，拿到host的第一时间就将direct-lvm模式给配置了。(当然也可以选择其他的storage driver)
2.查看当前devicemapper模式
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
|
[root@docker ~]
# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[root@docker ~]
# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.12.6
Storage Driver: devicemapper
Pool Name: docker-253:0-403318478-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data
file
:
/dev/loop0
Metadata
file
:
/dev/loop1
Data Space Used: 11.8 MB
Data Space Total: 107.4 GB
Data Space Available: 102.8 GB
Metadata Space Used: 581.6 kB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.147 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported:
true
Deferred Removal Enabled:
true
Deferred Deletion Enabled:
true
Deferred Deleted Device Count: 0
Data loop
file
:
/var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged
for
production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop
file
:
/var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.107-RHEL7 (2015-10-14)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume:
local
Network: null host bridge overlay
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Security Options: seccomp
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 2
Total Memory: 1.954 GiB
Name: docker
ID: HXUH:PR3W:6MQC:2GKX:URWU:QZLM:J6QS:HAXP:QAOP:ECZR:P4Z2:TF7Q
Docker Root Dir:
/var/lib/docker
Debug Mode (client):
false
Debug Mode (server):
false
Registry: https:
//index
.docker.io
/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
127.0.0.0
/8
Registries: docker.io (secure)
[root@docker ~]
#
|
基于docker info查询的结果可以看到当前模式为loop-lvm。
3.停止docker服务
|
1
|
[root@docker ~]
# systemctl stop docker
|
四.分配裸设备
本例以添加硬盘到docker宿主机，推荐使用外部共享存储的设备但不局限于此种方式，可根据自己的环境决定。
添加200GB的硬盘
创建一个Volume Group
将Volume Group挂给docker宿主机
创建VG
1.查看设备
|
1
2
3
4
5
6
7
8
|
[root@docker ~]
# fdisk -l /dev/sdb
Disk
/dev/sdb
: 214.7 GB, 214748364800 bytes, 419430400 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical
/physical
): 512 bytes / 512 bytes
I
/O
size (minimum
/optimal
): 512 bytes / 512 bytes
[root@docker ~]
#
|
2.创建PV
|
1
2
3
|
[root@docker ~]
# pvcreate /dev/sdb
Physical volume
"/dev/sdb"
successfully created.
[root@docker ~]
#
|
3.创建VG
|
1
2
3
|
[root@docker ~]
# vgcreate docker /dev/sdb
Volume group
"docker"
successfully created
[root@docker ~]
#
|
4.查看VG信息
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
[root@docker ~]
# vgs
VG
#PV #LV #SN Attr VSize VFree
docker 1 0 0 wz--n- <200.00g <200.00g
[root@docker ~]
# vgdisplay docker
--- Volume group ---
VG Name docker
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 1
VG Access
read
/write
VG Status resizable
MAX LV 0
Cur LV 0
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size <200.00 GiB
PE Size 4.00 MiB
Total PE 51199
Alloc PE / Size 0 / 0
Free PE / Size 51199 / <200.00 GiB
VG UUID m6or9g-k3Ff-s5eX-w9yz-MjyN-oaA5-l2Vwsw
[root@docker ~]
#
|
创建thinpool
-
创建pool
|
1
2
3
4
5
|
[root@docker ~]
# lvcreate --wipesignatures y -n thinpool docker -l 95%VG
Logical volume
"thinpool"
created.
[root@docker ~]
# lvcreate --wipesignatures y -n thinpoolmeta docker -l 1%VG
Logical volume
"thinpoolmeta"
created.
[root@docker ~]
#
|
数据LV大小为VG的95%,元数据LV大小为VG的1%,剩余的空间用来自动扩展。
2.将pool转换为thinpool
|
1
2
3
4
5
6
7
8
|
[root@docker ~]
# lvcreate --wipesignatures y -n thinpoolmeta docker -l 1%VG
Logical volume
"thinpoolmeta"
created.
[root@docker ~]
# lvconvert -y --zero n -c 512K --thinpool docker/thinpool --poolmetadata docker/thinpoolmeta
Thin pool volume with chunk size 512.00 KiB can address at most 126.50 TiB of data.
WARNING: Converting logical volume docker
/thinpool
and docker
/thinpoolmeta
to thin pool's data and metadata volumes with metadata wiping.
THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Converted docker
/thinpool_tdata
to thin pool.
[root@docker ~]
#
|
配置thinpool
-
配置池的自动扩展
|
1
2
3
4
5
6
|
[root@docker ~]
# cat /etc/lvm/profile/docker-thinpool.profile
activation {
thin_pool_autoextend_threshold=80
thin_pool_autoextend_percent=20
}
[root@docker ~]
#
|
2.应用配置变更
|
1
2
3
|
[root@docker ~]
# lvchange --metadataprofile docker-thinpool docker/thinpool
Logical volume docker
/thinpool
changed.
[root@docker ~]
#
|
3.状态监控检查
|
1
2
3
4
|
[root@docker ~]
# lvs -o+seg_monitor
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Monitor
thinpool docker twi-a-t--- <190.00g 0.00 0.01 monitored
[root@docker ~]
#
|
配置Docker
1.修改服务配置文件
|
1
2
|
[root@docker ~]
# vim /usr/lib/systemd/system/docker.service
--storage-driver=devicemapper --storage-opt=dm.thinpooldev=
/dev/mapper/docker-thinpool
--storage-opt dm.use_deferred_removal=
true
|
ExecStart后加入storage相关配置参数,如果配置了$OPTIONS也可以在对应的EnvironmentFile中加入。
2.清除graphdriver
|
1
|
[root@docker ~]
# rm -rf /var/lib/docker/*
|
之前已提醒数据备份，因为在这里清除graphdriver会将image,Container和volume所有数据都删除。如果不删除，则会遇到以下的错误导致docker服务起不来的。
|
1
|
Error starting daemon: error initializing graphdriver: devmapper: Base Device UUID and Filesystem verification failed: devicemapper: Error running deviceCreate (ActivateDevice) dm_task_run failed
|
启动docker服务
|
1
2
|
[root@docker ~]
# systemctl daemon-reload
[root@docker ~]
# systemctl start docker
|
检查devicemapper配置
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
|
[root@docker ~]
# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 17.12.0-ce
Storage Driver: devicemapper
Pool Name: docker-thinpool
Pool Blocksize: 524.3kB
Base Device Size: 10.74GB
Backing Filesystem: xfs
Udev Sync Supported:
true
Data Space Used: 20.45MB
Data Space Total: 204GB
Data Space Available: 204GB
Metadata Space Used: 266.2kB
Metadata Space Total: 2.143GB
Metadata Space Available: 2.143GB
Thin Pool Minimum Free Space: 20.4GB
Deferred Removal Enabled:
true
Deferred Deletion Enabled:
true
Deferred Deleted Device Count: 0
Library Version: 1.02.140-RHEL7 (2017-05-03)
Logging Driver: json-
file
Cgroup Driver: cgroupfs
Plugins:
Volume:
local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-
file
logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 89623f28b87a6004d4b785663257362d1658a729
runc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.954GiB
Name: docker
ID: IMMY:YLYX:LF5E:GZID:ACCP:4V43:2IPT:MCSD:DINH:MKFJ:DSDV:TWF4
Docker Root Dir:
/var/lib/docker
Debug Mode (client):
false
Debug Mode (server):
false
Registry: https:
//index
.docker.io
/v1/
Labels:
Experimental:
false
Insecure Registries:
127.0.0.0
/8
Live Restore Enabled:
false
[root@docker ~]
#
|
基于docker info查询的结果可以看到当前模式为direct-lvm。
配置 Docker 加速器
|
1
2
3
|
# curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://df98fb04.m.daocloud.io
# systemctl restart docker
# cat /etc/docker/daemon.json
|
测试
pull一个镜像看是否数据会写到thinpool里；
|
1
2
3
4
5
6
7
8
9
10
11
12
13
|
[root@docker ~]
# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
thinpool docker twi-a-t--- <190.00g 0.01 0.01
[root@docker ~]
# docker pull centos
Using default tag: latest
latest: Pulling from library
/centos
af4b0a2388c6: Pull complete
Digest: sha256:2671f7a3eea36ce43609e9fe7435ade83094291055f1c96d9d1d1d7c0b986a5d
Status: Downloaded newer image
for
centos:latest
[root@docker ~]
# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
thinpool docker twi-a-t--- <190.00g 0.13 0.01
[root@docker ~]
#
|
可以看到Data%在pull一个centos镜像后使用率由0.01变为0.13，说明direct-lvm配置成功且正常工作。
本文转自 dengaosky 51CTO博客，原文链接：http://blog.51cto.com/dengaosky/2066706，如需转载请自行联系原作者
