iscsi multipath
1、iscsi
1、target（server端）
1、准备逻辑卷作为iscsi的存储设备。
[root@master ~]# pvs
PV VG Fmt Attr PSize PFree
/dev/sdd1 share lvm2 a-- <6.00g 1020.00m
[root@master ~]# vgs
VG #PV #LV #SN Attr VSize VFree
share 1 1 0 wz--n- <6.00g 1020.00m
[root@master ~]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
sharedata share -wi-a----- 5.00g 2、安装iscsi服务、配置服务端
[root@master ~]# yum install targetcli -y
[root@master ~]# targetcli
targetcli shell version 2.1.53
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.
/> ls
o- / ............................................................................... [...]
o- backstores .................................................................... [...]
| o- block ........................................................ [Storage Objects: 0]
| o- fileio ....................................................... [Storage Objects: 0]
| o- pscsi ........................................................ [Storage Objects: 0]
| o- ramdisk ...................................................... [Storage Objects: 0]
o- iscsi .................................................................. [Targets: 0]
o- loopback ............................................................... [Targets: 0]
#1、创建块存储对象
/> backstores/block create server1storage /dev/share/sharedata
Created block storage object server1storage using /dev/share/sharedata.
/> #2、/dev/share/sharedata在iSCSI系统中的名字为westos:storage1，这里会创建默认端口
/> iscsi/ create iqn.2018-11.com.westos:storage1
Created target iqn.2018-11.com.westos:storage1.
Created TPG 1.
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port 3260.
/> #3、设备共享到网上的名字iqn.2018-11.com.westos:storage1,发起端与目标需要唯一名称进行标识
/> iscsi/iqn.2018-11.com.westos:storage1/tpg1/luns create /backstores/block/server1storage
Created LUN 0.
#4、名字与设备建立acl访问策略
/> /iscsi/iqn.2018-11.com.westos:storage1/tpg1/acls create iqn.2018-11.com.westos:storagekey
Created Node ACL for iqn.2018-11.com.westos:storagekey
Created mapped LUN 0.
#5、加密,创建访问控制列表,创建失败，原因第一步骤生成端口
/> iscsi/iqn.2018-11.com.westos:storage1/tpg1/portals create 10.10.10.6
Using default IP port 3260
Could not create NetworkPortal in configFS
#6、删除存在的端口
/> iscsi/iqn.2018-11.com.westos:storage1/tpg1/portals delete 0.0.0.0 3260
Deleted network portal 0.0.0.0:3260
#7、加密,创建访问控制列表,
/> iscsi/iqn.2018-11.com.westos:storage1/tpg1/portals create 10.10.10.6
Using default IP port 3260
Created network portal 10.10.10.6:3260.
2、initiator（clent端）
1、安装客户端、使用
[root@node-1 ~]# yum install iscsi-initiator-utils
[root@node-1 ~]# systemctl start iscsid
# 查找iSCSI服务器所提供的iSCSI目标
[root@node-1 ~]# iscsiadm -m discovery -t st -p 10.10.10.6
10.10.10.6:3260,1 iqn.2018-11.com.westos:storage1
#设置发起端的IQN
[root@node-1 ~]# vim /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2018-11.com.westos:storagekey
#登录服务器上的一个iscsi目标
[root@node-1 ~]# iscsiadm -m node -T iqn.2018-11.com.westos:storage1 -p 10.10.10.6 -l
Logging in to [iface: default, target: iqn.2018-11.com.westos:storage1, portal: 10.10.10.6,3260] (multiple)
Login to [iface: default, target: iqn.2018-11.com.westos:storage1, portal: 10.10.10.6,3260] successful.
# 查看会话
#-P 3 查看更详细信息
[root@node-1 ~]# iscsiadm -m session
tcp: [3] 10.10.10.6:3260,1 iqn.2018-11.com.westos:storage1 (non-flash)
#可以进行分区格式化挂载了
[root@node-1 ~]# fdisk -l
Disk /dev/sdc: 5368 MB, 5368709120 bytes, 10485760 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 33550336 bytes
设置自动挂载/etc/fstab
_netdev,表示开机挂载之前先启动网络服务（如若未先启动网络服务则会导致出错）
[root@master ~]# cat /etc/fstab /dev/sda1 /data xfs defaults,_netdev 0 0
3、卸载删除设备
#1、登出iscsi目标，暂时断开连接
[root@node-1 ~]# iscsiadm -m node -T iqn.2018-11.com.westos:storage1 -p 10.10.10.6 -u
Logging out of session [sid: 3, target: iqn.2018-11.com.westos:storage1, portal: 10.10.10.6,3260]
Logout of [sid: 3, target: iqn.2018-11.com.westos:storage1, portal: 10.10.10.6,3260] successful.
#2、删除iscsi目标的本地记录，以永久断开连接
[root@node-1 ~]# iscsiadm -m node -T iqn.2018-11.com.westos:storage1 -p 10.10.10.6 -o delete
#3、清理target端
[root@master ~]# targetcli
/> clearconfig confirm=True 2、multipath
1、安装multipath软件
[root@node-1 ~]# yum install device-mapper-multipath
[root@node-1 ~]# systemctl start multipathd #服务启动缺少配置文件会失败
[root@node-1 ~]# find / -name multipath.conf
/usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf
[root@node-1 ~]# cp /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf /etc/multipath.conf
[root@node-1 ~]# systemctl restart multipathd #安装成功
[root@node-1 ~]# lsmod |grep dm #多路径需要依赖内核模块
dm_multipath 27792 0
dm_mod 124501 1 dm_multipath将不需要聚合的磁盘进行黑名单
[root@node-1 ~]# vim /etc/multipath.conf
blacklist {
devnode "^rbd"}
2、查看multipath
wwid更稳定，一个存储盘格式化后wwid都不会改变，存储盘本身的uuid不变格式化后会发现存储盘分区有了uuid，也更了解了
在应用服务器上/ets/fstab里面只能使用uuid，不能使用wwid，因为分区格式化后才有uuid
[root@node-1 ~]# multipath -l
353333330000007d0 dm-0 Linux ,scsi_debug
size=8.0M features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=0 status=active
| `- 7:0:0:0 sdd 8:48 active undef running
`-+- policy='service-time 0' prio=0 status=enabled
`- 8:0:0:0 sde 8:64 active undef running3、使用multipath
# -r 重新载入 查看报错
[root@node-1 ~]# multipath -r -v3 |grep 353333330000036b0
May 13 17:44:23 | sdc: uid = 353333330000036b0 (udev)
353333330000036b0 6:0:0:0 sdc 8:32 1 undef undef Li
May 13 17:44:23 | wwid 353333330000036b0 not in wwids file, skipping sdc
#单盘加入多路径
[root@node-1 ~]# multipath -a 353333330000036b0
wwid '353333330000036b0' added
[root@node-1 ~]# multipath -r
create: 353333330000036b0 undef Linux ,scsi_debug
size=8.0M features='0' hwhandler='0' wp=undef
`-+- policy='service-time 0' prio=1 status=undef
`- 6:0:0:0 sdc 8:32 undef ready running
reload: 353333330000007d0 undef Linux ,scsi_debug
size=8.0M features='0' hwhandler='0' wp=undef
|-+- policy='service-time 0' prio=1 status=undef
| `- 7:0:0:0 sdd 8:48 active ready running
`-+- policy='service-time 0' prio=1 status=undef
`- 8:0:0:0 sde 8:64 active ready running
# 删除多路径设备
[root@node-1 ~]# multipath -f 353333330000007d0
# 删除所有多路径设备
[root@node-1 ~]# multipath -F
# 使用重载重新刷出来
[root@node-1 ~]# multipath -r
# -f删除之后，删除lsscsi设备
[root@node-1 ~]# lsscsi
[0:0:0:0] cd/dvd QEMU QEMU DVD-ROM 2.5+ /dev/sr0
[2:0:0:0] disk QEMU QEMU HARDDISK 2.5+ /dev/sda
[2:0:0:1] disk QEMU QEMU HARDDISK 2.5+ /dev/sdb
[6:0:0:0] disk Linux scsi_debug 0004 /dev/sdc
[7:0:0:0] disk Linux scsi_debug 0004 /dev/sdd
[8:0:0:0] disk Linux scsi_debug 0004 /dev/sde
[root@node-1 ~]# echo 1 > /sys/block/sdc/device/delete
4、multipath属性
[root@node-1 ~]# multipath -ll
mpatha (353333330000007d0) dm-0 Linux ,scsi_debug
size=8.0M features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active #路径组
| `- 7:0:0:0 sdd 8:48 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
`- 8:0:0:0 sde 8:64 active ready running
[root@con01 ~]# multipath -ll
360050767088108620000000000000047 dm-2 INSPUR ,MCS
size=20G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| |- 18:0:0:1 sdk 8:160 active ready running #路径组 按照优先级分组（group_by_prio）
| `- 17:0:0:1 sdj 8:144 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled #路径组
|- 16:0:0:1 sdi 8:128 active ready running
`- 15:0:0:1 sdh 8:112 active ready running
mpatha #友好名称，对应cat /etc/multipath.conf | grep user_friendly_names
[root@node-1 ~]# ll /dev/mapper/mpatha
lrwxrwxrwx 1 root root 7 May 13 20:21 /dev/mapper/mpatha -> ../dm-0
[root@node-1 ~]# ls /dev/dm-0
/dev/dm-0
Linux
# 厂商名称 > /etc/multipath.conf blacklist_exceptions.device.vendor
# > cat /sys/block/sdd/device/vendor
scsi_debug# cat /sys/block/sdd/device/model
features # 支持很多特性
hwhandler # hardware handler 硬件存储相关
wp # 操作权限
policy # 路径选择算法 > service-time（选择延时最短）
# > round-robin (轮询)
# > queue-length (io路径最少)
prio # 优先级
path_grouping_policy #如上12行 group_by_prio multibus(所有路径一个组) failover（一个路径一个组）
17:0:0:1 # host:channel:targetID:lunid