系统版本：CentOS Linux release 7.8.2003 (Core)
安装iscsi服务
yum -y install targetcli
systemctl enable target
systemctl start target
创建磁盘
targetcli
创建设备
/> cd /backstores/fileio
/backstores/fileio> create disk01 /disk/disk01.img 10G
backstores表示支持的存储对象的后端类型，主要由
block：block类型存储对象适用于本地块设备和逻辑设备
fileio：fileio类型存储对象适用于存储在本地磁盘上的常规文件（镜像文件，稀疏文件）
pscsi：pscsi类型存储对象适用于直接通过SCSI命令访问的资源
ramdisk：ramdisk类型存储对象适用于临时缓存设备，支持多session
创建目标
cd /iscsi
配置ISCSITarget命名（iqn）
/iscsi> create iqn.2021-10.snow.com:st01
创建lun
/iscsi> cd iqn.2021-10.snow.com:st01/tpg1/luns
/iscsi/iqn.20...t01/tpg1/luns> create /backstores/fileio/disk01
创建ACL允许ISCSI客户机连接：（输入客户端的iqn）
cd /iscsi/iqn.2021-10.snow.com:st01/tpg1/acls
/iscsi/iqn.20...t01/tpg1/acls> create iqn.1994-05.com.redhat:e087af342dee
保存配置
/> saveconfig
安装客户端
yum -y install iscsi-initiator-utils
systemctl enable iscsi
systemctl start iscsi
查看（iqn）
cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.1994-05.com.redhat:e087af342dee
发现
iscsiadm -m discovery -t st -p 192.168.0.15
192.168.0.15:3260,1 iqn.2021-10.snow.com:st01
登录
iscsiadm -m node -T iqn.2021-10.snow.com:st01 -p 192.168.0.15:3260 -l
服务端添加磁盘
cd /backstores/fileio
/backstores/fileio> create disk02 /disk/disk02.img 20G
/backstores/fileio> cd /iscsi/iqn.2021-10.snow.com:st01/tpg1/luns/
/iscsi/iqn.20...t01/tpg1/luns> create /backstores/fileio/disk02
客户端重新扫描磁盘：（发现20G新磁盘）
[root@hecs-66a2 host2]# lsscsi
[2:0:0:0] disk LIO-ORG disk01 4.0 /dev/sda
[2:0:0:0]中的数字2，host2代表的主机号。
[root@hecs-66a2 host2]# echo "- - -" > /sys/class/scsi_host/host2/scan
[root@hecs-66a2 host2]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 10G 0 disk
sdb 8:16 0 20G 0 disk《=====新增加的LUN!添加成功。。。
vda 253:0 0 40G 0 disk
└─vda1 253:1 0 40G 0 part /
多路径软件安装
yum install device-mapper-multipath -y
lsmod |grep multipath
modprobe dm-multipath
modprobe dm-round-robin
lsmod |grep multipath
生成默认配置文件
/sbin/mpathconf --enable
修改配置文件规则（注释find_multipaths）
sed -i 's/find_multipaths/#find_multipaths/' /etc/multipath.conf
否则单条路径磁盘，聚合失败。
find_multipaths如置为yes，则将为3种场景创建多路径设备（不在黑名单的两个路径的wwid相同；用户手动强制创建；一个路径的wwid与之前已经创建的多路径设备相同）
user_friendly_names:如为yes，则用/etc/multipath/bindings中的设置命名.
加载路径配置文件
multipath -r
查看聚合路径信息
multipath -ll
启动多路径软件
systemctl enable multipathd.service
systemctl restart multipathd.service
常用命令参考：
/usr/lib/udev/scsi_id -g -u /dev/sda(手动检查wwid)
multipath -r （修改multipath.conf配置文件之后重新加载）
multipath -ll （查看多路径状态）
multipath -v2 （格式化路径，检测路径，合并路径）
multipath -v3 （查看多路径详情blacklist、whitelist和设备wwid）
multipath -F （删除现有链路）
多路径测试：
添加多个ip地址：
ip a a 10.10.10.10/24 dev eth0
ip a a 20.20.20.20/24 dev eth0
扫描每条路径：
iscsiadm -m discovery -t st -p 10.10.10.10
iscsiadm -m node -T iqn.2021-10.snow.com:st01 -p 10.10.10.10:3260 -l
iscsiadm -m discovery -t st -p 20.20.20.20
iscsiadm -m node -T iqn.2021-10.snow.com:st01 -p 20.20.20.20:3260 -l
显示多路径情况：
mpathb (36001405a8111ea8e7e54c8fb25613092) dm-1 LIO-ORG ,disk02
size=20G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=0 status=active
|- 2:0:0:1 sdb 8:16 active undef running
|- 3:0:0:1 sdd 8:48 active undef running
`- 4:0:0:1 sdf 8:80 active undef running
mpatha (360014054170ebd76f8b4cd69600e1ba9) dm-0 LIO-ORG ,disk01
size=10G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=0 status=active
|- 2:0:0:0 sda 8:0 active ready running
|- 3:0:0:0 sdc 8:32 active ready running
`- 4:0:0:0 sde 8:64 active ready running
更改多路径磁盘权限：
参考模板：
/usr/share/doc/device-mapper-1.02.170/12-dm-permissions.rules
编写规则：
[root@hecs-66a2 ~]# cat /etc/udev/rules.d/99-oracle-asm.rules
ENV{DM_UUID}=="mpath-?*", OWNER:="grid", GROUP:="asmadmin", MODE:="660"
ENV{DM_UUID}=="mpath-?*", OWNER:="grid", GROUP:="asmadmin", MODE:="660",SYMLINK+="asmdisk_%k"
ENV{DM_UUID}=="mpath-?*", OWNER:="grid", GROUP:="asmadmin", MODE:="660",SYMLINK="asmdisk_%k"
重新加载规则
udevadm control -R
udevadm trigger
权限检查：（已经更改为grid:asmadmin）
[root@hecs-66a2 rules.d]# ll /dev/dm-*
brw-rw---- 1 grid asmadmin 252, 0 Oct 30 08:45 /dev/dm-0
brw-rw---- 1 grid asmadmin 252, 1 Oct 30 08:45 /dev/dm-1
[root@hecs-66a2 rules.d]# ll /dev/mapper/mpath*
lrwxrwxrwx 1 root root 7 Oct 30 08:45 /dev/mapper/mpatha -> ../dm-0
lrwxrwxrwx 1 root root 7 Oct 30 08:45 /dev/mapper/mpathb -> ../dm-1
SYMLINK+与 SYMLINK区别？
SYMLINK+
create a symbolic link 。
SYMLINK：
instead of calling it 。
- SYMLINK - a list of symbolic links which act as alternative names for the device node
As hinted above, udev only creates one true device node for one device. If you wish to provide alternate names for this device node, you use the symbolic link functionality. With the SYMLINK assignment, you are actually maintaining a list of symbolic links, all of which will be pointed at the real device node. To manipulate these links, we introduce a new operator for appending to lists: +=. You can append multiple symlinks to the list from any one rule by separating each one with a space.
KERNEL=="hdb", NAME="my_spare_disk"
The above rule says: match a device which was named by the kernel as hdb, and instead of calling it hdb, name the device node as my_spare_disk. The device node appears at /dev/my_spare_disk.
KERNEL=="hdb", DRIVER=="ide-disk", SYMLINK+="sparedisk"
The above rule says: match a device which was named by the kernel as hdb AND where the driver is ide-disk. Name the device node with the default name and create a symbolic link to it named sparedisk. Note that we did not specify a device node name, so udev uses the default. In order to preserve the standard /dev layout, your own rules will typically leave the NAME alone but create some SYMLINKs and/or perform other assignments.
KERNEL=="hdc", SYMLINK+="cdrom cdrom0"
The above rule is probably more typical of the types of rules you might be writing. It creates two symbolic links at /dev/cdrom and /dev/cdrom0, both of which point at /dev/hdc. Again, no NAME assignment was specified, so the default kernel name (hdc) is used.
Linux 识别新添加的LUN?
Dynamically adding storage for use with multipath I/O
echo "- - -" > /sys/class/scsi_host/host/scan
inux系统提供多重机制以重新扫描SCSI总线并重认系统中加入的SCSI设备。在2.4内核方案中，由于动态LUN扫描机制不具备一致性，往往需要中断I/O。
2.6内核里，LUN扫描有了显著改进并添加了动态LUN扫描机制。Linux目前缺乏像drvconfig或ioscan那样允许动态SCSI通道重配的命令。
Linux主机对磁盘设备进行重新配置的方式包括：
重启系统
卸载并重新加载HBA驱动模块
Echo /proc下的SCSI设备列表
通过/sys下的属性设置运行SCSI扫描
通过HBA厂商脚本运行SCSI扫描
cd /sys/class/scsi_host
echo '- - -' > host0/scan
操作系统版本redhat 6 , QLogic HBA驱动
echo 1 > /sys/class/fc_host/host1/issue_lip
echo 1 > /sys/class/fc_host/host2/issue_lip
echo "- - -" >/sys/class/scsi_host/host1/scan
echo "- - -" >/sys/class/scsi_host/host2/scan
利用HBA厂商的脚本执行scan，例如：
QLogic # ./ql-dynamic-tgt-lun-disc.sh --scan
Emulex # lun_scan all
https://driverdownloads.qlogic.com/QLogicDriverDownloads_UI/ResourceByOS.aspx?productid=924&oemid=65&oemcatid=58604
The simplest way to use this utility is to run
