一、简介
iSCSI（internet SCSI）技术由IBM公司研究开发，是一个供硬件设备使用的、可以在IP协议的上层运行的SCSI指令集，这种指令集合可以实现在IP网络上运行SCSI协议，使其能够在诸如高速千兆以太网上进行路由选择。iSCSI技术是一种新储存技术，该技术是将现有SCSI接口与以太网络(Ethernet)技术结合，使服务器可与使用IP网络的储存装置互相交换资料。
iSCSI是一种基于TCP/IP 的协议，用来建立和管理IP存储设备、主机和客户机等之间的相互连接，并创建存储区域网络（SAN）。SAN 使得SCSI 协议应用于高速数据传输网络成为可能，这种传输以数据块级别（block-level）在多个数据存储网络间进行。SCSI 结构基于C/S模式，其通常应用环境是：设备互相靠近，并且这些设备由SCSI 总线连接。
iSCSI 的主要功能是在TCP/IP 网络上的主机系统（启动器 initiator）和存储设备（目标器 target）之间进行大量数据的封装和可靠传输过程。iSCSI可分享的设备类型有很多，包括镜像文件（*.img）、分区（partition）、物理硬盘、raid设备、逻辑卷等
完整的iSCSI系统的拓扑结构如下：
Linux环境挂载iSCSI存储
1. 检查系统是否安装iSCSI initiator
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
|
[root@justin ~]
# rpm -qa|grep iscsi
[root@justin ~]
#
[root@justin ~]
# mount /dev/cdrom /mnt/
mount
: block device
/dev/sr0
is write-protected, mounting
read
-only
[root@justin ~]
# cd /mnt/
[root@justin Packages]
# find . -name *iscsi*
.
/iscsi-initiator-utils-6
.2.0.872-41.el6.i686.rpm
[root@justin Packages]
# yum list|grep iscsi
Unable to
read
consumer identity
iscsi-initiator-utils.i686 6.2.0.872-41.el6 dvd
[root@justin Packages]
# rpm -ivh iscsi-initiator-utils-6.2.0.872-41.el6.i686.rpm ^C
[root@justin Packages]
# yum -y install iscsi-initiator-utils iscsi-initiator-utils-devel
[root@localhost ~]
# rpm -qa|grep iscsi
iscsi-initiator-utils-6.2.0.873-14.el6.x86_64
iscsi-initiator-utils-devel-6.2.0.873-14.el6.x86_64
[root@localhost ~]
#
|
这个安装将iscsid、iscsiadm安装到/sbin目录下，它还将把默认的配置文件安装到/etc/iscsi目录下：
|
1
2
3
4
5
|
[root@localhost ~]
# cat /sbin/iscsi
iscsiadm iscsid iscsi-iname iscsistart iscsiuio
[root@localhost ~]
# cat /etc/iscsi/i
initiatorname.iscsi iscsid.conf
[root@localhost ~]
# cat /etc/iscsi/i
|
/etc/iscsi/iscsid.conf 所有刚发起的iSCSI session默认都将使用这个文件中的参数设定。
/etc/iscsi/initiatorname.iscsi 软件iSCSI initiator的intiator名称配置文件。
在iscsi启动的时候，iscsid和iscsiadm会读取这两个配置文件。
2. 查看Initiator IQNname
|
1
2
3
|
[root@justin Packages]
# more /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.1994-05.com.redhat:19bd3b75485f
[root@justin Packages]
#
|
3.启动iscsi的服务，并将iscsi写到开机自行加载。
|
1
2
3
4
5
6
7
8
|
[root@justin Packages]
# chkconfig --list|grep iscsi
iscsi 0:off 1:off 2:off 3:on 4:on 5:on 6:off
iscsid 0:off 1:off 2:off 3:on 4:on 5:on 6:off
[root@justin Packages]
# service iscsi start
[root@justin Packages]
# service iscsid start
[root@justin Packages]
# /sbin/iscsistart
iscsistart: InitiatorName not
set
. Exiting iscsistart
[root@APP src]
#
|
安装完成后使用service iscsi start 和service iscsid start无法启动尝试使用/sbin/iscsistart ，启动报：InitiatorName not set. Exiting iscsistart，解决：直接使用iscsiadm通过下面的第4步查询target，自动启动iscsi和iscsid，如果出现“iscsid dead but pid file exist”,输入命令：rm –fr /var/run/iscsid.pid
4. 查询ISCSI设备target的iqn号码
首先要在存储设备上划分磁盘阵列的磁盘空间（即Virtual disk），设置好iSCSI 目标方（target），
iscsiadm是基于命令行的iscsi管理工具，提供了对iscsi节点、会话、连接以及发现记录的操作。默认情况下，iscsi发起方和目标方之间通过端口3260连接
iscsiadm的使用说明可以查看/usr/share/doc/iscsi-initiator-utils-6.2.0.742/README，也可以运行man iscsiadm或iscsiadm --help
Discover targets at a given IP address:
iscsiadm --mode discoverydb --type sendtargets --portal 192.168.1.10 --discover
|
1
2
3
4
5
6
7
8
9
10
|
[root@localhost ~]
# iscsiadm -m discovery -t st -p 10.0.0.10
Starting iscsid: [ OK ]
iscsiadm: No portals found
[root@localhost ~]
# ps -ef|grep -i iscsi
root 2259 2 0 07:03 ? 00:00:00 [iscsi_eh]
root 2303 1 0 07:03 ? 00:00:00 iscsiuio
root 2308 1 0 07:03 ? 00:00:00 iscsid
root 2309 1 0 07:03 ? 00:00:00 iscsid
root 2314 2154 0 07:04 pts
/0
00:00:00
grep
-i iscsi
[root@localhost ~]
#
|
-m discovery //侦测target
-t sendtargets //通过iscsi协议
-p IP:port //指定target的IP和port，不写port的话，默认为3260
iscsiadm: No portals found
这时我们需要到存储上去允许我们接入，这里需要用到Initiator IQNname
|
1
2
3
|
[root@localhost ~]
# cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.1994-05.com.redhat:19bd3b75485f
[root@localhost ~]
#
|
|
1
2
3
|
[root@localhost ~]
# iscsiadm -m discovery -t st -p 10.0.0.10
10.0.0.10:3260,1 iqn.2001-05.com.equallogic:0-8b0906-8c1c82a0a-88971a5r64956574-vol-167new-2015-11-27-02:00:01.79876.1
[root@localhost ~]
#
|
此时会找到目标target iqn，类似10.0.0.10:3260,1 iqn.2001-05.com.equallogic:0-8b0906-8c1c82a0a-88971a5r64956574-vol-167new-2015-11-27-02:00:01.79876.1 ，被发现的目标也叫做节点
iscsiadm 侦测到的结果会写入/var/lib/iscsi/nodes/ 中，因此只需启动/etc/init.d/iscsi 就能够在下次开机时，自动连接到正确的target了。
|
1
2
3
4
|
[root@localhost ~]
# cd /var/lib/iscsi/nodes/
[root@localhost nodes]
# ls
iqn.2001-05.com.equallogic:0-8b0906-8c1c82a0a-88971a5r64956574-vol-167new-2015-11-27-02:00:01.79876.1
[root@localhost nodes]
#
|
侦测信息都写入了/var/lib/iscsi/nodes/iqn.2001-05.com.equallogic:0-8b0906-8c1c82a0a-88971a5r64956574-vol-167new-2015-11-27-02:00:01.79876.1/10.0.0.10,3260,1/default 文件中了。
5. 登录target
Login, must use a node record id found by the discovery:
iscsiadm --mode node --targetname iqn.2001-05.com.doe:test --portal 192.168.1.1:3260 --login
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
|
#登录全部扫描到的服务器
[root@justin Packages]
# iscsiadm -m node --login
#登录某一个，targetname为检测到的存储target name
[root@localhost 10.0.0.10,3260,1]
# iscsiadm -m node --targetname iqn.2001-05.com.equallogic:0-8b0906-8c1c82a0a-88971a5r64956574-vol-167new-2015-11-27-02:00:01.79876.1 -p 10.0.0.10 -l
Logging
in
to [iface: default, target: iqn.2001-05.com.equallogic:0-8b0906-8c1c82a0a-88971a5r64956574-vol-167new-2015-11-27-02:00:01.79876.1, portal: 10.0.0.10,3260] (multiple)
Login to [iface: default, target: iqn.2001-05.com.equallogic:0-8b0906-8c1c82a0a-88971a5r64956574-vol-167new-2015-11-27-02:00:01.79876.1, portal: 10.0.0.10,3260] successful.
#查看核对登录信息
[root@localhost 10.0.0.10,3260,1]
# iscsiadm -m session -o show
tcp: [1] 10.0.0.10:3260,1 iqn.2001-05.com.equallogic:0-8b0906-8c1c82a0a-88971a5r64956574-vol-167new-2015-11-27-02:00:01.79876.1 (non-flash)
[root@localhost 10.0.0.10,3260,1]
# fdisk -l
Disk
/dev/sda
: 53.7 GB, 53687091200 bytes
255 heads, 63 sectors
/track
, 6527 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical
/physical
): 512 bytes / 512 bytes
I
/O
size (minimum
/optimal
): 512 bytes / 512 bytes
Disk identifier: 0x0008019d
Device Boot Start End Blocks Id System
/dev/sda1
* 1 26 204800 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2
26 1070 8388608 82 Linux swap / Solaris
/dev/sda3
1070 6528 43834368 83 Linux
Disk
/dev/sdb
: 859.0 GB, 859003944960 bytes
255 heads, 63 sectors
/track
, 104434 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical
/physical
): 512 bytes / 512 bytes
I
/O
size (minimum
/optimal
): 512 bytes / 512 bytes
Disk identifier: 0xbc7a8d15
Device Boot Start End Blocks Id System
/dev/sdb1
1 104434 838866073+ 7 HPFS
/NTFS
[root@localhost 10.0.0.10,3260,1]
#
|
可以看到一个800多G的盘，因为这里挂载的是镜像，挂载lun步骤一样
也可简写
|
1
|
[root@justin Packages]
# iscsiadm -m node -p 10.0.0.10 -l
|
默认重启后也会自动挂载的，如果要在系统启动时自动登入可以如下操作：
|
1
2
|
[root@justin Packages]
# iscsiadm -m node -targetname iqn.1994-05.com.redhat:4ef28aa338b5 -p 192.168.100.252 --op update -n node.startup -v automatic
[root@justin Packages]
# iscsiadm -m node -T iqn.1994-05.com.redhat:4ef28aa338b5 -p 192.168.100.252 --op update -n node.startup -v automatic
|
iqn.1994-05.com.redhat:4ef28aa338b5是目标名 iqn号可以在/etc/iscsi/initiatorname.iscsi中查看
Logout:
iscsiadm --mode node --targetname iqn.2001-05.com.doe:test --portal 192.168.1.1:3260 --logout
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
[root@justin Packages]
# umount /mnt/data
[root@localhost ~]
# iscsiadm -m node --targetname iqn.2001-05.com.equallogic:0-8b0906-8c1c82a0a-88971a5r64956574-vol-167new-2015-11-27-02:00:01.79876.1 -p 10.0.0.10 -u
Logging out of session [sid: 1, target: iqn.2001-05.com.equallogic:0-8b0906-8c1c82a0a-88971a5r64956574-vol-167new-2015-11-27-02:00:01.79876.1, portal: 10.0.0.10,3260]
Logout of [sid: 1, target: iqn.2001-05.com.equallogic:0-8b0906-8c1c82a0a-88971a5r64956574-vol-167new-2015-11-27-02:00:01.79876.1, portal: 10.0.0.10,3260] successful.
[root@localhost ~]
# fdisk -l
Disk
/dev/sda
: 53.7 GB, 53687091200 bytes
255 heads, 63 sectors
/track
, 6527 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical
/physical
): 512 bytes / 512 bytes
I
/O
size (minimum
/optimal
): 512 bytes / 512 bytes
Disk identifier: 0x0008019d
Device Boot Start End Blocks Id System
/dev/sda1
* 1 26 204800 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2
26 1070 8388608 82 Linux swap / Solaris
/dev/sda3
1070 6528 43834368 83 Linux
[root@localhost ~]
#
|
6.显示已经映射结果磁盘阵列：iscsiadm –mode node
|
1
2
3
4
|
[root@justin Packages]
# iscsiadm -m node -o show
[root@justin Packages]
# iscsiadm -m node
10.0.0.10:3260,1 iqn.2001-05.com.equallogic:0-1cb196-d89a91b07-3e549b4f79c51ee0-vol-98.115
[root@SAMBA1 ~]
#
|
7.登入节点后以看到目标方的存储设备信息
在服务器上运行# fdisk –l可以看到目标方的存储设备信息
|
1
2
3
4
5
6
7
8
|
[root@justin Packages]
# fdisk -l
Disk
/dev/sdb
: 3489.8 GB, 3489862254592 bytes
255 heads, 63 sectors
/track
, 424284 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk
/dev/sdb
doesn't contain a valid partition table
|
/dev/sdb即为连接的iscsi存储设备，PS：如果SAN硬盘通过iscsiadm扫描后能看到三个，挂载其中的mapper映射即可
8.对iscsi设备分区、格式化、挂载
|
1
2
|
[root@justin Packages]
# fdisk /dev/sdb
[root@justin Packages]
# mkfs.ext4 /dev/sdb1
|
对iscsi存储设备分区，在开机自动挂载需要注意挂载选项不能使用defaults，应为_netdev，UUID要顶格写（Linux系统重启后，磁盘设备的名称可能会发生变化，从而引起文件系统不能挂载上来或者不能正确挂载，使用UUID的方式进行挂载可以解决这个问题，使用命令“tune2fs -l /dev/sdb”可以查到文件系统的UUID，也可以使用给文件系统设置卷标的方式来解决），例如：
|
1
2
|
[root@justin Packages]
# vi /etc/fstab
UUID=9b71318a-6609-4679-8a69-420c1fffea63
/mnt/data
ext4 _netdev 0 0
|
将/etc/fstab中所有的档案系统都挂载上：
|
1
|
[root@justin Packages]
# mount -a
|
用df查看文件系统是否已经挂载成功
|
1
|
[root@justin Packages]
# df -Th
|
存储容量大于2T，不能用不同分区 fdisk ，只能用GPT分区.
|
1
2
3
4
5
|
[root@justin Packages]
# parted /dev/sdb #对/dev/sdb进行分区
print
#打印信息，从中可以看出这个磁盘大小和分区格式
mklabel gpt
mkpart primary 0% 100%
print
#从中可以看出这个磁盘分区格式已经变了成GPT了
|
备注：如果重新安装OS后，就需要重新获取IQN name。
多路径功能配置
环境
存储：Dell Equallogic
存储IP：10.0.0.10
服务器：eth1 10.0.0.97 eth2 10.0.0.98
1．使用ping命令确保两块网卡到EQ的Group IP地址能通
宕掉eth1后eth2立马就可以通，此时再启动eth1后eth1就不通了，eth1、eth2反过来测试结果每次只有一个IP会通，Linux默认启用了反向路由检查，我们只需要关闭反向路由检查即可。
echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/eth2/rp_filter
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
|
[root@localhost ~]
# cat /proc/sys/net/ipv4/conf/all/rp_filter
0
[root@localhost ~]
# cat /proc/sys/net/ipv4/conf/eth0/rp_filter
1
[root@localhost ~]
# cat /proc/sys/net/ipv4/conf/eth1/rp_filter
1
[root@localhost ~]
# cat /proc/sys/net/ipv4/conf/eth2/rp_filter
1
[root@localhost ~]
# echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter
[root@localhost ~]
# echo 0 > /proc/sys/net/ipv4/conf/eth2/rp_filter
[root@localhost ~]
# ping 10.0.0.10 -I eth1
PING 10.0.0.10 (10.0.0.10) from 10.0.0.97 eth1: 56(84) bytes of data.
64 bytes from 10.0.0.10: icmp_seq=1 ttl=255
time
=0.126 ms
^C
--- 10.0.0.10
ping
statistics ---
1 packets transmitted, 1 received, 0% packet loss,
time
772ms
rtt min
/avg/max/mdev
= 0.126
/0
.126
/0
.126
/0
.000 ms
[root@localhost ~]
# ping 10.0.0.10 -I eth2
PING 10.0.0.10 (10.0.0.10) from 10.0.0.98 eth2: 56(84) bytes of data.
64 bytes from 10.0.0.10: icmp_seq=1 ttl=255
time
=0.136 ms
^C
--- 10.0.0.10
ping
statistics ---
1 packets transmitted, 1 received, 0% packet loss,
time
752ms
rtt min
/avg/max/mdev
= 0.136
/0
.136
/0
.136
/0
.000 ms
[root@localhost ~]
#
|
这个是临时的重启就丢失了，我们需要写入开机启动文件/etc/rc.d/rc.local里
echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/eth2/rp_filter
|
1
2
3
4
5
6
7
8
9
10
11
12
13
|
[root@localhost ~]
# vim /etc/rc.d/rc.local
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.
touch
/var/lock/subsys/local
echo
0 >
/proc/sys/net/ipv4/conf/all/rp_filter
echo
0 >
/proc/sys/net/ipv4/conf/eth1/rp_filter
echo
0 >
/proc/sys/net/ipv4/conf/eth2/rp_filter
[root@localhost ~]
#
|
或者直接修改内核文件/etc/sysctl.conf
|
1
2
3
4
5
6
7
|
[root@localhost ~]
# vim /etc/sysctl.conf
# Controls source route verification
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.eth1.rp_filter = 0
net.ipv4.conf.eth2.rp_filter = 0
[root@localhost ~]
# sysctl -p
|
此时我们通过iscsiadm -m discovery -t st -p 10.0.0.10 还是只能看到一条路径
|
1
2
3
|
root@localhost ~]
# iscsiadm -m discovery -t st -p 10.0.0.10
10.0.0.10:3260,1 iqn.2001-05.com.equallogic:0-8a0906-413c82a0a-36e6675ea9b58bc4-vol-98.115-2017-03-06-01:00:03.112094.5
[root@localhost ~]
#
|
此时我们需要建立iscsi端口，
|
1
2
3
4
5
|
[root@localhost ~]
# iscsiadm -m iface -I iscsi_eth1 --op=new
New interface iscsi_eth1 added
[root@localhost ~]
# iscsiadm -m iface -I iscsi_eth2 --op=new
New interface iscsi_eth2 added
[root@localhost ~]
#
|
绑定iscsi端口，建立iscsi_eth1和eht1、iscsi_eth2和eth2的映射关系.
|
1
2
3
4
5
|
[root@localhost ~]
# iscsiadm -m iface -I iscsi_eth1 --op=update -n iface.net_ifacename -v eth1
iscsi_eth1 updated.
[root@localhost ~]
# iscsiadm -m iface -I iscsi_eth2 --op=update -n iface.net_ifacename -v eth2
iscsi_eth2 updated.
[root@localhost ~]
#
|
也可以直接编辑iscsi_eth2和iscsi_eth1文件，把里面的iface.net_ifacename=default改成iface.net_ifacename=eth2和iface.net_ifacename=eth1即可。
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
|
[root@localhost ~]
# cat /var/lib/iscsi/ifaces/iscsi_eth1
# BEGIN RECORD 6.2.0-873.22.el6
iface.iscsi_ifacename = iscsi_eth1
iface.net_ifacename = eth1
iface.transport_name = tcp
iface.vlan_id = 0
iface.vlan_priority = 0
iface.iface_num = 0
iface.mtu = 0
iface.port = 0
iface.tos = 0
iface.ttl = 0
iface.tcp_wsf = 0
iface.tcp_timer_scale = 0
iface.def_task_mgmt_timeout = 0
iface.erl = 0
iface.max_receive_data_len = 0
iface.first_burst_len = 0
iface.max_outstanding_r2t = 0
iface.max_burst_len = 0
# END RECORD
[root@localhost ~]
#
|
确认iface配置
|
1
2
3
4
5
6
|
[root@info-
file
-server ~]
# iscsiadm -m iface
default tcp,<empty>,<empty>,<empty>,<empty>
iser iser,<empty>,<empty>,<empty>,<empty>
iscsi_eth2 tcp,<empty>,<empty>,eth2,<empty>
iscsi_eth1 tcp,<empty>,<empty>,eth1,<empty>
[root@info-
file
-server ~]
#
|
重启iscsi服务后就可以发现两条路径了
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
|
[root@localhost ~]
# service iscsi restart
Stopping iscsi: [ OK ]
Starting iscsi: iscsiadm: Could not login to [iface: default, target: iqn.2001-05.com.equallogic:0-8a0906-413c82a0a-36e6675ea9b58bc4-vol-98.115-2017-03-06-01:00:03.112094.5, portal: 10.0.0.10,3260].
iscsiadm: initiator reported error (24 - iSCSI login failed due to authorization failure)
iscsiadm: Could not log into all portals
[ OK ]
[root@info-
file
-server ~]
# iscsiadm -m discovery -t st -p 10.0.0.10
iscsiadm: This
command
will remove the record [iface: iscsi_eth2, target: iqn.2001-05.com.equallogic:0-8a0906-d20c82a0a-dfb6675fa0158cff-115test-2017-03-21-00:02:41.113343.1, portal: 10.0.0.10,3260], but a session is using it. Logout session
then
rerun
command
to remove record.
iscsiadm: This
command
will remove the record [iface: iscsi_eth1, target: iqn.2001-05.com.equallogic:0-8a0906-d20c82a0a-dfb6675fa0158cff-115test-2017-03-21-00:02:41.113343.1, portal: 10.0.0.10,3260], but a session is using it. Logout session
then
rerun
command
to remove record.
10.0.0.10:3260,1 iqn.2001-05.com.equallogic:0-8a0906-175c82a0a-b0e6675fa8f58d0e-98.115test
10.0.0.10:3260,1 iqn.2001-05.com.equallogic:0-8a0906-175c82a0a-b0e6675fa8f58d0e-98.115test
[root@info-
file
-server ~]
# iscsiadm -m node -l
Logging
in
to [iface: iscsi_eth2, target: iqn.2001-05.com.equallogic:0-8a0906-413c82a0a-36e6675ea9b58bc4-vol-98.115-2017-03-06-01:00:03.112094.5, portal: 10.0.0.10,3260] (multiple)
|