参考
- https://www.cnblogs.com/pengdonglin137/p/17115545.html
- https://blog.51cto.com/u_5041419/4728883
- Linux系统multipath Iscsi多路径实验
框图
步骤
上文中在ubuntu下搭建了iscsi存储，下面基于iscsi存储创建multipath环境。
- 给Host添加一块存储设备
如/dev/nvme0n2
- 配置tgt
cat /etc/tgt/conf.d/iscsi.conf
cat /etc/tgt/conf.d/iscsi.conf <target iqn.2023-02.pendl.com:disk1> backing-store /dev/nvme0n1 initiator-address 192.168.159.144 </target> <target iqn.2023-02.pendl.com:disk2> backing-store /dev/nvme0n2 </target>
上面第二个target是要创建的multipath iscsi存储，对应的是host上的/dev/nvme0n2，这里没有指定IP地址，所有可以通过Host的多个IP访问。
- 重启tgt，并查看状态
systemctl restart tgt systemctl status tgt
可以看到如下信息：
➜ x86 sudo systemctl status tgt.service ● tgt.service - (i)SCSI target daemon Loaded: loaded (/lib/systemd/system/tgt.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2023-02-13 17:35:23 PST; 2s ago Docs: man:tgtd(8) Process: 8403 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS) Process: 8404 ExecStartPost=/usr/sbin/tgt-admin -e -c /etc/tgt/targets.conf (code=exited, status=0/SUCCESS) Process: 8455 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v ready (code=exited, status=0/SUCCESS) Main PID: 8402 (tgtd) Status: "Starting event loop..." Tasks: 33 Memory: 1.6M CGroup: /system.slice/tgt.service └─8402 /usr/sbin/tgtd -f Feb 13 17:35:23 ubuntu systemd[1]: Starting (i)SCSI target daemon... Feb 13 17:35:23 ubuntu tgtd[8402]: tgtd: iser_ib_init(3431) Failed to initialize RDMA; load kernel modules? Feb 13 17:35:23 ubuntu tgtd[8402]: tgtd: work_timer_start(146) use timer_fd based scheduler Feb 13 17:35:23 ubuntu tgtd[8402]: tgtd: bs_init(387) use signalfd notification Feb 13 17:35:23 ubuntu tgtd[8402]: tgtd: device_mgmt(246) sz:18 params:path=/dev/nvme0n1 Feb 13 17:35:23 ubuntu tgtd[8402]: tgtd: bs_thread_open(409) 16 Feb 13 17:35:23 ubuntu tgtd[8402]: tgtd: device_mgmt(246) sz:18 params:path=/dev/nvme0n2 Feb 13 17:35:23 ubuntu tgtd[8402]: tgtd: bs_thread_open(409) 16 Feb 13 17:35:23 ubuntu systemd[1]: Started (i)SCSI target daemon.
- 验证target
sudo tgtadm --mode target --op show
可以看到如下信息：
// 这里省去了Target1的信息 Target 2: iqn.2023-02.pendl.com:disk2 System information: Driver: iscsi State: ready I_T nexus information: LUN information: LUN: 0 Type: controller SCSI ID: IET 00020000 SCSI SN: beaf20 Size: 0 MB, Block size: 1 Online: Yes Removable media: No Prevent removal: No Readonly: No SWP: No Thin-provisioning: No Backing store type: null Backing store path: None Backing store flags: LUN: 1 Type: disk SCSI ID: IET 00020001 SCSI SN: beaf21 Size: 21475 MB, Block size: 512 Online: Yes Removable media: No Prevent removal: No Readonly: No SWP: No Thin-provisioning: No Backing store type: rdwr Backing store path: /dev/nvme0n2 Backing store flags: Account information: ACL information: ALL
客户端
- 可以再给客户端增加一块网卡
为了简单，客户端只用了一块网卡，只是服务端有多块网卡。
- 发现target
root@ubuntu-vm:~# iscsiadm -m discovery -t st -p 192.168.159.130 192.168.159.130:3260,1 iqn.2023-02.pendl.com:disk2 root@ubuntu-vm:~# iscsiadm -m discovery -t st -p 192.168.159.144 192.168.159.144:3260,1 iqn.2023-02.pendl.com:disk1 192.168.159.144:3260,1 iqn.2023-02.pendl.com:disk2
- 登录
root@ubuntu-vm:~# iscsiadm -m node -l Logging in to [iface: default, target: iqn.2023-02.pendl.com:disk2, portal: 192.168.159.130,3260] (multiple) Logging in to [iface: default, target: iqn.2023-02.pendl.com:disk2, portal: 192.168.159.144,3260] (multiple) Logging in to [iface: default, target: iqn.2023-02.pendl.com:disk1, portal: 192.168.159.144,3260] (multiple) Login to [iface: default, target: iqn.2023-02.pendl.com:disk2, portal: 192.168.159.130,3260] successful. Login to [iface: default, target: iqn.2023-02.pendl.com:disk2, portal: 192.168.159.144,3260] successful. Login to [iface: default, target: iqn.2023-02.pendl.com:disk1, portal: 192.168.159.144,3260] successful.
登录以后，服务端也会有相应的变化：
Target 2: iqn.2023-02.pendl.com:disk2 System information: Driver: iscsi State: ready I_T nexus information: I_T nexus: 10 Initiator: iqn.2023-02.pendl.com:client alias: ubuntu-vm Connection: 0 IP Address: 192.168.159.130 I_T nexus: 11 Initiator: iqn.2023-02.pendl.com:client alias: ubuntu-vm Connection: 0 IP Address: 192.168.159.144 LUN information: LUN: 0 Type: controller SCSI ID: IET 00020000 SCSI SN: beaf20 Size: 0 MB, Block size: 1 Online: Yes Removable media: No Prevent removal: No Readonly: No SWP: No Thin-provisioning: No Backing store type: null Backing store path: None Backing store flags: LUN: 1 Type: disk SCSI ID: IET 00020001 SCSI SN: beaf21 Size: 21475 MB, Block size: 512 Online: Yes Removable media: No Prevent removal: No Readonly: No SWP: No Thin-provisioning: No Backing store type: rdwr Backing store path: /dev/nvme0n2 Backing store flags: Account information: ACL information: ALL
- 安装multipath软件包
apt-get install multipath-tools
- 查看multipath状态
root@ubuntu-vm:~# multipath -ll mpathc (360000000000000000e00000000020001) dm-0 IET ,VIRTUAL-DISK size=20G features='0' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=enabled |- 2:0:0:1 sda 8:0 active ready running `- 3:0:0:1 sdb 8:16 active ready running
- 分区格式化
root@ubuntu-vm:~# ls /dev/mapper/ -l total 0 crw------- 1 root root 10, 236 Feb 14 09:42 control lrwxrwxrwx 1 root root 7 Feb 14 10:24 mpathc -> ../dm-0
下面对/dev/mpathc进行分区格式化
root@ubuntu-vm:~# fdisk -l /dev/mapper/mpathc Disk /dev/mapper/mpathc: 20 GiB, 21474836480 bytes, 41943040 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes root@ubuntu-vm:~# fdisk /dev/mapper/mpathc Welcome to fdisk (util-linux 2.34). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Device does not contain a recognized partition table. Created a new DOS disklabel with disk identifier 0xf7a0076c. Command (m for help): n Partition type p primary (0 primary, 0 extended, 4 free) e extended (container for logical partitions) Select (default p): Using default response p. Partition number (1-4, default 1): First sector (2048-41943039, default 2048): Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-41943039, default 41943039): Created a new partition 1 of type 'Linux' and of size 20 GiB. Command (m for help): w The partition table has been altered. Syncing disks. root@ubuntu-vm:~# mkfs.ext4 /dev/mapper/mpathc mke2fs 1.45.5 (07-Jan-2020) Found a dos partition table in /dev/mapper/mpathc Proceed anyway? (y,N) y Creating filesystem with 5242880 4k blocks and 1310720 inodes Filesystem UUID: cdf9c090-5de1-41d9-a264-5ccdc45a27e2 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000 Allocating group tables: done Writing inode tables: done Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: done BASH 折叠 复制 全屏
- 挂载
mount /dev/mapper/mpathc ./demo
