ibping
先在一台服务器上使用ibping开启服务器模式，这台服务器的配置信息
[root@storage02 ~]# ibv_devices
device node GUID
------ ----------------
mlx4_0 0002c90300b382a0
irdma0 ae1f6bfffeec331c
irdma1 ae1f6bfffeec331d
[root@storage02 ~]# ibstat mlx4_0
CA 'mlx4_0'
CA type: MT4099
Number of ports: 2
Firmware version: 2.42.5000
Hardware version: 1
Node GUID: 0x0002c90300b382a0
System image GUID: 0x0002c90300b382a3
Port 1:
State: Down
Physical state: Polling
Rate: 10
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x02594868
Port GUID: 0x0002c90300b382a1
Link layer: InfiniBand
Port 2:
State: Active
Physical state: LinkUp
Rate: 56
Base lid: 5
LMC: 0
SM lid: 1
Capability mask: 0x0259486a
Port GUID: 0x0002c90300b382a2
Link layer: InfiniBand
[root@storage02 ~]# ibping -S -C mlx4_0 -P 2
然后另一台机器开启client模式
[root@storage03 ~]# ibv_devices
device node GUID
------ ----------------
mlx4_0 248a070300ea0390
irdma0 ae1f6bfffeec32f2
irdma1 ae1f6bfffeec32f3
[root@storage03 ~]# ibstat mlx4_0
CA 'mlx4_0'
CA type: MT4099
Number of ports: 1
Firmware version: 2.42.5000
Hardware version: 1
Node GUID: 0x248a070300ea0390
System image GUID: 0x248a070300ea0393
Port 1:
State: Active
Physical state: LinkUp
Rate: 56
Base lid: 4
LMC: 0
SM lid: 1
Capability mask: 0x0259486a
Port GUID: 0x248a070300ea0391
Link layer: InfiniBand
client端开始访问到server端
[root@storage03 ~]# ibping -G 0x0002c90300b382a2 -c 10
Pong from storage02.(none) (Lid 5): time 0.019 ms
Pong from storage02.(none) (Lid 5): time 0.021 ms
Pong from storage02.(none) (Lid 5): time 0.023 ms
Pong from storage02.(none) (Lid 5): time 0.017 ms
Pong from storage02.(none) (Lid 5): time 0.022 ms
Pong from storage02.(none) (Lid 5): time 0.022 ms
Pong from storage02.(none) (Lid 5): time 0.020 ms
Pong from storage02.(none) (Lid 5): time 0.019 ms
Pong from storage02.(none) (Lid 5): time 0.021 ms
Pong from storage02.(none) (Lid 5): time 0.021 ms
--- storage02.(none) (Lid 5) ibping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 10000 ms
rtt min/avg/max = 0.017/0.020/0.023 ms
qperf
qperf跟ibping类似需要，但是qperf在测试时，建议关闭防火墙systemctl stop firewalld
在其中一台机器启用server模式
[root@storage01 ~]# qperf
另一台机器，查看频道适配器的配置
[root@storage02 ~]# qperf -v -i mlx4_0:2 172.16.50.11 conf
conf:
loc_node = storage02
loc_cpu = 20 Cores: Mixed CPUs
loc_os = Linux 4.18.0-477.13.1.el8_8.x86_64
loc_qperf = 0.4.11
rem_node = storage01
rem_cpu = 20 Cores: Mixed CPUs
rem_os = Linux 4.18.0-477.13.1.el8_8.x86_64
rem_qperf = 0.4.11
也可以运行TCP带宽和延迟测试
[root@storage02 ~]# qperf 172.16.50.11 tcp_bw tcp_lat
tcp_bw:
bw = 3.96 GB/sec
tcp_lat:
latency = 11.1 us
也可以测试得到消息大小为1到64K的TCP延迟范围
[root@storage02 ~]# qperf 172.16.50.11 -oo msg_size:1:64K:*2 -vu tcp_lat
tcp_lat:
latency = 10.5 us
msg_size = 1 bytes
tcp_lat:
latency = 11.1 us
msg_size = 2 bytes
tcp_lat:
latency = 11.1 us
msg_size = 4 bytes
tcp_lat:
latency = 10.9 us
msg_size = 8 bytes
tcp_lat:
latency = 10.9 us
msg_size = 16 bytes
tcp_lat:
latency = 11.1 us
msg_size = 32 bytes
tcp_lat:
latency = 10.7 us
msg_size = 64 bytes
tcp_lat:
latency = 11.1 us
msg_size = 128 bytes
tcp_lat:
latency = 11.4 us
msg_size = 256 bytes
tcp_lat:
latency = 11.5 us
msg_size = 512 bytes
tcp_lat:
latency = 12.2 us
msg_size = 1 KiB (1,024)
tcp_lat:
latency = 13 us
msg_size = 2 KiB (2,048)
tcp_lat:
latency = 15.8 us
msg_size = 4 KiB (4,096)
tcp_lat:
latency = 17.2 us
msg_size = 8 KiB (8,192)
tcp_lat:
latency = 22.6 us
msg_size = 16 KiB (16,384)
tcp_lat:
latency = 38.8 us
msg_size = 32 KiB (32,768)
tcp_lat:
latency = 59.3 us
msg_size = 64 KiB (65,536)
perftest
perftest提供了很全面的RDMA测试工具，具体工具可以参考这里：
ib_send_lat latency test with send transactions
ib_send_bw bandwidth test with send transactions
- ib_write_lat latency test with RDMA write transactions
- ib_write_bw bandwidth test with RDMA write transactions
- ib_read_lat latency test with RDMA read transactions
- ib_read_bw bandwidth test with RDMA read transactions
- ib_atomic_lat latency test with atomic transactions
- ib_atomic_bw bandwidth test with atomic transactions
这里我以ib_read_lat作为示例，先在其中一台机器开启ib_read_lat
[root@storage03 ~]# ib_read_lat -d mlx4_0
************************************
* Waiting for client to connect... *
************************************
---------------------------------------------------------------------------------------
RDMA_Read Latency Test
Dual-port : OFF Device : mlx4_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: ON
ibv_wr* API : OFF
Mtu : 2048[B]
Link type : IB
Outstand reads : 16
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0x04 QPN 0x020f PSN 0x17e736 OUT 0x10 RKey 0x8010100 VAddr 0x0055f4c06e7000
remote address: LID 0x01 QPN 0x0215 PSN 0x79fcd6 OUT 0x10 RKey 0x28010100 VAddr 0x0055fa855bd000
---------------------------------------------------------------------------------------
然后另一台作为client去访问测试
[root@storage01 ~]# ib_read_lat -d mlx4_0 172.16.50.13
---------------------------------------------------------------------------------------
RDMA_Read Latency Test
Dual-port : OFF Device : mlx4_0
Number of qps : 1 Transport type : IB
Connection type : RC Using SRQ : OFF
PCIe relax order: ON
ibv_wr* API : OFF
TX depth : 1
Mtu : 2048[B]
Link type : IB
Outstand reads : 16
rdma_cm QPs : OFF
Data ex. method : Ethernet
---------------------------------------------------------------------------------------
local address: LID 0x01 QPN 0x0215 PSN 0x79fcd6 OUT 0x10 RKey 0x28010100 VAddr 0x0055fa855bd000
remote address: LID 0x04 QPN 0x020f PSN 0x17e736 OUT 0x10 RKey 0x8010100 VAddr 0x0055f4c06e7000
---------------------------------------------------------------------------------------
#bytes #iterations t_min[usec] t_max[usec] t_typical[usec] t_avg[usec] t_stdev[usec] 99% percentile[usec] 99.9% percentile[usec]
Conflicting CPU frequency values detected: 2400.000000 != 1873.463000. CPU Frequency is not max.
2 1000 1.76 6.12 1.79 1.80 0.06 1.88 6.12
---------------------------------------------------------------------------------------
