D3c14x fio测试

场景

命令

结果(400G)

单盘

4k随机读

fio -direct=1 -iodepth=256 -rw=randread -ioengine=libaio -bs=4k -numjobs=64 -time_based=1 -runtime=100 -norandommap -group_reporting -filename=/dev/nvme0n1p1 -name=test

read: IOPS=7985, BW=31.2MiB/s (32.7MB/s)(3121MiB/100063msec)

Run status group 0 (all jobs):

READ: bw=31.2MiB/s (32.7MB/s), 31.2MiB/s-31.2MiB/s (32.7MB/s-32.7MB/s), io=3121MiB (3273MB), run=100063-100063msec

Disk stats (read/write):

nvme0n1: ios=798544/972, merge=0/59, ticks=43618256/53026, in_queue=43192150, util=99.94%

双盘

4k随机读

fio -direct=1 -iodepth=256 -rw=randread -ioengine=libaio -bs=4k -numjobs=64 -time_based=1 -runtime=100 -norandommap -group_reporting -filename=/dev/nvme0n1p1:/dev/nvme1n1p1 -name=test

read: IOPS=15.9k, BW=62.2MiB/s (65.3MB/s)(6229MiB/100064msec)

Run status group 0 (all jobs):

READ: bw=62.2MiB/s (65.3MB/s), 62.2MiB/s-62.2MiB/s (65.3MB/s-65.3MB/s), io=6229MiB (6531MB), run=100064-100064msec

Disk stats (read/write):

nvme0n1: ios=796981/12140, merge=0/0, ticks=32597340/98841, in_queue=32387257, util=100.00%

nvme1n1: ios=797621/13668, merge=0/15, ticks=43272768/106252, in_queue=42899111, util=100.00%

4块盘

4k随机读

fio -direct=1 -iodepth=256 -rw=randread -ioengine=libaio -bs=4k -numjobs=64 -time_based=1 -runtime=100 -norandommap -group_reporting -filename=/dev/nvme0n1p1:/dev/nvme1n1p1:/dev/nvme2n1p1:/dev/nvme3n1p1 -name=test

read: IOPS=31.9k, BW=124MiB/s (130MB/s)(12.2GiB/100064msec)

Run status group 0 (all jobs):

READ: bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=12.2GiB (13.1GB), run=100064-100064msec

Disk stats (read/write):

nvme0n1: ios=796846/1626, merge=0/93, ticks=29686605/50296, in_queue=29355877, util=99.83%

nvme1n1: ios=796852/1611, merge=0/89, ticks=23880434/43818, in_queue=23595350, util=99.83%

nvme2n1: ios=796779/1658, merge=0/112, ticks=30865312/71818, in_queue=30385811, util=99.82%

nvme3n1: ios=796680/1703, merge=0/118, ticks=43407622/87969, in_queue=42973916, util=99.82%

单盘

4k随机写

fio -direct=1 -iodepth=256 -rw=randwrite -ioengine=libaio -bs=4k -numjobs=64 -time_based=1 -runtime=100 -norandommap -group_reporting -filename=/dev/nvme0n1p1 -name=test --allow_mounted_write=1

write: IOPS=3985, BW=15.6MiB/s (16.3MB/s)(1559MiB/100126msec)

Run status group 0 (all jobs):

WRITE: bw=15.6MiB/s (16.3MB/s), 15.6MiB/s-15.6MiB/s (16.3MB/s-16.3MB/s), io=1559MiB (1634MB), run=100126-100126msec

Disk stats (read/write):

nvme0n1: ios=126/399857, merge=0/138, ticks=12711/43733879, in_queue=43707645, util=100.00%

双盘

4k随机写

fio -direct=1 -iodepth=256 -rw=randwrite -ioengine=libaio -bs=4k -numjobs=64 -time_based=1 -runtime=100 -norandommap -group_reporting -filename=/dev/nvme0n1p1:/dev/nvme1n1p1 -name=test --allow_mounted_write=1

write: IOPS=7962, BW=31.1MiB/s (32.6MB/s)(3114MiB/100126msec)

Run status group 0 (all jobs):

WRITE: bw=31.1MiB/s (32.6MB/s), 31.1MiB/s-31.1MiB/s (32.6MB/s-32.6MB/s), io=3114MiB (3266MB), run=100126-100126msec

Disk stats (read/write):

nvme0n1: ios=140/409052, merge=0/70, ticks=9591/42625083, in_queue=42424075, util=100.00%

nvme1n1: ios=222/410971, merge=0/62, ticks=23066/42544896, in_queue=42438980, util=100.00%

4块盘

4k随机写

fio -direct=1 -iodepth=256 -rw=randwrite -ioengine=libaio -bs=4k -numjobs=64 -time_based=1 -runtime=100 -norandommap -group_reporting -filename=/dev/nvme0n1p1:/dev/nvme1n1p1:/dev/nvme2n1p1:/dev/nvme3n1p1 -name=test --allow_mounted_write=1

write: IOPS=15.0k, BW=62.4MiB/s (65.4MB/s)(6248MiB/100126msec)

Run status group 0 (all jobs):

WRITE: bw=62.4MiB/s (65.4MB/s), 62.4MiB/s-62.4MiB/s (65.4MB/s-65.4MB/s), io=6248MiB (6552MB), run=100126-100126msec

Disk stats (read/write):

nvme0n1: ios=159/399844, merge=0/40, ticks=15711/42661706, in_queue=42476301, util=99.98%

nvme1n1: ios=605/399838, merge=0/27, ticks=47265/33408132, in_queue=33272908, util=99.98%

nvme2n1: ios=872/399847, merge=0/22, ticks=68440/35584637, in_queue=35479516, util=99.98%

nvme3n1: ios=1481/399777, merge=0/24, ticks=136199/38670596, in_queue=38563640, util=99.97%

单盘

4k随机读写

fio -direct=1 -iodepth=256 -rw=randrw -ioengine=libaio -bs=4k -numjobs=64 -time_based=1 -runtime=100 -norandommap -group_reporting -filename=/dev/nvme0n1p1 -name=test --allow_mounted_write=1

read: IOPS=3549, BW=13.9MiB/s (14.5MB/s)(1388MiB/100072msec)

write: IOPS=3551, BW=13.9MiB/s (14.5MB/s)(1388MiB/100072msec)

Run status group 0 (all jobs):

READ: bw=13.9MiB/s (14.5MB/s), 13.9MiB/s-13.9MiB/s (14.5MB/s-14.5MB/s), io=1388MiB (1455MB), run=100072-100072msec

WRITE: bw=13.9MiB/s (14.5MB/s), 13.9MiB/s-13.9MiB/s (14.5MB/s-14.5MB/s), io=1388MiB (1456MB), run=100072-100072msec

Disk stats (read/write):

nvme0n1: ios=356332/355127, merge=0/7, ticks=21919546/21877968, in_queue=43472351, util=99.99%

双盘

4k随机读写

fio -direct=1 -iodepth=256 -rw=randrw -ioengine=libaio -bs=4k -numjobs=64 -time_based=1 -runtime=100 -norandommap -group_reporting -filename=/dev/nvme0n1p1:/dev/nvme1n1p1 -name=test --allow_mounted_write=1

read: IOPS=7071, BW=27.6MiB/s (28.0MB/s)(2764MiB/100070msec)

write: IOPS=7079, BW=27.7MiB/s (28.0MB/s)(2767MiB/100070msec); 0 zone resets

Run status group 0 (all jobs):

READ: bw=27.6MiB/s (28.0MB/s), 27.6MiB/s-27.6MiB/s (28.0MB/s-28.0MB/s), io=2764MiB (2899MB), run=100070-100070msec

WRITE: bw=27.7MiB/s (28.0MB/s), 27.7MiB/s-27.7MiB/s (28.0MB/s-28.0MB/s), io=2767MiB (2902MB), run=100070-100070msec

Disk stats (read/write):

nvme0n1: ios=354340/367142, merge=0/0, ticks=17247836/17338869, in_queue=34361110, util=100.00%

nvme1n1: ios=353777/367477, merge=0/7, ticks=21092416/21250225, in_queue=41900639, util=100.00%

4块盘

4k随机读写

fio -direct=1 -iodepth=256 -rw=randrw -ioengine=libaio -bs=4k -numjobs=64 -time_based=1 -runtime=100 -norandommap -group_reporting -filename=/dev/nvme0n1p1:/dev/nvme1n1p1:/dev/nvme2n1p1:/dev/nvme3n1p1 -name=test --allow_mounted_write=1

read: IOPS=14.2k, BW=55.5MiB/s (58.1MB/s)(5549MiB/100071msec)

write: IOPS=14.2k, BW=55.5MiB/s (58.1MB/s)(5550MiB/100071msec)

Run status group 0 (all jobs):

READ: bw=55.5MiB/s (58.1MB/s), 55.5MiB/s-55.5MiB/s (58.1MB/s-58.1MB/s), io=5549MiB (5819MB), run=100071-100071msec

WRITE: bw=55.5MiB/s (58.1MB/s), 55.5MiB/s-55.5MiB/s (58.1MB/s-58.1MB/s), io=5550MiB (5819MB), run=100071-100071msec

Disk stats (read/write):

nvme0n1: ios=354468/356030, merge=0/50, ticks=15199542/15291216, in_queue=30075891, util=99.98%

nvme1n1: ios=354632/356052, merge=0/57, ticks=11910074/11978364, in_queue=23455446, util=99.97%

nvme2n1: ios=355489/355220, merge=0/56, ticks=11406275/11445706, in_queue=22522634, util=99.87%

nvme3n1: ios=358151/354728, merge=0/52, ticks=18718678/18601360, in_queue=36936909, util=99.97%

单盘

512k随机读

fio -direct=1 -iodepth=256 -rw=randread -ioengine=libaio -bs=512k -numjobs=64 -time_based=1 -runtime=100 -norandommap -group_reporting -filename=/dev/nvme0n1p1 -name=test

read: IOPS=5313, BW=2657MiB/s (2786MB/s)(260GiB/100155msec)

Run status group 0 (all jobs):

READ: bw=2657MiB/s (2786MB/s), 2657MiB/s-2657MiB/s (2786MB/s-2786MB/s), io=260GiB (279GB), run=100155-100155msec

Disk stats (read/write):

nvme0n1: ios=532255/739, merge=0/81, ticks=42785617/820, in_queue=42521339, util=100.00%

双盘

512k随机读

fio -direct=1 -iodepth=256 -rw=randread -ioengine=libaio -bs=512k -numjobs=64 -time_based=1 -runtime=100 -norandommap -group_reporting -filename=/dev/nvme0n1p1:/dev/nvme1n1p1 -name=test

read: IOPS=10.3k, BW=5157MiB/s (5408MB/s)(504GiB/100159msec)

Run status group 0 (all jobs):

READ: bw=5157MiB/s (5408MB/s), 5157MiB/s-5157MiB/s (5408MB/s-5408MB/s), io=504GiB (542GB), run=100159-100159msec

Disk stats (read/write):

nvme0n1: ios=516191/41696, merge=0/0, ticks=43274212/4528, in_queue=43016549, util=100.00%

nvme1n1: ios=516673/1005774, merge=0/0, ticks=2808221/94176, in_queue=2565298, util=100.00%

4块盘

512k随机读

fio -direct=1 -iodepth=256 -rw=randread -ioengine=libaio -bs=512k -numjobs=64 -time_based=1 -runtime=100 -norandommap -group_reporting -filename=/dev/nvme0n1p1:/dev/nvme1n1p1:/dev/nvme2n1p1:/dev/nvme3n1p1 -name=test

read: IOPS=21.3k, BW=10.4GiB/s (11.2GB/s)(1041GiB/100141msec)

Run status group 0 (all jobs):

READ: bw=10.4GiB/s (11.2GB/s), 10.4GiB/s-10.4GiB/s (11.2GB/s-11.2GB/s), io=1041GiB (1117GB), run=100141-100141msec

Disk stats (read/write):

nvme0n1: ios=533223/175, merge=0/17, ticks=5070514/260, in_queue=4806422, util=99.78%

nvme1n1: ios=533347/161, merge=0/16, ticks=19774460/252, in_queue=19509790, util=99.88%

nvme2n1: ios=533486/0, merge=0/0, ticks=42684631/0, in_queue=42419650, util=99.93%

nvme3n1: ios=533329/44, merge=0/14, ticks=7897370/89, in_queue=7632897, util=99.82%

单盘

512k随机写

fio -direct=1 -iodepth=256 -rw=randwrite -ioengine=libaio -bs=512k -numjobs=64 -time_based=1 -runtime=100 -norandommap -group_reporting -filename=/dev/nvme0n1p1 -name=test --allow_mounted_write=1

write: IOPS=3985, BW=1993MiB/s (2090MB/s)(195GiB/100202msec)

Run status group 0 (all jobs):

WRITE: bw=1993MiB/s (2090MB/s), 1993MiB/s-1993MiB/s (2090MB/s-2090MB/s), io=195GiB (209GB), run=100202-100202msec

Disk stats (read/write):

nvme0n1: ios=334/408982, merge=0/148, ticks=36363/43563865, in_queue=43417266, util=99.94%

双盘

512k随机写

fio -direct=1 -iodepth=256 -rw=randwrite -ioengine=libaio -bs=512k -numjobs=64 -time_based=1 -runtime=100 -norandommap -group_reporting -filename=/dev/nvme0n1p1:/dev/nvme1n1p1 -name=test --allow_mounted_write=1

write: IOPS=7935, BW=3968MiB/s (4161MB/s)(388GiB/100158msec)

Run status group 0 (all jobs):

WRITE: bw=3968MiB/s (4161MB/s), 3968MiB/s-3968MiB/s (4161MB/s-4161MB/s), io=388GiB (417GB), run=100158-100158msec

Disk stats (read/write):

nvme0n1: ios=118/408673, merge=0/139, ticks=11509/42743671, in_queue=42512054, util=99.98%

nvme1n1: ios=216/410054, merge=0/137, ticks=17816/41033808, in_queue=40816428, util=99.96%

4块盘

512k随机写

fio -direct=1 -iodepth=256 -rw=randwrite -ioengine=libaio -bs=512k -numjobs=64 -time_based=1 -runtime=100 -norandommap -group_reporting -filename=/dev/nvme0n1p1:/dev/nvme1n1p1:/dev/nvme2n1p1:/dev/nvme3n1p1 -name=test --allow_mounted_write=1

write: IOPS=15.9k, BW=7940MiB/s (8326MB/s)(777GiB/100182msec)

Run status group 0 (all jobs):

WRITE: bw=7940MiB/s (8326MB/s), 7940MiB/s-7940MiB/s (8326MB/s-8326MB/s), io=777GiB (834GB), run=100182-100182msec

Disk stats (read/write):

nvme0n1: ios=296/413459, merge=0/99, ticks=20817/35004780, in_queue=34824756, util=99.98%

nvme1n1: ios=295/398527, merge=0/95, ticks=2992/9574845, in_queue=9375689, util=99.62%

nvme2n1: ios=4782/398688, merge=0/104, ticks=19110/14337248, in_queue=14156407, util=99.67%

nvme3n1: ios=320/413815, merge=0/241, ticks=34549/43348323, in_queue=43176012, util=99.98%

单盘

512k随机读写

fio -direct=1 -iodepth=256 -rw=randrw -ioengine=libaio -bs=512k -numjobs=64 -time_based=1 -runtime=100 -norandommap -group_reporting -filename=/dev/nvme0n1p1 -name=test --allow_mounted_write=1

read: IOPS=3539, BW=1770MiB/s (1856MB/s)(173GiB/100104msec)

write: IOPS=3540, BW=1770MiB/s (1856MB/s)(173GiB/100104msec)

Run status group 0 (all jobs):

READ: bw=1770MiB/s (1856MB/s), 1770MiB/s-1770MiB/s (1856MB/s-1856MB/s), io=173GiB (186GB), run=100104-100104msec

WRITE: bw=1770MiB/s (1856MB/s), 1770MiB/s-1770MiB/s (1856MB/s-1856MB/s), io=173GiB (186GB), run=100104-100104msec

Disk stats (read/write):

nvme0n1: ios=354198/364125, merge=0/137, ticks=21751311/21094387, in_queue=42494011, util=99.95%

2块盘

512k随机读写

fio -direct=1 -iodepth=256 -rw=randrw -ioengine=libaio -bs=512k -numjobs=64 -time_based=1 -runtime=100 -norandommap -group_reporting -filename=/dev/nvme0n1p1:/dev/nvme1n1p1 -name=test --allow_mounted_write=1

read: IOPS=7069, BW=3535MiB/s (3706MB/s)(346GiB/100130msec)

write: IOPS=7077, BW=3539MiB/s (3710MB/s)(346GiB/100130msec)

Run status group 0 (all jobs):

READ: bw=3535MiB/s (3706MB/s), 3535MiB/s-3535MiB/s (3706MB/s-3706MB/s), io=346GiB (371GB), run=100130-100130msec

WRITE: bw=3539MiB/s (3710MB/s), 3539MiB/s-3539MiB/s (3710MB/s-3710MB/s), io=346GiB (372GB), run=100130-100130msec

Disk stats (read/write):

nvme0n1: ios=354503/366476, merge=0/46, ticks=18166277/17452301, in_queue=35261969, util=99.98%

nvme1n1: ios=354261/364829, merge=0/45, ticks=21571712/20898547, in_queue=42110725, util=99.92%

4块盘

512k随机读写

fio -direct=1 -iodepth=256 -rw=randrw -ioengine=libaio -bs=512k -numjobs=64 -time_based=1 -runtime=100 -norandommap -group_reporting -filename=/dev/nvme0n1p1:/dev/nvme1n1p1:/dev/nvme2n1p1:/dev/nvme3n1p1 -name=test --allow_mounted_write=1

read: IOPS=12.8k, BW=6394MiB/s (6705MB/s)(625GiB/100103msec)

write: IOPS=12.8k, BW=6394MiB/s (6705MB/s)(625GiB/100103msec)

Run status group 0 (all jobs):

READ: bw=6394MiB/s (6705MB/s), 6394MiB/s-6394MiB/s (6705MB/s-6705MB/s), io=625GiB (671GB), run=100103-100103msec

WRITE: bw=6394MiB/s (6705MB/s), 6394MiB/s-6394MiB/s (6705MB/s-6705MB/s), io=625GiB (671GB), run=100103-100103msec

Disk stats (read/write):

nvme0n1: ios=319526/487617, merge=0/62, ticks=9625659/8070640, in_queue=17355706, util=99.98%

nvme1n1: ios=320708/336142, merge=0/217, ticks=16152010/23259754, in_queue=39090397, util=99.97%

nvme2n1: ios=320597/320252, merge=0/59, ticks=6426223/5375672, in_queue=11485608, util=96.65%

nvme3n1: ios=321173/320126, merge=0/51, ticks=3137990/2874921, in_queue=5698905, util=96.35%