公司在测试环境加了一批新的服务器，因为还没有急着上架就简单的使用FIO工具测试了下服务器磁盘在不同RAID级别下的IOPS的值，然后比对一下差距，增加一下经验。
废话就不说了，这里简单描述下服务器的状态：
|
1
2
3
4
5
6
|
品牌：DELL
型号：PowerEdge R720
CPU：2 * Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz
内存：4 * 8G
阵列卡：PERC H310 Mini (Embedded)
硬盘：6 * 300G 15k SAS
|
本次主要使用FIO分别对RAID 0，RAID 5，RAID 10，RAID 50和单盘做了IOPS的测试，下面首先介绍FIO的安装和使用，后面即为实验的输出和总结信息。
FIO工具
一、FIO 简述
FIO是测试IOPS的非常好的工具，用来对硬件进行压力测试和验证，支持13种不同的I/O引擎，包括：sync,mmap, libaio, posixaio, SG v3, splice, null, network, syslet, guasi, solarisaio等。
fio 官网地址：http://freecode.com/projects/fio/
二、FIO 安装
|
1
2
3
4
5
6
7
8
9
|
[root@kvm-phy27-jz ~]
# yum install -y gcc gcc-c++ libaio-devel
[root@kvm-phy27-jz ~]
# cd /usr/local/src
[root@kvm-phy27-jz src]
# wget http://brick.kernel.dk/snaps/fio-2.1.10.tar.gz
[root@kvm-phy27-jz ~]
# tar -zxf fio-2.1.10.tar.gz
[root@kvm-phy27-jz ~]
# cd fio-2.1.10
[root@kvm-phy27-jz fio-2.1.10]
# make
[root@kvm-phy27-jz fio-2.1.10]
# make install
[root@kvm-phy27-jz fio-2.1.10]
# which fio
/usr/local/bin/fio
|
三、FIO参数
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
filename=
/test
.xxx
# 测试文件名称，通常选择需要测试的盘的data目录
direct=1
# 测试过程绕过机器自带的buffer，使测试结果更真实
rw=randwrite
# 测试随机写的I/O
rw=randrw
# 测试随机写和读的I/O
bs=16k
# 单次io的块文件大小为16k
bsrange=512-2048
# 同上，提定数据块的大小范围
size=200g
# 本次的测试文件大小为200g，以每次4k的io进行测试
numjobs=30
# 本次的测试线程为30
runtime=1000
# 测试时间为1000秒，如果不写则一直将5g文件分4k每次写完为止
ioengine=psync
# io引擎使用pync方式
rwmixwrite=30
# 在混合读写的模式下，写占30%
group_reporting
# 关于显示结果的，汇总每个进程的信息
lockmem=1g
# 只使用1g内存进行测试
zero_buffers
# 用0初始化系统buffer
nrfiles=8
# 每个进程生成文件的数量
|
四、有关fio命令输出含义
|
1
2
3
4
5
6
7
8
9
10
11
|
read
# 顺序读
write
# 顺序写
rw,readwrite
# 顺序混合读写
randwrite
# 随机写
randread
# 随机读
randrw
# 随机混合读写
bw
# 带宽，单位KB/s
iops
# 每秒钟的IO数
runt
# 总运行时间
lat (msec)
# 延迟(毫秒)，msec为毫秒，usec为微秒
|
五、常用FIO命令
|
1
2
3
4
5
6
7
8
|
顺序读：
fio -filename=
/test
.xxx -direct=1 -iodepth 1 -thread -rw=
read
-ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=1000 -group_reporting -name=nolinux_test
随机写：
fio -filename=
/test
.xxx -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=1000 -group_reporting -name=nolinux_test
顺序写：
fio -filename=
/test
.xxx -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=1000 -group_reporting -name=nolinux_test
混合随机读写：
fio -filename=
/test
.xxx -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=100 -group_reporting -name=nolinux_test -ioscheduler=noop
|
OK，下面就是不同级别的RAID的IOPS情况：
这一系列的测试都是基于上面的服务器配置，系统使用CentOS6.6，经过简单优化，磁盘的文件系统为EXT4。
一、单盘情况
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
|
[root@kvm-phy27-jz fio-2.1.10]
# fio -filename=/test.xxx -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=100 -group_reporting -name=nolinux_test
nolinux_test: (g=0): rw=randrw, bs=16K-16K
/16K-16K
, ioengine=psync, iodepth=1
...
nolinux_test: (g=0): rw=randrw, bs=16K-16K
/16K-16K
, ioengine=psync, iodepth=1
fio 2.0.7
Starting 30 threads
nolinux_test: Laying out IO
file
(s) (1
file
(s) / 204800MB)
Jobs: 22 (f=22): [mmmmmmmmmmm_mmmm__mm_m_m_EmmmE] [14.4%
done
] [4521K
/2129K
/s
] [276
/130
iops] [eta 09m:59s]
nolinux_test: (groupid=0, jobs=30): err= 0: pid=2413
read
: io=444528KB, bw=4437.8KB
/s
, iops=277 , runt=100170msec
clat (msec): min=3 , max=1070 , avg=101.45, stdev=89.86
lat (msec): min=3 , max=1070 , avg=101.45, stdev=89.86
clat percentiles (msec):
| 1.00th=[ 9], 5.00th=[ 17], 10.00th=[ 23], 20.00th=[ 34],
| 30.00th=[ 45], 40.00th=[ 59], 50.00th=[ 76], 60.00th=[ 96],
| 70.00th=[ 121], 80.00th=[ 153], 90.00th=[ 212], 95.00th=[ 273],
| 99.00th=[ 429], 99.50th=[ 515], 99.90th=[ 709], 99.95th=[ 791],
| 99.99th=[ 922]
bw (KB
/s
) : min= 18, max= 375, per=3.40%, avg=150.67, stdev=51.94
write: io=194288KB, bw=1939.6KB
/s
, iops=121 , runt=100170msec
clat (usec): min=221 , max=246125 , avg=15053.58, stdev=14396.12
lat (usec): min=223 , max=246127 , avg=15056.39, stdev=14396.24
clat percentiles (usec):
| 1.00th=[ 247], 5.00th=[ 1480], 10.00th=[ 2384], 20.00th=[ 5728],
| 30.00th=[ 9664], 40.00th=[11840], 50.00th=[13760], 60.00th=[15680],
| 70.00th=[17792], 80.00th=[20864], 90.00th=[25984], 95.00th=[31104],
| 99.00th=[45824], 99.50th=[95744], 99.90th=[205824], 99.95th=[220160],
| 99.99th=[238592]
bw (KB
/s
) : min= 10, max= 335, per=4.05%, avg=78.50, stdev=51.26
lat (usec) : 250=0.41%, 500=0.43%, 750=0.33%, 1000=0.10%
lat (msec) : 2=0.95%, 4=2.48%, 10=5.93%, 20=18.04%, 50=24.99%
lat (msec) : 100=19.59%, 250=22.31%, 500=4.04%, 750=0.35%, 1000=0.05%
lat (msec) : 2000=0.01%
cpu : usr=0.42%, sys=2.22%, ctx=2520168, majf=5, minf=166643
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=27783
/w
=12143
/d
=0, short=r=0
/w
=0
/d
=0
Run status group 0 (all jobs):
READ: io=444528KB, aggrb=4437KB
/s
, minb=4437KB
/s
, maxb=4437KB
/s
, mint=100170msec, maxt=100170msec
WRITE: io=194288KB, aggrb=1939KB
/s
, minb=1939KB
/s
, maxb=1939KB
/s
, mint=100170msec, maxt=100170msec
Disk stats (
read
/write
):
sda: ios=27782
/12215
, merge=0
/25
, ticks=2591396
/91815
, in_queue=2684776, util=100.00%
|
二、6块盘做RAID 0
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
|
[root@kvm-phy27-jz fio-2.1.10]
# fio -filename=/test.xxx -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=100 -group_reporting -name=nolinux_test
nolinux_test: (g=0): rw=randrw, bs=16K-16K
/16K-16K
, ioengine=psync, iodepth=1
...
nolinux_test: (g=0): rw=randrw, bs=16K-16K
/16K-16K
, ioengine=psync, iodepth=1
fio 2.0.7
Starting 30 threads
nolinux_test: Laying out IO
file
(s) (1
file
(s) / 204800MB)
Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [3.0%
done
] [22685K
/9444K
/s
] [1384
/576
iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [4.0%
done
] [22783K
/10197K
/s
] [1390
/622
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [5.0%
done
] [22489K
/9755K
/s
] [1372
/595
iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [6.0%
done
] [22063K
/9444K
/s
] [1346
/576
iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [7.0%
done
] [22734K
/9771K
/s
] [1387
/596
iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [8.0%
done
] [22047K
/8740K
/s
] [1345
/533
iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [9.0%
done
] [21376K
/9706K
/s
] [1304
/592
iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [10.0%
done
] [23487K
/9951K
/s
] [1433
/607
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [11.0%
done
] [22325K
/8609K
/s
] [1362
/525
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [12.0%
done
] [23078K
/10164K
/s
] [1408
/620
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [13.9%
done
] [23716K
/10229K
/s
] [1447
/624
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [14.9%
done
] [23978K
/9886K
/s
] [1463
/603
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [15.8%
done
] [23471K
/9836K
/s
] [1432
/600
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [16.8%
done
] [23782K
/10638K
/s
] [1451
/649
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [17.8%
done
] [23160K
/10082K
/s
] [1413
/615
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [18.8%
done
] [22243K
/9345K
/s
] [1357
/570
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [19.8%
done
] [21392K
/9133K
/s
] [1305
/557
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [20.8%
done
] [22505K
/9706K
/s
] [1373
/592
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [21.8%
done
] [23045K
/10458K
/s
] [1406
/638
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [22.8%
done
] [22800K
/10000K
/s
] [1391
/610
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [23.8%
done
] [23127K
/10049K
/s
] [1411
/613
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [24.8%
done
] [22767K
/9656K
/s
] [1389
/589
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [25.7%
done
] [23372K
/10295K
/s
] [1426
/628
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [26.7%
done
] [23176K
/9820K
/s
] [1414
/599
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [27.7%
done
] [22636K
/9345K
/s
] [1381
/570
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [28.7%
done
] [22570K
/9886K
/s
] [1377
/603
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [29.7%
done
] [20885K
/9886K
/s
] [1274
/603
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [30.7%
done
] [22276K
/9673K
/s
] [1359
/590
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [31.7%
done
] [21916K
/9411K
/s
] [1337
/574
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [32.7%
done
] [24420K
/10098K
/s
] [1490
/616
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [33.7%
done
] [22865K
/9444K
/s
] [1395
/576
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [34.7%
done
] [21850K
/9869K
/s
] [1333
/602
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [35.6%
done
] [22783K
/9869K
/s
] [1390
/602
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [36.6%
done
] [22030K
/8953K
/s
] [1344
/546
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [37.6%
done
] [21948K
/9460K
/s
] [1339
/577
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [38.6%
done
] [20181K
/8920K
/s
] [1231
/544
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [39.6%
done
] [23471K
/9722K
/s
] [1432
/593
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [40.6%
done
] [23814K
/10458K
/s
] [1453
/638
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [41.6%
done
] [24764K
/10704K
/s
] [1511
/653
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [42.6%
done
] [22963K
/10016K
/s
] [1401
/611
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [43.6%
done
] [21376K
/10360K
/s
] [1304
/632
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [44.6%
done
] [21163K
/8854K
/s
] [1291
/540
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [45.5%
done
] [21703K
/9460K
/s
] [1324
/577
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [46.5%
done
] [23602K
/9525K
/s
] [1440
/581
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [47.5%
done
] [22129K
/10688K
/s
] [1350
/652
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [48.5%
done
] [23602K
/10327K
/s
] [1440
/630
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [49.5%
done
] [22603K
/10229K
/s
] [1379
/624
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [50.5%
done
] [23045K
/10229K
/s
] [1406
/624
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [51.5%
done
] [21327K
/9706K
/s
] [1301
/592
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [52.5%
done
] [22560K
/10158K
/s
] [1377
/620
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [53.5%
done
] [24977K
/10393K
/s
] [1524
/634
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [54.5%
done
] [21998K
/9116K
/s
] [1342
/556
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [55.4%
done
] [22701K
/9951K
/s
] [1385
/607
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [56.4%
done
] [22849K
/10377K
/s
] [1394
/633
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [57.4%
done
] [21916K
/9509K
/s
] [1337
/580
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [58.4%
done
] [22063K
/9869K
/s
] [1346
/602
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [59.4%
done
] [20950K
/9525K
/s
] [1278
/581
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [60.4%
done
] [23945K
/10409K
/s
] [1461
/635
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [61.4%
done
] [24166K
/10321K
/s
] [1475
/630
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [62.4%
done
] [22446K
/9584K
/s
] [1370
/585
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [63.4%
done
] [25247K
/10190K
/s
] [1541
/622
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [64.4%
done
] [22921K
/9469K
/s
] [1399
/578
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [65.3%
done
] [23494K
/9863K
/s
] [1434
/602
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [66.3%
done
] [21381K
/9551K
/s
] [1305
/583
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [67.3%
done
] [21086K
/8224K
/s
] [1287
/502
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [68.3%
done
] [23658K
/9912K
/s
] [1444
/605
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [69.3%
done
] [21823K
/9011K
/s
] [1332
/550
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [70.3%
done
] [23183K
/9240K
/s
] [1415
/564
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [71.3%
done
] [21266K
/8454K
/s
] [1298
/516
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [72.3%
done
] [22577K
/10027K
/s
] [1378
/612
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [73.3%
done
] [22374K
/9869K
/s
] [1365
/602
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [74.3%
done
] [21588K
/8642K
/s
] [1317
/527
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [75.2%
done
] [22200K
/9945K
/s
] [1355
/607
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [76.2%
done
] [21954K
/10256K
/s
] [1340
/626
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [77.2%
done
] [21676K
/9469K
/s
] [1323
/578
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [78.2%
done
] [21823K
/10190K
/s
] [1332
/622
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [79.2%
done
] [21381K
/9748K
/s
] [1305
/595
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [80.2%
done
] [20873K
/9240K
/s
] [1274
/564
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [81.2%
done
] [23576K
/10698K
/s
] [1439
/653
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [82.2%
done
] [23003K
/9945K
/s
] [1404
/607
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [83.2%
done
] [21397K
/10403K
/s
] [1306
/635
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [84.2%
done
] [23396K
/9469K
/s
] [1428
/578
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [85.1%
done
] [21801K
/9280K
/s
] [1330
/566
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [86.1%
done
] [20737K
/9280K
/s
] [1265
/566
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [87.1%
done
] [22554K
/9444K
/s
] [1376
/576
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [88.1%
done
] [24158K
/10311K
/s
] [1474
/629
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [89.1%
done
] [21736K
/9345K
/s
] [1326
/570
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [90.1%
done
] [22669K
/9362K
/s
] [1383
/571
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [91.1%
done
] [20541K
/9296K
/s
] [1253
/567
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [92.1%
done
] [23838K
/10240K
/s
] [1455
/625
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [93.1%
done
] [22085K
/9388K
/s
] [1348
/573
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [94.1%
done
] [22773K
/9109K
/s
] [1390
/556
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [95.0%
done
] [21561K
/9306K
/s
] [1316
/568
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [96.0%
done
] [20840K
/8519K
/s
] [1272
/520
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [97.0%
done
] [22822K
/9797K
/s
] [1393
/598
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [98.0%
done
] [22069K
/9797K
/s
] [1347
/598
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [99.0%
done
] [22167K
/10534K
/s
] [1353
/643
iops] [eta 0Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [100.0%
done
] [23248K
/10633K
/s
] [1419
/649
iops] [eta 00m:00s]
nolinux_test: (groupid=0, jobs=30): err= 0: pid=2480
read
: io=2146.2MB, bw=21948KB
/s
, iops=1371 , runt=100132msec
clat (usec): min=296 , max=588194 , avg=20452.68, stdev=30073.79
lat (usec): min=296 , max=588195 , avg=20453.01, stdev=30073.79
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 8],
| 30.00th=[ 9], 40.00th=[ 10], 50.00th=[ 12], 60.00th=[ 15],
| 70.00th=[ 18], 80.00th=[ 24], 90.00th=[ 40], 95.00th=[ 63],
| 99.00th=[ 165], 99.50th=[ 212], 99.90th=[ 326], 99.95th=[ 371],
| 99.99th=[ 453]
bw (KB
/s
) : min= 72, max= 1346, per=3.37%, avg=738.97, stdev=201.50
write: io=950496KB, bw=9492.5KB
/s
, iops=593 , runt=100132msec
clat (usec): min=176 , max=131922 , avg=3208.13, stdev=3135.80
lat (usec): min=177 , max=131927 , avg=3210.90, stdev=3135.89
clat percentiles (usec):
| 1.00th=[ 223], 5.00th=[ 243], 10.00th=[ 338], 20.00th=[ 1336],
| 30.00th=[ 2352], 40.00th=[ 2864], 50.00th=[ 3248], 60.00th=[ 3632],
| 70.00th=[ 4048], 80.00th=[ 4512], 90.00th=[ 5280], 95.00th=[ 5984],
| 99.00th=[ 7776], 99.50th=[ 8768], 99.90th=[26496], 99.95th=[86528],
| 99.99th=[129536]
bw (KB
/s
) : min= 21, max= 960, per=3.37%, avg=320.15, stdev=144.88
lat (usec) : 250=1.88%, 500=1.96%, 750=0.78%, 1000=0.58%
lat (msec) : 2=2.50%, 4=15.39%, 10=35.36%, 20=23.54%, 50=13.16%
lat (msec) : 100=3.06%, 250=1.58%, 500=0.20%, 750=0.01%
cpu : usr=1.43%, sys=10.07%, ctx=11331053, majf=0, minf=166561
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=137355
/w
=59406
/d
=0, short=r=0
/w
=0
/d
=0
Run status group 0 (all jobs):
READ: io=2146.2MB, aggrb=21947KB
/s
, minb=21947KB
/s
, maxb=21947KB
/s
, mint=100132msec, maxt=100132msec
WRITE: io=950496KB, aggrb=9492KB
/s
, minb=9492KB
/s
, maxb=9492KB
/s
, mint=100132msec, maxt=100132msec
Disk stats (
read
/write
):
sda: ios=137228
/59403
, merge=0
/19
, ticks=2562622
/89096
, in_queue=2652072, util=99.99%
|
三、6块盘做RAID 5
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
|
[root@kvm-phy27-jz fio-2.1.10]
# fio -filename=/test.xxx -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=100 -group_reporting -name=nolinux_test
nolinux_test: (g=0): rw=randrw, bs=16K-16K
/16K-16K
, ioengine=psync, iodepth=1
...
nolinux_test: (g=0): rw=randrw, bs=16K-16K
/16K-16K
, ioengine=psync, iodepth=1
fio 2.0.7
Starting 30 threads
nolinux_test: Laying out IO
file
(s) (1
file
(s) / 204800MB)
Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [100.0%
done
] [3077K
/1456K
/s
] [187
/88
iops] [eta 00m:00s]
nolinux_test: (groupid=0, jobs=30): err= 0: pid=3013
read
: io=324576KB, bw=3242.7KB
/s
, iops=202 , runt=100097msec
clat (msec): min=1 , max=345 , avg=101.77, stdev=64.98
lat (msec): min=1 , max=345 , avg=101.77, stdev=64.98
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 10],
| 30.00th=[ 79], 40.00th=[ 100], 50.00th=[ 115], 60.00th=[ 128],
| 70.00th=[ 141], 80.00th=[ 157], 90.00th=[ 178], 95.00th=[ 196],
| 99.00th=[ 235], 99.50th=[ 249], 99.90th=[ 277], 99.95th=[ 285],
| 99.99th=[ 314]
bw (KB
/s
) : min= 24, max= 212, per=3.34%, avg=108.17, stdev=28.20
write: io=140704KB, bw=1405.7KB
/s
, iops=87 , runt=100097msec
clat (msec): min=3 , max=307 , avg=106.48, stdev=65.12
lat (msec): min=3 , max=307 , avg=106.48, stdev=65.13
clat percentiles (msec):
| 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 12],
| 30.00th=[ 84], 40.00th=[ 105], 50.00th=[ 120], 60.00th=[ 133],
| 70.00th=[ 147], 80.00th=[ 161], 90.00th=[ 182], 95.00th=[ 202],
| 99.00th=[ 239], 99.50th=[ 251], 99.90th=[ 289], 99.95th=[ 289],
| 99.99th=[ 310]
bw (KB
/s
) : min= 21, max= 278, per=4.17%, avg=58.56, stdev=35.44
lat (msec) : 2=0.30%, 4=4.24%, 10=13.41%, 20=6.98%, 50=0.69%
lat (msec) : 100=13.61%, 250=60.28%, 500=0.48%
cpu : usr=0.45%, sys=2.42%, ctx=1953443, majf=0, minf=166561
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=20286
/w
=8794
/d
=0, short=r=0
/w
=0
/d
=0
Run status group 0 (all jobs):
READ: io=324576KB, aggrb=3242KB
/s
, minb=3242KB
/s
, maxb=3242KB
/s
, mint=100097msec, maxt=100097msec
WRITE: io=140704KB, aggrb=1405KB
/s
, minb=1405KB
/s
, maxb=1405KB
/s
, mint=100097msec, maxt=100097msec
Disk stats (
read
/write
):
sda: ios=20258
/8866
, merge=0
/32
, ticks=153891
/101181
, in_queue=255029, util=99.54%
|
四、6块盘做RAID 10
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
|
[root@kvm-phy27-jz fio-2.1.10]
# fio -filename=/test.xxx -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=200G -numjobs=30 -
runtime=100 -group_reporting -name=nolinux_test
nolinux_test: (g=0): rw=randrw, bs=16K-16K
/16K-16K
, ioengine=psync, iodepth=1
...
nolinux_test: (g=0): rw=randrw, bs=16K-16K
/16K-16K
, ioengine=psync, iodepth=1
fio 2.0.7
Starting 30 threads
nolinux_test: Laying out IO
file
(s) (1
file
(s) / 204800MB)
Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [3.0%
done
] [15613K
/7438K
/s
] [953
/454
iops] [eta 01m:Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [4.0%
done
] [16596K
/7323K
/s
] [1013
/447
iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [5.0%
done
] [16482K
/6897K
/s
] [1006
/421
iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [5.9%
done
] [17235K
/7389K
/s
] [1052
/451
iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [6.9%
done
] [16646K
/7208K
/s
] [1016
/440
iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [7.9%
done
] [17547K
/6799K
/s
] [1071
/415
iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [8.9%
done
] [17711K
/7995K
/s
] [1081
/488
iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [9.9%
done
] [17448K
/6684K
/s
] [1065
/408
iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [10.9%
done
] [17252K
/6799K
/s
] [1053
/415
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [11.9%
done
] [17104K
/7323K
/s
] [1044
/447
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [12.9%
done
] [17039K
/7061K
/s
] [1040
/431
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [13.9%
done
] [17219K
/7127K
/s
] [1051
/435
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [14.9%
done
] [17874K
/6930K
/s
] [1091
/423
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [15.8%
done
] [17170K
/6848K
/s
] [1048
/418
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [16.8%
done
] [14614K
/5750K
/s
] [892
/351
iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [17.8%
done
] [16629K
/7536K
/s
] [1015
/460
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [18.8%
done
] [17776K
/7634K
/s
] [1085
/466
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [19.8%
done
] [16596K
/7176K
/s
] [1013
/438
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [20.8%
done
] [16252K
/7389K
/s
] [992
/451
iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [21.8%
done
] [14696K
/6619K
/s
] [897
/404
iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [22.8%
done
] [15745K
/6619K
/s
] [961
/404
iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [23.8%
done
] [13008K
/5554K
/s
] [794
/339
iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [24.8%
done
] [17809K
/6864K
/s
] [1087
/419
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [25.7%
done
] [17252K
/7634K
/s
] [1053
/466
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [26.7%
done
] [17006K
/6799K
/s
] [1038
/415
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [27.7%
done
] [18415K
/7340K
/s
] [1124
/448
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [28.7%
done
] [17022K
/7553K
/s
] [1039
/461
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [29.7%
done
] [16596K
/7045K
/s
] [1013
/430
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [30.7%
done
] [15171K
/7192K
/s
] [926
/439
iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [31.7%
done
] [15777K
/7061K
/s
] [963
/431
iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [32.7%
done
] [16564K
/7749K
/s
] [1011
/473
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [33.7%
done
] [15843K
/7454K
/s
] [967
/455
iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [34.7%
done
] [16613K
/7798K
/s
] [1014
/476
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [35.6%
done
] [17563K
/7880K
/s
] [1072
/481
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [36.6%
done
] [15974K
/6946K
/s
] [975
/424
iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [37.6%
done
] [16269K
/7454K
/s
] [993
/455
iops] [eta 01mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [38.6%
done
] [17088K
/7176K
/s
] [1043
/438
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [39.6%
done
] [16859K
/7471K
/s
] [1029
/456
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [40.6%
done
] [16662K
/6733K
/s
] [1017
/411
iops] [eta 01Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [41.6%
done
] [17039K
/7487K
/s
] [1040
/457
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [42.6%
done
] [15810K
/7716K
/s
] [965
/471
iops] [eta 00mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [43.6%
done
] [17858K
/7241K
/s
] [1090
/442
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [44.6%
done
] [16252K
/7225K
/s
] [992
/441
iops] [eta 00mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [45.5%
done
] [17416K
/7569K
/s
] [1063
/462
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [46.5%
done
] [16482K
/7127K
/s
] [1006
/435
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [47.5%
done
] [17301K
/7569K
/s
] [1056
/462
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [48.5%
done
] [16941K
/7438K
/s
] [1034
/454
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [49.5%
done
] [16039K
/7536K
/s
] [979
/460
iops] [eta 00mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [50.5%
done
] [17399K
/7290K
/s
] [1062
/445
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [51.5%
done
] [17383K
/7487K
/s
] [1061
/457
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [52.5%
done
] [17039K
/7127K
/s
] [1040
/435
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [53.5%
done
] [16662K
/7602K
/s
] [1017
/464
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [54.5%
done
] [17104K
/7847K
/s
] [1044
/479
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [55.4%
done
] [16531K
/7127K
/s
] [1009
/435
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [56.4%
done
] [16072K
/7012K
/s
] [981
/428
iops] [eta 00mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [57.4%
done
] [15925K
/7454K
/s
] [972
/455
iops] [eta 00mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [58.4%
done
] [16711K
/7667K
/s
] [1020
/468
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [59.4%
done
] [17498K
/7602K
/s
] [1068
/464
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [60.4%
done
] [16859K
/7143K
/s
] [1029
/436
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [61.4%
done
] [17088K
/6602K
/s
] [1043
/403
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [62.4%
done
] [17104K
/6799K
/s
] [1044
/415
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [63.4%
done
] [17301K
/7208K
/s
] [1056
/440
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [64.4%
done
] [17596K
/7061K
/s
] [1074
/431
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [65.3%
done
] [17661K
/7127K
/s
] [1078
/435
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [66.3%
done
] [16924K
/6930K
/s
] [1033
/423
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [67.3%
done
] [17629K
/6815K
/s
] [1076
/416
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [68.3%
done
] [17514K
/7258K
/s
] [1069
/443
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [69.3%
done
] [17432K
/7585K
/s
] [1064
/463
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [70.3%
done
] [16662K
/7585K
/s
] [1017
/463
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [71.3%
done
] [16596K
/7028K
/s
] [1013
/429
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [72.3%
done
] [17334K
/7585K
/s
] [1058
/463
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [73.3%
done
] [17825K
/7258K
/s
] [1088
/443
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [74.3%
done
] [17891K
/7536K
/s
] [1092
/460
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [75.2%
done
] [15925K
/7405K
/s
] [972
/452
iops] [eta 00mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [76.2%
done
] [17186K
/7340K
/s
] [1049
/448
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [77.2%
done
] [16269K
/7520K
/s
] [993
/459
iops] [eta 00mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [78.2%
done
] [16318K
/7389K
/s
] [996
/451
iops] [eta 00mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [79.2%
done
] [17154K
/7110K
/s
] [1047
/434
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [80.2%
done
] [16351K
/7667K
/s
] [998
/468
iops] [eta 00mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [81.2%
done
] [17006K
/7897K
/s
] [1038
/482
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [82.2%
done
] [18022K
/7258K
/s
] [1100
/443
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [83.2%
done
] [16236K
/7323K
/s
] [991
/447
iops] [eta 00mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [84.2%
done
] [16465K
/7602K
/s
] [1005
/464
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [85.1%
done
] [17039K
/7159K
/s
] [1040
/437
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [86.1%
done
] [17350K
/7258K
/s
] [1059
/443
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [87.1%
done
] [17432K
/7454K
/s
] [1064
/455
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [88.1%
done
] [16777K
/7421K
/s
] [1024
/453
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [89.1%
done
] [17121K
/7323K
/s
] [1045
/447
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [90.1%
done
] [16531K
/7356K
/s
] [1009
/449
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [91.1%
done
] [17137K
/7585K
/s
] [1046
/463
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [92.1%
done
] [17612K
/7143K
/s
] [1075
/436
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [93.1%
done
] [16596K
/7274K
/s
] [1013
/444
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [94.1%
done
] [17694K
/7454K
/s
] [1080
/455
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [95.0%
done
] [16793K
/6979K
/s
] [1025
/426
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [96.0%
done
] [18169K
/7634K
/s
] [1109
/466
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [97.0%
done
] [17399K
/6619K
/s
] [1062
/404
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [98.0%
done
] [17842K
/6897K
/s
] [1089
/421
iops] [eta 00Jobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [99.0%
done
] [16334K
/6897K
/s
] [997
/421
iops] [eta 00mJobs: 30 (f=30): [mmmmmmmmmmmmmmmmmmmmmmmmmmmmmm] [100.0%
done
] [16875K
/7356K
/s
] [1030
/449
iops] [eta 00m:00s]
nolinux_test: (groupid=0, jobs=30): err= 0: pid=28011
read
: io=1606.9MB, bw=16445KB
/s
, iops=1027 , runt=100055msec
clat (usec): min=207 , max=623956 , avg=27266.40, stdev=26421.57
lat (usec): min=207 , max=623957 , avg=27266.74, stdev=26421.57
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 6], 10.00th=[ 8], 20.00th=[ 10],
| 30.00th=[ 12], 40.00th=[ 15], 50.00th=[ 19], 60.00th=[ 24],
| 70.00th=[ 31], 80.00th=[ 42], 90.00th=[ 60], 95.00th=[ 77],
| 99.00th=[ 124], 99.50th=[ 153], 99.90th=[ 229], 99.95th=[ 285],
| 99.99th=[ 392]
bw (KB
/s
) : min= 30, max= 1021, per=3.35%, avg=550.46, stdev=118.04
write: io=706368KB, bw=7059.9KB
/s
, iops=441 , runt=100055msec
clat (usec): min=206 , max=380138 , avg=4432.19, stdev=6592.84
lat (usec): min=208 , max=380139 , avg=4434.78, stdev=6592.91
clat percentiles (usec):
| 1.00th=[ 235], 5.00th=[ 294], 10.00th=[ 502], 20.00th=[ 1816],
| 30.00th=[ 3024], 40.00th=[ 3632], 50.00th=[ 4192], 60.00th=[ 4768],
| 70.00th=[ 5344], 80.00th=[ 6112], 90.00th=[ 7392], 95.00th=[ 8768],
| 99.00th=[12224], 99.50th=[14400], 99.90th=[69120], 99.95th=[146432],
| 99.99th=[366592]
bw (KB
/s
) : min= 22, max= 731, per=3.35%, avg=236.33, stdev=107.84
lat (usec) : 250=0.87%, 500=2.13%, 750=0.87%, 1000=0.52%
lat (msec) : 2=1.95%, 4=8.84%, 10=29.78%, 20=22.78%, 50=22.25%
lat (msec) : 100=8.56%, 250=1.37%, 500=0.06%, 750=0.01%
cpu : usr=1.15%, sys=7.45%, ctx=8559338, majf=0, minf=166568
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=102836
/w
=44148
/d
=0, short=r=0
/w
=0
/d
=0
Run status group 0 (all jobs):
READ: io=1606.9MB, aggrb=16444KB
/s
, minb=16444KB
/s
, maxb=16444KB
/s
, mint=100055msec, maxt=100055msec
WRITE: io=706368KB, aggrb=7059KB
/s
, minb=7059KB
/s
, maxb=7059KB
/s
, mint=100055msec, maxt=100055msec
Disk stats (
read
/write
):
sda: ios=102742
/44197
, merge=0
/22
, ticks=2546701
/103319
, in_queue=2650955, util=99.98%
|
五、6块盘做RAID 50
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
|
[root@kvm-phy27-jz fio-2.1.10]
# fio -filename=/test.xxx -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=100 -group_reporting -name=nolinux_test
nolinux_test: (g=0): rw=randrw, bs=16K-16K
/16K-16K
, ioengine=psync, iodepth=1
...
nolinux_test: (g=0): rw=randrw, bs=16K-16K
/16K-16K
, ioengine=psync, iodepth=1
fio 2.0.7
Starting 30 threads
nolinux_test: Laying out IO
file
(s) (1
file
(s) / 204800MB)
Jobs: 8 (f=8): [_m___mm__mmm____m__m__________] [4.4%
done
] [4435K
/2127K
/s
] [270
/129
iops] [eta 36m:39s] s]
nolinux_test: (groupid=0, jobs=30): err= 0: pid=2660
read
: io=468304KB, bw=4678.9KB
/s
, iops=292 , runt=100091msec
clat (usec): min=521 , max=212024 , avg=71598.16, stdev=42935.63
lat (usec): min=521 , max=212025 , avg=71598.60, stdev=42935.59
clat percentiles (msec):
| 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 6], 20.00th=[ 10],
| 30.00th=[ 57], 40.00th=[ 71], 50.00th=[ 81], 60.00th=[ 89],
| 70.00th=[ 98], 80.00th=[ 109], 90.00th=[ 122], 95.00th=[ 135],
| 99.00th=[ 157], 99.50th=[ 165], 99.90th=[ 180], 99.95th=[ 190],
| 99.99th=[ 206]
bw (KB
/s
) : min= 26, max= 283, per=3.33%, avg=155.83, stdev=32.69
write: io=200320KB, bw=2001.4KB
/s
, iops=125 , runt=100091msec
clat (msec): min=2 , max=212 , avg=72.30, stdev=43.07
lat (msec): min=2 , max=212 , avg=72.31, stdev=43.07
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 11],
| 30.00th=[ 58], 40.00th=[ 71], 50.00th=[ 81], 60.00th=[ 90],
| 70.00th=[ 99], 80.00th=[ 109], 90.00th=[ 123], 95.00th=[ 135],
| 99.00th=[ 157], 99.50th=[ 167], 99.90th=[ 184], 99.95th=[ 192],
| 99.99th=[ 208]
bw (KB
/s
) : min= 24, max= 351, per=3.75%, avg=75.13, stdev=43.18
lat (usec) : 750=0.01%
lat (msec) : 2=0.23%, 4=4.20%, 10=15.54%, 20=3.47%, 50=3.88%
lat (msec) : 100=44.91%, 250=27.75%
cpu : usr=0.53%, sys=2.99%, ctx=2748999, majf=5, minf=166631
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=29269
/w
=12520
/d
=0, short=r=0
/w
=0
/d
=0
Run status group 0 (all jobs):
READ: io=468304KB, aggrb=4678KB
/s
, minb=4678KB
/s
, maxb=4678KB
/s
, mint=100091msec, maxt=100091msec
WRITE: io=200320KB, aggrb=2001KB
/s
, minb=2001KB
/s
, maxb=2001KB
/s
, mint=100091msec, maxt=100091msec
Disk stats (
read
/write
):
sda: ios=29273
/12617
, merge=0
/23
, ticks=212137
/99979
, in_queue=312063, util=99.54%
|
各种级别RAID的总结如下表：
| RAID级别 | read(iops) | write(iops) |
| 单盘，无RAID | 277 | 121 |
| 0 | 1371 | 593 |
| 5 | 202 | 87 |
| 10 |
1027 | 441 |
| 50 | 292 | 125 |
OK，以上就是本次针对相同的服务器进行不同磁盘阵列下的IOPS测试情况。
虽然实验之前没有对系统做过多的深度优化，但是由于本次的所有测试都是在相同环境下进行，因此干扰也不会很大。由于RAID 5的原理，测试出来的效果和磁盘阵列卡的缓存关系更密切一点，不过暂时只有这种阵列卡可以测试了。从不同RAID级别的IOPS中可以看出，如果对空间需求不是很高的情况下，使用RAID 10还是不错的选择。虽然RAID 10更倾向于数据库服务器去使用，不过做虚拟化的母机使用效果还会很不错的。
最后，相对而言本次的测试还是有一定的对比效果的，希望能对51博友有所帮助。
本文转自 aaao 51CTO博客，原文链接：http://blog.51cto.com/nolinux/1662942，如需转载请自行联系原作者
