如何对Exadata的I/O进行性能校准测试

前段时间看到MACLEAN LIU分享了一篇不错的关于Exadata I/O性能校准的文章：Calibrate测试Exadata IO，这篇文章基本涵盖到Exadata I/O性能校准的方法。刚好最近又有人问我这方面的问题，所以在这里再班门弄斧做一些补充，存在重复的地方请无视之。

在Exadata上如果需要对I/O进行性能校准或者测试，通常有如下几种方法：

1. Oracle的I/O性能校准工具orion；

2. Oracle Database 11g引入的的DBMS_RESOURCE_MANAGER.CALIBRATE_IO包；

3. EXADATA Cell端cellsrv中的calibrate命令；

当然有人说通过dd进行测试也可以得到磁盘的IOPS和MBPS。 在没有任何工具的前提下，当然dd也可以用来进行大致的估算，但是其结果通常是不可靠的，所以这里就不把dd例如这个范畴。以下分别对这几种方式做一些简单的介绍和对比。

• orion

./orion_x86_64�0�2�0�2–run <workload_mode>

simple - tests random 8K small IOs at various loads, then random 1M large IOs at various loads.
normal - tests combinations of random 8K small IOs and random 1M large IOs
advanced - run the workload specified by the user using optional parameters
dss - run with random 1M large IOs at increasing loads to determine the maximum throughput
oltp - run with random 8K small IOs at increasing loads to determine the maximum IOPS

• calibrate_io

• timed_statistics�0�2必须设置为TRUE；
• 异步I/O已经启用；如果使用文件系统，则FILESYSTEMIO_OPTIONS�0�2需要设置为SETALL。可以通过以下命令来查询数据文件的异步I/O是否已经启用：

• COL NAME FORMAT A50
  SELECT NAME,ASYNCH_IO FROM V$DATAFILE F,V$IOSTAT_FILE I
  WHERE  F.FILE#=I.FILE_NO
  AND    FILETYPE_NAME='DATA FILE';

SET SERVEROUTPUT ON
DECLARE
lat INTEGER;
iops INTEGER;
mbps INTEGER;
BEGIN
DBMS_RESOURCE_MANAGER.CALIBRATE_IO (1, 10,
iops, mbps, lat);
DBMS_OUTPUT.PUT_LINE ('max_iops = ' || iops); DBMS_OUTPUT.PUT_LINE ('latency = ' || lat); DBMS_OUTPUT.PUT_LINE ('max_mbps = ' || mbps);
END;
/

测试完成以后，会在底下显示测试的结果。当然Oracle也提供了相应的动态性能视图供查询：

select * from gv$io_calibration_status;
select * from DBA_RSRC_IO_CALIBRATE;

• cellcli calibrate

在停止cellsrv的服务的情况下，使用root用户在cellcli中运行calibrate。如果需要使用在线模式，则必须在后面加上force参数。以下引用的数据来在自MACLEAN LIU的blog：

CellCLI> calibrate force;

Calibration will take a few minutes…
Aggregate random read throughput across all hard disk LUNs: 1921 MBPS
Aggregate random read throughput across all flash disk LUNs: 4164.33 MBPS
Aggregate random read IOs per second (IOPS) across all hard disk LUNs: 4971
Aggregate random read IOs per second (IOPS) across all flash disk LUNs: 145195
Controller read throughput: 1919.64 MBPS
Calibrating hard disks (read only) …
LUN 0_0 on drive [28:0 ] random read throughput: 168.12 MBPS, and 430 IOPS
LUN 0_1 on drive [28:1 ] random read throughput: 164.23 MBPS, and 423 IOPS
LUN 0_10 on drive [28:10 ] random read throughput: 170.80 MBPS, and 433 IOPS
LUN 0_11 on drive [28:11 ] random read throughput: 168.32 MBPS, and 421 IOPS
LUN 0_2 on drive [28:2 ] random read throughput: 170.07 MBPS, and 431 IOPS
LUN 0_3 on drive [28:3 ] random read throughput: 169.82 MBPS, and 421 IOPS
LUN 0_4 on drive [28:4 ] random read throughput: 165.17 MBPS, and 417 IOPS
LUN 0_5 on drive [28:5 ] random read throughput: 166.82 MBPS, and 429 IOPS
LUN 0_6 on drive [28:6 ] random read throughput: 170.85 MBPS, and 430 IOPS
LUN 0_7 on drive [28:7 ] random read throughput: 168.42 MBPS, and 429 IOPS
LUN 0_8 on drive [28:8 ] random read throughput: 169.78 MBPS, and 428 IOPS
LUN 0_9 on drive [28:9 ] random read throughput: 168.77 MBPS, and 430 IOPS
Calibrating flash disks (read only, note that writes will be significantly slower) …
LUN 1_0 on drive [FLASH_1_0] random read throughput: 271.01 MBPS, and 19808 IOPS
LUN 1_1 on drive [FLASH_1_1] random read throughput: 270.24 MBPS, and 19821 IOPS
LUN 1_2 on drive [FLASH_1_2] random read throughput: 270.41 MBPS, and 19844 IOPS
LUN 1_3 on drive [FLASH_1_3] random read throughput: 270.37 MBPS, and 19812 IOPS
LUN 2_0 on drive [FLASH_2_0] random read throughput: 272.32 MBPS, and 20634 IOPS
LUN 2_1 on drive [FLASH_2_1] random read throughput: 272.12 MBPS, and 20635 IOPS
LUN 2_2 on drive [FLASH_2_2] random read throughput: 272.28 MBPS, and 20676 IOPS
LUN 2_3 on drive [FLASH_2_3] random read throughput: 272.43 MBPS, and 20669 IOPS
LUN 4_0 on drive [FLASH_4_0] random read throughput: 271.13 MBPS, and 19802 IOPS
LUN 4_1 on drive [FLASH_4_1] random read throughput: 271.90 MBPS, and 19799 IOPS
LUN 4_2 on drive [FLASH_4_2] random read throughput: 271.42 MBPS, and 19798 IOPS
LUN 4_3 on drive [FLASH_4_3] random read throughput: 272.25 MBPS, and 19808 IOPS
LUN 5_0 on drive [FLASH_5_0] random read throughput: 272.22 MBPS, and 19824 IOPS
LUN 5_1 on drive [FLASH_5_1] random read throughput: 272.44 MBPS, and 19823 IOPS
LUN 5_2 on drive [FLASH_5_2] random read throughput: 271.83 MBPS, and 19808 IOPS
LUN 5_3 on drive [FLASH_5_3] random read throughput: 271.73 MBPS, and 19837 IOPS
CALIBRATE results are within an acceptable range.
Calibration has finished.

这里的calibrate是针对单个cell节点进行的测试，而Exadata datasheet上给出的IOPS和MBPS的数据都是来自此命令的测试结果。（先测出单个Cell节点的各项指标，然后乘以cell节点数）同时以上在进行测试IOPS的时候，使用的I/O请求是8K大小的随机读；在测试MBPS的时候，则使用的是1M大小的随机读。同时上述测试既给出整个Cell节点总体的IOPS和MBPS，又给出了Cell节点中单个LUN的IOPS和MBPS。那Exadata为什么不对所有的Cell节点一起测试，然后得到一个总的测试结果呢？因为不同的Cell节点之间互相访问需要通过infiniband网络，如果测试同时进行， 则瓶颈可能出在infiniband上，导致测试的结果不准确。

calibrate专为Exadata设计，所以只能在Exadata的Cell端运行。使用它有如下好处：

1. 既能得到一个Cell节点总体的性能数据，又可以得到单个磁盘或者闪盘的性能数据；
2. 安装Exadata时候可�0�2以用来检测磁盘/闪盘的I/O性能，排除掉可能存在缺陷的磁盘/闪盘；
3. 在出现I/O性能问题的时候，可以用它来对Cell的I/O进行校准，判断是否属于硬件级别的问题。

以上