2-Node, 11.2.0.4 RAC with ASM. Normal redundancy diskgroups. When the storage link is broken, so that Node-A can not access Storage-B and Node-b can not access Storage-A
测试环境
Release 11.2.0.4.0
磁盘组信息
SQL> @c
DG_NAME DG_STATE TYPE DSK_NO DSK_NAME PATH MOUNT_S FAILGROUP STATE
--------------- ---------- ------ ---------- ---------- ------------------------------------------------------------ ------- -------------------- --------
CRS MOUNTED NORMAL 0 CRS_0000 /dev/oracleasm/disks/VOTE01 CACHED CRS_0000 NORMAL
CRS MOUNTED NORMAL 1 CRS_0001 /dev/oracleasm/disks/VOTE02 CACHED CRS_0001 NORMAL
CRS MOUNTED NORMAL 3 CRS_0003 /dev/oracleasm/disks/VOTEZC CACHED SYSFG3 NORMAL
DATA MOUNTED NORMAL 0 DATA_0000 /dev/oracleasm/disks/DATA01 CACHED AA NORMAL
DATA MOUNTED NORMAL 1 DATA_0001 /dev/oracleasm/disks/DATA02 CACHED BB NORMAL
DISK_NUMBER NAME PATH HEADER_STATUS OS_MB TOTAL_MB FREE_MB REPAIR_TIMER V FAILGRO
----------- ---------- ------------------------------ -------------------- ---------- ---------- ---------- ------------ - -------
0 DATA_0000 /dev/oracleasm/disks/DATA01 MEMBER 1019 1019 656 0 N REGULAR
0 CRS_0000 /dev/oracleasm/disks/VOTE01 MEMBER 2055 2055 1657 0 Y REGULAR
3 CRS_0003 /dev/oracleasm/disks/VOTEZC MEMBER 2055 2055 2021 0 Y QUORUM
1 CRS_0001 /dev/oracleasm/disks/VOTE02 MEMBER 2047 2047 1457 0 Y REGULAR
1 DATA_0001 /dev/oracleasm/disks/DATA02 MEMBER 1019 1019 656 0 N REGULAR
GROUP_NUMBER NAME COMPATIBILITY DATABASE_COMPATIBILITY V
------------ ---------- ------------------------------------------------------------ ------------------------------------------------------------ -
2 DATA 11.2.0.0.0 11.2.0.0.0 N
1 CRS 11.2.0.0.0 11.2.0.0.0 Y存储链路断开
ora01 只能看到data01 磁盘
ora02只能看到data02 磁盘
ora 01 asm 日志
Mon May 27 09:49:31 2019
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Mon May 27 09:49:55 2019
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Mon May 27 09:50:55 2019
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Mon May 27 09:51:43 2019
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Mon May 27 09:52:05 2019
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12930.trc:
ORA-15300: File +data/ora-cluster/OCRFILE/REGISTRY.255.1009297099 is incompatible with this operation
Mon May 27 09:52:21 2019
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_12732.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Mon May 27 09:52:48 2019
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_13215.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_13215.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_13215.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_13215.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_13215.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_13215.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Mon May 27 09:53:04 2019
WARNING: Read Failed. group:2 disk:1 AU:1 offset:0 size:4096
WARNING: Read Failed. group:2 disk:1 AU:1 offset:4096 size:4096
ERROR: no read quorum in group: required 2, found 0 disks
ERROR: Could not read PST for grp 2. Force dismounting the disk group.
Mon May 27 09:53:04 2019
NOTE: cache dismounting (not clean) group 2/0x4F1BECB7 (DATA)
NOTE: messaging CKPT to quiesce pins Unix process pid: 13254, image: oracle@ora01 (B000)
Mon May 27 09:53:04 2019
NOTE: halting all I/Os to diskgroup 2 (DATA)
Mon May 27 09:53:04 2019
NOTE: LGWR doing non-clean dismount of group 2 (DATA)
NOTE: LGWR sync ABA=6.353 last written ABA 6.353
Mon May 27 09:53:04 2019
kjbdomdet send to inst 2
detach from dom 2, sending detach message to inst 2
Mon May 27 09:53:05 2019
ERROR: ORA-15130 in COD recovery for diskgroup 2/0x4f1becb7 (DATA)
ERROR: ORA-15130 thrown in RBAL for group number 2
Mon May 27 09:53:05 2019
List of instances:
1 2
Dirty detach reconfiguration started (new ddet inc 1, cluster inc 12)
Global Resource Directory partially frozen for dirty detach
* dirty detach - domain 2 invalid = TRUE
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_rbal_7577.trc:
ORA-15130: diskgroup "DATA" is being dismounted
2 GCS resources traversed, 0 cancelled
Dirty Detach Reconfiguration complete
Mon May 27 09:53:05 2019
WARNING: dirty detached from domain 2
NOTE: cache dismounted group 2/0x4F1BECB7 (DATA)
NOTE: cache deleting context for group DATA 2/0x4f1becb7
SQL> alter diskgroup DATA dismount force /* ASM SERVER:1327230135 */
GMON dismounting group 2 at 32 for pid 34, osid 13254
NOTE: Disk DATA_0000 in mode 0x7f marked for de-assignment
NOTE: Disk DATA_0001 in mode 0x7f marked for de-assignment
NOTE:Waiting for all pending writes to complete before de-registering: grpnum 2
Mon May 27 09:53:35 2019
SUCCESS: diskgroup DATA was dismounted
SUCCESS: alter diskgroup DATA dismount force /* ASM SERVER:1327230135 */
Mon May 27 09:53:35 2019
NOTE: diskgroup resource ora.DATA.dg is offline
SUCCESS: ASM-initiated MANDATORY DISMOUNT of group DATAora02 asm 日志
Mon May 27 09:53:13 2019
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_12421.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA01"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_12421.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA01"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_12421.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA01"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_12421.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA01"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_12421.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA01"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_12421.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA01"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
WARNING: cache failed reading from group=2(DATA) fn=1 blk=8 count=1 from disk= 0 (DATA_0000) kfkist=0x20 status=0x02 osderr=0x0 file=kfc.c line=11596
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_12421.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA01"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
ORA-15080: synchronous I/O operation to a disk failed
WARNING: cache succeeded reading from group=2(DATA) fn=1 blk=8 count=1 from disk= 1 (DATA_0001) kfkist=0x20 status=0x01 osderr=0x0 file=kfc.c line=11637
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_12421.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA01"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_12421.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA01"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
WARNING: cache failed reading from group=2(DATA) fn=8 blk=0 count=1 from disk= 0 (DATA_0000) kfkist=0x20 status=0x02 osderr=0x0 file=kfc.c line=11596
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_12421.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA01"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
ORA-15080: synchronous I/O operation to a disk failed
Mon May 27 09:53:13 2019
NOTE: repairing group 2 file 1 extent 0
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_r000_12428.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA01"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
WARNING: cache succeeded reading from group=2(DATA) fn=8 blk=0 count=1 from disk= 1 (DATA_0001) kfkist=0x20 status=0x01 osderr=0x0 file=kfc.c line=11637
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_r000_12428.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA01"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
WARNING: block repair initiating disk offline
NOTE: disk 65534 is not a part of group 2 anymore so cannot be offlined.
NOTE: process _r000_+asm2 (12428) initiating offline of disk 0.3916345468 (DATA_0000) with mask 0x7e in group 2
NOTE: checking PST: grp = 2
Mon May 27 09:53:13 2019
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_12421.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA01"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_12421.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA01"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
WARNING: cache failed reading from group=2(DATA) fn=8 blk=60 count=1 from disk= 0 (DATA_0000) kfkist=0x20 status=0x02 osderr=0x0 file=kfc.c line=11596
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_12421.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA01"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
ORA-15080: synchronous I/O operation to a disk failed
NOTE: repairing group 2 file 8 extent 0
WARNING: cache succeeded reading from group=2(DATA) fn=8 blk=60 count=1 from disk= 1 (DATA_0001) kfkist=0x20 status=0x01 osderr=0x0 file=kfc.c line=11637
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_r001_12430.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA01"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_r001_12430.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA01"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Mon May 27 09:53:14 2019
NOTE: repairing group 2 file 8 extent 0
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_r002_12432.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA01"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
WARNING: block repair initiating disk offline
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_r002_12432.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA01"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_12421.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA01"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_12421.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA01"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
WARNING: cache failed reading from group=2(DATA) fn=8 blk=120 count=1 from disk= 0 (DATA_0000) kfkist=0x20 status=0x02 osderr=0x0 file=kfc.c line=11596
WARNING: block repair initiating disk offline
Mon May 27 09:53:14 2019
GMON checking disk modes for group 2 at 7 for pid 28, osid 12428
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_12421.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/DATA01"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
ORA-15080: synchronous I/O operation to a disk failed
WARNING: cache succeeded reading from group=2(DATA) fn=8 blk=120 count=1 from disk= 1 (DATA_0001) kfkist=0x20 status=0x01 osderr=0x0 file=kfc.c line=11637
NOTE: checking PST for grp 2 done.
NOTE: initiating PST update: grp = 2, dsk = 0/0xe96eb07c, mask = 0x6a, op = clear
GMON updating disk modes for group 2 at 8 for pid 28, osid 12428
NOTE: group DATA: updated PST location: disk 0001 (PST copy 0)
Mon May 27 09:53:15 2019
Received dirty detach msg from inst 1 for dom 2
Mon May 27 09:53:15 2019
NOTE: repairing group 2 file 8 extent 0
NOTE: disk 65534 is not a part of group 2 anymore so cannot be offlined.
NOTE: disk 65534 is not a part of group 2 anymore so cannot be offlined.
WARNING: block repair initiating disk offline
Mon May 27 09:53:15 2019
List of instances:
1 2
Dirty detach reconfiguration started (new ddet inc 1, cluster inc 12)
Global Resource Directory partially frozen for dirty detach
* dirty detach - domain 2 invalid = TRUE
26 GCS resources traversed, 0 cancelled
Dirty Detach Reconfiguration complete
Mon May 27 09:53:16 2019
NOTE: SMON starting instance recovery for group DATA domain 2 (mounted)
NOTE: SMON skipping disk 0 (mode=00000015)
NOTE: F1X0 found on disk 1 au 2 fcn 0.690
NOTE: PST enabling heartbeating (grp 2)
NOTE: disk 65534 is not a part of group 2 anymore so cannot be offlined.
NOTE: PST update grp = 2 completed successfully
NOTE: initiating PST update: grp = 2, dsk = 0/0xe96eb07c, mask = 0x7e, op = clear
GMON updating disk modes for group 2 at 9 for pid 28, osid 12428
NOTE: group DATA: updated PST location: disk 0001 (PST copy 0)
NOTE: cache closing disk 0 of grp 2: DATA_0000
NOTE: starting recovery of thread=1 ckpt=6.354 group=2 (DATA)
NOTE: disk 65534 is not a part of group 2 anymore so cannot be offlined.
NOTE: disk 65534 is not a part of group 2 anymore so cannot be offlined.
NOTE: ASM recovery sucessfully read ACD from one mirror side
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_smon_10850.trc:
ORA-15062: ASM disk is globally closed
ORA-15062: ASM disk is globally closed
NOTE: SMON waiting for thread 1 recovery enqueue
NOTE: SMON about to begin recovery lock claims for diskgroup 2 (DATA)
SUCCESS: extent 0 of file 8 group 2 repaired - all online mirror sides found readable, no repair required
SUCCESS: extent 0 of file 8 group 2 repaired - all online mirror sides found readable, no repair required
NOTE: waiting for instance recovery of group 2
NOTE: disk 65534 is not a part of group 2 anymore so cannot be offlined.
NOTE: PST update grp = 2 completed successfully
SUCCESS: extent 0 of file 8 group 2 repaired - all online mirror sides found readable, no repair required
SUCCESS: extent 0 of file 1 group 2 repaired - all online mirror sides found readable, no repair required
NOTE: SMON successfully validated lock domain 2
NOTE: advancing ckpt for group 2 (DATA) thread=1 ckpt=6.354
NOTE: SMON did instance recovery for group DATA domain 2
Mon May 27 09:53:39 2019
WARNING: Disk 0 (DATA_0000) in group 2 will be dropped in: (12960) secs on ASM inst 2
Mon May 27 09:56:42 2019
WARNING: Disk 0 (DATA_0000) in group 2 will be dropped in: (12775) secs on ASM inst 2
Mon May 27 09:59:45 2019
WARNING: Disk 0 (DATA_0000) in group 2 will be dropped in: (12592) secs on ASM inst 2
Mon May 27 10:02:49 2019
WARNING: Disk 0 (DATA_0000) in group 2 will be dropped in: (12409) secs on ASM inst 2
Mon May 27 10:05:52 2019
WARNING: Disk 0 (DATA_0000) in group 2 will be dropped in: (12226) secs on ASM inst 2
Mon May 27 10:08:55 2019
WARNING: Disk 0 (DATA_0000) in group 2 will be dropped in: (12043) secs on ASM inst 2ASM 状态信息
ORA01
DG_NAME DG_STATE TYPE DSK_NO DSK_NAME PATH MOUNT_S FAILGROUP STATE
--------------- ---------- ------ ---------- ---------- ------------------------------------------------------------ ------- -------------------- --------
CRS MOUNTED NORMAL 0 CRS_0000 /dev/oracleasm/disks/VOTE01 CACHED CRS_0000 NORMAL
CRS MOUNTED NORMAL 1 CRS_0001 /dev/oracleasm/disks/VOTE02 CACHED CRS_0001 NORMAL
CRS MOUNTED NORMAL 3 CRS_0003 /dev/oracleasm/disks/VOTEZC CACHED SYSFG3 NORMAL
DATA DISMOUNTED 0 /dev/oracleasm/disks/DATA01 CLOSED NORMAL
DISK_NUMBER NAME PATH HEADER_STATUS OS_MB TOTAL_MB FREE_MB REPAIR_TIMER V FAILGRO
----------- ---------- ------------------------------ -------------------- ---------- ---------- ---------- ------------ - -------
0 /dev/oracleasm/disks/DATA01 MEMBER 1019 0 0 0 N REGULAR
0 CRS_0000 /dev/oracleasm/disks/VOTE01 MEMBER 2055 2055 1657 0 Y REGULAR
3 CRS_0003 /dev/oracleasm/disks/VOTEZC MEMBER 2055 2055 2021 0 Y QUORUM
1 CRS_0001 /dev/oracleasm/disks/VOTE02 MEMBER 2047 2047 1457 0 Y REGULAR
GROUP_NUMBER NAME COMPATIBILITY DATABASE_COMPATIBILITY V
------------ ---------- ------------------------------------------------------------ ------------------------------------------------------------ -
0 DATA 0.0.0.0.0 0.0.0.0.0 N
1 CRS 11.2.0.0.0 11.2.0.0.0 Yora02
DG_NAME DG_STATE TYPE DSK_NO DSK_NAME PATH MOUNT_S FAILGROUP STATE
--------------- ---------- ------ ---------- ---------- ------------------------------------------------------------ ------- -------------------- --------
CRS MOUNTED NORMAL 0 CRS_0000 /dev/oracleasm/disks/VOTE01 CACHED CRS_0000 NORMAL
CRS MOUNTED NORMAL 1 CRS_0001 /dev/oracleasm/disks/VOTE02 CACHED CRS_0001 NORMAL
CRS MOUNTED NORMAL 3 CRS_0003 /dev/oracleasm/disks/VOTEZC CACHED SYSFG3 NORMAL
DATA MOUNTED NORMAL 0 DATA_0000 MISSING AA NORMAL
DATA MOUNTED NORMAL 1 DATA_0001 /dev/oracleasm/disks/DATA02 CACHED BB NORMAL
DISK_NUMBER NAME PATH HEADER_STATUS OS_MB TOTAL_MB FREE_MB
----------- ---------- ------------------------------ -------------------- ---------- ---------- ----------
0 DATA_0000 UNKNOWN 0 1019 656
0 CRS_0000 /dev/oracleasm/disks/VOTE01 MEMBER 2055 2055 1657
3 CRS_0003 /dev/oracleasm/disks/VOTEZC MEMBER 2055 2055 2021
1 DATA_0001 /dev/oracleasm/disks/DATA02 MEMBER 1019 1019 656
1 CRS_0001 /dev/oracleasm/disks/VOTE02 MEMBER 2047 2047 1457
