AI 助理
备案控制台
开发者社区 云计算 文章 正文

asm Normal redundancy storage link is broken(19C)

2019-06-25 3102
版权
版权声明：
本文内容由阿里云实名注册用户自发贡献，版权归原作者所有，阿里云开发者社区不拥有其著作权，亦不承担相应法律责任。具体规则请查看《 阿里云开发者社区用户服务协议》和 《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容，填写 侵权投诉表单进行举报，一经查实，本社区将立刻删除涉嫌侵权内容。
简介： asm Normal redundancy storage link is broken，Storage Split

测试模拟环境

360_20190625093602067

create diskgroup dg1  normal redundancy 
failgroup aa disk '/dev/oracleasm/disks/VOTE01'  # 存储A
failgroup bb disk '/dev/oracleasm/disks/VOTE02'  # 存储B
quorum failgroup zc disk '/dev/oracleasm/disks/DISK01'    ## 仲裁存储
ATTRIBUTE 'compatible.asm' = '18.0.0.0.0', 'compatible.rdbms' = '18.0.0.0.0';

数据库版本


SQL*Plus: Release 18.0.0.0.0 - Production on Tue Jun 25 12:03:41 2019
Version 18.3.0.0.0

prod04 crs log:

2019-06-25 11:49:19.879 [OCSSD(4103)]CRS-1615: No I/O has completed after 50% of the maximum interval. If this persists, voting file /dev/oracleasm/disks/VOTE02 will be considered not functional in 99140 milliseconds.
2019-06-25 11:49:23.789 [OCSSD(4103)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 103770 msecs.
2019-06-25 11:49:23.791 [OCSSD(4103)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 103720 msecs.
2019-06-25 11:49:31.793 [OCSSD(4103)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 111780 msecs.
2019-06-25 11:49:31.795 [OCSSD(4103)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 111730 msecs.
2019-06-25 11:49:39.798 [OCSSD(4103)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 119780 msecs.
2019-06-25 11:49:39.799 [OCSSD(4103)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 119730 msecs.
2019-06-25 11:49:47.802 [OCSSD(4103)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 127780 msecs.
2019-06-25 11:49:47.803 [OCSSD(4103)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 127740 msecs.
2019-06-25 11:49:49.418 [OCSSD(4103)]CRS-1649: An I/O error occurred for voting file: /dev/oracleasm/disks/VOTE02; details at (:CSSNM00059:) in /u01/app/grid_base/diag/crs/prod04/crs/trace/ocssd.trc.
2019-06-25 11:49:49.418 [OCSSD(4103)]CRS-1649: An I/O error occurred for voting file: /dev/oracleasm/disks/VOTE02; details at (:CSSNM00060:) in /u01/app/grid_base/diag/crs/prod04/crs/trace/ocssd.trc.
2019-06-25 11:49:51.308 [OCSSD(4103)]CRS-1650: Configuration change 1561434591:0 rejected by node 2 prod05; details at (:CSSNM000066:) in /u01/app/grid_base/diag/crs/prod04/crs/trace/ocssd.trc
2019-06-25 11:49:51.352 [OCSSD(4103)]CRS-1601: CSSD Reconfiguration complete. Active nodes are prod04 prod05 .
2019-06-25 11:49:53.541 [CRSD(4667)]CRS-2878: Failed to restart resource 'ora.DG1.dg'
2019-06-25 11:49:53.543 [CRSD(4667)]CRS-2769: Unable to failover resource 'ora.DG1.dg'.
2019-06-25 11:49:54.575 [OCSSD(4103)]CRS-1650: Configuration change 1561434594:0 rejected by node 2 prod05; details at (:CSSNM000066:) in /u01/app/grid_base/diag/crs/prod04/crs/trace/ocssd.trc
2019-06-25 11:49:54.620 [OCSSD(4103)]CRS-1601: CSSD Reconfiguration complete. Active nodes are prod04 prod05 .
2019-06-25 11:50:09.811 [OCSSD(4103)]CRS-1614: No I/O has completed after 75% of the maximum interval. If this persists, voting file /dev/oracleasm/disks/VOTE02 will be considered not functional in 49210 milliseconds.
2019-06-25 11:50:39.821 [OCSSD(4103)]CRS-1613: No I/O has completed after 90% of the maximum interval. If this persists, voting file /dev/oracleasm/disks/VOTE02 will be considered not functional in 19200 milliseconds.
2019-06-25 11:50:59.827 [OCSSD(4103)]CRS-1604: CSSD voting file is offline: /dev/oracleasm/disks/VOTE02; details at (:CSSNM00058:) in /u01/app/grid_base/diag/crs/prod04/crs/trace/ocssd.trc.
2019-06-25 11:50:59.830 [OCSSD(4103)]CRS-1672: The number of voting files currently available 2 has fallen to the minimum number of voting files required 2. Further reduction in voting files will result in eviction and loss of functionality

prod05 asm log:

2019-06-25T11:49:50.946702+08:00
WARNING: Write Failed. group:2 disk:1 AU:1 offset:1044480 size:4096
path:/dev/oracleasm/disks/VOTE02
     incarnation:0xf0f0dce6 asynchronous result:'I/O error'
     subsys:System krq:0x7fb6e08d3c38 bufp:0x7fb6df381000 osderr1:0x69b5 osderr2:0x0
     IO elapsed time: 0 usec Time waited on I/O: 0 usec
WARNING: Hbeat write to PST disk 1.4042317030 in group 2 failed. [2]
2019-06-25T11:49:50.987326+08:00
NOTE: checking PST: grp = 2
2019-06-25T11:49:50.988005+08:00
GMON checking disk modes for group 2 at 28 for pid 41, osid 57148
2019-06-25T11:49:50.994499+08:00
NOTE: checking PST for grp 2 done.
2019-06-25T11:49:50.994807+08:00
NOTE: sending set offline flag message (1518616090) to 1 disk(s) in group 2
2019-06-25T11:49:50.995444+08:00
WARNING: Disk 1 (DG1_0001) in group 2 mode 0x7f is now being offlined
2019-06-25T11:49:51.006032+08:00
NOTE: initiating PST update: grp 2 (DG1), dsk = 1/0xf0f0dce6, mask = 0x6a, op = clear mandatory
2019-06-25T11:49:51.006479+08:00
GMON updating disk modes for group 2 at 29 for pid 41, osid 57148
WARNING: Write Failed. group:2 disk:1 AU:1 offset:4096 size:4096
path:/dev/oracleasm/disks/VOTE02
     incarnation:0xf0f0dce6 synchronous result:'I/O error'
     subsys:System krq:0x7fb6df36b668 bufp:0x7fb6df34f000 osderr1:0x534b4950 osderr2:0x0
     IO elapsed time: 0 usec Time waited on I/O: 0 usec
2019-06-25T11:49:51.139504+08:00
 Received dirty detach msg from inst 2 for dom 2
2019-06-25T11:49:51.168873+08:00
NOTE: Attempting voting file refresh on diskgroup DG1
NOTE: Refresh completed on diskgroup DG1. Found 3 voting file(s).
NOTE: Voting file relocation is required in diskgroup DG1
2019-06-25T11:49:51.182641+08:00
List of instances (total 2) :
 1 2
My inst 1   
Dirty detach reconfiguration started (new ddet inc 1, cluster inc 4)
 Global Resource Directory partially frozen for dirty detach
dirty detach - domain 2 invalid = TRUE 
 0 GCS resources traversed, 0 cancelled
2019-06-25T11:49:51.235840+08:00
NOTE: Attempting voting file relocation on diskgroup DG1
2019-06-25T11:49:51.249004+08:00
Dirty Detach Reconfiguration complete (total time 0.1 secs) 
2019-06-25T11:49:51.251495+08:00
NOTE: SMON detected lock domain 2 invalid at system inc 4 06/25/19 11:49:51
2019-06-25T11:49:51.372440+08:00
NOTE: Failed voting file relocation on diskgroup DG1
2019-06-25T11:49:51.478897+08:00
NOTE: SMON starting instance recovery of group DG1 domain 2 inc 4 (mounted) at 06/25/19 11:49:51
NOTE: 06/25/19 11:49:51 DG1.F1X0 found on disk 0 au 10 fcn 0.0 datfmt 1
NOTE: SMON skipping disk 1 (mode=00000015)
NOTE: instance recovery of group DG1 will recover thread=2 ckpt=2.0 domain=2 inc#=4 instnum=2
NOTE: SMON waiting for thread 2 recovery enqueue
NOTE: SMON about to begin recovery lock claims for diskgroup 2 (DG1)
validate pdb 2, flags x0, valid 0, pdb flags xa4 
* validated domain 2, flags = 0xa0
2019-06-25T11:49:51.515918+08:00
NOTE: SMON successfully validated lock domain 2 inc 4 (DG1)
2019-06-25T11:49:51.575068+08:00
Errors in file /u01/app/grid_base/diag/asm/+asm/+ASM1/trace/+ASM1_smon_4494.trc:
ORA-15025: could not open disk "/dev/oracleasm/disks/VOTE02"
ORA-27041: unable to open file
Linux-x86_64 Error: 6: No such device or address
Additional information: 3
WARNING: Write Failed. group:2 disk:1 AU:60 offset:0 size:4096
path:Unknown disk
     incarnation:0xf0f0dce6 asynchronous result:'I/O error'
     subsys:Unknown library krq:0x7fef485bd680 bufp:0x7fef47736000 osderr1:0x434c5344 osderr2:0x0
     IO elapsed time: 0 usec Time waited on I/O: 0 usec
NOTE: advancing ckpt for group 2 (DG1) thread=2 ckpt=2.0 domain inc# 4
NOTE: cache initiating offline of disk 1 group DG1
NOTE: initiating PST update: grp 2 (DG1), dsk = 1/0xf0f0dce6, mask = 0x6a, op = clear mandatory
2019-06-25T11:49:51.620068+08:00
GMON updating disk modes for group 2 at 30 for pid 23, osid 4494
2019-06-25T11:49:51.623397+08:00
NOTE: PST update grp = 2 completed successfully 
NOTE: ospid 4494 initiating cluster wide offline of the following disks from gn 2: 1
2019-06-25T11:49:51.629048+08:00
NOTE: disk 1 (DG1_0001) in group 2 (DG1) is locally offline for writes
2019-06-25T11:49:51.633752+08:00
NOTE: ospid 4494 finished cluster wide offline, gn 2
NOTE: successfully wrote at least one mirror side for diskgroup DG1
2019-06-25T11:49:51.634353+08:00
Errors in file /u01/app/grid_base/diag/asm/+asm/+ASM1/trace/+ASM1_smon_4494.trc:
ORA-15080: synchronous I/O operation failed to write block 10752 of disk 1 in disk group DG1
2019-06-25T11:49:51.634652+08:00
NOTE: SMON did instance recovery for group DG1 domain 2
2019-06-25T11:49:52.442330+08:00
NOTE: PST update grp = 2 completed successfully 
NOTE: initiating PST update: grp 2 (DG1), dsk = 1/0xf0f0dce6, mask = 0x7e, op = clear mandatory
2019-06-25T11:49:52.443344+08:00
GMON updating disk modes for group 2 at 31 for pid 41, osid 57148
2019-06-25T11:49:52.449953+08:00
NOTE: cache closing disk 1 of grp 2: DG1_0001
2019-06-25T11:49:52.459736+08:00
NOTE: cache closing disk 1 of grp 2: (not open) DG1_0001
2019-06-25T11:49:52.461747+08:00
NOTE: PST update grp = 2 completed successfully 
2019-06-25T11:49:53.999944+08:00
WARNING: Started Drop Disk Timeout for Disk 1 (DG1_0001) in group 2 with a value 43200
WARNING: Disk 1 (DG1_0001) in group 2 will be dropped in: (43200) secs on ASM inst 1
2019-06-25T11:49:54.475008+08:00
NOTE: Attempting voting file refresh on diskgroup DG1
NOTE: Refresh completed on diskgroup DG1. Found 3 voting file(s).
NOTE: Voting file relocation is required in diskgroup DG1
NOTE: Attempting voting file relocation on diskgroup DG1
NOTE: Failed voting file relocation on diskgroup DG1
2019-06-25T11:59:05.055729+08:00
WARNING: Disk 1 (DG1_0001) in group 2 will be dropped in: (42649) secs on ASM inst 1

prod05 crs log:

2019-06-25 11:49:19.272 [OCSSD(7735)]CRS-1615: No I/O has completed after 50% of the maximum interval. If this persists, voting file /dev/oracleasm/disks/VOTE01 will be considered not functional in 99990 milliseconds.
2019-06-25 11:49:24.324 [OCSSD(7735)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 103750 msecs.
2019-06-25 11:49:24.325 [OCSSD(7735)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 104060 msecs.
2019-06-25 11:49:32.329 [OCSSD(7735)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 111750 msecs.
2019-06-25 11:49:32.331 [OCSSD(7735)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 112070 msecs.
2019-06-25 11:49:40.334 [OCSSD(7735)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 119760 msecs.
2019-06-25 11:49:40.336 [OCSSD(7735)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 120070 msecs.
2019-06-25 11:49:48.340 [OCSSD(7735)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvWorkerThread_0 not scheduled for 127760 msecs.
2019-06-25 11:49:48.342 [OCSSD(7735)]CRS-1719: Cluster Synchronization Service daemon (CSSD) clssnmvDiskPingThread_0 not scheduled for 128080 msecs.
2019-06-25 11:49:49.811 [OCSSD(7735)]CRS-1649: An I/O error occurred for voting file: /dev/oracleasm/disks/VOTE01; details at (:CSSNM00059:) in /u01/app/grid_base/diag/crs/prod05/crs/trace/ocssd.trc.
2019-06-25 11:49:49.812 [OCSSD(7735)]CRS-1649: An I/O error occurred for voting file: /dev/oracleasm/disks/VOTE01; details at (:CSSNM00060:) in /u01/app/grid_base/diag/crs/prod05/crs/trace/ocssd.trc.
2019-06-25 11:49:51.313 [OCSSD(7735)]CRS-1641: Rejecting configuration change 1561434591:0 because only 1 of the required 0 voting files of the new configuration were located; details at (:CSSNM00026:) in /u01/app/grid_base/diag/crs/prod05/crs/trace/ocssd.trc
2019-06-25 11:49:51.383 [OCSSD(7735)]CRS-1601: CSSD Reconfiguration complete. Active nodes are prod04 prod05 .
2019-06-25 11:49:53.380 [ORAAGENT(8566)]CRS-5017: The resource action "ora.DG1.dg start" encountered the following error: 
2019-06-25 11:49:53.380+ORA-15032: not all alterations performed
ORA-15017: diskgroup "DG1" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DG1"
. For details refer to "(:CLSN00107:)" in "/u01/app/grid_base/diag/crs/prod05/crs/trace/crsd_oraagent_grid.trc".
2019-06-25 11:49:54.596 [OCSSD(7735)]CRS-1641: Rejecting configuration change 1561434594:0 because only 1 of the required 0 voting files of the new configuration were located; details at (:CSSNM00026:) in /u01/app/grid_base/diag/crs/prod05/crs/trace/ocssd.trc
2019-06-25 11:49:54.650 [OCSSD(7735)]CRS-1601: CSSD Reconfiguration complete. Active nodes are prod04 prod05 .
2019-06-25 11:49:58.836 [OHASD(2593)]CRS-2771: Maximum restart attempts reached for resource 'ora.storage'; will not restart.
2019-06-25 11:50:10.232 [OCSSD(7735)]CRS-1614: No I/O has completed after 75% of the maximum interval. If this persists, voting file /dev/oracleasm/disks/VOTE01 will be considered not functional in 49030 milliseconds.
2019-06-25 11:50:40.243 [OCSSD(7735)]CRS-1613: No I/O has completed after 90% of the maximum interval. If this persists, voting file /dev/oracleasm/disks/VOTE01 will be considered not functional in 19020 milliseconds.
2019-06-25 11:51:00.250 [OCSSD(7735)]CRS-1604: CSSD voting file is offline: /dev/oracleasm/disks/VOTE01; details at (:CSSNM00058:) in /u01/app/grid_base/diag/crs/prod05/crs/trace/ocssd.trc.
2019-06-25 11:51:00.251 [OCSSD(7735)]CRS-1672: The number of voting files currently available 2 has fallen to the minimum number of voting files required 2. Further reduction in voting files will result in eviction and loss of functionality
2019-06-25 11:55:29.161 [OHASD(2593)]CRS-2771: Maximum restart attempts reached for resource 'ora.storage'; will not restart.
2019-06-25 12:00:59.804 [ORAROOTAGENT(3154)]CRS-5019: All OCR locations are on ASM disk groups [DG1], and none of these disk groups are mounted. Details are at "(:CLSN00140:)" in "/u01/app/grid_base/diag/crs/prod05/crs/trace/ohasd_orarootagent_root.trc".

prod06 asm log:

2019-06-25T11:49:51.029835+08:00
WARNING: Disk 1 (DG1_0001) in group 2 mode 0x7f is now being offlined
WARNING: Read Failed. group:2 disk:0 AU:1 offset:0 size:4096
path:/dev/oracleasm/disks/VOTE01
     incarnation:0xf0f07ed3 synchronous result:'I/O error'
     subsys:System krq:0x7f57466038e0 bufp:0x7f574199a000 osderr1:0x69b5 osderr2:0x0
     IO elapsed time: 0 usec Time waited on I/O: 0 usec
WARNING: Read Failed. group:2 disk:0 AU:1 offset:4096 size:4096
path:/dev/oracleasm/disks/VOTE01
     incarnation:0xf0f07ed3 synchronous result:'I/O error'
     subsys:System krq:0x7f5746603d20 bufp:0x7f5741997000 osderr1:0x69b5 osderr2:0x0
     IO elapsed time: 0 usec Time waited on I/O: 0 usec
2019-06-25T11:49:51.051804+08:00
ERROR: no read quorum in group: required 1932620872, found 6143 disks
ERROR: Could not read PST for grp 2. Force dismounting the disk group.
2019-06-25T11:49:51.093245+08:00
NOTE: cache dismounting (not clean) group 2/0x38008E3F (DG1)
NOTE: messaging CKPT to quiesce pins Unix process pid: 70764, image: oracle@prod05 (B000)
2019-06-25T11:49:51.097051+08:00
NOTE: halting all I/Os to diskgroup 2 (DG1)
2019-06-25T11:49:51.099532+08:00
NOTE: LGWR doing non-clean dismount of group 2 (DG1) thread 2
NOTE: LGWR sync ABA=1.10750 last written ABA 1.10750
2019-06-25T11:49:51.100491+08:00
NOTE: initiating dirty detach from lock domain 2
2019-06-25T11:49:51.169386+08:00
kjbdomdet send to inst 1
detach from dom 2, sending detach message to inst 1
2019-06-25T11:49:51.231898+08:00
List of instances (total 2) :
 1 2
My inst 2   
Dirty detach reconfiguration started (new ddet inc 1, cluster inc 4)
 Global Resource Directory partially frozen for dirty detach
dirty detach - domain 2 invalid = TRUE 
 0 GCS resources traversed, 0 cancelled
Dirty Detach Reconfiguration complete (total time 0.0 secs) 
2019-06-25T11:49:51.300226+08:00
WARNING: dirty detached from lock domain 2
NOTE: CKPT released recovery enqueue for thread 2 group 2 (DG1)
2019-06-25T11:49:51.333377+08:00
NOTE: cache dismounted group 2/0x38008E3F (DG1)
2019-06-25T11:49:51.336638+08:00
SQL> alter diskgroup DG1 dismount force /* ASM SERVER:939560511 */ 
2019-06-25T11:49:51.338336+08:00
NOTE: client +ASM2:+ASM:prod-cluster no longer has group 1 (DATA) mounted
2019-06-25T11:49:51.545472+08:00

* instance 1 validates domain 2 
2019-06-25T11:49:51.658363+08:00
NOTE: disk 1 (DG1_0001) in group 2 (DG1) is locally offline for reads
WARNING: Read Failed. group:2 disk:0 AU:1 offset:0 size:4096
path:/dev/oracleasm/disks/VOTE01
     incarnation:0xf0f07ed3 synchronous result:'I/O error'
     subsys:System krq:0x7f57466038e0 bufp:0x7f574199a000 osderr1:0x69b5 osderr2:0x0
     IO elapsed time: 0 usec Time waited on I/O: 0 usec
WARNING: Read Failed. group:2 disk:0 AU:1 offset:4096 size:4096
path:/dev/oracleasm/disks/VOTE01
     incarnation:0xf0f07ed3 synchronous result:'I/O error'
     subsys:System krq:0x7f5746603d20 bufp:0x7f5741997000 osderr1:0x69b5 osderr2:0x0
     IO elapsed time: 0 usec Time waited on I/O: 0 usec
2019-06-25T11:49:51.663499+08:00
ERROR: no read quorum in group: required 1932620872, found 6143 disks
ERROR: Could not read PST for grp 2. Force dismounting the disk group.
NOTE: disk 1 (DG1_0001) in group 2 (DG1) is locally offline for writes
WARNING: Read Failed. group:2 disk:0 AU:1 offset:0 size:4096
path:/dev/oracleasm/disks/VOTE01
     incarnation:0xf0f07ed3 synchronous result:'I/O error'
     subsys:System krq:0x7f57466038e0 bufp:0x7f574199a000 osderr1:0x69b5 osderr2:0x0
     IO elapsed time: 0 usec Time waited on I/O: 0 usec
WARNING: Read Failed. group:2 disk:0 AU:1 offset:4096 size:4096
path:/dev/oracleasm/disks/VOTE01
     incarnation:0xf0f07ed3 synchronous result:'I/O error'
     subsys:System krq:0x7f5746603d20 bufp:0x7f5741997000 osderr1:0x69b5 osderr2:0x0
     IO elapsed time: 0 usec Time waited on I/O: 0 usec
2019-06-25T11:49:51.667890+08:00
ERROR: no read quorum in group: required 1932620872, found 6143 disks
ERROR: Could not read PST for grp 2. Force dismounting the disk group.
2019-06-25T11:49:52.317435+08:00
ERROR: ORA-15130 in COD recovery for diskgroup 2/0x38008e3f (DG1)
ERROR: ORA-15130 thrown in RBAL for group number 2
2019-06-25T11:49:52.328473+08:00
Errors in file /u01/app/grid_base/diag/asm/+asm/+ASM2/trace/+ASM2_rbal_9706.trc:
ORA-15130: diskgroup "DG1" is being dismounted
2019-06-25T11:49:52.357451+08:00
Errors in file /u01/app/grid_base/diag/asm/+asm/+ASM2/trace/+ASM2_rbal_9706.trc:
ORA-15130: diskgroup "DG1" is being dismounted
2019-06-25T11:49:52.359170+08:00
NOTE: client +ASM2:+ASM:prod-cluster no longer has group 2 (DG1) mounted
NOTE: cache deleting context for group DG1 2/0x38008e3f
2019-06-25T11:49:52.471944+08:00
GMON dismounting group 2 at 12 for pid 45, osid 70764
2019-06-25T11:49:52.474613+08:00
NOTE: Disk DG1_0000 in mode 0x7f marked for de-assignment
NOTE: Disk DG1_0001 in mode 0x1 marked for de-assignment
NOTE: Disk DISK01 in mode 0x7f marked for de-assignment
2019-06-25T11:49:52.490181+08:00
SUCCESS: diskgroup DG1 was dismounted
NOTE: Notify CRSD to check and update diskgroup resource state
NOTE: diskgroup resource ora.DG1.dg state is update
2019-06-25T11:49:52.556236+08:00
SUCCESS: alter diskgroup DG1 dismount force /* ASM SERVER:939560511 */
SUCCESS: ASM-initiated MANDATORY DISMOUNT of group DG1
2019-06-25T11:49:52.874415+08:00
SQL> ALTER DISKGROUP DG1 MOUNT  /* asm agent *//* {0:4:10} */ 
2019-06-25T11:49:52.930608+08:00
NOTE: cache registered group DG1 2/0x38008E42
NOTE: cache began mount (not first) of group DG1 2/0x38008E42
NOTE: Assigning number (2,1) to disk (/dev/oracleasm/disks/VOTE02)
NOTE: Assigning number (2,2) to disk (/dev/oracleasm/disks/DISK01)
2019-06-25T11:49:53.222251+08:00
ERROR: no read quorum in group: required 1932620872, found 6143 disks
2019-06-25T11:49:53.224797+08:00
NOTE: cache dismounting (clean) group 2/0x38008E42 (DG1)
NOTE: messaging CKPT to quiesce pins Unix process pid: 10073, image: oracle@prod05 (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: LGWR not being messaged to dismount
NOTE: cache dismounted group 2/0x38008E42 (DG1)
NOTE: cache ending mount (fail) of group DG1 number=2 incarn=0x38008e42
NOTE: cache deleting context for group DG1 2/0x38008e42
2019-06-25T11:49:53.329552+08:00
GMON dismounting group 2 at 14 for pid 41, osid 10073
2019-06-25T11:49:53.330351+08:00
NOTE: Disk DG1_0001 in mode 0x8 marked for de-assignment
NOTE: Disk DISK01 in mode 0x8 marked for de-assignment
ERROR: diskgroup DG1 was not mounted
2019-06-25T11:49:53.372223+08:00
ASM Health Checker found 1 new failures
2019-06-25T11:49:53.375295+08:00
WARNING: Disk Group DG1 containing voting files is not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup "DG1" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DG1"

2019-06-25T11:49:53.377036+08:00
ERROR: ALTER DISKGROUP DG1 MOUNT  /* asm agent *//* {0:4:10} */

结论，数据库2个节点正常运行，第二个节点磁盘组被umount(alter diskgroup DG1 dismount force / ASM SERVER:939560511 /)

PROD04:

[grid@prod04 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
ora.DATA.GHCHKPT.advm
               OFFLINE OFFLINE      prod04                   STABLE
               OFFLINE OFFLINE      prod05                   STABLE
ora.DATA.dg
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
ora.DG1.dg
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  OFFLINE      prod05                   STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
ora.chad
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
ora.data.ghchkpt.acfs
               OFFLINE OFFLINE      prod04                   STABLE
               OFFLINE OFFLINE      prod05                   STABLE
ora.helper
               ONLINE  ONLINE       prod04                   IDLE,STABLE
               ONLINE  ONLINE       prod05                   IDLE,STABLE
ora.net1.network
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
ora.ons
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
ora.proxy_advm
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       prod04                   STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       prod05                   169.254.10.230 10.16
                                                             8.0.202,STABLE
ora.asm
      1        ONLINE  ONLINE       prod04                   Started,STABLE
      2        ONLINE  ONLINE       prod05                   Started,STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       prod04                   STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       prod05                   Open,STABLE
ora.prod04.vip
      1        ONLINE  ONLINE       prod04                   STABLE
ora.prod05.vip
      1        ONLINE  ONLINE       prod05                   STABLE
ora.qosmserver
      1        ONLINE  ONLINE       prod04                   STABLE
ora.rhpserver
      1        OFFLINE OFFLINE                               STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       prod04                   STABLE
--------------------------------------------------------------------------------
[grid@prod04 ~]$ 
[grid@prod04 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   6a99161549b74fe7bf1ed9c2ac8c32e2 (/dev/oracleasm/disks/VOTE01) [DG1]
 2. OFFLINE  23763a2e80bf4fcabffc35e320ce2e99 (/dev/oracleasm/disks/VOTE02) [DG1]
 3. ONLINE   7c6f5cf2b71c4f5dbfa91b2abfd08d3c (/dev/oracleasm/disks/DISK01) [DG1]
Located 3 voting disk(s).

PROD05:

[grid@prod05 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
ora.DATA.GHCHKPT.advm
               OFFLINE OFFLINE      prod04                   STABLE
               OFFLINE OFFLINE      prod05                   STABLE
ora.DATA.dg
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
ora.DG1.dg
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  OFFLINE      prod05                   STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
ora.chad
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
ora.data.ghchkpt.acfs
               OFFLINE OFFLINE      prod04                   STABLE
               OFFLINE OFFLINE      prod05                   STABLE
ora.helper
               ONLINE  ONLINE       prod04                   IDLE,STABLE
               ONLINE  ONLINE       prod05                   IDLE,STABLE
ora.net1.network
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
ora.ons
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
ora.proxy_advm
               ONLINE  ONLINE       prod04                   STABLE
               ONLINE  ONLINE       prod05                   STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       prod04                   STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       prod05                   169.254.10.230 10.16
                                                             8.0.202,STABLE
ora.asm
      1        ONLINE  ONLINE       prod04                   Started,STABLE
      2        ONLINE  ONLINE       prod05                   Started,STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       prod04                   STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       prod05                   Open,STABLE
ora.prod04.vip
      1        ONLINE  ONLINE       prod04                   STABLE
ora.prod05.vip
      1        ONLINE  ONLINE       prod05                   STABLE
ora.qosmserver
      1        ONLINE  ONLINE       prod04                   STABLE
ora.rhpserver
      1        OFFLINE OFFLINE                               STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       prod04                   STABLE
--------------------------------------------------------------------------------
[grid@prod05 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. OFFLINE  6a99161549b74fe7bf1ed9c2ac8c32e2 (/dev/oracleasm/disks/VOTE01) [DG1]
 2. ONLINE   23763a2e80bf4fcabffc35e320ce2e99 (/dev/oracleasm/disks/VOTE02) [DG1]
 3. ONLINE   7c6f5cf2b71c4f5dbfa91b2abfd08d3c (/dev/oracleasm/disks/DISK01) [DG1]
Located 3 voting disk(s).
文章标签：
服务网格
关键词：
服务网格normal
snowofsummer
目录
相关文章
snowofsummer
|
SQL Oracle 关系型数据库
asm normal diskgroup ,strong split test
normal disk strong spilte.
snowofsummer
1227 0
snowofsummer
asm Normal redundancy storage link is broken,
asm storage splite
snowofsummer
1173 0
snowofsummer
|
关系型数据库 Oracle 数据库
asm normal 模式扩容
asm disk group ,normal ,FAILGROUP add disk
snowofsummer
1294 0
技术小能手
|
SQL 数据建模 关系型数据库
深入分析：12C ASM Normal冗余中PDB文件块号与AU关系与恢复
在 10G 和 11G 中，DBA 可以根据文件名，确定这个文件在 ASM 磁盘组上的分布，然后 dd 出来每一个 AU，最后拼凑成一个完成的数据文件。
技术小能手
1625 0
snowofsummer
|
7月前
|
Oracle 关系型数据库
oracle asm 磁盘显示offline
oracle asm 磁盘显示offline
snowofsummer
356 2
1436047922066202
|
2月前
|
存储 Oracle 关系型数据库
数据库数据恢复—Oracle ASM磁盘组故障数据恢复案例
Oracle数据库数据恢复环境&故障： Oracle ASM磁盘组由4块磁盘组成。Oracle ASM磁盘组掉线 ，ASM实例不能mount。 Oracle数据库故障分析&恢复方案： 数据库数据恢复工程师对组成ASM磁盘组的磁盘进行分析。对ASM元数据进行分析发现ASM存储元数据损坏，导致磁盘组无法挂载。
1436047922066202
66 1
1436047922066202
|
7月前
|
存储 Oracle 关系型数据库
【数据库数据恢复】Oracle数据库ASM磁盘组掉线的数据恢复案例
oracle数据库ASM磁盘组掉线，ASM实例不能挂载。数据库管理员尝试修复数据库，但是没有成功。
1436047922066202
165 1
【数据库数据恢复】Oracle数据库ASM磁盘组掉线的数据恢复案例
小麦苗
|
SQL Oracle 关系型数据库
Oracle ASM磁盘和磁盘组的常用SQL语句
Oracle ASM磁盘和磁盘组的常用SQL语句
小麦苗
290 0
snowofsummer
|
文字识别 Oracle NoSQL
oracle 11g 单机asm配置
oracle 11g 单机asm配置
snowofsummer
675 0
IT邦德
|
Oracle 关系型数据库
❤️Oracle ASM加磁盘及剔盘操作❤️
❤️Oracle ASM加磁盘及剔盘操作❤️
IT邦德
343 0

热门文章

最新文章

  • 1
    Kubernetes上的服务网格 Istio - 分布式追踪篇
  • 2
    动态代理解释-JDK,CGLIB,JAVASSIST,ASM
  • 3
    AIX平台下磁盘的PVID对ASM磁盘的破坏
  • 4
    ASM REACTING TO PARTITION ERRORS [ID 1062954.1]
  • 5
    RAC 下使用DBCA 建立ASM出错（ORA-15032，ORA-15063）
  • 6
    使用eBPF加速阿里云服务网格ASM
  • 7
    AIX添加ASM的裸盘
  • 8
    针对11.2 RAC丢失OCR和Votedisk所在ASM Diskgroup的恢复手段
  • 9
    单实例ASM无法创建磁盘组问题处理
  • 10
    asm数据文件迁移(asm–>asm)
  • 1
    探索微服务架构下的服务网格（Service Mesh）实践
    209
  • 2
    Golang深入浅出之-Go语言中的服务网格（Service Mesh）原理与应用
    472
  • 3
    探索微服务架构下的服务网格(Service Mesh)实践之路
    102
  • 4
    探索微服务架构下的服务网格(Service Mesh)实践
    92
  • 5
    【专栏】随着技术发展，未来将探索服务网格、容器化和云原生技术，以提升微服务架构的效能
    115
  • 6
    asm.js 的异步脚本
    58
  • 7
    使用ACME CA为ASM网关签发证书
    237
  • 8
    Resized LUNS Not Recoginzed by ASM Diskgroup (Doc ID 2973897.1)
    53
  • 9
    How To Resize An ASM Disk? (Doc ID 470209.1)
    55
  • 10
    OpenKruise 是一个基于 Istio 的云原生服务网格
    80

    • 相关课程

    更多
  • 深度解读服务网格的未来
  • Nacos 服务网格生态
  • 服务网格技术最佳实践

    • 相关电子书

    更多
  • 基于英特尔®架构的阿里云服务网格ASM技术加速应用服务加密通
  • 阿里云总监课第五期第四节：从容器到云原生–服务网格
  • MongoShake -- Multi Active-Active and Cross-Region Disaster Recoverable MongoDB Service
    • 下一篇
    DataWorks售前咨询