版本：11.2.0.4.0
Reference : OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE) (Doc ID 428681.1)
CRS-4256 CRS-4602 While Replacing Voting Disk (文档 ID 1475588.1)
How to Restore ASM Based OCR After Complete Loss of the CRS Diskgroup on Linux/Unix Systems (文档 ID 1062983.1)
环境检查：
确认ocr,votedisk 磁盘位置。
[grid@prod01 ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 309717e185584f38bf0380d7e53e95ee (/dev/asm-diskb) [DATA]
2. ONLINE 791a04adb2534f28bf4954cfa12289a5 (/dev/asm-diskc) [DATA]
3. ONLINE af884b0a019d4f77bf3c0269106a3c95 (/dev/asm-diskd) [DATA]
Located 3 voting disk(s).
[grid@prod01 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2828
Available space (kbytes) : 259292
ID : 671775796
Device/File Name : +DATA
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user模拟破坏磁盘
[root@prod01 gpnpd]# dd if=/dev/zero of=/dev/asm-diskb bs=100M count=10
10+0 records in
10+0 records out
1048576000 bytes (1.0 GB) copied, 1.72446 s, 608 MB/s
[root@prod01 gpnpd]# dd if=/dev/zero of=/dev/asm-diskc bs=100M count=10
10+0 records in
10+0 records out
1048576000 bytes (1.0 GB) copied, 1.75642 s, 597 MB/s
[root@prod01 gpnpd]# dd if=/dev/zero of=/dev/asm-diskd bs=100M count=10
10+0 records in
10+0 records out
1048576000 bytes (1.0 GB) copied, 1.91177 s, 548 MB/s
[root@prod01 gpnpd]# 检查集群状态
[root@prod02 ~]# /u01/app/11.2.0/grid/bin/crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.重建磁盘组，
在一个节点操作,其它节点处于关闭状态。
/u01/app/11.2.0/grid/bin/crsctl start crs -nocrs -excl
建立data磁盘组，修改 asm_diskstring参数。
[root@prod01 ~]# /u01/app/11.2.0/grid/bin/crsctl start crs -nocrs -excl
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'prod01'
CRS-2676: Start of 'ora.mdnsd' on 'prod01' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'prod01'
CRS-2676: Start of 'ora.gpnpd' on 'prod01' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'prod01'
CRS-2672: Attempting to start 'ora.gipcd' on 'prod01'
CRS-2676: Start of 'ora.cssdmonitor' on 'prod01' succeeded
CRS-2676: Start of 'ora.gipcd' on 'prod01' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'prod01'
CRS-2672: Attempting to start 'ora.diskmon' on 'prod01'
CRS-2676: Start of 'ora.diskmon' on 'prod01' succeeded
CRS-2676: Start of 'ora.cssd' on 'prod01' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'prod01'
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'prod01'
CRS-2672: Attempting to start 'ora.ctssd' on 'prod01'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'prod01' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'prod01'
CRS-2676: Start of 'ora.drivers.acfs' on 'prod01' succeeded
CRS-2676: Start of 'ora.ctssd' on 'prod01' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'prod01' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'prod01'
CRS-2676: Start of 'ora.asm' on 'prod01' succeeded
[grid@prod01 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0 Production on Wed Nov 21 13:59:13 2018
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> create diskgroup DATA normal redundancy
DISK '/dev/asm-diskb','/dev/asm-diskc','/dev/asm-diskd'
attribute 'compatible.asm'='11.2'; 2 3
Diskgroup created.
SQL> alter system set asm_diskstring='/dev/*';
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
[grid@prod01 ~]$
恢复ocr,votedisk
ocrconfig -showbackup
root:
/u01/app/11.2.0/grid/bin/ocrconfig -restore /u01/app/11.2.0/grid/cdata/prod-cluster/backup_20181121_134213.ocr
grid
[grid@prod01 ~]$ ocrconfig -showbackup
PROT-26: Oracle Cluster Registry backup locations were retrieved from a local copy
prod02 2018/11/21 09:00:45 /u01/app/11.2.0/grid/cdata/prod-cluster/backup00.ocr
prod02 2018/11/21 05:00:45 /u01/app/11.2.0/grid/cdata/prod-cluster/backup01.ocr
prod02 2018/11/21 01:00:44 /u01/app/11.2.0/grid/cdata/prod-cluster/backup02.ocr
prod02 2018/11/20 01:00:42 /u01/app/11.2.0/grid/cdata/prod-cluster/day.ocr
prod02 2018/11/12 21:00:25 /u01/app/11.2.0/grid/cdata/prod-cluster/week.ocr
PROT-25: Manual backups for the Oracle Cluster Registry are not available
#root 恢复ocr
[root@prod01 ~]# /u01/app/11.2.0/grid/bin/ocrconfig -restore /u01/app/11.2.0/grid/cdata/prod-cluster/backup_20181121_134213.ocr
#root 恢复votedisk
[root@prod01 bin]# ./crsctl replace votedisk +DATA
Successful addition of voting disk 96e6e83cf3364f9abf13c9310830d798.
Successful addition of voting disk 3d317981258b4fcfbf64f119f41a7f28.
Successful addition of voting disk 3aa046d1930c4f39bf9bcf3297554d85.
Successfully replaced voting disk group with +DATA.
CRS-4266: Voting file(s) successfully replaced验证集群状态
关闭集群# crsctl stop crs -f
启动所有节点集群：# crsctl start crs
[root@prod01 bin]# ./crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE prod01
ONLINE ONLINE prod02
ora.DG1.dg
ONLINE OFFLINE prod01
ONLINE OFFLINE prod02
ora.LISTENER.lsnr
ONLINE ONLINE prod01
ONLINE ONLINE prod02
ora.asm
ONLINE ONLINE prod01 Started
ONLINE ONLINE prod02 Started
ora.gsd
OFFLINE OFFLINE prod01
OFFLINE OFFLINE prod02
ora.net1.network
ONLINE ONLINE prod01
ONLINE ONLINE prod02
ora.ons
ONLINE ONLINE prod01
ONLINE ONLINE prod02
ora.registry.acfs
ONLINE ONLINE prod01
ONLINE ONLINE prod02
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE prod01
ora.cvu
1 OFFLINE OFFLINE
ora.oc4j
1 OFFLINE OFFLINE
ora.prod01.vip
1 ONLINE ONLINE prod01
ora.prod02.vip
1 ONLINE ONLINE prod02
ora.scan1.vip
1 ONLINE ONLINE prod01
[root@prod01 bin]# ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 96e6e83cf3364f9abf13c9310830d798 (/dev/asm-diskb) [DATA]
2. ONLINE 3d317981258b4fcfbf64f119f41a7f28 (/dev/asm-diskc) [DATA]
3. ONLINE 3aa046d1930c4f39bf9bcf3297554d85 (/dev/asm-diskd) [DATA]
Located 3 voting disk(s).
[root@prod01 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2816
Available space (kbytes) : 259304
ID : 671775796
Device/File Name : +DATA
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
[root@prod01 bin]#asm spfile恢复
create spfile='+DATA' from memory.
[grid@prod01 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0 Production on Wed Nov 21 14:24:06 2018
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> show parameter spfile;
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string
SQL> show parameter asm_disk
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
asm_diskgroups string
asm_diskstring string
SQL> alter system set asm_diskstring='/dev/*';
System altered.
SQL> show parameter asm_disk
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
asm_diskgroups string
asm_diskstring string /dev/*
SQL> create spfile='+data' from memory;
File created.
SQL>CRS-4602错误故障：
[root@prod01 bin]# ./crsctl replace votedisk +DATA
CRS-4602: Failed 27 to add voting file 1d4482d87eea4f84bfb02d760f922b85.
CRS-4602: Failed 27 to add voting file 6dc3d69ccf204f28bf54faa59949aa32.
CRS-4602: Failed 27 to add voting file 5c44d63828734fbebf1cf9c70052cab6.
Failed to replace voting disk group with +DATA.
CRS-4000: Command Replace failed, or completed with errors.
[root@prod01 bin]#
由于asm spfile文件丢失导致asm_diskstring参数为空，导致该问题：
[grid@prod01 prod01]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0 Production on Wed Nov 21 14:13:18 2018
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> alter system set asm_diskstring='/dev/*';
System altered.
SQL>
