asm 磁盘组
当链路恢复之后，磁盘状态显示MISSING（CRS_0000，OCR_0000）。
[grid@prod02 ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.4.0 Production on Tue Sep 17 14:49:06 2019
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> @c
DG_NAME DG_STATE TYPE DSK_NO DSK_NAME PATH MOUNT_S FAILGROUP STATE
--------------- ---------- ------ ---------- ---------- -------------------------------------------------- ------- -------------------- --------
CRS MOUNTED NORMAL 2 CRS_0002 /dev/oracleasm/disks/DISK03 CACHED ZCDISK NORMAL
CRS MOUNTED NORMAL 5 CRS_0005 /dev/oracleasm/disks/VOTEDB02 CACHED CRS_0001 NORMAL
CRS MOUNTED NORMAL 9 CRS_0009 MISSING CRS_0000 NORMAL
OCR MOUNTED NORMAL 0 OCR_0000 MISSING OCR_0000 NORMAL
OCR MOUNTED NORMAL 1 OCR_0001 /dev/oracleasm/disks/NODE02DATA01 CACHED OCR_0001 NORMAL
DISK_NUMBER NAME PATH HEADER_STATUS OS_MB TOTAL_MB FREE_MB REPAIR_TIMER V FAILGRO
----------- ---------- -------------------------------------------------- -------------------- ---------- ---------- ---------- ------------ - -------
0 /dev/oracleasm/disks/NODE01DATA01 MEMBER 5120 0 0 0 N REGULAR
1 /dev/oracleasm/disks/VOTEDB01 MEMBER 2048 0 0 0 Y REGULAR
0 OCR_0000 UNKNOWN 0 5120 3185 12410 N REGULAR
9 CRS_0009 UNKNOWN 0 2048 1584 12410 N REGULAR
1 OCR_0001 /dev/oracleasm/disks/NODE02DATA01 MEMBER 5120 5120 3185 0 N REGULAR
5 CRS_0005 /dev/oracleasm/disks/VOTEDB02 MEMBER 2048 2048 1648 0 Y REGULAR
2 CRS_0002 /dev/oracleasm/disks/DISK03 MEMBER 5115 5115 5081 0 Y QUORUM
7 rows selected.
GROUP_NUMBER NAME COMPATIBILITY DATABASE_COMPATIBILITY V
------------ ---------- ------------------------------------------------------------ ------------------------------------------------------------ -
1 OCR 11.2.0.0.0 11.2.0.0.0 N
2 CRS 11.2.0.0.0 11.2.0.0.0 Y
SQL>恢复方法（在存活节点操作）
[grid@prod02 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0 Production on Tue Sep 17 14:51:45 2019
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> ALTER DISKGROUP crs online all;
Diskgroup altered.
SQL> ALTER DISKGROUP ocr online all;
Diskgroup altered.
SQL>磁盘组状态显示：
[grid@prod02 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0 Production on Tue Sep 17 14:52:55 2019
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> @c
DG_NAME DG_STATE TYPE DSK_NO DSK_NAME PATH MOUNT_S FAILGROUP STATE
--------------- ---------- ------ ---------- ---------- -------------------------------------------------- ------- -------------------- --------
CRS MOUNTED NORMAL 2 CRS_0002 /dev/oracleasm/disks/DISK03 CACHED ZCDISK NORMAL
CRS MOUNTED NORMAL 5 CRS_0005 /dev/oracleasm/disks/VOTEDB02 CACHED CRS_0001 NORMAL
CRS MOUNTED NORMAL 9 CRS_0009 /dev/oracleasm/disks/VOTEDB01 CACHED CRS_0000 NORMAL
OCR MOUNTED NORMAL 0 OCR_0000 /dev/oracleasm/disks/NODE01DATA01 CACHED OCR_0000 NORMAL
OCR MOUNTED NORMAL 1 OCR_0001 /dev/oracleasm/disks/NODE02DATA01 CACHED OCR_0001 NORMAL
DISK_NUMBER NAME PATH HEADER_STATUS OS_MB TOTAL_MB FREE_MB REPAIR_TIMER V FAILGRO
----------- ---------- -------------------------------------------------- -------------------- ---------- ---------- ---------- ------------ - -------
0 OCR_0000 /dev/oracleasm/disks/NODE01DATA01 MEMBER 5120 5120 3184 0 N REGULAR
1 OCR_0001 /dev/oracleasm/disks/NODE02DATA01 MEMBER 5120 5120 3184 0 N REGULAR
9 CRS_0009 /dev/oracleasm/disks/VOTEDB01 MEMBER 2048 2048 1552 0 Y REGULAR
5 CRS_0005 /dev/oracleasm/disks/VOTEDB02 MEMBER 2048 2048 1648 0 Y REGULAR
2 CRS_0002 /dev/oracleasm/disks/DISK03 MEMBER 5115 5115 5081 0 Y QUORUM
GROUP_NUMBER NAME COMPATIBILITY DATABASE_COMPATIBILITY V
------------ ---------- ------------------------------------------------------------ ------------------------------------------------------------ -
1 OCR 11.2.0.0.0 11.2.0.0.0 N
2 CRS 11.2.0.0.0 11.2.0.0.0 Y
SQL>注意事项！！！，否则不能执行online 操作。
磁盘组属性11.2.0.0.0（DATABASE_COMPATIBILITY，COMPATIBILITY）
集群软件和数据库不需要操作，自动恢复。
[grid@prod01 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.CRS.dg
ONLINE ONLINE prod01
ONLINE ONLINE prod02
ora.LISTENER.lsnr
ONLINE ONLINE prod01
ONLINE ONLINE prod02
ora.OCR.dg
ONLINE ONLINE prod01
ONLINE ONLINE prod02
ora.asm
ONLINE ONLINE prod01 Started
ONLINE ONLINE prod02 Started
ora.gsd
OFFLINE OFFLINE prod01
OFFLINE OFFLINE prod02
ora.net1.network
ONLINE ONLINE prod01
ONLINE ONLINE prod02
ora.ons
ONLINE ONLINE prod01
ONLINE ONLINE prod02
ora.registry.acfs
ONLINE ONLINE prod01
ONLINE ONLINE prod02
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE prod02
ora.cvu
1 ONLINE ONLINE prod02
ora.oc4j
1 ONLINE ONLINE prod02
ora.ora.db
1 ONLINE ONLINE prod01 Open
2 ONLINE ONLINE prod02 Open
ora.prod01.vip
1 ONLINE ONLINE prod01
ora.prod02.vip
1 ONLINE ONLINE prod02
ora.scan1.vip
1 ONLINE ONLINE prod02
[grid@prod01 ~]$prod01 grid log:
2019-09-17 14:49:12.688:
[/u01/app/11.2.0/grid/bin/cssdagent(42554)]CRS-5818:Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:15:28} in /u01/app/11.2.0/grid/log/prod01/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2019-09-17 14:49:13.787:
[ohasd(36900)]CRS-2878:Failed to restart resource 'ora.cssd'
2019-09-17 14:49:13.788:
[ohasd(36900)]CRS-2769:Unable to failover resource 'ora.cssd'.
2019-09-17 14:49:15.267:
[cssd(44154)]CRS-1713:CSSD daemon is started in clustered mode
2019-09-17 14:49:26.061:
[cssd(44154)]CRS-1707:Lease acquisition for node prod01 number 1 completed
2019-09-17 14:49:27.365:
[cssd(44154)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/VOTEDB02; details in /u01/app/11.2.0/grid/log/prod01/cssd/ocssd.log.
2019-09-17 14:49:27.373:
[cssd(44154)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/DISK03; details in /u01/app/11.2.0/grid/log/prod01/cssd/ocssd.log.
2019-09-17 14:49:29.932:
[cssd(44154)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod01 prod02 .
2019-09-17 14:49:31.272:
[ohasd(36900)]CRS-2878:Failed to restart resource 'ora.cssd'
2019-09-17 14:49:32.293:
[ctssd(44276)]CRS-2403:The Cluster Time Synchronization Service on host prod01 is in observer mode.
2019-09-17 14:49:32.600:
[ctssd(44276)]CRS-2407:The new Cluster Time Synchronization Service reference node is host prod02.
2019-09-17 14:49:32.602:
[ctssd(44276)]CRS-2401:The Cluster Time Synchronization Service started on host prod01.
2019-09-17 14:49:34.282:
[ohasd(36900)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2019-09-17 14:49:34.282:
[ohasd(36900)]CRS-2769:Unable to failover resource 'ora.diskmon'.
2019-09-17 14:49:41.580:
[/u01/app/11.2.0/grid/bin/oraagent.bin(37016)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/prod01/agent/ohasd/oraagent_grid/oraagent_grid.log"
2019-09-17 14:49:42.635:
[ctssd(44276)]CRS-2409:The clock on host prod01 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2019-09-17 14:49:46.814:
[/u01/app/11.2.0/grid/bin/oraagent.bin(37016)]CRS-5011:Check of resource "+ASM" failed: details at "(:CLSN00006:)" in "/u01/app/11.2.0/grid/log/prod01/agent/ohasd/oraagent_grid/oraagent_grid.log"
2019-09-17 14:49:58.993:
[ohasd(36900)]CRS-2878:Failed to restart resource 'ora.asm'
2019-09-17 14:50:00.024:
[ohasd(36900)]CRS-2878:Failed to restart resource 'ora.crsd'
2019-09-17 14:50:05.917:
[crsd(44448)]CRS-1012:The OCR service started on node prod01.
2019-09-17 14:50:05.943:
[evmd(44295)]CRS-1401:EVMD started on node prod01.
2019-09-17 14:50:07.250:
[crsd(44448)]CRS-1201:CRSD started on node prod01.
2019-09-17 14:50:08.757:
[/u01/app/11.2.0/grid/bin/oraagent.bin(44611)]CRS-5011:Check of resource "ora" failed: details at "(:CLSN00007:)" in "/u01/app/11.2.0/grid/log/prod01/agent/crsd/oraagent_oracle/oraagent_oracle.log"
2019-09-17 14:52:02.203:
[cssd(44154)]CRS-1605:CSSD voting file is online: /dev/oracleasm/disks/VOTEDB01; details in /u01/app/11.2.0/grid/log/prod01/cssd/ocssd.log.
2019-09-17 14:52:02.203:
[cssd(44154)]CRS-1626:A Configuration change request completed successfully
2019-09-17 14:52:02.217:
[cssd(44154)]CRS-1601:CSSD Reconfiguration complete. Active nodes are prod01 prod02 .prod01 asm log:
Tue Sep 17 14:49:52 2019
NOTE: No asm libraries found in the system
MEMORY_TARGET defaulting to 1128267776.
* instance_number obtained from CSS = 1, checking for the existence of node 0...
* node 0 does not exist. instance_number = 1
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Initial number of CPU is 8
Number of processor cores in the system is 8
Number of processor sockets in the system is 8
Private Interface 'eth1:1' configured from GPnP for use as a private interconnect.
[name='eth1:1', type=1, ip=169.254.237.99, mac=00-0c-29-1e-dd-e5, net=169.254.0.0/16, mask=255.255.0.0, use=haip:cluster_interconnect/62]
Public Interface 'eth0' configured from GPnP for use as a public interface.
[name='eth0', type=1, ip=192.168.0.230, mac=00-0c-29-1e-dd-db, net=192.168.0.0/24, mask=255.255.255.0, use=public/1]
CELL communication is configured to use 0 interface(s):
CELL IP affinity details:
NUMA status: non-NUMA system
cellaffinity.ora status: N/A
CELL communication will use 1 IP group(s):
Grp 0:
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/11.2.0/grid/dbs/arch
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
NOTE: Volume support enabled
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options.
ORACLE_HOME = /u01/app/11.2.0/grid
System name: Linux
Node name: prod01
Release: 2.6.32-642.el6.x86_64
Version: #1 SMP Wed Apr 13 00:51:26 EDT 2016
Machine: x86_64
VM name: VMWare Version: 6
Using parameter settings in server-side spfile +CRS/prod-cluster01/asmparameterfile/registry.253.993388635
System parameters with non-default values:
large_pool_size = 12M
instance_type = "asm"
remote_login_passwordfile= "EXCLUSIVE"
asm_diskstring = "/dev/oracleasm/disks/*"
asm_diskgroups = "VOTE"
asm_diskgroups = "OCR"
asm_power_limit = 1
_asm_storagemaysplit = TRUE
diagnostic_dest = "/u01/app/grid"
Cluster communication is configured to use the following interface(s) for this instance
169.254.237.99
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
Tue Sep 17 14:49:53 2019
PMON started with pid=2, OS id=44340
Tue Sep 17 14:49:53 2019
PSP0 started with pid=3, OS id=44342
Tue Sep 17 14:49:54 2019
VKTM started with pid=4, OS id=44345 at elevated priority
VKTM running at (1)millisec precision with DBRM quantum (100)ms
Tue Sep 17 14:49:54 2019
GEN0 started with pid=5, OS id=44349
Tue Sep 17 14:49:54 2019
DIAG started with pid=6, OS id=44351
Tue Sep 17 14:49:54 2019
PING started with pid=7, OS id=44353
Tue Sep 17 14:49:54 2019
DIA0 started with pid=8, OS id=44355
Tue Sep 17 14:49:54 2019
LMON started with pid=9, OS id=44357
Tue Sep 17 14:49:54 2019
LMD0 started with pid=10, OS id=44359
* Load Monitor used for high load check
* New Low - High Load Threshold Range = [7680 - 10240]
Tue Sep 17 14:49:54 2019
LMS0 started with pid=11, OS id=44361 at elevated priority
Tue Sep 17 14:49:54 2019
LMHB started with pid=12, OS id=44365
Tue Sep 17 14:49:54 2019
MMAN started with pid=13, OS id=44367
Tue Sep 17 14:49:54 2019
DBW0 started with pid=14, OS id=44369
Tue Sep 17 14:49:54 2019
LGWR started with pid=15, OS id=44371
Tue Sep 17 14:49:54 2019
CKPT started with pid=16, OS id=44373
Tue Sep 17 14:49:54 2019
SMON started with pid=17, OS id=44375
Tue Sep 17 14:49:54 2019
RBAL started with pid=18, OS id=44377
Tue Sep 17 14:49:54 2019
GMON started with pid=19, OS id=44379
Tue Sep 17 14:49:54 2019
MMON started with pid=20, OS id=44381
Tue Sep 17 14:49:54 2019
MMNL started with pid=21, OS id=44383
lmon registered with NM - instance number 1 (internal mem no 0)
Reconfiguration started (old inc 0, new inc 40)
ASM instance
List of instances:
1 2 (myinst: 1)
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
* allocate domain 2, invalid = TRUE
* allocate domain 1, invalid = TRUE
* domain 0 valid = 1 according to instance 2
* domain 2 valid = 1 according to instance 2
* domain 1 valid = 1 according to instance 2
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Submitted all GCS remote-cache requests
Fix write in gcs resources
Reconfiguration complete
Tue Sep 17 14:49:55 2019
LCK0 started with pid=22, OS id=44386
ORACLE_BASE not set in environment. It is recommended
that ORACLE_BASE be set in the environment
Tue Sep 17 14:49:57 2019
SQL> ALTER DISKGROUP ALL MOUNT /* asm agent call crs *//* {0:9:7} */
NOTE: Diskgroups listed in ASM_DISKGROUPS are
VOTE
OCR
NOTE: Diskgroup used for Voting files is:
CRS
Diskgroup with spfile:CRS
Diskgroup used for OCR is:OCR
NOTE: cache registered group OCR number=1 incarn=0xf62b4c86
NOTE: cache began mount (not first) of group OCR number=1 incarn=0xf62b4c86
NOTE: cache registered group CRS number=2 incarn=0x801b4c85
NOTE: cache began mount (not first) of group CRS number=2 incarn=0x801b4c85
NOTE: cache registered group VOTE number=3 incarn=0xe5ab4c87
NOTE: cache began mount (first) of group VOTE number=3 incarn=0xe5ab4c87
NOTE: Assigning number (1,1) to disk (/dev/oracleasm/disks/NODE02DATA01)
NOTE: Assigning number (1,0) to disk (/dev/oracleasm/disks/NODE01DATA01)
NOTE: cache closing disk 0 of grp 1: (not open) OCR_0000
NOTE: Assigning number (2,5) to disk (/dev/oracleasm/disks/VOTEDB02)
NOTE: Assigning number (2,9) to disk (/dev/oracleasm/disks/VOTEDB01)
NOTE: Assigning number (2,2) to disk (/dev/oracleasm/disks/DISK03)
NOTE: cache closing disk 9 of grp 2: (not open) CRS_0009
ERROR: no read quorum in group: required 2, found 0 disks
NOTE: cache dismounting (clean) group 3/0xE5AB4C87 (VOTE)
NOTE: messaging CKPT to quiesce pins Unix process pid: 44399, image: oracle@prod01 (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: lgwr not being msg'd to dismount
NOTE: cache dismounted group 3/0xE5AB4C87 (VOTE)
NOTE: cache ending mount (fail) of group VOTE number=3 incarn=0xe5ab4c87
NOTE: cache deleting context for group VOTE 3/0xe5ab4c87
GMON dismounting group 3 at 4 for pid 23, osid 44399
ERROR: diskgroup VOTE was not mounted
GMON querying group 1 at 5 for pid 23, osid 44399
NOTE: cache closing disk 0 of grp 1: (not open) OCR_0000
NOTE: cache opening disk 1 of grp 1: OCR_0001 path:/dev/oracleasm/disks/NODE02DATA01
NOTE: F1X0 found on disk 1 au 2 fcn 0.7444
NOTE: cache mounting (not first) normal redundancy group 1/0xF62B4C86 (OCR)
kjbdomatt send to inst 2
NOTE: attached to recovery domain 1
NOTE: redo buffer size is 256 blocks (1053184 bytes)
NOTE: LGWR attempting to mount thread 1 for diskgroup 1 (OCR)
Process LGWR (pid 44371) is running at high priority QoS for Exadata I/O
NOTE: LGWR found thread 1 closed at ABA 6.290
NOTE: LGWR mounted thread 1 for diskgroup 1 (OCR)
NOTE: LGWR opening thread 1 at fcn 0.7444 ABA 7.291
NOTE: cache mounting group 1/0xF62B4C86 (OCR) succeeded
NOTE: cache ending mount (success) of group OCR number=1 incarn=0xf62b4c86
GMON querying group 2 at 6 for pid 23, osid 44399
NOTE: cache closing disk 9 of grp 2: (not open) CRS_0009
NOTE: cache opening disk 2 of grp 2: CRS_0002 path:/dev/oracleasm/disks/DISK03
NOTE: cache opening disk 5 of grp 2: CRS_0005 path:/dev/oracleasm/disks/VOTEDB02
NOTE: F1X0 found on disk 5 au 2 fcn 0.197448
NOTE: cache mounting (not first) normal redundancy group 2/0x801B4C85 (CRS)
kjbdomatt send to inst 2
NOTE: attached to recovery domain 2
NOTE: redo buffer size is 256 blocks (1053184 bytes)
NOTE: LGWR attempting to mount thread 2 for diskgroup 2 (CRS)
NOTE: LGWR found thread 2 closed at ABA 25.705
NOTE: LGWR mounted thread 2 for diskgroup 2 (CRS)
NOTE: LGWR opening thread 2 at fcn 0.197448 ABA 26.706
NOTE: cache mounting group 2/0x801B4C85 (CRS) succeeded
NOTE: cache ending mount (success) of group CRS number=2 incarn=0x801b4c85
NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 1
SUCCESS: diskgroup OCR was mounted
NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 2
SUCCESS: diskgroup CRS was mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup "VOTE" cannot be mounted
ORA-15040: diskgroup is incomplete
ERROR: ALTER DISKGROUP ALL MOUNT /* asm agent call crs *//* {0:9:7} */
SQL> ALTER DISKGROUP ALL ENABLE VOLUME ALL /* asm agent *//* {0:9:7} */
SUCCESS: ALTER DISKGROUP ALL ENABLE VOLUME ALL /* asm agent *//* {0:9:7} */
WARNING: failed to online diskgroup resource ora.OCR.dg (unable to communicate with CRSD/OHASD)
Tue Sep 17 14:50:00 2019
NOTE: [crsd.bin@prod01 (TNS V1-V3) 44448] opening OCR file
Starting background process ASMB
Tue Sep 17 14:50:00 2019
ASMB started with pid=25, OS id=44471
Tue Sep 17 14:50:00 2019
NOTE: client +ASM1:+ASM registered, osid 44473, mbr 0x0
NOTE: Attempting voting file refresh on diskgroup CRS
NOTE: Refresh completed on diskgroup CRS
. Found 2 voting file(s).
NOTE: Voting file relocation is required in diskgroup CRS
NOTE: Attempting voting file relocation on diskgroup CRS
NOTE: Successful voting file relocation on diskgroup CRS
WARNING: failed to online diskgroup resource ora.CRS.dg (unable to communicate with CRSD/OHASD)
Tue Sep 17 14:50:37 2019
NOTE: client ora1:ora registered, osid 44859, mbr 0x1
Tue Sep 17 14:51:08 2019
ALTER SYSTEM SET local_listener=' (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.0.232)(PORT=1521))))' SCOPE=MEMORY SID='+ASM1';
Tue Sep 17 14:51:51 2019
NOTE: disk validation pending for group 2/0x801b4c85 (CRS)
NOTE: Attempting voting file refresh on diskgroup CRS
NOTE: Refresh completed on diskgroup CRS
. Found 2 voting file(s).
NOTE: Voting file relocation is required in diskgroup CRS
NOTE: Found /dev/oracleasm/disks/VOTEDB01 for disk CRS_0009
WARNING: ignoring disk in deep discovery
SUCCESS: validated disks for 2/0x801b4c85 (CRS)
NOTE: membership refresh pending for group 2/0x801b4c85 (CRS)
NOTE: Attempting voting file relocation on diskgroup CRS
NOTE: Successful voting file relocation on diskgroup CRS
Tue Sep 17 14:51:57 2019
GMON querying group 2 at 9 for pid 18, osid 44377
NOTE: cache opening disk 9 of grp 2: CRS_0009 path:/dev/oracleasm/disks/VOTEDB01
SUCCESS: refreshed membership for 2/0x801b4c85 (CRS)
NOTE: Attempting voting file refresh on diskgroup CRS
NOTE: Refresh completed on diskgroup CRS
. Found 2 voting file(s).
NOTE: Voting file relocation is required in diskgroup CRS
NOTE: Attempting voting file relocation on diskgroup CRS
NOTE: voting file allocation on grp 2 disk CRS_0009
Tue Sep 17 14:52:02 2019
NOTE: Successful voting file relocation on diskgroup CRS
NOTE: Voting File refresh pending for group 2/0x801b4c85 (CRS)
NOTE: disk validation pending for group 1/0xf62b4c86 (OCR)
NOTE: Found /dev/oracleasm/disks/NODE01DATA01 for disk OCR_0000
WARNING: ignoring disk in deep discovery
SUCCESS: validated disks for 1/0xf62b4c86 (OCR)
NOTE: Attempting voting file refresh on diskgroup CRS
NOTE: Refresh completed on diskgroup CRS
. Found 3 voting file(s).
Tue Sep 17 14:52:08 2019
NOTE: group OCR: updated PST location: disk 0001 (PST copy 0)
NOTE: group OCR: updated PST location: disk 0000 (PST copy 1)
NOTE: membership refresh pending for group 1/0xf62b4c86 (OCR)
GMON querying group 1 at 10 for pid 18, osid 44377
NOTE: group OCR: updated PST location: disk 0001 (PST copy 0)
NOTE: group OCR: updated PST location: disk 0000 (PST copy 1)
NOTE: cache opening disk 0 of grp 1: OCR_0000 path:/dev/oracleasm/disks/NODE01DATA01
SUCCESS: refreshed membership for 1/0xf62b4c86 (OCR)
NOTE: Voting File refresh pending for group 1/0xf62b4c86 (OCR)
Tue Sep 17 14:52:17 2019
NOTE: Attempting voting file refresh on diskgroup OCR
NOTE: Refresh completed on diskgroup OCR. No voting file found.prod01 db log:
Tue Sep 17 14:50:29 2019
Adjusting the default value of parameter parallel_max_servers
from 320 to 120 due to the value of parameter processes (150)
Starting ORACLE instance (normal)
************************ Large Pages Information *******************
Per process system memlock (soft) limit = UNLIMITED
Total Shared Global Region in Large Pages = 0 KB (0%)
Large Pages used by this instance: 0 (0 KB)
Large Pages unused system wide = 0 (0 KB)
Large Pages configured system wide = 0 (0 KB)
Large Page size = 2048 KB
RECOMMENDATION:
Total System Global Area size is 770 MB. For optimal performance,
prior to the next instance restart:
1. Increase the number of unused large pages by
at least 385 (page size 2048 KB, total size 770 MB) system wide to
get 100% of the System Global Area allocated with large pages
********************************************************************
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Initial number of CPU is 8
Number of processor cores in the system is 8
Number of processor sockets in the system is 8
Private Interface 'eth1:1' configured from GPnP for use as a private interconnect.
[name='eth1:1', type=1, ip=169.254.237.99, mac=00-0c-29-1e-dd-e5, net=169.254.0.0/16, mask=255.255.0.0, use=haip:cluster_interconnect/62]
Public Interface 'eth0' configured from GPnP for use as a public interface.
[name='eth0', type=1, ip=192.168.0.230, mac=00-0c-29-1e-dd-db, net=192.168.0.0/24, mask=255.255.255.0, use=public/1]
Public Interface 'eth0:1' configured from GPnP for use as a public interface.
[name='eth0:1', type=1, ip=192.168.0.232, mac=00-0c-29-1e-dd-db, net=192.168.0.0/24, mask=255.255.255.0, use=public/1]
CELL communication is configured to use 0 interface(s):
CELL IP affinity details:
NUMA status: non-NUMA system
cellaffinity.ora status: N/A
CELL communication will use 1 IP group(s):
Grp 0:
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.2.0/db_1/dbs/arch
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options.
ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1
System name: Linux
Node name: prod01
Release: 2.6.32-642.el6.x86_64
Version: #1 SMP Wed Apr 13 00:51:26 EDT 2016
Machine: x86_64
VM name: VMWare Version: 6
Using parameter settings in server-side pfile /u01/app/oracle/product/11.2.0/db_1/dbs/initora1.ora
System parameters with non-default values:
processes = 150
spfile = "+OCR/ora/spfileora.ora"
sga_target = 768M
control_files = "+OCR/ora/controlfile/current.260.1019221637"
db_block_size = 8192
compatible = "11.2.0.4.0"
cluster_database = TRUE
db_create_file_dest = "+OCR"
thread = 1
undo_tablespace = "UNDOTBS1"
instance_number = 1
remote_login_passwordfile= "EXCLUSIVE"
db_domain = ""
dispatchers = "(PROTOCOL=TCP) (SERVICE=oraXDB)"
remote_listener = "scan:1521"
audit_file_dest = "/u01/app/oracle/admin/ora/adump"
audit_trail = "DB"
db_name = "ora"
open_cursors = 300
pga_aggregate_target = 256M
diagnostic_dest = "/u01/app/oracle"
Cluster communication is configured to use the following interface(s) for this instance
169.254.237.99
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
Tue Sep 17 14:50:35 2019
PMON started with pid=2, OS id=44805
Tue Sep 17 14:50:35 2019
PSP0 started with pid=3, OS id=44807
Tue Sep 17 14:50:36 2019
VKTM started with pid=4, OS id=44809 at elevated priority
VKTM running at (1)millisec precision with DBRM quantum (100)ms
Tue Sep 17 14:50:36 2019
GEN0 started with pid=5, OS id=44813
Tue Sep 17 14:50:36 2019
DIAG started with pid=6, OS id=44815
Tue Sep 17 14:50:36 2019
DBRM started with pid=7, OS id=44817
Tue Sep 17 14:50:36 2019
PING started with pid=8, OS id=44819
Tue Sep 17 14:50:36 2019
ACMS started with pid=9, OS id=44821
Tue Sep 17 14:50:36 2019
DIA0 started with pid=10, OS id=44823
Tue Sep 17 14:50:36 2019
LMON started with pid=11, OS id=44825
Tue Sep 17 14:50:36 2019
LMD0 started with pid=12, OS id=44827
* Load Monitor used for high load check
* New Low - High Load Threshold Range = [7680 - 10240]
Tue Sep 17 14:50:36 2019
LMS0 started with pid=13, OS id=44829 at elevated priority
Tue Sep 17 14:50:36 2019
LMS1 started with pid=14, OS id=44833 at elevated priority
Tue Sep 17 14:50:37 2019
RMS0 started with pid=15, OS id=44837
Tue Sep 17 14:50:37 2019
LMHB started with pid=16, OS id=44839
Tue Sep 17 14:50:37 2019
MMAN started with pid=17, OS id=44841
Tue Sep 17 14:50:37 2019
DBW0 started with pid=18, OS id=44843
Tue Sep 17 14:50:37 2019
LGWR started with pid=19, OS id=44845
Tue Sep 17 14:50:37 2019
CKPT started with pid=20, OS id=44847
Tue Sep 17 14:50:37 2019
SMON started with pid=21, OS id=44849
Tue Sep 17 14:50:37 2019
RECO started with pid=22, OS id=44851
Tue Sep 17 14:50:37 2019
RBAL started with pid=23, OS id=44853
Tue Sep 17 14:50:37 2019
ASMB started with pid=24, OS id=44855
Tue Sep 17 14:50:37 2019
MMON started with pid=25, OS id=44857
Tue Sep 17 14:50:37 2019
MMNL started with pid=26, OS id=44861
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
NOTE: initiating MARK startup
starting up 1 shared server(s) ...Starting background process MARK
Tue Sep 17 14:50:37 2019
MARK started with pid=28, OS id=44865
NOTE: MARK has subscribed
lmon registered with NM - instance number 1 (internal mem no 0)
Reconfiguration started (old inc 0, new inc 16)
List of instances:
1 2 (myinst: 1)
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
* domain 0 valid according to instance 2
* domain 0 valid = 1 according to instance 2
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Submitted all GCS remote-cache requests
Fix write in gcs resources
Reconfiguration complete
Tue Sep 17 14:50:38 2019
LCK0 started with pid=31, OS id=44873
Starting background process RSMN
Tue Sep 17 14:50:38 2019
RSMN started with pid=32, OS id=44875
ORACLE_BASE not set in environment. It is recommended
that ORACLE_BASE be set in the environment
Tue Sep 17 14:50:39 2019
ALTER SYSTEM SET local_listener=' (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.0.232)(PORT=1521))' SCOPE=MEMORY SID='ora1';
ALTER DATABASE MOUNT /* db agent *//* {1:13741:2} */
NOTE: Loaded library: System
SUCCESS: diskgroup OCR was mounted
NOTE: dependency between database ora and diskgroup resource ora.OCR.dg is established
Successful mount of redo thread 1, with mount id 1637464745
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Lost write protection disabled
Completed: ALTER DATABASE MOUNT /* db agent *//* {1:13741:2} */
ALTER DATABASE OPEN /* db agent *//* {1:13741:2} */
Picked broadcast on commit scheme to generate SCNs
Thread 1 opened at log sequence 6
Current log# 2 seq# 6 mem# 0: +OCR/ora/onlinelog/group_2.262.1019221641
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
minact-scn: Inst 1 is a slave inc#:16 mmon proc-id:44857 status:0x2
minact-scn status: grec-scn:0x0000.00000000 gmin-scn:0x0000.00000000 gcalc-scn:0x0000.00000000
[44877] Successfully onlined Undo Tablespace 2.
Undo initialization finished serial:0 start:11652414 end:11652714 diff:300 (3 seconds)
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is WE8MSWIN1252
No Resource Manager plan active
Starting background process GTX0
Tue Sep 17 14:50:47 2019
GTX0 started with pid=36, OS id=44919
Starting background process RCBG
Tue Sep 17 14:50:47 2019
RCBG started with pid=37, OS id=44921
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Tue Sep 17 14:50:47 2019
QMNC started with pid=38, OS id=44923
Tue Sep 17 14:50:50 2019
Completed: ALTER DATABASE OPEN /* db agent *//* {1:13741:2} */
Tue Sep 17 14:50:50 2019
Starting background process CJQ0
Tue Sep 17 14:50:50 2019
CJQ0 started with pid=33, OS id=44946
Tue Sep 17 14:52:09 2019
NOTE: Found /dev/oracleasm/disks/NODE01DATA01 for disk OCR_0000
SUCCESS: disk OCR_0000 (0.3915103355) replaced in diskgroup OCR
NOTE: disk 0 (OCR_0000) in group 1 (OCR) is online for writes
Tue Sep 17 14:52:29 2019
NOTE: disk 0 (OCR_0000) in group 1 (OCR) is online for reads
Tue Sep 17 14:55:48 2019
Starting background process SMCO
Tue Sep 17 14:55:48 2019
SMCO started with pid=30, OS id=45629 
