Oracle RAC Study之--Oracle 10g RAC 故障之--CRS启动失败
系统环境：
OS: linux EL5
CRS： CRS 10.2.0.1
Oracle：Oracle 10.2.0.1
1、crs 启动失败
[root@node2 ~]# crsctl check crs
|
1
2
3
|
Failure
1
contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM
|
2、查看系统日志
[root@node2 ~]# tail /var/log/messages
|
1
2
3
4
5
6
7
8
9
10
|
Nov
2
09
:
45
:
07
node2 logger: Cluster Ready Services waiting
on
dependencies. Diagnostics
in
/tmp/crsctl
.2928.
Nov
2
09
:
46
:
07
node2 logger: Cluster Ready Services waiting
on
dependencies. Diagnostics
in
/tmp/crsctl
.2962.
Nov
2
09
:
46
:
07
node2 logger: Cluster Ready Services waiting
on
dependencies. Diagnostics
in
/tmp/crsctl
.3172.
Nov
2
09
:
46
:
07
node2 logger: Cluster Ready Services waiting
on
dependencies. Diagnostics
in
/tmp/crsctl
.2928.
Nov
2
09
:
47
:
07
node2 logger: Cluster Ready Services waiting
on
dependencies. Diagnostics
in
/tmp/crsctl
.2962.
Nov
2
09
:
47
:
07
node2 logger: Cluster Ready Services waiting
on
dependencies. Diagnostics
in
/tmp/crsctl
.2928.
Nov
2
09
:
47
:
07
node2 logger: Cluster Ready Services waiting
on
dependencies. Diagnostics
in
/tmp/crsctl
.3172.
Nov
2
09
:
48
:
07
node2 logger: Cluster Ready Services waiting
on
dependencies. Diagnostics
in
/tmp/crsctl
.2962.
Nov
2
09
:
48
:
07
node2 logger: Cluster Ready Services waiting
on
dependencies. Diagnostics
in
/tmp/crsctl
.2928.
Nov
2
09
:
48
:
07
node2 logger: Cluster Ready Services waiting
on
dependencies. Diagnostics
in
/tmp/crsctl
.3172.
|
[root@node2 ~]# cat /tmp/crsctl.3172
OCR initialization failed accessing OCR device: PROC-26: Error while accessing the physical storage Operating System error [Permission denied] [13]
[root@node2 ~]#
3、ocr 对应的裸设备权限出错
[root@node2 ~]# ls -l /dev/raw
total 0
crw------- 1 root root 162, 1 Nov 2 09:41 raw1
crw------- 1 root root 162, 2 Nov 2 09:41 raw2
[root@node2 ~]#
4、修改raw 权限配置文件，系统启动时自动应用权限
[root@node2 ~]# cat /etc/udev/rules.d/65-raw-permissions.rules
# Set permissions of raw bindings to Oracle Clusterware devices
KERNEL=="raw1", OWNER="oracle", GROUP="dba", MODE="660"
KERNEL=="raw2", OWNER="oracle", GROUP="dba", MODE="660"
[root@node2 ~]#
5、重新启动udev 服务
[root@node1 rules.d]# start_udev
Starting udev: [ OK ]
[root@node1 rules.d]# ls -l /dev/raw
total 0
crw-rw---- 1 oracle dba 162, 1 Nov 2 09:55 raw1
crw-rw---- 1 oracle dba 162, 2 Nov 2 09:55 raw2
[root@node1 rules.d]#
6、重新启动crs ，系统正常
|
1
2
3
4
5
6
7
8
9
10
|
[root@node2 ~]# crsctl stop crs
Stopping resources.
Successfully stopped CRS resources
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
[root@node2 ~]# crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
|
[root@node2 ~]# tail /var/log/messages
|
1
2
3
4
5
6
7
8
9
10
|
Nov
2
09
:
52
:
07
node2 logger: Cluster Ready Services waiting
on
dependencies. Diagnostics
in
/tmp/crsctl
.2928.
Nov
2
09
:
52
:
08
node2 logger: Cluster Ready Services waiting
on
dependencies. Diagnostics
in
/tmp/crsctl
.3172.
Nov
2
09
:
53
:
08
node2 logger: Cluster Ready Services waiting
on
dependencies. Diagnostics
in
/tmp/crsctl
.2962.
Nov
2
09
:
53
:
08
node2 logger: Cluster Ready Services waiting
on
dependencies. Diagnostics
in
/tmp/crsctl
.3172.
Nov
2
09
:
53
:
08
node2 logger: Cluster Ready Services waiting
on
dependencies. Diagnostics
in
/tmp/crsctl
.2928.
Nov
2
09
:
54
:
10
node2 logger: Cluster Ready Services completed waiting
on
dependencies.
Nov
2
09
:
54
:
10
node2
last
message repeated
2
times
Nov
2
09
:
55
:
10
node2 logger: Running CRSD
with
TZ =
Nov
2
09
:
56
:
01
node2 logger: Oracle CSSD graceful shutdown
Nov
2
09
:
56
:
32
node2 logger: Cluster Ready Services completed waiting
on
dependencies.
|
[root@node2 ~]# crsctl check crs
|
1
2
3
|
CSS appears healthy
CRS appears healthy
EVM appears healthy
|
[root@node2 ~]# crs_stat -t
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
Name Type Target State Host
------------------------------------------------------------
ora....SM1.asm application ONLINE ONLINE node1
ora....E1.lsnr application ONLINE ONLINE node1
ora.node1.gsd application ONLINE ONLINE node1
ora.node1.ons application ONLINE ONLINE node1
ora.node1.vip application ONLINE ONLINE node1
ora....SM2.asm application ONLINE OFFLINE
ora....E2.lsnr application ONLINE ONLINE node2
ora.node2.gsd application ONLINE ONLINE node2
ora.node2.ons application ONLINE ONLINE node2
ora.node2.vip application ONLINE ONLINE node2
ora.prod.db application ONLINE ONLINE node2
ora....d1.inst application ONLINE ONLINE node1
ora....d2.inst application ONLINE OFFLINE
ora...._taf.cs application OFFLINE OFFLINE
ora....od1.srv application OFFLINE OFFLINE
ora....od2.srv application OFFLINE OFFLINE
|
@至此，CRS启动成功！
