系统环境：
操作系统：RedHat EL5.5
集群软件： GI 11G
数据库软件：Oracle 11.2.0.1
故障原因：
由于新节点（node3）是从原有的老节点（node2），克隆而来，在添加新节点时，忘了修改新节点的主机名，导致出现以下故障：
1、新节点CRS service启动正常
[root@node3 ~]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
2、listener 资源启动失败
[root@node3 ~]# crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.DG1.dg ora....up.type ONLINE ONLINE node1
ora....ER.lsnr ora....er.type ONLINE ONLINE node1
ora....N1.lsnr ora....er.type ONLINE ONLINE node1
ora....VOTE.dg ora....up.type ONLINE ONLINE node1
ora.RCY.dg ora....up.type ONLINE ONLINE node1
ora.asm ora.asm.type ONLINE ONLINE node1
ora.eons ora.eons.type ONLINE ONLINE node1
ora.gsd ora.gsd.type ONLINE ONLINE node1
ora....network ora....rk.type ONLINE ONLINE node1
ora....SM1.asm application ONLINE ONLINE node1
ora....E1.lsnr application ONLINE ONLINE node1
ora.node1.gsd application ONLINE ONLINE node1
ora.node1.ons application ONLINE ONLINE node1
ora.node1.vip ora....t1.type ONLINE ONLINE node1
ora....SM2.asm application ONLINE ONLINE node2
ora....E2.lsnr application ONLINE ONLINE node2
ora.node2.gsd application ONLINE ONLINE node2
ora.node2.ons application ONLINE ONLINE node2
ora.node2.vip ora....t1.type ONLINE ONLINE node2
ora....SM3.asm application ONLINE ONLINE node3
ora....E3.lsnr application OFFLINE OFFLINE
ora.node3.gsd application ONLINE ONLINE node3
ora.node3.ons application ONLINE ONLINE node3
ora.oc4j ora.oc4j.type ONLINE ONLINE node2
ora.ons ora.ons.type ONLINE ONLINE node1
ora.pmydb.db ora....se.type OFFLINE OFFLINE
ora....taf.svc ora....ce.type OFFLINE OFFLINE
ora.prod.db ora....se.type OFFLINE OFFLINE
ora....ry.acfs ora....fs.type ONLINE ONLINE node1
ora.scan1.vip ora....ip.type ONLINE ONLINE node1
[root@node3 ~]# crs_stat |grep lsn
NAME=ora.LISTENER.lsnr
NAME=ora.LISTENER_SCAN1.lsnr
NAME=ora.node1.LISTENER_NODE1.lsnr
NAME=ora.node2.LISTENER_NODE2.lsnr
NAME=ora.node3.LISTENER_NODE3.lsnr
NAME=ora.node3.LISTENER_NODE3.lsnr 服务启动失败
3、listener 资源故障解决方法：
手工启动：
[root@node3 ~]# crs_start ora.node3.LISTENER_NODE3.lsnr -f
CRS-2527: Unable to start 'ora.LISTENER.lsnr' because it has a 'hard' dependency on 'ora.cluster_vip_net1.type'
CRS-2525: All instances of the resource 'ora.node1.vip' are already running; relocate is not allowed because the force option was not specified
CRS-2525: All instances of the resource 'ora.node2.vip' are already running; relocate is not allowed because the force option was not specified
CRS-0222: Resource 'ora.node3.LISTENER_NODE3.lsnr' has dependency error.
从以上日志可以看出，listener 在启动时，缺少vip的支持（node3 vip service）
[root@node3 ~]# crs_stat |grep vip
NAME=ora.node1.vip
TYPE=ora.cluster_vip_net1.type
NAME=ora.node2.vip
TYPE=ora.cluster_vip_net1.type
NAME=ora.scan1.vip
TYPE=ora.scan_vip.type
缺少node vip service！
添加vip service：
[root@node3 ~]# srvctl add -h
The SRVCTL add command adds the configuration and the Oracle Clusterware application to the OCR for the cluster database, named instances, named services, or for the named nodes.
Usage: srvctl add vip -n <node_name> -k <network_number> -A <name|ip>/<netmask>/[if1[|if2...]] [-v]
[root@node1 ~]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost
192.168.8.41 node1
192.168.8.43 node1-vip
10.1.1.1 node1-priv
192.168.8.42 node2
192.168.8.44 node2-vip
10.1.1.2 node2-priv
192.168.8.45 scan_ip
192.168.8.46 node3
192.168.8.47 node3-vip
10.1.1.3 node3-priv
[root@node3 ~]# srvctl add vip -n node3 -A 192.168.8.47/255.255.255.0/eth0 -k 0
PRCN-2049 : The network attributes specified (network number: 0, subnet: 192.168.8.0, adapters: eth0) conflict with an already registered network (network number: 1, subnet: 192.168.8.0, adapters: eth0)
[root@node3 ~]# srvctl add vip -n node3 -A 192.168.8.47/255.255.255.0/eth0 -k 1
[root@node3 ~]#
添加node3 vip service 成功！
[root@node3 ~]# crs_stat |grep vip
NAME=ora.node1.vip
TYPE=ora.cluster_vip_net1.type
NAME=ora.node2.vip
TYPE=ora.cluster_vip_net1.type
NAME=ora.node3.vip
TYPE=ora.cluster_vip_net1.type
NAME=ora.scan1.vip
TYPE=ora.scan_vip.type
[root@node3 ~]#
启动ora.node3.LISTENER_NODE3.lsnr service：
[root@node3 ~]# crs_stat |grep lsn
NAME=ora.LISTENER.lsnr
NAME=ora.LISTENER_SCAN1.lsnr
NAME=ora.node1.LISTENER_NODE1.lsnr
NAME=ora.node2.LISTENER_NODE2.lsnr
NAME=ora.node3.LISTENER_NODE3.lsnr
[root@node3 ~]# crs_start -f ora.node3.LISTENER_NODE3.lsnr
Attempting to start `ora.node3.vip` on member `node3`
Start of `ora.node3.vip` on member `node3` succeeded.
Attempting to start `ora.LISTENER.lsnr` on member `node3`
Start of `ora.LISTENER.lsnr` on member `node3` succeeded.
[root@node3 ~]#
node3 Grid 日志：
[root@node3 node3]# pwd
/u01/11.2.0/grid/log/node3
[root@node3 node3]# tail -f alertnode3.log
[gpnpd(7901)]CRS-2332:Error pushing GPnP profile to "mdns:service:gpnp._tcp.local.://node1:12802/agent=gpnpd,cname=node-cluster,host=node1,pid=3431/gpnpd h:node1 c:node-cluster".
2014-04-17 14:47:42.399
[ctssd(8044)]CRS-2408:The clock on host node3 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2014-04-17 14:48:14.444
[ctssd(8044)]CRS-2408:The clock on host node3 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2014-04-17 14:48:18.952
[cssd(7954)]CRS-1601:CSSD Reconfiguration complete. Active nodes are node1 node2 node3 .
[ctssd(8044)]CRS-2408:The clock on host node3 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2014-04-17 14:49:34.501
[ctssd(8044)]CRS-2408:The clock on host node3 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2014-04-17 14:49:58.515
[ctssd(8044)]CRS-2408:The clock on host node3 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2014-04-17 14:50:22.552
[ctssd(8044)]CRS-2408:The clock on host node3 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2014-04-17 14:50:46.595
[ctssd(8044)]CRS-2408:The clock on host node3 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
4、添加Oracle 节点
CRS资源已经启动成功，但在node1 上添加 node3 的Oracle 节点时，报以下错误！
重新启动node的CRS service ,仍然不能解决，最后尝试在node2 上添加node3 的Oracle新节点，却成功（node1 不能添加，还是个不解之谜；可能是在添加CRS节点时是在node1上执行，添加出现了错误（在node3 运行root.sh时）导致node不能正常识别node3的新节点）！
node3 记录以下日志：
[root@node3 node3]# tail -f alertnode3.log
[gpnpd(7901)]CRS-2332:Error pushing GPnP profile to "mdns:service:gpnp._tcp.local.://node1:12802/agent=gpnpd,cname=node-cluster,host=node1,pid=3431/gpnpd h:node1 c:node-cluster".
2014-04-17 14:47:42.399
