rac11g错误：ORA-15097: cannot SHUTDOWN ASM instance with connected client

背景：

想停rac环境，之前10g都是用crs_stop -all 来停止，所以，这次也这样做的，没想到报了如下错误：

[grid@rac2 ~]$ crs_stop -all
CRS-2500: Cannot stop resource 'ora.gsd' as it is not running
CRS-2500: Cannot stop resource 'ora.oc4j' as it is not running
Attempting to stop `ora.ons` on member `rac2`
Attempting to stop `ora.LISTENER.lsnr` on member `rac2`
Attempting to stop `ora.eons` on member `rac2`
CRS-2789: Cannot stop resource 'ora.gsd' as it is not running on server 'rac2'
Attempting to stop `ora.eons` on member `rac1`
Attempting to stop `ora.LISTENER.lsnr` on member `rac1`
Attempting to stop `ora.DATA.dg` on member `rac2`
Attempting to stop `ora.FRA.dg` on member `rac2`
Attempting to stop `ora.OCR_VOTE.dg` on member `rac2`
Attempting to stop `ora.registry.acfs` on member `rac2`
Attempting to stop `ora.test.db` on member `rac2`
CRS-2789: Cannot stop resource 'ora.gsd' as it is not running on server 'rac1'
Attempting to stop `ora.DATA.dg` on member `rac1`
Attempting to stop `ora.FRA.dg` on member `rac1`
Attempting to stop `ora.OCR_VOTE.dg` on member `rac1`
Attempting to stop `ora.registry.acfs` on member `rac1`
Attempting to stop `ora.test.db` on member `rac1`
Stop of `ora.LISTENER.lsnr` on member `rac2` succeeded.
CRS-2789: Cannot stop resource 'ora.LISTENER.lsnr' as it is not running on server 'rac2'
Attempting to stop `ora.rac2.vip` on member `rac2`
Stop of `ora.rac2.vip` on member `rac2` succeeded.
Stop of `ora.ons` on member `rac2` succeeded.
CRS-2789: Cannot stop resource 'ora.ons' as it is not running on server 'rac2'
Attempting to stop `ora.net1.network` on member `rac2`
Stop of `ora.net1.network` on member `rac2` succeeded.
Stop of `ora.LISTENER.lsnr` on member `rac1` succeeded.
Attempting to stop `ora.ons` on member `rac1`
Attempting to stop `ora.rac1.vip` on member `rac1`
Attempting to stop `ora.LISTENER_SCAN1.lsnr` on member `rac1`
Stop of `ora.registry.acfs` on member `rac2` succeeded.
Stop of `ora.rac1.vip` on member `rac1` succeeded.
Stop of `ora.LISTENER_SCAN1.lsnr` on member `rac1` succeeded.
Attempting to stop `ora.scan1.vip` on member `rac1`
Stop of `ora.scan1.vip` on member `rac1` succeeded.
Stop of `ora.ons` on member `rac1` succeeded.
CRS-2789: Cannot stop resource 'ora.ons' as it is not running on server 'rac1'
Attempting to stop `ora.net1.network` on member `rac1`
Stop of `ora.net1.network` on member `rac1` succeeded.
Stop of `ora.registry.acfs` on member `rac1` succeeded.
Stop of `ora.eons` on member `rac2` succeeded.
Stop of `ora.OCR_VOTE.dg` on member `rac1` succeeded.
Stop of `ora.eons` on member `rac1` succeeded.
Stop of `ora.test.db` on member `rac2` succeeded.
Stop of `ora.OCR_VOTE.dg` on member `rac2` succeeded.
Stop of `ora.test.db` on member `rac1` succeeded.
CRS-0216: Could not stop resource 'ora.LISTENER.lsnr rac2 1'.


CRS-0216: Could not stop resource 'ora.asm rac1 1'.
CRS-0216: Could not stop resource 'ora.asm rac2 1'.
CRS-0216: Could not stop resource 'ora.gsd'.
CRS-0216: Could not stop resource 'ora.gsd rac1 1'.
CRS-0216: Could not stop resource 'ora.gsd rac2 1'.
CRS-0216: Could not stop resource 'ora.oc4j'.
CRS-0216: Could not stop resource 'ora.ons rac1 1'.
CRS-0216: Could not stop resource 'ora.ons rac2 1'.

以上这些进程都没有被关闭，现在看下就跟下面这样了：

[grid@rac2 bin]$ crs_stat -t
Name           Type           Target    State     Host        
------------------------------------------------------------
ora.DATA.dg    ora....up.type OFFLINE   OFFLINE               
ora.FRA.dg     ora....up.type OFFLINE   OFFLINE               
ora....ER.lsnr ora....er.type OFFLINE   OFFLINE               
ora....N1.lsnr ora....er.type OFFLINE   OFFLINE               
ora....VOTE.dg ora....up.type OFFLINE   OFFLINE               
ora.asm        ora.asm.type   OFFLINE   ONLINE    rac2        
ora.eons       ora.eons.type  OFFLINE   OFFLINE               
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE               
ora....network ora....rk.type OFFLINE   OFFLINE               
ora.oc4j       ora.oc4j.type  OFFLINE   OFFLINE               
ora.ons        ora.ons.type   OFFLINE   OFFLINE               
ora....SM1.asm application    OFFLINE   ONLINE    rac1        
ora....C1.lsnr application    OFFLINE   OFFLINE               
ora.rac1.gsd   application    OFFLINE   OFFLINE               
ora.rac1.ons   application    OFFLINE   OFFLINE               
ora.rac1.vip   ora....t1.type OFFLINE   OFFLINE               
ora....SM2.asm application    OFFLINE   ONLINE    rac2        
ora....C2.lsnr application    OFFLINE   OFFLINE               
ora.rac2.gsd   application    OFFLINE   OFFLINE               
ora.rac2.ons   application    OFFLINE   OFFLINE               
ora.rac2.vip   ora....t1.type OFFLINE   OFFLINE               
ora....ry.acfs ora....fs.type OFFLINE   OFFLINE               
ora.scan1.vip  ora....ip.type OFFLINE   OFFLINE               
ora.test.db    ora....se.type OFFLINE   OFFLINE  

然后再尝试手工关闭asm：

[grid@rac2 bin]$ srvctl stop asm -n rac1
PRCR-1014 : Failed to stop resource ora.asm
PRCR-1065 : Failed to stop resource ora.asm
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
ORA-15097: cannot SHUTDOWN ASM instance with connected client
CRS-2675: Stop of 'ora.asm' on 'rac1' failed

CRS-2675: Stop of 'ora.asm' on 'rac1' failed

原因及解决方案：

原来在Oracle 10g 环境下，我们停RAC的顺序是：

       （1）数据库

       （2）ASM

       （3）CRS

       但是在Oracle 11gR2下，如果是用OUI来安装的话，除了数据文件之外，OCR 也是放在ASM里的。  所以这里就遇到了问题。 OCR放在ASM里，这样CRSD 就会成为ASM的客户端。 如果我们像Oracle 10g里那样直接停ASM的时候，就会报上面的错误。 所以，在11gR2下面，要停ASM实例的话，只能和CRS一起停。

用以下方法停asm：在root用户下执行：crsctl stop crs

注意：不要直接kill 掉ASM 进程，或者用shutdown abort 来关闭ASM 实例。 这样CRS 也会被挂掉。

所以，我们之前10g rac停止的步骤，到了srvctl stop asm -n rac1 这一行，就没法执行了。。

srvctl stop instance -d test -i test1 
srvctl stop instance -d test -i test2 
srvctl stop asm -n rac1 
srvctl stop asm -n rac2 
srvctl stop nodeapps -n rac1 
srvctl stop nodeapps -n rac2 

所以，一般11g rac的启动和关闭步骤，要按照下面的顺序进行：

一 关闭rac
1，确认srvctl 和ps -ef|grep smon
[grid@rac1 ~]$ srvctl status database -d RACDB
实例 RACDB1 正在节点 rac1 上运行
实例 RACDB2 正在节点 rac2 上运行
[grid@rac1 ~]$ ps -ef|grep smon
oracle    3676     1  0 06:05 ?        00:00:02 ora_smon_RACDB1
grid     12840     1  0 01:54 ?        00:00:00 asm_smon_+ASM1
grid     27890 27621  0 07:52 pts/3    00:00:00 grep smon

2,将数据库关闭并再次确认
[grid@rac1 ~]$ srvctl stop database -d RACDB
[grid@rac1 ~]$  ps -ef|grep smon

3,使用root 帐号关闭ASM
[grid@rac1 ~]$ su -
口令：
[root@rac1 ~]# cd /home/grid

[root@rac1 grid]# sh .bash_profile
4，使用crs_stat 确认集群各项资源和服务运行状态
[root@rac1 bin]# /u01/app/11.2.0/grid/bin/crs_stat -t -v
5，使用crsctl 指令关闭集群
[root@rac1 bin]# /u01/app/11.2.0/grid/bin/crsctl stop cluster -all
6，使用crs_stat 确认集群各项资源和服务运行状态
[root@rac1 bin]# /u01/app/11.2.0/grid/bin/crs_stat -t -v
[root@rac2 ~]# /u01/app/11.2.0/grid/bin/crs_stat -t -v
CRS-0184: Cannot communicate with the CRS daemon.
 说明顺利关闭

二 。RAC 开启
1，root 执行grid 下面的环境变量 (可以不执行直接到/u01/app/11.2.0/grid/bin/模式下)
2，[root@rac1 ~]# /u01/app/11.2.0/grid/bin/crsctl stop cluster -all
3，开启集群
  [root@rac1 ~]# /u01/app/11.2.0/grid/bin/crsctl start cluster -all
  检查状态
  [root@rac2 ~]# /u01/app/11.2.0/grid/bin/crs_stat -t -v
4，使用srvcel 确认数据库状态

 [root@rac1 ~]# /u01/app/11.2.0/grid/bin/srvctl status database -d RACDB
实例 RACDB1 没有在 rac1 节点上运行
实例 RACDB2 没有在 rac2 节点上运行
5，打开RAC 
[root@rac1 ~]# /u01/app/11.2.0/grid/bin/srvctl start database -d RACDB
 确认状态
[root@rac2 ~]# /u01/app/11.2.0/grid/bin/srvctl status database -d RACDB
实例 RACDB1 正在节点 rac1 上运行
实例 RACDB2 正在节点 rac2 上运行
 
6,打开OEM

[root@rac1 ~]# /u01/app/11.2.0/grid/bin/emctl start  RACDB