ORA-17500 ODM err的问题排查

今天在一套环境中做系统检查的时候，发现alert日志中有一段ODM的错误。
日志内容大体如下，可以看到是在半夜4点多报的错误。
Clearing Resource Manager plan via parameter
Fri Aug 22 02:00:52 2014
ALTER SYSTEM ARCHIVE LOG
Fri Aug 22 02:00:52 2014
Thread 1 advanced to log sequence 6934 (LGWR switch)
  Current log# 3 seq# 6934 mem# 0: /u01/oracle/PETCUS1/oracnt01/redolog_A3/redo/redo03A.log
  Current log# 3 seq# 6934 mem# 1: /u01/oracle/PETCUS1/oracnt02/redolog_B3/redo/redo03B.log
Archived Log entry 6933 added for thread 1 sequence 6933 ID 0x4a0d6000 dest 1:
Fri Aug 22 04:27:37 2014
Control file backup creation failed.
Errors in file /u01/oracle/PETCUS1/oradmp/diag/rdbms/petcus1/PETCUS1/trace/PETCUS1_mmon_5584.trc:
ORA-17500: ODM err:ODM ERROR V-41-4-1-83-9 Bad file descriptor
Fri Aug 22 04:27:37 2014
Errors in file /u01/oracle/PETCUS1/oradmp/diag/rdbms/petcus1/PETCUS1/trace/PETCUS1_ora_10695.trc:
ORA-00245: control file backup operation failed
Fri Aug 22 05:03:01 2014
ALTER SYSTEM ARCHIVE LOG
Fri Aug 22 05:03:01 2014
Thread 1 advanced to log sequence 6935 (LGWR switch)
  Current log# 2 seq# 6935 mem# 0: /u01/oracle/PETCUS1/oracnt01/redolog_A2/redo/redo02A.log
  Current log# 2 seq# 6935 mem# 1: /u01/oracle/PETCUS1/oracnt02/redolog_B2/redo/redo02B.log
Archived Log entry 6934 added for thread 1 sequence 6934 ID 0x4a0d6000 dest 1:
Fri Aug 22 08:05:01 2014
ALTER SYSTEM ARCHIVE LOG
Fri Aug 22 08:05:01 2014
Thread 1 advanced to log sequence 6936 (LGWR switch)
  Current log# 4 seq# 6936 mem# 0: /u01/oracle/PETCUS1/oracnt01/redolog_A4/redo/redo04A.log
  Current log# 4 seq# 6936 mem# 1: /u01/oracle/PETCUS1/oracnt02/redolog_B4/redo/redo04B.log
Archived Log entry 6935 added for thread 1 sequence 6935 ID 0x4a0d6000 dest 1:
Fri Aug 22 11:02:05 2014
ALTER SYSTEM ARCHIVE LOG
Fri Aug 22 11:02:05 2014
Thread 1 advanced to log sequence 6937 (LGWR switch)
  Current log# 1 seq# 6937 mem# 0: /u01/oracle/PETCUS1/oracnt01/redolog_A1/redo/redo01A.log
  Current log# 1 seq# 6937 mem# 1: /u01/oracle/PETCUS1/oracnt02/redolog_B1/redo/redo01B.log
Archived Log entry 6936 added for thread 1 sequence 6936 ID 0x4a0d6000 dest 1:
Fri Aug 22 14:06:47 2014
ALTER SYSTEM ARCHIVE LOG
Fri Aug 22 14:06:47 2014
Thread 1 advanced to log sequence 6938 (LGWR switch)
  Current log# 3 seq# 6938 mem# 0: /u01/oracle/PETCUS1/oracnt01/redolog_A3/redo/redo03A.log
  Current log# 3 seq# 6938 mem# 1: /u01/oracle/PETCUS1/oracnt02/redolog_B3/redo/redo03B.log
Archived Log entry 6937 added for thread 1 sequence 6937 ID 0x4a0d6000 dest 1:
Fri Aug 22 17:02:40 2014
ALTER SYSTEM ARCHIVE LOG
Fri Aug 22 17:02:40 2014
Thread 1 advanced to log sequence 6939 (LGWR switch)
  Current log# 2 seq# 6939 mem# 0: /u01/oracle/PETCUS1/oracnt01/redolog_A2/redo/redo02A.log
  Current log# 2 seq# 6939 mem# 1: /u01/oracle/PETCUS1/oracnt02/redolog_B2/redo/redo02B.log
Archived Log entry 6938 added for thread 1 sequence 6938 ID 0x4a0d6000 dest 1:
Fri Aug 22 19:13:42 2014
Thread 1 advanced to log sequence 6940 (LGWR switch)
  Current log# 4 seq# 6940 mem# 0: /u01/oracle/PETCUS1/oracnt01/redolog_A4/redo/redo04A.log
  Current log# 4 seq# 6940 mem# 1: /u01/oracle/PETCUS1/oracnt02/redolog_B4/redo/redo04B.log
Fri Aug 22 19:13:50 2014
Archived Log entry 6939 added for thread 1 sequence 6939 ID 0x4a0d6000 dest 1:
Fri Aug 22 19:18:16 2014
Control file backup creation failed.
Errors in file /u01/oracle/PETCUS1/oradmp/diag/rdbms/petcus1/PETCUS1/trace/PETCUS1_ckpt_5573.trc:
ORA-17500: ODM err:ODM ERROR V-41-4-1-83-9 Bad file descriptor
Fri Aug 22 19:18:25 2014
Errors in file /u01/oracle/PETCUS1/oradmp/diag/rdbms/petcus1/PETCUS1/trace/PETCUS1_ora_12261.trc:
ORA-00245: control file backup operation failed

关于ODM，在以前的一个章节中详细讨论过。http://blog.itpub.net/23718752/viewspace-1252507/
启用了ODM对于系统的io提升是很明显的。
如果启用了odm，在数据库启动的时候会有如下的提示信息： 

diagnostic_dest          = "/u01/oracle/PETCUS1/oradmp"
Oracle instance running with ODM: Veritas 6.0.100.000 ODM Library, Version 2.0 
Sat Aug 16 17:25:59 2014
PMON started with pid=2, OS id=5545 
Sat Aug 16 17:25:59 2014
PSP0 started with pid=3, OS id=5547 

看这个错误，已经过去了挺长时间了。直接看trace也找不到思路，就从metalink中找一些思路。
其中有一篇文章提到的问题很类似
After Upgrade To 11.2.0.2 We Recieve Ora-00245 During Autobackup Of The Controlfile. (Doc ID 1308378.1)

我们这套库刚好就是升级到11.2.0.2的，这个条件满足，但是根据文章的提示说是使用rman来做的备份。
这一点通过ash很可能找不到任何线索。看报错的前后提示的日志吧。
 less /u01/oracle/PETCUS1/oradmp/diag/rdbms/petcus1/PETCUS1/trace/PETCUS1_ora_10695.trc
Trace file /dbccbsPT1/oracle/PETCUS1/oradmp/diag/rdbms/petcus1/PETCUS1/trace/PETCUS1_ora_10695.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORACLE_HOME = /opt/app/oracle/dbccbspt1/product/11.2.0
System name:    Linux
Node name:      ccbdbpt3
Release:        2.6.18-308.el5
Version:        #1 SMP Fri Jan 27 17:17:51 EST 2012
Machine:        x86_64
Instance name: PETCUS1
Redo thread mounted by this instance: 1
Oracle process number: 54
Unix process pid: 10695, image: oracle@ccbdbpt3 (TNS V1-V3)

*** 2014-08-22 02:00:53.024
*** SESSION ID:(2655.1985) 2014-08-22 02:00:53.024
*** CLIENT ID:() 2014-08-22 02:00:53.024
*** SERVICE NAME:(SYS$USERS) 2014-08-22 02:00:53.024
*** MODULE NAME:(rman@ccbdbpt3 (TNS V1-V3)) 2014-08-22 02:00:53.024
*** ACTION NAME:(0000001 FINISHED70) 2014-08-22 02:00:53.024
 
Initial buffer sizes: read 1024K, overflow 832K, change 805K
ORA-00245: control file backup operation failed


有了这些信息，就能肯定这个问题是由于一个bug导致的。看来这个问题也不算优先级很高的。可以通过workaround来完成。要不就打patch

SOLUTION

The solution is to apply the fix for Bug 10317487 which will be included in PatchSet 11.2.0.3.
One-off patches for 11.2.0.2 might be available as Patch 10317487

A workaround is to make the controlfile backup via SQL*Plus instead of RMAN

__OR__

Configure the Controlfile backup to a filesystem which is not using ODM.