今天在查找相关call stack的内容时，很偶然看到一个哥们写的关于在metalink中搜索相关oracle bug中描述相关call stack的方式，可以利用这种方式对相关dump文件的call stack同样在metalink中进行搜索。查看是否是相关bug引起的。具体方式和列子如下(请注意精彩的在最后)：
###########################
据库环境
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
ORACLE_HOME = /oracle/product/10.2.0
System name: HP-UX
Node name: SD7pp6
Release: B.11.31
Version: U
Machine: ia64
Instance name: gtsdb
一个数据库的UDUMP老是满，各个连接老是在进行DUMP
*** ACTION NAME:() 2009-11-19 13:20:50.978
*** MODULE NAME:(dbspicao10@SD7pp6(TNS V1-V3)) 2009-11-19 13:20:50.978
*** SERVICE NAME:(gtsdb) 2009-11-19 13:20:50.978
*** SESSION ID:(1469.12060) 2009-11-19 13:20:50.978
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
ksedst()+64 call ksedst1() C00000019A4AA348 ?
000000001 ?
$cold_kteinicnt1()+ call ksedst() C00000019A4AA348 ?
608 C000000000000EA5 ?
40000000032FEA00 ?
000000000 ? 000000000 ?
000000000 ? 000000000 ?
000000000 ?
ktsapsblk()+1408 call $cold_kteinicnt1() 9FFFFFFFFFFEE850 ?
000002000 ? 000000000 ?
9FFFFFFFFFFEE804 ?
9FFFFFFFFFFEE680 ?
000000002 ? 000000000 ?
C00000016451E014 ?
$cold_spefcifa()+30 call ktsapsblk() 9FFFFFFFFFFF19E0 ?
56 000000006 ? 000000000 ?
000000400 ? 000000000 ?
00000006B ? 000000000 ?
000000008 ?
spefmccallstd()+720 call $cold_spefcifa() 9FFFFFFFFFFF0DA0 ?
9FFFFFFF7F36E758 ?
9FFFFFFF7F36E740 ?
9FFFFFFFFFFEF2C8 ?
9FFFFFFF7F36E750 ?
C000000000000797 ?
60000000000D45D0 ?
400000000396B500 ?
pextproc()+128 call spefmccallstd() 9FFFFFFFFFFF1A40 ?
9FFFFFFFFFFF0CB0 ?
9FFFFFFFFFFF0DC0 ?
9FFFFFFFFFFEF2C8 ?
000000000 ?
peftrusted()+288 call pextproc() 9FFFFFFFFFFF1A40 ?
9FFFFFFFFFFF0CB0 ?
9FFFFFFFFFFF0DC0 ?
9FFFFFFFFFFF0D20 ?
psdexsp()+448 call peftrusted() 9FFFFFFFFFFF1A40 ?
60000000000D45D0 ?
C000000000000E22 ?
400000000396C030 ?
00001E33B ?
9FFFFFFFFFFF1AB0 ?
rpiswu2()+960 call psdexsp() 600000000004E8C0 ?
9FFFFFFFFFFEF2E0 ?
60000000000D45D0 ?
9FFFFFFFFFFEF860 ?
C000000000000EA3 ?
4000000002E3A920 ?
psdextp()+816 call rpiswu2() 9FFFFFFFFFFF0540 ?
40000000031E8110 ?
00001E22F ?
9FFFFFFFFFFEFFE0 ?
600000000004F818 ?
C000000000000FA7 ?
000000000 ?
9FFFFFFFFFFF0620 ?
pefccal()+1120 call psdextp() 600000000004F818 ?
9FFFFFFFFFFF0660 ?
9FFFFFFFFFFF0660 ?
9FFFFFFFFFFF0B90 ?
4000000001B72CC0 ?
60000000000E2700 ?
4000000001B6FD10 ?
9FFFFFFFFFFF0570 ?
pefcal()+432 call pefccal() 9FFFFFFFFFFF1A40 ?
9FFFFFFFFFFF0BB0 ?
60000000000D45D0 ?
9FFFFFFFFFFF1420 ?
C000000000000D22 ?
9FFFFFFF7F1C1850 ?
9FFFFFFFFFFF1ADC ?
9FFFFFFFFFFF0EF0 ?
pevm_FCAL()+288 call pefcal() 9FFFFFFFFFFF1A40 ?
9FFFFFFFFFFF1440 ?
60000000000D45D0 ?
9FFFFFFFFFFF19C0 ?
C000000000000715 ?
9FFFFFFFFFFF1440 ?
600000000004F818 ?
pfrinstr_FCAL()+176 call pevm_FCAL() 9FFFFFFF7F1C1848 ?
9FFFFFFFFFFF1A58 ?
60000000000D45D0 ?
pfrrun_no_tool()+19 call pfrinstr_FCAL() 9FFFFFFF7F1C1848 ?
2 C000000274E4BC4A ?
9FFFFFFF7F1C18B0 ?
pfrrun()+13376 call pfrrun_no_tool() 9FFFFFFF7F1C1848 ?
000002001 ?
9FFFFFFF7F1C18B0 ?
60000000000D45D0 ?
C00000000000099B ?
40000000030414E0 ?
9FFFFFFF7F1C1C98 ?
9FFFFFFF7F1C1910 ?
plsql_run()+1328 call pfrrun() 9FFFFFFFFFFF1B50 ?
9FFFFFFFFFFF1B40 ?
60000000000D45D0 ?
9FFFFFFFFFFF2740 ?
9FFFFFFFFFFF2740 ?
C000000000000DA2 ?
4000000002AC7390 ?
peidxr_run()+496 call plsql_run() 9FFFFFFF7F300200 ?
000000010 ?
9FFFFFFF7F320218 ?
9FFFFFFFFFFF2750 ?
60000000000D45D0 ?
9FFFFFFFFFFF3260 ?
peidxexe()+128 call peidxr_run() 9FFFFFFF7F36E028 ?
000000010 ?
9FFFFFFF7F320218 ?
9FFFFFFFFFFF3270 ?
60000000000D45D0 ?
9FFFFFFFFFFF37F0 ?
C000000000000491 ?
400000000318D060 ?
kkxdexe()+608 call peidxexe() 9FFFFFFF7F1C15F8 ?
60000000000D45D0 ?
C000000000000F26 ?
4000000003186480 ?
9FFFFFFF7F1C18C0 ?
9FFFFFFF7F1C1600 ?
9FFFFFFF7F1C1670 ?
9FFFFFFF7F1C1848 ?
kkxmpexe()+384 call kkxdexe() 9FFFFFFF7F1C15F8 ?
9FFFFFFF7F32CD10 ?
9FFFFFFF7F32CBA8 ?
60000000000D45D0 ?
C000000000000C1E ?
40000000039A1160 ?
kgmexec()+752 call kkxmpexe() 9FFFFFFFFFFF3DD0 ?
C00000000000132E ?
40000000039A14F0 ?
9FFFFFFF7F1C15F8 ?
00001854B ?
9FFFFFFF7F32CBA8 ?
9FFFFFFF7F32CD00 ?
9FFFFFFFFFFF3850 ?
evapls()+1264 call kgmexec() 600000000004E8C0 ?
000000001 ?
9FFFFFFFFFFF4420 ?
9FFFFFFFFFFF43F0 ?
C0000002525873A0 ?
600000000004F818 ?
9FFFFFFFFFFF3E90 ?
000000000 ?
evaopn2()+1056 call evapls() C000000256CC2BF8 ?
9FFFFFFFFFFF4420 ?
60000000000D45D0 ?
9FFFFFFFFFFF4A10 ?
C00000000000112A ?
4000000002E1C860 ?
00001858F ?
9FFFFFFFFFFF4430 ?
$cold_evamul()+160 call evaopn2() C000000256CC2BF8 ?
9FFFFFFFFFFF4A40 ?
60000000000D45D0 ?
9FFFFFFFFFFF5000 ?
C000000000000816 ?
4000000003753E20 ?
qesaAggNonDistSS()+ call $cold_evamul() C000000256CC2CB0 ?
896
qerhjInnerProbeHash call qesaAggNonDistSS() C00000021CEEAE00 ?
Table()+928 000007FFF ?
60000000000D45D0 ?
C000000000000A9A ?
4000000002DCFE80 ?
kdstf0000101km()+66 call qerhjInnerProbeHash 9FFFFFFFFFFF5BC0 ?
08 Table() 000007FFF ?
60000000000D45D0 ?
C000000000001838 ?
4000000002DCD740 ?
00001810B ?
9FFFFFFFFFFF5C38 ?
000000003 ?
kdsttgr()+65328 call kdstf0000101km() 9FFFFFFF7F32A9B0 ?
000000000 ?
4000000001B63850 ?
9FFFFFFFFFFF5BC0 ?
000007FFF ?
9FFFFFFFFFFF524F ?
C00000018018A044 ?
00000001B ?
qertbFetch()+3792 call kdsttgr() 9FFFFFFF7F32A9B0 ?
000000000 ?
C00000021CEEB4A0 ?
9FFFFFFF7F32A358 ?
C00000021CEEB510 ?
4000000001B63850 ?
9FFFFFFFFFFF5BC0 ?
000007FFF ?
rwsfcd()+256 call qertbFetch() 9FFFFFFFFFFF5BB0 ?
4000000002C149A0 ?
000018233 ?
9FFFFFFFFFFF5680 ?
qerhjFetch()+896 call rwsfcd() 9FFFFFFF7F32B060 ?
4000000001B63850 ?
9FFFFFFFFFFF5BC0 ?
000007FFF ?
60000000000D45D0 ?
qergsFetch()+864 call qerhjFetch() C00000021CEEB068 ?
4000000001B8B400 ?
C00000021CEEAE00 ?
000007FFF ?
60000000000D45D0 ?
C000000000000EA5 ?
4000000002F36050 ?
000018271 ?
opifch2()+9632 call qergsFetch() C00000021CEEAE00 ?
4000000001B931D0 ?
9FFFFFFFFFFF5E70 ?
000000002 ?
60000000000D45D0 ?
C000000000001F46 ?
4000000002E19320 ?
000018333 ?
opiefn0()+672 call opifch2() 9FFFFFFFFFFF6E30 ?
40000000030445A0 ?
000010217 ?
9FFFFFFFFFFF5CE0 ?
60000000000D45D0 ?
C000000000000F26 ?
600000000004E8C0 ?
040002D89 ?
kpoal8()+10256 call opiefn0() C0000000000015B3 ?
9FFFFFFF7F3B190A ?
9FFFFFFFFFFF6E88 ?
9FFFFFFFFFFF6F90 ?
9FFFFFFFFFFF6F70 ?
9FFFFFFFFFFF6F74 ?
000000005 ? 000000020 ?
opiodr()+2128 call kpoal8() 9FFFFFFFFFFF7650 ?
C000000000001530 ?
000000000 ?
9FFFFFFFFFFF6F70 ?
60000000000D45D0 ?
9FFFFFFF7F3C3758 ?
ttcpip()+1680 call opiodr() 00000005E ? 000000017 ?
4000000001AAB0F0 ?
0000046B0 ?
9FFFFFFFFFFF7660 ?
opitsk()+2336 call ttcpip() 600000000005A4C0 ?
000000001 ?
9FFFFFFFFFFF9D30 ?
000000001 ?
9FFFFFFFFFFF9EA0 ?
9FFFFFFFFFFF9C94 ?
4000000001B90E90 ?
000000000 ?
opiino()+1840 call opitsk() 000000000 ? 000000000 ?
60000000000D45D0 ?
400000000279B470 ?
0000180CD ?
4000000001AAB108 ?
opiodr()+2128 call opiino() 00000003C ?
9FFFFFFFFFFFC6F0 ?
9FFFFFFFFFFFEE90 ?
9FFFFFFFFFFFBBB0 ?
60000000000D45D0 ?
C000000000001530 ?
opidrv()+1088 call opiodr() 00000003C ? 000000004 ?
4000000001AAABA0 ?
0000046B0 ?
9FFFFFFFFFFFC700 ?
60000000000D45D0 ?
sou2o()+336 call opidrv() 00000003C ?
9FFFFFFFFFFFEE90 ?
60000000000E24E0 ?
opimai_real()+224 call sou2o() 9FFFFFFFFFFFEEB0 ?
00000003C ? 000000004 ?
9FFFFFFFFFFFEE90 ?
main()+368 call opimai_real() 000000000 ?
9FFFFFFFFFFFEEE0 ?
main_opd_entry()+80 call main() 000000002 ?
9FFFFFFFFFFFF390 ?
60000000000D45D0 ?
C000000000000004 ?
alert日志无任何报错信息。这个时候，就只有借助伟大的metalink
metalink有个习惯，就是很多BUG，都会把错误的函数链用这样的方式列出来ksedst <- $cold_kteinicnt1 <- 608 <- ktsapsblk <- $cold_spefcifa
由于这个DUMP有其调用的函数方法名Call Stack，于是就用这个到metalink去搜
搜索 ksedst $cold_kteinicnt1 ktsapsblk $cold_spefcifa
接可以查到BUG信息了 Trace files generated on by dbspicao module after upgrade to 10.2.0.4 [ID 741820.1]
Applies to:
Oracle Server - Enterprise Edition - Version: 10.2.0.4 to 10.2.0.4
This problem can occur on any platform.
Symptoms
After upgrade to 10.2.0.4, huge trace files are generated in udump directory without any error message in alert.log. These trace files can be generated periodically every 10-15 minutes, or another period.
They don't contain any error message, but have the following call stack:
ksedst <- $cold_kteinicnt1 <- 608 <- ktsapsblk <- $cold_spefcifa
<- spefmccallstd <- pextproc <- peftrusted <- psdexsp <- rpiswu2
<- psdextp <- pefccal <- pefcal <- pevm_FCAL <- pfrinstr_FCAL
<- pfrrun_no_tool <- pfrrun <- plsql_run <- peidxr_run <- peidxexe
<- kkxdexe <- kkxmpexe <- kgmexwi <- kgmexec <- evapls
<- evaopn2 <- $cold_evamul <- qesaAggNonDistSS <- 896 <- qerhjWalkHashBucket
<- qerhjInnerProbeHash <- Table <- kdstf0000101km <- kdsttgr <- qertbFetch
<- rwsfcd <- qerhjFetch <- qergsFetch <- opifch2 <- opiefn0
<- opiefn <- opiodr <- ttcpip <- opitsk <- opiino
<- opiodr <- opidrv <- sou2o <- opimai_real <- main
<- main_opd_entry
The module that generates them is dbspicao.
