KJC: Wait for msg sends to complete是一种只会发生在RAC环境中的等待事件，目前对于该等待事件已知的信息不多。 KJ意为Kernel Lock Management Layer，Messaging Enhancements (KJC)。 猜测该等待事件为global enqueue message的网络传输等待,前台进程和后台进程如LMS、LMD均可能发生该等待事件。
select name,parameter1,parameter2,parameter3,wait_class from v$event_name where name like '%KJC%';
NAME PARAMETER1
---------------------------------------------------------------- ----------------------------------------------------------------
PARAMETER2 PARAMETER3
---------------------------------------------------------------- ----------------------------------------------------------------
WAIT_CLASS
----------------------------------------------------------------
KJC: Wait for msg sends to complete msg
dest|rcvr mtype
Other
SQL> select * from gV$system_Event where event like '%KJC%';
INST_ID EVENT TOTAL_WAITS
---------- ---------------------------------------------------------------- -----------
TOTAL_TIMEOUTS TIME_WAITED AVERAGE_WAIT TIME_WAITED_MICRO TOTAL_WAITS_FG TOTAL_TIMEOUTS_FG
-------------- ----------- ------------ ----------------- -------------- -----------------
TIME_WAITED_FG AVERAGE_WAIT_FG TIME_WAITED_MICRO_FG EVENT_ID WAIT_CLASS_ID WAIT_CLASS#
-------------- --------------- -------------------- ---------- ------------- -----------
WAIT_CLASS
----------------------------------------------------------------
2 KJC: Wait for msg sends to complete 415
0 1 0 7638 163 0
0 0 2494 4079369678 1893977003 0
Other
1 KJC: Wait for msg sends to complete 93
0 0 0 2094 37 0
0 0 615 4079369678 1893977003 0
Other
注意到该等待事件的类别是other而非concurrency,若发生该等待事件我们应当关心的是网络性能
netstat -s or ifconfig?
以下为具体的等待发生场景:
PARSING IN CURSOR #140438030304224 len=169 dep=2 uid=0 oct=3 lid=0
tim=1310310630713104 hv=2411583655 ad='b5a49898' sqlid='637txtu7vvq57'
select service_name, service_name_hash,wait_class_id, wait_class#, wait_class,total_waits, time_waited from
gv$service_wait_class where inst_id = USERENV('Instance')
END OF STMT
PARSE #140438030304224:c=6999,e=7685,p=0,cr=5,cu=0,mis=1,r=0,dep=2,og=4,plh=809657567,tim=1310310630713101
CLOSE #140438030304224:c=0,e=20,dep=2,type=0,tim=1310310630713388
WAIT #140438029567384: nam='row cache lock' ela= 530 cache id=8 mode=0 request=3 obj#=6623 tim=1310310630716555
EXEC #140438032451672:c=0,e=61,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=4,plh=2853959010,tim=1310310630716708
FETCH #140438032451672:c=0,e=49,p=0,cr=4,cu=0,mis=0,r=1,dep=2,og=4,plh=2853959010,tim=1310310630716809
CLOSE #140438032451672:c=0,e=10,dep=2,type=3,tim=1310310630716877
WAIT #140438029567384: nam='KJC: Wait for msg sends to complete' ela= 18
msg=3235372280 dest|rcvr=65536 mtype=12 obj#=6623 tim=1310310630719146
EXEC #140438032420328:c=0,e=190,p=0,cr=1,cu=5,mis=0,r=1,dep=1,og=4,plh=0,tim=1310310630781989
WAIT #140438032420328: nam='KJC: Wait for msg sends to complete' ela= 55
msg=3235374760 dest|rcvr=65537 mtype=32 obj#=74418 tim=1310310630782830
这里的msg猜测可能是递增的message id
mytpe为该msg的类型
obj#为对象号
KJC: Wait for msg sends to complete可能的相关stack call:
LMS's stack:
ksedms ksdxfdmp ksdxcb sspuser sigacthandler libc_poll
kslwait ksxprcv kjctr_rksxp kjctsrw kjccwcrm kjccqmg
kjbmsclose kjbclose kclpdc kclpdcl kclpbi kjmsm
ksbrdp opirip opidrv sou2o main start
LMS4 was waiting for "KJC: Wait for msg sends to complete"
'KJC: Wait for msg sends to complete', at one of the stack trace
below
o kjccqmg => kjctcwcrm
o kjccsmg => kjctcwcrm
o an Oracle process is spinning in kjctcwcrm() on 9.0, 9.2 or 10.1
o an Oracle process got ORA-600[kjccgmb:1] on 9.0, 9.2 or 10.1
o Got ORA-7445[] at an unknown function called from kjbmmsg2res()
o A LMS process hung in kjctcwcrm() on shutdown
LMS may hang waiting for 'kjc: wait for msg sends to complete'
during shutdown of an instance with another session
reporting ORA-600 [kjccgmb:1]
IAGNOSTIC ANALYSIS:
--------------------
Let the customer tracing the lmon, lmd when doing the shutdown immediate,
find out:
In sur11_lmon_5948.trc
WAIT #0: nam='ges LMD to shutdown' ela= 496057 p1=0 p2=0 p3=0
WAIT #0: nam='KJC: Wait for msg sends to complete' ela= 49213 p1=0 p2=0
p3=0
WAIT #0: nam='KJC: Wait for msg sends to complete' ela= 49811 p1=0 p2=0
p3=0
WAIT #0: nam='KJC: Wait for msg sends to complete' ela= 49772 p1=0 p2=0
p3=0
WAIT #0: nam='KJC: Wait for msg sends to complete' ela= 49813 p1=0 p2=0
p3=0
WAIT #0: nam='KJC: Wait for msg sends to complete' ela= 49779 p1=0 p2=0
p3=0
WAIT #0: nam='KJC: Wait for msg sends to complete' ela= 49787 p1=0 p2=0
p3=0
WAIT #0: nam='KJC: Wait for msg sends to complete' ela= 49779 p1=0 p2=0
LMS4's stack:
ksedms ksdxfdmp ksdxcb sspuser sigacthandler libc_poll
kslwait ksxprcv kjctr_rksxp kjctsrw kjccwcrm kjccqmg
kjbmsclose kjbclose kclpdc kclpdcl kclpbi kjmsm
ksbrdp opirip opidrv sou2o main start
LMS4 was encountered ORA-4031 during shutting down,
#0 0x0000000004f5bdbe in kjccqmg ()
#1 0x0000000005172378 in kjbmflushsc ()
#2 0x0000000005175351 in kjbmfto ()
#3 0x0000000004ff425f in kjmsm ()
#4 0x00000000022677a7 in ksbrdp ()
#5 0x000000000248c8a3 in opirip ()
#6 0x000000000171eefd in opidrv ()
#7 0x0000000001d95f5f in sou2o ()
#8 0x0000000000a079ba in opimai_real ()
#9 0x0000000001d9b408 in ssthrdmain ()
#10 0x0000000000a078a1 in main ()
(gdb) bt
#0 0x00000000053907aa in kclpbi ()
#1 0x0000000004ff46dc in kjmsm ()
#2 0x00000000022677a7 in ksbrdp ()
#3 0x000000000248c8a3 in opirip ()
#4 0x000000000171eefd in opidrv ()
#5 0x0000000001d95f5f in sou2o ()
#6 0x0000000000a079ba in opimai_real ()
#7 0x0000000001d9b408 in ssthrdmain ()
#8 0x0000000000a078a1 in main ()
(gdb) bt
#0 0x0000000004f5bdbe in kjccqmg ()
#1 0x00000000051a232a in kjbmswrite ()
#2 0x00000000051014f1 in kjbrswrite ()
#3 0x00000000050fec8d in kjbrwreq ()
#4 0x0000000005189fcc in kjbmpwrite ()
#5 0x0000000004fe98fe in kjmxmpm ()
#6 0x0000000004feceee in kjmpmsgi ()
#7 0x0000000004ff2d8e in kjmsm ()
#8 0x00000000022677a7 in ksbrdp ()
#9 0x000000000248c8a3 in opirip ()
#10 0x000000000171eefd in opidrv ()
#11 0x0000000001d95f5f in sou2o ()
#12 0x0000000000a079ba in opimai_real ()
#13 0x0000000001d9b408 in ssthrdmain ()
#14 0x0000000000a078a1 in main ()
以下列出了与该'KJC: Wait for msg sends to complete'相关的Bug信息
| Bug | Affected Version | Description |
| Bug 5075434 Small performance overhead in RAC (waits for "KJC: Wait for msg sends to complete") | 10.2.0.2 | RAC session can suffer a small performance overhead due to waiting for a timeout on "KJC: Wait for msg sends to complete". |
| Bug 11687519: PZ99 PROCESS SPIN WITH "KJC: WAIT FOR MSG SENDS TO COMPLETE" | 10.2.0.3 | Abstract: PZ99 PROCESS SPIN WITH "KJC: WAIT FOR MSG SENDS TO COMPLETE" |
| Bug 4672266: LMS WAITING FOR 'KJC: WAIT FOR MSG SENDS TO COMPLETE' WHILE SHUTDOWN | 9.2.0.4 | Abstract: LMS WAITING FOR 'KJC: WAIT FOR MSG SENDS TO COMPLETE' WHILE SHUTDOWN LMS2 has been waiting for the event 'KJC: Wait for msg sends to complete' while SHUTDOWN IMMEDIATE. My customer decided to issue SHUTDOWN ABORT to stop the instance in one hour. |
| Bug 4672266 LMS hang waiting for 'kjc: wait for msg sends to complete' during shutdown | 9.2.0.4 9.2.0.7 10.1.0.5 10.2.0.2 | LMS may hang waiting for 'kjc: wait for msg sends to complete' during shutdown of an instance with another session reporting ORA-600 [kjccgmb:1] |
| Bug 2447621 - Unneccessary waits for "KJC: WAIT FOR MSG SEND COMPLETION" | <9.2 | Unneccessary waits for "KJC: Wait for message completion" can occur in a RAC environment. |
| Bug 5347566: SHUTDOWN IMMEDIATE VERY SLOW WITH 'KJC: WAIT FOR MSG SENDS TO COMPLETE' | 9.2.0.7 | Exactly similar as Bug 5289823 and Bug 4672266 LMS hang waiting for 'kjc: wait for msg sends to complete' during shutdown |
| Bug 5084699: RACGIMON WAITING FOR 'KJC: WAIT FOR MSG SENDS TO COMPLETE' | 10.2.0.1 | racgimon on one of the 4 nodes cluster has been exhibiting process spin. It appears to wait for 15 minutes every 20 minutes on "KJC: Wait for msg sends to complete" in the following PL/SQL execution:'begin dbms_lock.allocate_unique(:a1, :a2); end;' |
| Bug 3561040: SHUTDOWN HANG DUE TO LMS PROCESS WAITING FOR KJC: WAIT FOR MSG SENDS TO COMPLETE | 9.2.0.4 | LMS process is waiting fo r |
本文转自maclean_007 51CTO博客，原文链接:http://blog.51cto.com/maclean/1277844
