前言
当媒体服务器发行故障后，先后通过分析SIP信令及媒体服务器抓包及服务器日志，定位媒体服务器网元出现故障。
通过abrt抓取进程异常的core文件，使用sipp模拟大呼进行故障场景复现。采用valgrind进行内存泄漏定位。最终定位问题并修复该问题。
提示：以下是本篇文章正文内容
一、abrt获取core文件
上上周，负责江苏现场交付的同事联系我，客户有反馈通话过程中存在通话中断的现象。
我先让现场的抓取了客户的通话信令，发现通话中断时，收到了核心网交换机发送了拆线的指令。原因值是RAP/RTCP TIME OUT。
然后我让现场的同事采用gdb命令在媒体服务器进行网卡抓包，并进行实际拨测，当通话中断时，停止网络抓包。
通过网络包分析，发现媒体服务器存在媒体包丢包情况。
于是判断媒体服务器存在问题了，通过分析进城的运行日志，发现存在重启现象。当中午发生时会导致媒体包丢包，导致通话中断。
至此问题陡然升级，先联系现场配置abrt服务（service arbtd restart），当进程重启时，抓取到coredump。
经和客户沟通后，先下掉媒体服务器，通话旁路。
二、gdb分析coredump文件
gdb ms coredump
然后输入where命令，frame及相关的print等命令，逐一查看相关变量。
三、sipp模拟呼叫
于是编写sipp的uac和uas的模拟桩，uac模拟主叫，uas模拟被叫。同时uac和uas的模拟桩中加入rtp流，让场景变得更真实。
uac脚本如下：
<?xml version="1.0" encoding="ISO-8859-1" ?> <!-- This program is free software; you can redistribute it and/or --> <!-- modify it under the terms of the GNU General Public License as --> <!-- published by the Free Software Foundation; either version 2 of the --> <!-- License, or (at your option) any later version. --> <!-- --> <!-- This program is distributed in the hope that it will be useful, --> <!-- but WITHOUT ANY WARRANTY; without even the implied warranty of --> <!-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the --> <!-- GNU General Public License for more details. --> <!-- --> <!-- You should have received a copy of the GNU General Public License --> <!-- along with this program; if not, write to the --> <!-- Free Software Foundation, Inc., --> <!-- 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA --> <!-- --> <!-- Sipp default 'uac' scenario. --> <!-- --> <scenario name="Basic Sipstone UAC"> <!-- In client mode (sipp placing calls), the Call-ID MUST be --> <!-- generated by sipp. To do so, use [call_id] token. --> <send> <![CDATA[ INVITE sip:[field1]@[remote_ip]:[remote_port] SIP/2.0 Via: SIP/2.0/[transport] [local_ip]:[local_port] From: sipp <sip:[field0]@[local_ip]:[local_port]>;tag=[call_number] To: sut <sip:[field1]@[remote_ip]:[remote_port]> Call-ID: [call_id] Cseq: [cseq] INVITE Contact: sip:sipp@[local_ip]:[local_port] Route: <sip:tagscpxf03.js.ctcims.cn:[remote_port];call=ldmp;lr> Route: <sip:[local_ip]:8063;lr> Max-Forwards: 70 P-Asserted-Identity: <sip:[field0]@cintel.com.cn> Subject: Performance Test Content-Type: application/sdp Content-Length: [len] v=0 o=- 53655765 2353687637 IN IP4 [media_ip] s=SIPP c=IN IP4 [media_ip] t=0 0 m=audio [media_port] RTP/AVP 115 103 102 101 8 96 97 18 0 b=AS:80 b=RS:600 b=RR:2000 a=rtpmap:115 AMR-WB/16000/1 a=fmtp:115 mode-change-capability=2;max-red=0 a=rtpmap:103 AMR-WB/16000/1 a=fmtp:103 octet-align=1;mode-change-capability=2;max-red=0 a=rtpmap:102 AMR/8000/1 a=fmtp:102 mode-change-capability=2;max-red=0 a=rtpmap:101 AMR/8000/1 a=fmtp:101 octet-align=1;mode-change-capability=2;max-red=0 a=rtpmap:8 PCMA/8000 a=rtpmap:96 telephone-event/16000 a=fmtp:96 0-15 a=rtpmap:97 telephone-event/8000 a=fmtp:97 0-15 a=curr:qos local none a=curr:qos remote none a=des:qos mandatory local sendrecv a=des:qos optional remote sendrecv a=sendrecv a=maxptime:240 a=ptime:20 a=rtpmap:18 G729/8000 a=fmtp:18 annexb=no a=rtpmap:0 PCMU/8000 ]]> </send> <recv response="100" > </recv> <recv response="183" > </recv> <send> <![CDATA[ PRACK sip:[field1]@[remote_ip]:[remote_port] SIP/2.0 Via: SIP/2.0/[transport] [local_ip]:[local_port] From: <sip:[field0]@[local_ip]:[local_port]>;tag=[call_number] To: <sip:[field1]@192.168.2.228:8700>[peer_tag_param] Call-ID: [call_id] CSeq: 2 PRACK RAck: 1 1 INVITE Contact: <sip:[local_ip]@[local_ip]:[local_port]> Max-Forwards: 70 Content-Length: 0 ]]> </send> <recv response="200" > </recv> <nop> <action> <!--exec play_pcap_audio="pcap/dtmf_2833_0.pcap"/--> <exec play_pcap_audio="pcap/amrwb1_115.pcap"/> </action> </nop> <send> <![CDATA[ UPDATE sip:[remote_ip]:[remote_port] SIP/2.0 Via: SIP/2.0/[transport] [local_ip]:[local_port] [last_Call-ID:] From: sipp <sip:[field0]@[local_ip]:[local_port]>;tag=[call_number] To: sut <sip:[field1]@[remote_ip]:[remote_port]>[peer_tag_param] Allow: INVITE,ACK,CANCEL,BYE,UPDATE,PRACK,MESSAGE,REFER,NOTIFY,INFO,OPTIONS Cseq: 3 UPDATE Contact: <sip:[local_ip]@[local_ip]:[local_port]> Require: precondition Supported: timer Content-Length: [len] Content-Type: application/sdp v=0 o=HuaweiATS9900 171392092 171392093 IN IP4 5.2.169.97 s=SBC call c=IN IP4 [media_ip] b=AS:49 b=RS:600 b=RR:2000 t=0 0 m=audio [media_port] RTP/AVP 115 102 8 96 18 0 97 b=AS:80 b=RS:600 b=RR:2000 a=rtpmap:115 AMR-WB/16000/1 a=fmtp:115 mode-change-capability=2;max-red=0 a=rtpmap:102 AMR/8000 a=fmtp:102 mode-change-capability=2;max-red=0 a=rtpmap:8 PCMA/8000 a=rtpmap:96 telephone-event/16000 a=fmtp:96 0-15 a=curr:qos local sendrecv a=curr:qos remote none a=des:qos mandatory local sendrecv a=des:qos mandatory remote sendrecv a=sendrecv a=maxptime:240 a=ptime:20 a=rtpmap:18 G729/8000 a=fmtp:18 annexb=no a=rtpmap:0 PCMU/8000 a=rtpmap:97 telephone-event/8000 a=fmtp:97 0-15 ]]> </send> <recv response="200"> </recv> <nop> <action> <!--exec play_pcap_audio="pcap/dtmf_2833_0.pcap"/--> <exec play_pcap_audio="pcap/g711a.pcap"/> </action> </nop> <recv response="180"> </recv> <recv response="200"> </recv> <send> <![CDATA[ ACK sip:[remote_ip]:[remote_port];transport=udp;Hpt=8f28_16;CxtId=3;TRC=ffffffff-ffffffff SIP/2.0 Via: SIP/2.0/[transport] [local_ip]:[local_port] From: sipp <sip:[field0]@[local_ip]:[local_port]>;tag=[call_number] To: sut <sip:[field1]@[remote_ip]:[remote_port]>[peer_tag_param] Call-ID: [call_id] Cseq: [cseq] ACK Contact: sip:sipp@[local_ip]:[local_port] Max-Forwards: 70 Subject: Performance Test Content-Length: 0 ]]> </send> <nop> <action> <!--exec play_pcap_audio="pcap/dtmf_2833_0.pcap"/--> <exec play_pcap_audio="pcap/amr102.pcap"/> </action> </nop> <!-- This delay can be customized by the -d command-line option --> <!-- or by adding a 'milliseconds = "value"' option here. --> <pause milliseconds = "500"/> <send> <![CDATA[ INVITE sip:[field1]@[remote_ip]:[remote_port] SIP/2.0 Via: SIP/2.0/[transport] [local_ip]:[local_port] From: sipp <sip:[field0]@[local_ip]:[local_port]>;tag=[call_number] To: sut <sip:[field1]@[remote_ip]:[remote_port]>[peer_tag_param] Call-ID: [call_id] Cseq: 4 INVITE Contact: sip:sipp@[local_ip]:[local_port] Max-Forwards: 66 P-Early-Media: supported Subject: Performance Test Session-Expires: 1800;refresher=uas Min-SE: 600 Accept-Contact: *;+g.3gpp.icsi-ref="urn%3Aurn-7%3A3gpp-service.ims.icsi.mmtel";video Content-Type: application/sdp Content-Length: [len] v=0 o=HuaweiATS9900 171392092 171392094 IN IP4 5.2.169.97 s=SBC call c=IN IP4 [media_ip] b=AS:2211 b=RS:8600 b=RR:8000 t=0 0 m=audio [media_port] RTP/AVP 115 102 8 96 18 0 97 b=AS:80 b=RS:600 b=RR:2000 a=rtpmap:115 AMR-WB/16000/1 a=fmtp:104 mode-change-capability=2;max-red=0 a=rtpmap:102 AMR/8000 a=fmtp:102 mode-change-capability=2;max-red=0 a=rtpmap:8 PCMA/8000 a=rtpmap:96 telephone-event/16000 a=fmtp:96 0-15 a=sendrecv a=maxptime:240 a=ptime:20 a=rtpmap:18 G729/8000 a=fmtp:18 annexb=no a=rtpmap:0 PCMU/8000 a=rtpmap:97 telephone-event/8000 a=fmtp:97 0-15 m=video 52998 RTP/AVP 114 113 b=AS:2162 b=RS:8000 b=RR:6000 a=rtpmap:114 H264/90000 a=fmtp:114 profile-level-id=42C01F;packetization-mode=1;sar-understood=16;sar-supported=1;sprop-parameter-sets=Z0LAH9oC0ChoBtChNQ==,aM4G4g== a=imageattr:114 send a=rtpmap:113 H264/90000 a=fmtp:113 profile-level-id=42C01F;packetization-mode=0;sar-understood=16;sar-supported=1;sprop-parameter-sets=Z0LAH9oC0ChoBtChNQ==,aM4G4g== a=imageattr:113 send a=rtcp-fb:* nack a=rtcp-fb:* nack pli a=rtcp-fb:* ccm fir a=rtcp-fb:* ccm tmmbr a=sendrecv a=tcap:1 RTP/AVPF a=pcfg:1 t=1 a=extmap:2 urn:3gpp:video-orientation ]]> </send> <recv response="100" ></recv> <recv response="200" ></recv> <send> <![CDATA[ ACK sip:[remote_ip]:[remote_port];transport=udp;Hpt=8f28_16;CxtId=3;TRC=ffffffff-ffffffff SIP/2.0 Via: SIP/2.0/[transport] [local_ip]:[local_port] From: sipp <sip:[field0]@[local_ip]:[local_port]>;tag=[call_number] To: sut <sip:[field1]@[remote_ip]:[remote_port]>[peer_tag_param] Call-ID: [call_id] Cseq: 4 ACK Contact: sip:sipp@[local_ip]:[local_port] Max-Forwards: 70 Subject: Performance Test Content-Length: 0 ]]> </send> <nop> <action> <!--exec play_pcap_audio="pcap/dtmf_2833_0.pcap"/--> <exec play_pcap_audio="pcap/amrwb1_115.pcap"/> </action> </nop> <pause milliseconds = "30000"/> <!-- The 'crlf' option inserts a blank line in the statistics report. --> <send> <![CDATA[ BYE sip:[field1]@[remote_ip]:[remote_port] SIP/2.0 Via: SIP/2.0/[transport] [local_ip]:[local_port] From: sipp <sip:[field0]@[local_ip]:[local_port]>;tag=[call_number] To: sut <sip:[field1]@[remote_ip]:[remote_port]>[peer_tag_param] Call-ID: [call_id] Cseq: 1 BYE Contact: sip:sipp@[local_ip]:[local_port] Max-Forwards: 70 Subject: Performance Test Content-Length: 0 ]]> </send> <recv response="200" crlf="true"> </recv> <!-- definition of the response time repartition table (unit is ms) --> <ResponseTimeRepartition value="10, 20, 30, 40, 50, 100, 150, 200"/> <!-- definition of the call length repartition table (unit is ms) --> <CallLengthRepartition value="10, 50, 100, 500, 1000, 5000, 10000"/> </scenario>
uac的启动脚本如下：
./sipp -sf uac_jsfhgk.xml -inf calling_number_graylist_speech_template_performance.txt -i 192.168.2.179 -p 5788 192.168.2.112:5062 -m 10000 -r 10 -trace_screen -screen_overwrite true -screen_file uac_jsfhgk.log
uas脚本如下：
<?xml version="1.0" encoding="ISO-8859-1" ?> <!-- This program is free software; you can redistribute it and/or --> <!-- modify it under the terms of the GNU General Public License as --> <!-- published by the Free Software Foundation; either version 2 of the --> <!-- License, or (at your option) any later version. --> <!-- --> <!-- This program is distributed in the hope that it will be useful, --> <!-- but WITHOUT ANY WARRANTY; without even the implied warranty of --> <!-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the --> <!-- GNU General Public License for more details. --> <!-- --> <!-- You should have received a copy of the GNU General Public License --> <!-- along with this program; if not, write to the --> <!-- Free Software Foundation, Inc., --> <!-- 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA --> <!-- --> <!-- Sipp default 'uas' scenario. --> <!-- --> <scenario name="Basic UAS responder"> <!-- By adding rrs="true" (Record Route Sets), the route sets --> <!-- are saved and used for following messages sent. Useful to test --> <!-- against stateful SIP proxies/B2BUAs. --> <recv request="INVITE" crlf="true"> <action> <ereg regexp=".*" search_in="hdr" header="CSeq:" assign_to="cseq_invite" /> <ereg regexp=".*" search_in="hdr" header="Via:" assign_to="via_invite" /> <!--ereg regexp="<.*>" search_in="hdr" header="Contact:" assign_to="uac_addr" /--> <!--ereg regexp=".*" search_in="hdr" header="From:" assign_to="from_invite" /--> </action> </recv> <!-- The '[last_*]' keyword is replaced automatically by the --> <!-- specified header if it was present in the last message received --> <!-- (except if it was a retransmission). If the header was not --> <!-- present or if no message has been received, the '[last_*]' --> <!-- keyword is discarded, and all bytes until the end of the line --> <!-- are also discarded. --> <!-- --> <!-- If the specified header was present several times in the --> <!-- message, all occurences are concatenated (CRLF seperated) --> <!-- to be used in place of the '[last_*]' keyword. --> <send> <![CDATA[ SIP/2.0 100 Trying [last_Via:] [last_From:] To: <sip:[field1]@[remote_ip]:[remote_port]>;tag=[call_number] [last_Call-ID:] [last_CSeq:] Contact: <sip:[local_ip]:[local_port];transport=[transport]> Content-Length: 0 ]]> </send> <send> <![CDATA[ SIP/2.0 183 Session Progress [last_Via:] [last_From:] To: <sip:[field1]@[remote_ip]:[remote_port]>;tag=[call_number] [last_Call-ID:] [last_CSeq:] Require: 100rel,precondition RSeq: 1 P-Charging-Vector: icid-value="pbac22bhw.19e.627a.20211025080346";orig-ioi=js.ims.mnc011.mcc460.3gppnetwork.org;term-ioi=js.chinamobile.com Contact: <sip:[local_ip]:[local_port];transport=[transport]> Content-Length: [len] Content-Type: application/sdp v=0 o=- 1635149026 1635149026 IN IP4 5.92.12.134 s=SBC call c=IN IP4 [media_ip] b=AS:41 b=RR:1837 b=RS:600 t=0 0 m=audio [media_port] RTP/AVP 115 96 b=AS:41 b=RR:1837 b=RS:600 a=rtpmap:115 AMR-WB/16000/1 a=fmtp:115 mode-change-capability=2;max-red=0 a=ptime:20 a=maxptime:240 a=sendrecv a=curr:qos local none a=curr:qos remote none a=des:qos mandatory local sendrecv a=des:qos mandatory remote sendrecv a=conf:qos remote sendrecv a=rtpmap:96 telephone-event/16000 a=fmtp:96 0-15 ]]> </send> <recv request="PRACK"></recv> <send> <![CDATA[ SIP/2.0 200 OK [last_Via:] [last_From:] To: <sip:01082012002@[remote_ip]:[remote_port]>;tag=[call_number] [last_Call-ID:] CSeq: 2 PRACK Contact: <sip:[local_ip]:[local_port];transport=[transport]> Content-Length: [len] ]]> </send> <nop> <action> <!--exec play_pcap_audio="pcap/dtmf_2833_0.pcap"/--> <exec play_pcap_audio="pcap/amrwb1_115.pcap"/> </action> </nop> <recv request="UPDATE"></recv> <send> <![CDATA[ SIP/2.0 200 OK [last_Via:] [last_Call-ID:] [last_From:] [last_To:] [last_CSeq:] Allow: INVITE,ACK,OPTIONS,BYE,CANCEL,REGISTER,INFO,UPDATE,PRACK,REFER,SUBSCRIBE,NOTIFY,MESSAGE Contact: <sip:[field1]@[local_ip]:[local_port];transport=udp;Hpt=8ec2_16;CxtId=3;TRC=ffffffff-ffffffff;srti=d0_1025>;video;+g.3gpp.icsi-ref="urn%3Aurn-7%3A3gpp-service.ims.icsi.mmtel";+g.3gpp.mid-call;+g.3gpp.srvcc-alerting Server: Ericsson MTAS - CXP2010134/1 R20G11 Supported: timer,100rel,histinfo,join,norefersub,replaces Session-Expires: 1800;refresher=uas P-Access-Network-Info: 3GPP-E-UTRAN;utran-cell-id-3gpp="460115E13675381B";access-domain=bac01.+86591.fj.ctcims.cn;network-provided P-Charging-Vector: icid-value=sg6.psbc12ber.fz.fj.node.epc.-1608-718164-185570-946;orig-ioi=ericsson.com;term-ioi=fj.ctcims.cn;icid-generated-at=sg6.psbc12ber.fz.fj.node.epc.mnc011.mcc460.3gppnetwork.org Supported: timer P-Access-Network-Info: 3GPP-E-UTRAN;utran-cell-id-3gpp=4600069167E8F712;sbc-domain=sbc7.0791.100.jx.chinamobile.com;ue-ip=[local_ip];ue-port=[local_port];network-provided P-Charging-Vector: icid-value="cdpsbc21bhw.19c.4185.20180803081658";orig-ioi=cdscscf6bhw.sc.chinamobile.com;term-ioi=jx.chinamobile.com P-Early-Media: gated Content-Length: [len] Content-Type: application/sdp P-Charging-Vector: icid-value=sg9.psbc21ber.nc.jx.node.epc.mnc0-1589-273998-798645;orig-ioi=ericsson.com;term-ioi=js.ctcims.cn P-Early-Media: sendrecv P-Access-Network-Info: IEEE-802.11;"location-info=5.253.86.82" v=0 o=- 1635149026 1635149027 IN IP4 5.92.12.134 s=SBC call c=IN IP4 [media_ip] b=AS:41 b=RR:1837 b=RS:600 t=0 0 m=audio [media_port] RTP/AVP 8 96 b=AS:41 b=RR:1837 b=RS:600 a=rtpmap:8 PCMA/8000 a=ptime:20 a=maxptime:240 a=sendrecv a=curr:qos local sendrecv a=curr:qos remote sendrecv a=des:qos mandatory local sendrecv a=des:qos mandatory remote sendrecv a=rtpmap:96 telephone-event/16000 a=fmtp:96 0-15 ]]> </send> <nop> <action> <!--exec play_pcap_audio="pcap/dtmf_2833_0.pcap"/--> <exec play_pcap_audio="pcap/g711a.pcap"/> </action> </nop> <pause milliseconds = "6000"/> <send> <![CDATA[ SIP/2.0 180 Ringing Via:[$via_invite] [last_Call-ID:] [last_From:] [last_To:] CSeq: 1 INVITE Allow: INVITE,UPDATE,BYE,PRACK,INFO,OPTIONS,CANCEL,SUBSCRIBE,ACK,REFER,NOTIFY,REGISTER,PUBLISH,MESSAGE Call-Info: <tel:+8613665171969>;purpose=call-completion;m=NR Contact: <sip:[local_ip]:[local_port];transport=[transport]> P-Asserted-Identity: <tel:+8613665171969> P-Charging-Vector: icid-value="pbac22bhw.19e.627a.20211025080346";orig-ioi=js.ims.mnc011.mcc460.3gppnetwork.org;term-ioi=js.chinamobile.com P-Early-Media: inactive Content-Length: 0 ]]> </send> <pause milliseconds = "6000"/> <send> <![CDATA[ SIP/2.0 200 OK Via:[$via_invite] [last_From:] To: <sip:[field1]@[remote_ip]:[remote_port]>;tag=[call_number] [last_Call-ID:] CSeq: [$cseq_invite] Contact: <sip:[local_ip]:[local_port];transport=[transport]> Content-Type: application/sdp Content-Length: [len] v=0 o=- 1635149026 1635149027 IN IP4 5.92.12.134 s=SBC call c=IN IP4 [media_ip] b=AS:41 b=RR:1837 b=RS:600 t=0 0 m=audio [media_port] RTP/AVP 102 96 b=AS:41 b=RR:1837 b=RS:600 a=rtpmap:102 AMR/8000 a=fmtp:102 mode-change-capability=2;max-red=0 a=ptime:20 a=maxptime:240 a=sendrecv a=curr:qos local sendrecv a=curr:qos remote sendrecv a=des:qos mandatory local sendrecv a=des:qos mandatory remote sendrecv a=rtpmap:96 telephone-event/16000 a=fmtp:96 0-15 ]]> </send> <recv request="ACK" rtd="true" crlf="true"> </recv> <nop> <action> <!--exec play_pcap_audio="pcap/dtmf_2833_0.pcap"/--> <exec play_pcap_audio="pcap/amr102.pcap"/> </action> </nop> <recv request="INVITE"> </recv> <send> <![CDATA[ SIP/2.0 100 Trying [last_Via:] [last_From:] To: <sip:[field1]@[remote_ip]:[remote_port]>;tag=[call_number] [last_Call-ID:] [last_CSeq:] Contact: <sip:[local_ip]:[local_port];transport=[transport]> Content-Length: 0 ]]> </send> <send> <![CDATA[ SIP/2.0 200 OK [last_Via:] [last_From:] To: <sip:[field1]@[remote_ip]:[remote_port]>;tag=[call_number] [last_Call-ID:] [last_CSeq:] Contact: <sip:[local_ip]:[local_port];transport=[transport]> Content-Type: application/sdp Content-Length: [len] v=0 o=- 1635149026 1635149027 IN IP4 5.92.12.134 s=SBC call c=IN IP4 [media_ip] b=AS:41 b=RR:1837 b=RS:600 t=0 0 m=audio [media_port] RTP/AVP 115 96 b=AS:41 b=RR:1837 b=RS:600 a=rtpmap:115 AMR-WB/16000/1 a=fmtp:115 mode-change-capability=2;max-red=0 a=ptime:20 a=maxptime:240 a=sendrecv a=curr:qos local sendrecv a=curr:qos remote sendrecv a=des:qos mandatory local sendrecv a=des:qos mandatory remote sendrecv a=rtpmap:96 telephone-event/16000 a=fmtp:96 0-15 ]]> </send> <recv request="ACK"> </recv> <nop> <action> <!--exec play_pcap_audio="pcap/dtmf_2833_0.pcap"/--> <exec play_pcap_audio="pcap/amrwb1_115.pcap"/> </action> </nop> <recv request="BYE"> </recv> <send> <![CDATA[ SIP/2.0 200 OK [last_Via:] [last_From:] To: <sip:[field1]@[remote_ip]:[remote_port]>;tag=[call_number] [last_Call-ID:] [last_CSeq:] Contact: <sip:[local_ip]:[local_port];transport=[transport]> Content-Type: application/sdp Content-Length: 0 ]]> </send> <!-- Keep the call open for a while in case the 200 is lost to be --> <!-- able to retransmit it if we receive the BYE again. --> <pause milliseconds="0"/> <!-- definition of the response time repartition table (unit is ms) --> <ResponseTimeRepartition value="10, 20, 30, 40, 50, 100, 150, 200"/> <!-- definition of the call length repartition table (unit is ms) --> <CallLengthRepartition value="10, 50, 100, 500, 1000, 5000, 10000"/> </scenario>
uas的启动脚本如下：
./sipp -sf uas_jsfhgk.xml -i 192.168.2.179 -p 8063 -trace_screen -screen_overwrite true -screen_file uas_jsfhgk.log -inf calling_number_graylist_speech_template_performance.txt
三、valrind工具
通过10 caps的连续呼叫测试，媒体服务器的故障现象开始复现，确定了代码的大致位置后。开始对媒体服务器的代码部分进行修复加锁。加锁完成后，进程不重启了，但是发现有内存泄漏，通过top命令查看，进程占用的内存一直在增长。 于是启用了valgrind工具，用valgrind启动媒体服务进程，uac单呼一次，然后分析valgrind日志。 通过valgrind的日志发现，提示调用的avcodec_send_frame存在内存泄漏，但是我查了代码，已经调用了av_frame_free释放了内存，于是我判断应该不是这个地方存在内存泄漏。通过百度查询avcodec_send_frame内存泄漏问题时，我注意到一个专家写了一句话“当av_read_frame返回值不小于0时，内部会进行缓存操作，需要外界释放。
总结
提示：这里对文章进行总结：
当媒体服务器发行故障后，先后通过分析SIP信令及媒体服务器抓包及服务器日志，定位媒体服务器网元出现故障。
通过abrt抓取进程异常的core文件，使用sipp模拟大呼进行故障场景复现。采用valgrind进行内存泄漏定位。最终定位问题并修复该问题。
希望这个抢救的案例，能够给其他同行起到借鉴作用。
