Captagent is utilizing 100% CPU when sip-parse = true about captagent

[adubovikov]

I have checked, but unfortunately couldn't reproduce the issue. For me it looks like bad SIP parsing, Can you please crate a pcap dump with all SIP messages from start until high CPU usage ? After I will check again. Thank you.

[marrold]

Hi Alexandr,

That's plausible. I'm afraid sending a full PCAP isn't really possible as
it contains sensitive data as I'm sure you can imagine. Would increasing
the verbosity on the captagent logs help?

I could try capturing an offending packet this evening when there is less
traffic, to make it more straight forward to redact any numbers / IP's etc

Thanks

On Fri, Jan 23, 2015 at 2:52 PM, Alexandr Dubovikov <
[email protected]> wrote:

I have checked, but unfortunately couldn't reproduce the issue. For me it
looks like bad SIP parsing, Can you please crate a pcap dump with all SIP
messages from start until high CPU usage ? After I will check again. Thank
you.

—
Reply to this email directly or view it on GitHub
#5 (comment).

[adubovikov]

please check last git.
i did on workaround, please check syslog/log file if you will see something like this:
"TOO MANY LOOPS [10]"

thank you.

[marrold]

Thanks, I will test and report back.
On 24 Jan 2015 13:00, "Alexandr Dubovikov" [email protected] wrote:

please check last git.
i did on workaround, please check syslog/log file if you will see
something like this:
"TOO MANY LOOPS [10]"

thank you.

—
Reply to this email directly or view it on GitHub
#5 (comment).

[marrold]

100% CPU reoccurred today. Please see the logs- http://pastebin.com/raw.php?i=nHSCKdk5

Also, captagent died with a segfault, possibly due to 100% CPU usage -

Jan 26 10:38:09 CLL-Tracing kernel: [926734.544000] captagent[6523]: segfault at 7fb51c12230f ip 00007fb4248a9b8e sp 00007fb4239c05c8 error 4 in libc-2.19.so[7fb424811000+1bb000]

I don't have the packet capture for this occurrence, I can grab one next time if required?

Thanks

[adubovikov]

no, it's ok for now. I see that len is bigger than message itself. I have added more debug, can you please pull git again and check one more time ?

[marrold]

Pulled and running, Thanks

On Mon, Jan 26, 2015 at 3:08 PM, Alexandr Dubovikov <
[email protected]> wrote:

no, it's ok for now. I see that len is bigger than message itself. I have
added more debug, can you please pull git again and check one more time ?

—
Reply to this email directly or view it on GitHub
#5 (comment).

[marrold]

Please see a small selection of the latest log showing the common messages. As you believe it's a len issue I've been careful to use the same amount of characters when redacting numbers / IP's

http://pastebin.com/raw.php?i=GnraNnMW

Thanks

[marrold]

Hi, is there any update on this? I noticed it's no longer utilising 100% CPU due to the added 'break', but curious if a fix is on its way.

Thanks

[adubovikov]

Yes, we will update tomorrow. Sorry but this week was very busy.

On 8 February 2015 at 22:15, marrold [email protected] wrote:

Hi, is there any update on this? I noticed it's no longer utilising 100%
CPU due to the added 'break', but curious if a fix is on its way.

Thanks

—
Reply to this email directly or view it on GitHub
#5 (comment).

[marrold]

No need to apologies, appreciated as always.

On Sun, Feb 8, 2015 at 9:15 PM, Alexandr Dubovikov <[email protected]

wrote:

Yes, we will update tomorrow. Sorry but this week was very busy.

On 8 February 2015 at 22:15, marrold [email protected] wrote:

Hi, is there any update on this? I noticed it's no longer utilising 100%
CPU due to the added 'break', but curious if a fix is on its way.

Thanks

—
Reply to this email directly or view it on GitHub
<#5 (comment)
.

—
Reply to this email directly or view it on GitHub
#5 (comment).

[adubovikov]

so. please take the last git. Also I see that this TCP messages are broken. Probably it was bad message len, but it's hard to say.

anyway I am waiting on your feedback. Thank you!

[marrold]

Thanks, I will test tomorrow. Is there anyway to disregard broken TCP and
parse anyway?
On 9 Feb 2015 20:41, "Alexandr Dubovikov" [email protected] wrote:

so. please take the last git. Also I see that this TCP messages are
broken. Probably it was bad message len, but it's hard to say.

anyway I am waiting on your feedback. Thank you!

—
Reply to this email directly or view it on GitHub
#5 (comment).

[adubovikov]

hard to say. If only some lines of SDP are missed, it's ok, but if RURI or several important headers are missed - this is complete different story. Anyway, kamailio will drop it :-)

[marrold]

What I am trying to understand is, why wasn't the below example (from the
previous pastebin) parsed? It looks RFC compliant. Was this a TCP LEN vs
Actual Data LEN issue? Will the latest code mean such SIP messages are
parsed?

Thanks

Jan 27 18:25:44 CLL-Tracing captagent[27177]: [ERR] proto_uni.c:328 TOO
MANY LOOP LEN [1386] vs NEWLEN: [0] vs SKIP: [1383] vs PARSED: [0]
Jan 27 18:25:44 CLL-Tracing captagent[27177]: [ERR] proto_uni.c:329 PACKET

INVITE sip:[email protected]:5060;transport=tcp SIP/2.0
Via: SIP/2.0/TCP 11.111.1.111:5060
;branch=z9hG4bK-524287-1---d5a2e626e94acc0d;rport
Via: SIP/2.0/UDP 11.111.1.111:5061
;branch=z9hG4bK-6x3eokgxtich7y4w;rport=5061
Max-Forwards: 69
Record-Route: sip:11.111.1.111:5060;transport=tcp;lr;drr
Record-Route: sip:11.111.1.111:5060;lr;transport=UDP;drr
Contact: "Anonymous"sip:11.111.1.111:5061
To: sip:[email protected]
From: [phone] sip:[email protected];tag=nkgcuhqzbwycb74v.o
Call-ID: [email protected]
CSeq: 781 INVITE
Expires: 300
Content-Disposition: session
Content-Type: application/sdp
User-Agent: Sippy
P-Asserted-Identity: [phone] <sip:[email protected]
;user=phone>
h323-conf-id: 235399844-542757103-3421031868-2401107647
Portasip-3264-action: offer 1
cisco-GUID: 235399844-542757103-3421031868-2401107647
Content-Length: 445

v=0
o=Sippy 2712874956379544410 0 IN IP4 11.111.1.111
s=Cisco SDP 0
t=0 0
m=audio 64644 RTP/AVP 8 18 0 101
c=IN IP4 11.111.1.111
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
a=sqn:0
a=cdsc: 1 audio RTP/AVP 8 18 0 101
a=cdsc: 5 image udptl t38
a=cpar: a=T38FaxVersion:0
a=cpar: a=T38FaxRateManagement:transferredTCF
a=cpar: a=T38FaxMaxDatagram:160
a=cpar: a=T38FaxUdpEC:t38UDPRedundancy
a=X-sqn:0
a=X-cap: 1 image udptl t38

On Mon, Feb 9, 2015 at 9:05 PM, Alexandr Dubovikov <[email protected]

wrote:

hard to say. If only some lines of SDP are missed, it's ok, but if RURI or
several important headers are missed - this is complete different story.
Anyway, kamailio will drop it :-)

—
Reply to this email directly or view it on GitHub
#5 (comment).

[adubovikov]

the message len is 1386, but we have only 1383, somethere 3 characters are gone.
anyway, with the new git, this message will be parsed.

[adubovikov]

any feedback ? Can I close the issue ?

[marrold]

I spotted one entry in the logs regarding the loop, which is much less than
before. I was going to leave it 24 hours before providing feed back.

There still seems to be some issues with TCP packets, which may or may not
be related.

I don't suppose someone has created a HEP dissector for Wireshark?

Thanks

On Tue, Feb 10, 2015 at 8:48 PM, Alexandr Dubovikov <
[email protected]> wrote:

any feedback ? Can I close the issue ?

—
Reply to this email directly or view it on GitHub
#5 (comment).

[adubovikov]

yeah, we planned do it long time ago, but unfortunately no time for this. If you can do it, we will be very appreciated!!!

[marrold]

I plan to give it a go but can't guarantee any success!

In the mean time, should I raise a new issue for the missing TCP packets
once I've gathered some examples? I'm happy to wait as you've been busy.

On Tue, Feb 10, 2015 at 9:13 PM, Alexandr Dubovikov <
[email protected]> wrote:

yeah, we planned do it long time ago, but unfortunately no time for this.
If you can do it, we will be very appreciated!!!

—
Reply to this email directly or view it on GitHub
#5 (comment).

[adubovikov]

sure, contact me any time. Currently we have some requests, but we always find a bit time to help our users :-)

[marrold]

Please find the latest log here- https://gist.github.com/marrold/00e7e647ae0ece7e46f8

Looks like only 3 packets got caught in the loop out of thousands, so this is much better.

Thanks

[adubovikov]

ok. looks like it was fixed. Please check the last git.

thank you very much!

[marrold]

Pulled and running. Thanks for your assistance.

On Mon, Feb 16, 2015 at 6:16 PM, Alexandr Dubovikov <
[email protected]> wrote:

Closed #5 #5.

—
Reply to this email directly or view it on GitHub
#5 (comment).

[adubovikov]

you are welcome. thank for bug reporting.