process terminated, reason: "ev_io_stop: Assertion `("libev: ev_io_stop called with illegal fd..." about stud

[gyepisam]

I am unable to produce this error. Can you provide some information on your stud build options (using ssl cache?), libev version and some info on the client setup please?

I ran 5 httperf instances, each with 1000 concurrent connections fetching a 48M file and found other issues (which I'll be addressing), but not that one.

Thanks!

[verterok]

Hi,
I'm using libev 4.04 and openssl 1.0.0d.
And starting it with: ./stud --tls -b 127.0.0.1,21100 -f 0.0.0.0,21111 -n 2 -B 1000 server.pem

The client is a python based client (using twisted) that open an ssl connection (isn't http, but a inhouse protocol over tcp) and keep it open for some random time.

[jamwt]

So, just glancing at one of my machines, I have a -n 16 instance of stud running for about 4 weeks that's used about 10 CPU-days in that time, and I haven't run into that. But... it's possible my connections are not as long-lived as yours--mine seldom last more than an hour or two.

How long are these connections being held open? Any more details about OS and version? amd64 or x86? etc

[jamwt]

Also, related to the platform questions, do you know which libev backend is being used on your stud build? epoll, kqueue, select, etc. Something about that error makes me wonder if you exceeded a max fd count for a particular backend (select, etc). Mine are all using epoll on linux.

[verterok]

Hi,

I these tests, long-lived means N minutes...not even close to an hour. The max time a connection is held open is 30 minutes, but there are also connections that only live 1 minute.
I'm using a vm (kvm): Ubuntu 10.04 amd64, with openssl-1.0.0d and libev-4.04 (both backported from oneiric archive)
I don't know how to tell which libev backend is being used....but I needed to bump the max fd count in order to make it work, I was getting a different assertion error, so I'm using a wrapper shell script that increase the ulimit to 8192 before starting stud.

Regards,

[jamwt]

So, sorry if I'm stating the obvious here and you've already accounted for this, but if you're using 4k clients (and 1024 * 4 or 4,000?) it's entirely possible you are still exceeding the 8192 fd limit, since at the very minimum each client needs two fds (one to the frontend and one to the backend). Do you still have the problem with the ulimit set absurdly high, like 20k?

[verterok]

oh, excellent point! I'll bump the ulimit and get back with the results.

If it's the ulimit issue, It would be nice (if it's possible, I know nothing about libev) if stud doesn't die like this when the max fd count is reached.

Thanks!

[jamwt]

Since I haven't heard anything back, I'll mark this closed.