During batch feed, OSError: [Errno 24] Too many open files about pyvespa

[neo-anderson]

I tried ulimit -n 10000. It fixed the issue completely. Ingesting with 100 connections without an issue 👍. Thanks!

[thigm85]

This seems to be an OS limitation. I was once running a script on an AWS EC2 instance and I had to increase the open file soft limit when using app.feed_batch.

I increased the limit with ulimit -n 10000 and checked with ulimit -Sn. It worked after that.

If the above does not work, try to reduce the number of async connections via the connections parameter:

app.feed_batch(..., connections, ...)

[neo-anderson]

I didn't change ulimit, but tried reducing the number of connections like you suggested and it worked! The default number of connections (100) was fine for a local docker deployment. Had to reduce it to ingest the same data into Vespa Cloud. Thanks for your help.

[thigm85]

Glad it worked. Did you check what the highest value that worked was? I might consider changing the default connections value.

[neo-anderson]

Hi @thigm85 , connections set to 100 and 50 didn't work for me, but setting it to 20 worked. I dont know if it's the max value that would work though. Cheers!

[thigm85]

Thanks @neo-anderson

[bratseth]

@lesters put this in the doc somewhere and/or change default?

[thigm85]

@bratseth @lesters I can create a troubleshooting section and include this. It was on my to-do list anyway.

[bratseth]

Yes, great!