Comments (9)
I tried ulimit -n 10000
. It fixed the issue completely. Ingesting with 100 connections without an issue 👍. Thanks!
from pyvespa.
This seems to be an OS limitation. I was once running a script on an AWS EC2 instance and I had to increase the open file soft limit when using app.feed_batch
.
I increased the limit with ulimit -n 10000
and checked with ulimit -Sn
. It worked after that.
If the above does not work, try to reduce the number of async connections via the connections
parameter:
app.feed_batch(..., connections, ...)
from pyvespa.
I didn't change ulimit, but tried reducing the number of connections like you suggested and it worked! The default number of connections (100) was fine for a local docker deployment. Had to reduce it to ingest the same data into Vespa Cloud. Thanks for your help.
from pyvespa.
Glad it worked. Did you check what the highest value that worked was? I might consider changing the default connections
value.
from pyvespa.
Hi @thigm85 , connections set to 100 and 50 didn't work for me, but setting it to 20 worked. I dont know if it's the max value that would work though. Cheers!
from pyvespa.
Thanks @neo-anderson
from pyvespa.
@lesters put this in the doc somewhere and/or change default?
from pyvespa.
@bratseth @lesters I can create a troubleshooting section and include this. It was on my to-do list anyway.
from pyvespa.
Yes, great!
from pyvespa.
Related Issues (20)
- FR: expand feed iterable callback args HOT 1
- update only supports assign updates
- Add timeout-parameters to `VespaDeployment`
- Better documentation of restrictions with regards to VespaDocker volume mounts
- document v1 paths should be urlencoded
- Add support for conditional writes
- Integrate `vespacli` python package HOT 1
- pyvespa assumes vespaengine/vespa is pulled HOT 2
- Replace old logos with new in notebooks
- Async data operations does not apply retry strategy
- Improve user feedback on error from container logs
- Generate application packages from existing Vespa applications HOT 5
- Can't feed result of get operation into feed delete/update HOT 1
- Convert notebooks to use new prepend syntax
- Make docker dependency optional HOT 6
- Deploy the Vespa application with docker for bge-m3 HOT 1
- Refine retry-strategy (429-errors) for VespaSync
- Make VespaCloud (optionally) create key/cert with `vespacli` HOT 1
- Make it more obvious how to retrieve a reference to `Vespa` without redeploying HOT 1
- deploy_to_prod return value should be Vespa HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pyvespa.