Comments (12)
I think it makes a lot of sense to set a lock there.
from circus.
maybe we should rather queue calls there instead of locking with an async scheduler in the ioloop triggering the calls? That will remove the need of locks and would scale better imo.
We indeed need to handle the message very fast there, and using a lock could slow the process too much.
from circus.
afaict, call() sends a message and grabs it's reply, and functions that use call() look at the reply. so i think we might have to deal with the Lock, because before we can return we have to send and get a reply. i haven't done a lot of python async programming, so I might be missing something obvious, sorry :)
from circus.
sory for the spam, i had a cache issue there. Anyway let me know if the change fix the issue for you since I wasn't able to reproduce it with 100 processes around. We may still need a way to block on call anyway but it should already been handled in the recv command since the iostream is internally using a queue.
from circus.
@benoitc I think you should rename cast into "async_call" because the semantics is not super clear. Also, what happens if I do want to get back the async result ? I think we need for that new API an optional callback for this.
I also agree with @fetep that the call() api needs a lock
from circus.
well casting is a a synchronous call without waiting an answer. That's diferrent from an async call. If you look the patch I kept the call for when it's needed. Not sure why you would need a callback there. If you want a callback thing you could use an IOStream.
Did you test it without the lock? Basically the IOstream use a queue, so the lock shouldn't be needed so the call should be OK there. Also I would love to have a test to reproduce the problem. I'm afraid that if we need a lock there it's hiding a scalability problem. We should rather make sure nothing locks.
from circus.
what do you mean by 'casting' exactly ? "broadcasting" ?
how is that different from an async call, and how would I get back my result in async call without a callback ?
re test: pete gave one I believe, so it would be a good thing to have it in our test case indeed.
last: what's your problem with locks, I don't understand it
from circus.
i mean casting as to throw away... Which is exactly what we do When you don't need to get the result you use this method.
The problem with locking is that you're locking everything in the ioloop. We shouldn't have to do that there. Like I said the messages falling in the handle functions are queued before, so in theory the call should be OKish. I'm not against the lock as a temporary solution anyway but would be good to know if casting send like my patch does isn't enough.
from circus.
if you throw away the result, I am not sure to understand why a queue is needed at all.
I think we're talking about many different things in this ticket now.
I think we need to focus on fixing the bug first, then discuss the various semantics separately (sync vs async vs throw away)
I propose that we convert Pete's example into a test, make sure we can reliably reproduce the bug, then discuss the possible fix. then open another bug about what kind of extra APIs are needed for our messaging needs.
from circus.
not sure to follow. I didn't say it was needed.
The patch I submitted should fix that issue. Explanation coming with explained the reasoning. All the rest is an explanation about the pyzmq internals and why we shouldn't lock.
from circus.
let me restate :
1/ until we have a test case that reproduce the bug, we're just making assumption.
2/ if the lock fix is really slowing down thing, let's have a test that benchmark this part of the code to measure its speed.
from circus.
I commited @benoitc fix but I am quite unhappy with the output.
I think we're moving into a position where we don't really know what's going on in the flapping messages. I think we should make this story better
from circus.
Related Issues (20)
- circusctl unable to set selected options
- Alternative ways to discuss circus library HOT 1
- Fallback for os env vars HOT 1
- Plugin HttpObserver broken since 0.17 because of change in AsyncHTTPClient from Tornado 5
- Trouble accessing web dashboard
- util.py: load_virtualenv broken with Python 3.10
- Hang caused by circus and psutil using conflicting methods to identify a process
- a pypy venv issue
- arbiter is already running arbiter_start_watchers command HOT 2
- Add Python3.9 and PyPy to support list HOT 3
- Copying environment variables with dollar sign + capital letters
- Make a new release supporting Python 3.10 HOT 3
- Move away from `nose` as the testing framework
- Add official support for Python 3.10 HOT 1
- Adopt PEP 621
- Consider removing `papa` dependency HOT 1
- Move `tests` outside of the main package HOT 2
- ConflictError arbiter is already running arbiter_start_watchers command HOT 1
- Util.py/load_virtualenv picks incorrect python version if py_ver=None
- DeprecationWarning: 'pipes' is deprecated and slated for removal in Python 3.13
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from circus.