Comments (55)
There is no difference between emitting from the main process or from an external process. The difference is in how you create the SocketIO
object. In the main process, you initialize the SocketIO
by passing the app
instance, while in the external processes you do not pass app
. In both cases you must pass the connection url for the queue. Once the object is created, however, emitting works in the same way.
from flask-socketio.
Hi, @miguelgrinberg. Here is what I have now.
an html to render, to show the message(from your example).
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Simple</title>
<script type="text/javascript" src="//code.jquery.com/jquery-1.4.2.min.js"></script>
<script type="text/javascript" src="//cdnjs.cloudflare.com/ajax/libs/socket.io/1.3.5/socket.io.min.js"></script>
<script type="text/javascript" charset="utf-8">
$(document).ready(function() {
var socket = io.connect('http://' + document.domain + ':' + location.port);
socket.on('redis', function (msg) {
$('#log').append('<br>' + $('<div/>').text('Received: ' + msg.data).html());
});
});
</script>
</head>
<body>
<h2>Receive:</h2>
<div id="hh"></div>
<div id="log"></div>
</body>
</html>
fapp.py:
from flask import Flask, render_template
from flask_socketio import SocketIO
app = Flask(__name__)
sio = SocketIO(app, message_queue='redis://')
@app.route('/')
def index():
return render_template('s.html')
@sio.on('connect')
def test_connect():
sio.emit('redis', {'data': 'you will see this if success'})
if __name__ == '__main__':
sio.run(app)
this demo will work if no queue is added, but hangs if I add message_queue para. Did I miss something?
from flask-socketio.
The celery worker runs in a different process, so it can't see the socketio.emit()
function. I think an approach that you can take is to have a websocket connection in your application, and this connection can talk to the celery task and report its state back to the client.
from flask-socketio.
This might be worth a read:
from flask-socketio.
Thanks, that is an interesting implementation.
from flask-socketio.
Is it possible to integrate this redis pub/sub strategy with Flask-SocketIO?
I want to deploy Flask with Flask-SocketIO in at least two servers to provide high availability.
from flask-socketio.
@iurisilvio You can't use multiple servers with gevent-socketio, as it keeps session data in memory.
from flask-socketio.
Thanks for your answer.
You can use sticky sessions to send the user always to the same server and use redis to share messages between server nodes. It is common practice: http://socket.io/docs/using-multiple-nodes/#sticky-load-balancing
I'm looking for alternatives. The gevent-socketio module has an example with redis:
I want to integrate something like this example in Flask-SocketIO, but I was unable to do it.
from flask-socketio.
I'm going to be honest, I haven't spent a lot of time looking into this, but as I understand it, the sticky sessions only help for the less efficient transport mechanisms supported by gevent-socketio such as long-polling, which involve multiple request cycles. I don't think this is a problem for the WebSocket transport, since the connection is established and remains until one of the parts ends it.
But the reason why using multiple servers is hard or not possible is not related to that. The problem is that the state of the server, including the list of connected clients, is stored in memory. If you use multiple servers, even with sticky sessions, each server will have a partial view of who is connected, so broadcasting will not reach all users but only those that reside in the same server from where the message was sent.
Do you know of a way in gevent-socketio to store the server state in a shared storage such as redis? I don't think this is possible, but if it is, then I can probably do the same in Flask-SocketIO and you will be able to have multiple servers.
from flask-socketio.
I understand your concerns, but I think it is possible. All redis examples I found out there do not share connections. Each server maintain some connections and pub/sub to redis to share messages between servers.
It is a hard problem to solve, maybe I will just use an external socketio server. I read socket.io-redis code, it is simple but handle only broadcast messages.
I will try to subclass your GenericNamespace
to provide some redis communication between servers.
from flask-socketio.
I agree, it is a hard problem to solve, and as I said before, it's not a problem that gevent-socketio tries to solve, so you will have to build something on top of it, or find a replacement.
Also consider that I have followed the same approach in Flask-SocketIO, my data structures are also held in memory.
I did not mean to discourage you, by the way, this is just a software problem, so it is something that definitely can be done, but I thought your question was about doing it with Flask-SocketIO and gevent-socketio, and I think that is not possible without making changes to these projects.
from flask-socketio.
Hi all, maybe this fork will be of some help (adds support for redis stores). gevent-socketio seems to be abandoned anyway...
https://github.com/ryesoft/gevent-socketio/
from flask-socketio.
@gordol thanks for the link, very interesting. I'll see if I can make this project compatible with the redis extension.
from flask-socketio.
That would be really awesome! If you need any help or testers, let me know. I'm already using redis for my sessions and worker queue, and I've been wanting to experiment with porting this library to use the redis fork of gevent-socketio.
from flask-socketio.
Did you guys get any further on this? I think in the mean time I will just call the application via webhooks when celery processes tasks.
This seems like a horrible solution of course... But I should say thanks so much - you have massively helped me in learning flask!
from flask-socketio.
Now that release 1.0 is out, I am working on a Redis backend, that will enable an external process (such as a Celery worker) to emit to a socket.io client, simply by writing to a Redis queue. This will also enable the use of multiple socket.io server workers.
from flask-socketio.
from flask-socketio.
This isn't quite a complete solution, going off memory from a previous project, but should give a general idea of how I went about about this... basically just spawning a redis pubsub listener when socket client connects, and it's really pretty much that simple.
One can take this and go deeper with it as well, with channels for each user in your system, if you need to be able to emit messages to only certain users, etc... just a matter of spawning more event listeners on specific channels or namespaces.
I usually do a lot of other stuff too, like use redis to store buffered stream data from socket channels that are shared across multiple socket clients, and I use redis as a lightweight worker queue instead of celery, but not sure that's relevant here. I use socketIO as a message system, to push new updates when data changes, but also use it to push down bulk snapshots of data upon connection. Redis queues are perfect for this with lrange and rpush if that's what you're after. So then you can leverage redis for pubsub and for caching.
Anyway, I digress... here's a very basic redis pubsub implementation that works w/ socketIO.
redis_channel_key = 'socketIO'
@socketio.on('connect', namespace='/example')
def example_connect():
print(str(datetime.now())+' - Client connected')
@copy_current_request_context
def event_listener(namespace):
redis_channel = '%s%s' % (redis_channel_key, namespace)
pubsub = redis.pubsub()
pubsub.subscribe(redis_channel)
print(str(datetime.now())+' - Subscribed to redis channel: %s and emitting to namespace: %s' % (redis_channel, namespace))
for m in pubsub.listen():
if m['type'] == 'message':
data = json.loads(m['data'])
emit(data['name'], data['payload'])
print(str(datetime.now())+' - Spawning event listener')
spawn(event_listener, namespace)
@celery.task
def async_emit(channel, data, namespace):
redis_channel = '%s%s' % (redis_channel_key, namespace)
data = json.dumps(dict(name=channel, payload=data))
print(str(datetime.now())+' - Publishing to channel: %s on namespace: %s' % (channel, namespace))
redis.publish(redis_channel, data)
@celery.task
def dummy_task
async_emit.delay('foo', dict(testing=True), namespace='/example')
async_emit.delay('bar', dict(foo='bar'), namespace='/example')
#now you can leverage redis pubsub to emit socket messages from celery across multiple workers
dummy_task.delay()
from flask-socketio.
Also just to hit the OP question, the approach I used for celery compatability is to spawn a background thread (see the flask-socketio example app) which listens for tasks and emits as necessary.
from flask-socketio.
Here is a progress update on this request.
Communication through a message queue is now implemented in package python-socketio, through the use of Kombu, which provides a common API to work with several message queues including Redis and RabbitMQ.
The use case requested in this issue is only one of the configurations I was interested in supporting. The other is to allow the Socket.IO server to be horizontally scaled, with each server managing a subset of the clients. Broadcasting and callback functions are coordinated among all the servers through the message queue.
I'm now starting to work on integrating the new stuff in this package, so I expect a new release soon.
from flask-socketio.
This is awesome, great work!
from flask-socketio.
Any updates regarding this @miguelgrinberg? Thanks!
from flask-socketio.
The changes are already committed to the master branch. I'm not 100% sure the API will not change, but what's committed appears to work just fine. The docs have been updated, also. You have to have a Redis, RabbitMQ or similar message queue running to do this.
Emitting from the Celery worker is done using the KombuManager
class from the python-socketio package, there is an example in the docs of that package (http://python-socketio.readthedocs.org/en/latest/#using-a-message-queue). This I think I will be improving a bit, so that you don't have to use a separate package, but that I haven't done yet.
from flask-socketio.
Great, thanks! I've seen the docs, but do you think you could you point me out some example, or more documentation on how to implement this? (What I'm trying to get here is a simple system for facebook-like notifications, for the twitter-like application from your tutorials).
Thanks!
from flask-socketio.
I don't have any examples yet for Flask-SocketIO. The python-socketio docs have the example to send from an external Python process using the KombuManager
class. That will work just fine, but for this extension I will be adding a wrapper to the other stuff. I expect this done in the next week or so, and at that point I'll include an example.
from flask-socketio.
@jfloff The documentation (file docs/index.rst in this repo) is now looking pretty good. I may still make some additional changes here an there, but the message queue sections are mostly complete now.
from flask-socketio.
All the work to support the message queue workflows is now complete. This includes the support for multiple worker servers behind a load balancer such as nginx, and the ability to emit events to clients from a processes other than a server (a Celery worker, for example). Closing this issue. A release will be made available shortly.
from flask-socketio.
This is awesome Miguel, good work!
On Jan 9, 2016, at 6:54 PM, Miguel Grinberg [email protected] wrote:
All the work to support the message queue workflows is now complete. This includes the support for multiple worker servers behind a load balancer such as nginx, and the ability to emit events to clients from a processes other than a server (a Celery worker, for example). Closing this issue. A release will be made available shortly.
—
Reply to this email directly or view it on GitHub.
from flask-socketio.
@miguelgrinberg I would like to check if there is any simple example code on emitting the events from external process ?
The example in the document is a little bit hard to catch
What I understand about this is that SocketIO is listening to a message queue. But how to put message into the queue ? and how to let the server emit events once a message is put into the queue ?
Thanks for the help
from flask-socketio.
@miguelgrinberg So I do not need to send anything to the message queue itself ?
from flask-socketio.
No, all you need to do is have the queue running. Flask-SocketIO knows how to communicate over it.
from flask-socketio.
@miguelgrinberg great , I have fixed it
thanks !
from flask-socketio.
Hi, all. I recently try this thread to emit from external process. Did i do something wrong?
Flask:
socketio = SocketIO(app, message_queue='redis://localhost:6379/')
socketio.run(app, port=80)
external process:
from flask_socketio import SocketIO
socketio = SocketIO(message_queue='redis://localhost:6379/')
# socketio = SocketIO(message_queue='redis://)
socketio.emit('redis', {'data': 'foo'})
there's nothing in my 'redis' channel when emit from external. Can someone show me how to do the right thing? TIA
from flask-socketio.
@wiwengweng You have to specify which redis db it is receiving it
like so
socketio = SocketIO(message_queue='redis://localhost:6379/4')
from flask-socketio.
@wiwengweng the first argument of the emit call is not the redis channel, it is the Socket.IO event. Flask-SocketIO uses its own private channel for processes to share information, applications are not supposed to write to the redis queue directly for Socket.IO purposes. Of course you can use the queue for your own reasons, that would be separate from what Flask-SocketIO uses it for.
from flask-socketio.
@miguelgrinberg Thanks, I am sorry if I make you misunderstand my demo. In fact 'redis' is a redis channel name. Here's the full sample I create.
fapp.py
from flask import Flask, render_template
from flask_socketio import SocketIO
app = Flask(__name__)
sio = SocketIO(app, message_queue='redis://localhost:6379/4')
@app.route('/')
def index():
return render_template('index.html')
@sio.on('redis')
def message(sid, data):
print('message ', data)
if __name__ == '__main__':
sio.run(app)
external:
pub.py
from flask_socketio import SocketIO
socketio = SocketIO(message_queue='redis://localhost:6379/4')
socketio.emit('redis', {'data': 'foo'})
I add the db id '4' as @ripitrust told me too. I start fapp.py and then run pub.py. I think I can see the message print in the console, but still I see nothing. Hope you help with this. TIA.
from flask-socketio.
Regarding connection URLs: If you are running redis on the same host as your server, and you are using all default options, then your connection string can be just redis://
. If your redis set up is not the default, then the connection URL is in the format redis://[:password]@[hostname]:[port]/[db-number]
.
I think you are misunderstanding what it means to emit from an external process. The recipient of those events is not the server, it is the client. When you emit a redis
event on the external process, the server's redis
handler will not be invoked. Communication is always from the server to the client or viceversa. If an external process is doing an emit, it does so on behalf of the server. If you need an external process to communicate with the SocketIO server, then that external process needs to be a SocketIO client, for that you need a different package, mine is only a server.
from flask-socketio.
Thanks. I will modify the demo using an html page, then back~
from flask-socketio.
If you are using eventlet or gevent, then you need to monkey patch the Python library for the redis package to work.
from flask-socketio.
No, this demo use pure flask-socketio to go. Did you think this wired?
from flask-socketio.
And you have redis running, and you are connecting to the right URL for it?
from flask-socketio.
Yes, redis is running on the same server. But one thing. I am running this demo on Windows. Did it matter?
c:\Program Files\Redis>redis-cli.exe
127.0.0.1:6379>
127.0.0.1:6379>
server started:
C:\Python27\python.exe D:/center/web/fapp.py
(5676) wsgi starting up on http://127.0.0.1:5000
(5676) accepted ('127.0.0.1', 64614)
(5676) accepted ('127.0.0.1', 64613)
the first request will be accepted, but if I refresh the page, it hangs.
from flask-socketio.
Hi, @miguelgrinberg, I just start the redis cli client and find that my app is connected to redis. I am starting ubuntu to see if this hangs too. Any of your advice will be appreciating. :)
from flask-socketio.
As I run this example on my ubuntu server, I can see the emit message then, and also the queue shows the message too in redis:
wyl@ubuntu:~$ redis-cli
127.0.0.1:6379> PSUBSCRIBE *
Reading messages... (press Ctrl-C to quit)
1) "psubscribe"
2) "*"
3) (integer) 1
1) "pmessage"
2) "*"
3) "flask-socketio"
4) "(dp0\nS'skip_sid'\np1\nNsS'room'\np2\nNsS'namespace'\np3\nS'/'\np4\nsS'event'\np5\nS'redis'\np6\nsS'callback'\np7\nNsS'data'\np8\n(dp9\ng8\nS'you will see this if success'\np10\nssS'method'\np11\nS'emit'\np12\ns."
I feel kind of exciting and upset, did it mean flask-socketio not suit for windows?
from flask-socketio.
Hmm. Not sure why it does not work for you on Windows. I've used redis on Windows many times and never had any problem. I'll see if I can reproduce this problem later when I have a Windows machine at hand.
from flask-socketio.
I hope the html and fapp.py code above will help you reproduce this. I do more tests, and find redis will get the message only when I restart redis, but demo page cannot receive the message. If now I restart flask, redis will stop receive message and flask hangs again.
I try this sample on my own pc, and will try on another pure windows again, and back to you.
windows redis version 3.2.100
from flask-socketio.
Are you sure you are not seeing some odd interaction with the firewall? Try disabling it completely for your test.
from flask-socketio.
yes, firewall is always disabled on my PC, so...
from flask-socketio.
now I change to redis 2.8.4, which worked on Ubuntu. Still I can receive the message on redis, but the client cannot receive the emit message.
I suppose the following output is the right thing redis should show:
D:\redis64-2.8.4>redis-cli.exe
psubscribe *
psubscribe
*
1
pmessage
*
flask-socketio
(dp0
S'skip_sid'
p1
NsS'room'
p2
NsS'namespace'
p3
S'/'
p4
sS'event'
p5
S'redis'
p6
sS'callback'
p7
NsS'data'
p8
(dp9
g8
S'you will see this if success'
p10
ssS'method'
p11
S'emit'
p12
s.
from flask-socketio.
The client receiving the message or not has not really much to do with redis, since the client does not directly connect to the queue.
In this situation, the Flask server is the redis client who subscribes to these messages. Are you configuring the queue properly on the side of the server? You can run the server with additional logging to see more runtime information.
from flask-socketio.
right before I just thought to create a venv to run and that makes sence. Maybe there is package conflicts? If I run the demo with few packages:
Flask 0.11.1
Flask-SocketIO 2.7.2
Jinja2 2.8
MarkupSafe 0.23
Werkzeug 0.11.11
click 6.6
itsdangerous 0.24
pip 7.1.0
python-engineio 1.0.3
python-socketio 1.6.0
redis 2.10.5
setuptools 18.0.1
six 1.10.0
wheel 0.24.0
demo will run without error
but if I just mass the demo with my default python env packages, hangs. So wired.
c:\Program Files\Redis>pip list
amqp (1.4.9)
anyjson (0.3.3)
argh (0.26.2)
backports-abc (0.4)
billiard (3.3.0.23)
celery (3.1.23)
certifi (2016.9.26)
colorama (0.3.7)
coverage (4.2)
decorator (4.0.10)
eventlet (0.19.0)
Flask (0.10.1)
Flask-HTTPAuth (3.1.2)
Flask-SocketIO (2.7.1)
Flask-SQLAlchemy (2.0)
Flask-Testing (0.5.0)
Flask-WTF (0.12)
ftputil (3.3.1)
gevent (1.1.1)
graphviz (0.4.10)
greenlet (0.4.10)
gunicorn (19.6.0)
iptools (0.6.1)
itsdangerous (0.24)
Jinja2 (2.7.3)
jsonify (0.5)
keyring (9.3.1)
kombu (3.0.35)
MarkupSafe (0.23)
netifaces (0.10.5)
passlib (1.6.5)
pathtools (0.1.2)
pbr (1.10.0)
pip (8.1.2)
psutil (4.2.0)
psycogreen (1.0)
psycopg2 (2.6.2)
pycallgraph (1.0.1)
pycrypto (2.6.1)
pyftpsync (1.0.3)
python-engineio (1.0.3)
python-openid (2.2.5)
python-socketio (1.6.0)
pytz (2016.4)
pywin32 (220)
pywin32-ctypes (0.0.1)
PyYAML (3.11)
redis (2.10.5)
requests (2.10.0)
setuptools (16.0)
simplejson (3.9.0)
singledispatch (3.4.0.3)
six (1.9.0)
SQLAlchemy (1.0.13)
sqlalchemy-migrate (0.10.0)
sqlparse (0.1.19)
Tempita (0.5.2)
Unipath (1.1)
vboxapi (1.0)
watchdog (0.8.3)
Werkzeug (0.10.4)
WMI (1.4.9)
wol (0.2)
wolframalpha (2.4)
WTForms (2.1)
xmltodict (0.10.2)
from flask-socketio.
@wiwengweng Remember when I said this, near the top of this thread?
If you are using eventlet or gevent, then you need to monkey patch the Python library for the redis package to work.
You are using eventlet it seems, so to use Redis you need to monkey patch the python standard library.
from flask-socketio.
well, I am not sure how did eventlet get used. The demo is clear that it runs only depending on flask and flask-socketio. I run this code on pycharm IDE. Will this bother my env?? btw, how can I run the server with additional logging? Thanks
from flask-socketio.
If you install eventlet or gevent in your virtualenv, then Flask-SocketIO will use them (by default, you can disable this by selecting an async_mode
explicitly). See the description of the async_mode
argument in https://flask-socketio.readthedocs.io/en/latest/#flask_socketio.SocketIO. Running a Flask-SocketIO server without eventlet or gevent makes no practical sense other than for quick tests.
how can I run the server with additional logging?
This is the engineio_logger
argument, also documented in the page I linked above.
from flask-socketio.
Great!! I also check your doc during our great fire wall is enabled(sorry, maybe you won't care what is GFW). Yes, you're right. In my real project we are using gevent, so I explicitly enable async_mode='gevent', in my project, everything goes~ Thanks. I will continue following your project to check out more amazing features~! Thank you again!
from flask-socketio.
Related Issues (20)
- TypeError: <engineio.packet.Packet object at 0xabec08e0> is not a byte HOT 1
- Add support for redis sentinel cluster HOT 2
- Test cookie handling API changed in Werkzeug 2.3 HOT 1
- Unable to connect to namespaces other than the main namespace HOT 1
- Cannot start multiple threads HOT 1
- Support for once to listen for an event only 1 time (Like official socket.io server has) HOT 1
- Error when using gevent with allow_unsafe_werkzeug HOT 2
- Option to redirect http to https instead of raising errors HOT 1
- Strange Behavior with RabbitMQ and Flask-SocketIO HOT 4
- get_received(namespace=None) does not returns the messages HOT 5
- Add type hints HOT 9
- Socket.IO test client does not run background task HOT 7
- gevent - cannot enter into function. HOT 1
- Multiple SocketIO instances connected to the same Flask server HOT 1
- python-engineio and python-socketio version matching with Flask-SocketIO >= 5.3.0 HOT 1
- Flask-SocketIO cannot run in ws HOT 1
- pyinstaller Packaging error HOT 1
- How do I intercept event events and authenticate them in a unified manner?
- Waiting for client events leads to timeouts
- Sending events in a separate thread with delays does not send HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flask-socketio.