ssl.SSLEOFError: EOF occurred in violation of protocol about websocket-client

[bubbleboy14]

@gborrageiro-ld @andrea-mucci @engn33r

I put together a websocket throughput test in dez, which is an async network stack built on rel. The new test is in the latest version, 0.10.10.1, which can be installed thusly:

pip install dez

To run the test, first run this:

dez_test wsecho

That starts up a pretty basic websocket echo server (code) with a couple extra features.

Then, in a different terminal, run:

dez_test wsc

That (code) will prompt you to select a test configuration (high throughput in upstream/downstream/both/neither directions). It will then start up a WebSocketApp (with rel dispatcher) and proceed to direct heavy traffic in the specified direction. Anyway, it runs fine for me in every configuration.

Is anyone able to reproduce the issue using this test? Feel free to modify the code. Or is there a different websocket server that has this issue? For whatever reason, it doesn't seem to be a problem with dez.

Anyway, this shouldn't be hard to fix, if anyone can come up with a reliable repro - thoughts and test cases welcome!

[bubbleboy14]

@gborrageiro-ld thanks for the detailed report! Do you get the same SSLEOFError with the default (unspecified) dispatcher - that is, connected to the same endpoint, but without rel?

Also, since it looks like it's throwing this error in the course of trying to send a pong, I'm wondering whether the same problem is still encountered if you don't specify ping_interval and ping_timeout.

[gborrageiro-ld]

hi @bubbleboy14,

I have two streams, one authenticated and the other is unauthenticated. I depend on a dispatcher like rel to handle multiple streams; what alternatives would you suggest?
If I remove the ping/pong, then the exception does eventually occur, yes.

Thanks
Gabriel

[gborrageiro-ld]

I removed rel and let the busy, unauthenticated stream run, and I do not see this segmentation fault.

[bubbleboy14]

Hey @gborrageiro-ld I created a standalone test from the example code you provided:

import rel, websocket

BASEURL = "wss://fstream.binance.com/stream?streams=btcusdt@bookTicker/btcusdt@depth20@100ms/btcusdt@aggTrade/btcusdt@markPrice@1s/btcusdt@"
variants = ["bookTicker", "forceOrder"]

def onmsg(ws, msg):
    print("received:", msg)

for v in variants:
    ws = websocket.WebSocketApp(BASEURL + v, on_message=onmsg)
    ws.run_forever(dispatcher=rel, reconnect=3, ping_interval=180, ping_timeout=60)

rel.dispatch()

I've been running it for a while, and I haven't gotten that SSLEOFError. Does it take a long time? Are you able to reproduce the issue with this test case? If not, could you modify it to produce the error?

Anyway, thanks a million for the excellent report, we should be able to track this one down. LMK how it goes!

[andrea-mucci]

Hi,

I have the same error but with a different approach

i have a client that only send data to a server

.....
self.socket = websocket.WebSocket()
self.socket.connect(self.url)

#this is the method that send a command
def command(content):
     self.socket.send(....)

nothing special, the problem is that if i start to send messages from a loop, like:

for item in items:
     __method_that_send_ws_message()

in a certin point a get a failure:

  File "/home/andrea/PycharmProjects/ArchAI/archai/venv/lib/python3.10/site-packages/websocket/_core.py", line 283, in send
    return self.send_frame(frame)
  File "/home/andrea/PycharmProjects/ArchAI/archai/venv/lib/python3.10/site-packages/websocket/_core.py", line 311, in send_frame
    l = self._send(data)
  File "/home/andrea/PycharmProjects/ArchAI/archai/venv/lib/python3.10/site-packages/websocket/_core.py", line 525, in _send
    return send(self.sock, data)
  File "/home/andrea/PycharmProjects/ArchAI/archai/venv/lib/python3.10/site-packages/websocket/_socket.py", line 170, in send
    return _send()
  File "/home/andrea/PycharmProjects/ArchAI/archai/venv/lib/python3.10/site-packages/websocket/_socket.py", line 147, in _send
    return sock.send(data)
  File "/usr/lib/python3.10/ssl.py", line 1206, in send
    return self._sslobj.write(data)
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2426)

the strange thing is that before the error i have already sent a lot of messages, the problem come if a send the messages from a loop in a method

[bubbleboy14]

@andrea-mucci hm, interesting. the python ssl docs say:

A subclass of SSLError raised when the SSL connection has been terminated abruptly. Generally, you shouldn’t try to reuse the underlying transport when this error is encountered.

i don't know enough about the inner workings (of python's ssl module) to say how wide an array of circumstances (beyond a simple connection drop) can lead to this.

however, since you and @gborrageiro-ld both describe high throughput situations, i wonder whether, perhaps, some buffer is filling up... i mean, it really shouldn't be something like that. do you have any thoughts on this, @engn33r ?

i notice you're both running py3.10 - this may be related.

@andrea-mucci are you using the rel dispatcher?

can anyone provide a concise repro?

[engn33r]

Checking any search engine for "python EOF occurred in violation of protocol" shows this a common error message and often related to the SSL library. I suggest sharing the OS that you are using (Windows, Mac, *nix) and the output of these commands when such an SSL error is encountered:

openssl version
python3 -c "import ssl; print(ssl.OPENSSL_VERSION); import websocket; print(websocket.__version__)"
python3 -V

[gborrageiro-ld]

I think SSL error might be a red herring. I think that rel crashes when any of the WS streams are very busy / return a lot of data quickly. Dr Gabriel Borrageiro Senior Quantitative Portfolio Manager Laser Digital

…

________________________________ From: engn33r ***@***.***> Sent: Sunday, September 10, 2023 10:42:14 PM To: websocket-client/websocket-client ***@***.***> Cc: Gabriel Borrageiro ***@***.***>; Mention ***@***.***> Subject: Re: [websocket-client/websocket-client] ssl.SSLEOFError: EOF occurred in violation of protocol (Issue #942) Checking any search engine for "python EOF occurred in violation of protocol" shows this a common error message and often related to the SSL library. I suggest sharing the OS that you are using (Windows, Mac, *nix) and the output of these commands NkdkJdXPPEBannerStart External Sender - From: (engn33r ***@***.***>)<https://spambrella.cloud-protect.net/email-details/?k=k1&payload=53616c7465645f5f08b5bebb288f6c82050ba47dc706c640491676361fdbfebe870235497ff17e2d625e0d1bb885d2bc84362ff4d68126208494c2b3f49994d73e127e014b7ec09005cfbfe7c9d0ce4f8954bf801efd12e74be0f9acf06ac56c1bd536557cec982a96910cacd6638226b31a3ad68c7bf734e01f60d1cbbed945168bee6641a3b4fb6380ccf34843dc31580a27d34afee617b32b4b06d4e181a9c3ff8d45c56745ccb3206d71863e05236e71ce12d0a890d3239b60d177c34d3200129fdf785c6fffabe9fa68cf74a6983c5ad65e3c7659f6376fd7a38170a6eb22eae7eb73098329242f6710334ce832> This message came from outside your organisation. Learn More<https://spambrella.cloud-protect.net/email-details/?k=k1&payload=53616c7465645f5f08b5bebb288f6c82050ba47dc706c640491676361fdbfebe870235497ff17e2d625e0d1bb885d2bc84362ff4d68126208494c2b3f49994d73e127e014b7ec09005cfbfe7c9d0ce4f8954bf801efd12e74be0f9acf06ac56c1bd536557cec982a96910cacd6638226b31a3ad68c7bf734e01f60d1cbbed945168bee6641a3b4fb6380ccf34843dc31580a27d34afee617b32b4b06d4e181a9c3ff8d45c56745ccb3206d71863e05236e71ce12d0a890d3239b60d177c34d3200129fdf785c6fffabe9fa68cf74a6983c5ad65e3c7659f6376fd7a38170a6eb22eae7eb73098329242f6710334ce832> Do not click any links, open attachments, or reply if you do not believe that the email is legitimate. If in doubt, report using the report email button NkdkJdXPPEBannerEnd Checking any search engine for "python EOF occurred in violation of protocol" shows this a common error message and often related to the SSL library. I suggest sharing the OS that you are using (Windows, Mac, *nix) and the output of these commands when such an SSL error is encountered: openssl version python3 -c "import ssl; print(ssl.OPENSSL_VERSION)" python3 -V — Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_websocket-2Dclient_websocket-2Dclient_issues_942-23issuecomment-2D1712908531&d=DwMCaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=jHSk2w3iI_5gQOIDQ6L3stXnXofiJhZ43_UzHLOdT71DfGoMStAgy-CWLDRnfG0F&m=qeezOWT66tnJ_zU1nMrfOnRIR-Wx4UxLwCdf_CPE-uRGJe0TjGUJgQ23qKkfnBMS&s=puPJ7Iw8BC63gyzHP63bDkizLXunvr-wi0YV4_JEiKI&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_BBUWXOSPFLPDG2GGNXVSZRTXZYCYNANCNFSM6AAAAAA3N3IBRU&d=DwMCaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=jHSk2w3iI_5gQOIDQ6L3stXnXofiJhZ43_UzHLOdT71DfGoMStAgy-CWLDRnfG0F&m=qeezOWT66tnJ_zU1nMrfOnRIR-Wx4UxLwCdf_CPE-uRGJe0TjGUJgQ23qKkfnBMS&s=fcKTaWE24qfVR_IblDmlezwbfIZrEj0ajnN1lqDAMVQ&e=>. You are receiving this because you were mentioned.Message ID: ***@***.***>

________________________________ This email has been scanned for spam and viruses. Click here<https://spambrella.cloud-protect.net/index01.php?mod_id=11&mod_option=logitem&report=1&type=easyspam&k=k1&payload=53616c7465645f5fcf3902304698b850c506a635de29d9eabe2be8aa3ca3141a76e3a475c4459967c81634555c7d345c7a116d8b267bd8b135d6f19f9b8a0b614d5df62b68fdb3021ec75525c0ad65083521332fa9facd72bc09ade37a74d6e0bb486c78af1f0ee9d4144dd69f753aab113fe318341f0e3c4ec1fc7df2d92b7d78508de6e548608fa274af9dc3821d181c9dfea7dca6ba2c7483390f5036a965971501bccd96745a275da8f00f16ccc7> to report this email as spam. Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.

[bubbleboy14]

@engn33r @gborrageiro-ld @andrea-mucci

I just encountered this issue myself! This patch fixed it for me:

#961

Does it fix things for you guys?

It's just two small changes in read() (in run_forever()): 1) check for SSLEOFError; 2) pass bool(reconnect) to handleDisconnect().

@engn33r, these seem like logical tweaks, right? Fixes the issue for me, anyway.

LMK what you think!