Support for websocket? about smooth

[PerMalmberg]

@squonk11 I'm actually currently thinking about how to best implement minimal, but functional, websocket support. If you have suggestions on how, feel free to contribute. PRs are accepted.

[squonk11]

Unfortunately I am not a professional SW engineer but a privately interested enthusiast. I implemented already the websocket protocol into this repository. But since this is Arduino I am asking myself if it is easier to port this to esp-idf or implement websocket into "smooth".
I had a short look into your sources but until now I did not get the point how things work together. I see that you are using some quite sophisticated features of C++ which I am currently not so familiar with (as I said: I am not a professional SW engineer). Somewhere there must be the analysis of the received header - here the detection of the websocket connection must be implemented. Then the websocket connection must not be closed after the response - here it is also important to understand how this can be implemented.
Maybe you have some explanations which makes it easier for me to understand your sourecode?

[PerMalmberg]

some quite sophisticated features of C++

Not really, mainly template and polymorphism. I'm sure you can understand it given time and a bit of help :)

The main classes of interest are likely application::network::http::HTTPProtocol and application::network::http::HTTPServerClient. As everything is templated all the way down to the core::network::Socket class, I'm thinking its probably best to extend the HTTPProtocol such that it can operate in two modes - regular HTTP verbs and Websocket. This can probably be done using either inheritance or composition; I'm thinking that the later is the better option since Websockets really aren't the same protocol has regular HTTP.

HTTPProtocol::data_received is likely the place to handle eventual upgrade-requests and once detected, switch "modes" and start delivering data in a format suitable for websockets to the applications. Currently HTTPPacket is used for this, so either that has to be changed, or some other way has to be found. An important part of the Smooth framework is that everything is event based so this will have to be too, meaning that just as there may be several HTTPPacket received by the application (e.g. a file is POST:ed by the client) in a single requests, websockets may have to split data into several parts too (unless Websockets guarantee a max packet size?)

Keep-alive is already property of the HTTPSServerClient so that functionality is easily achieved.

I'll be thinking about this a bit more while doing some gardening. Btw, I'm usually on IRC in #ESP32 @ freenode, under nick "permal" if you want to chat.

[PerMalmberg]

So I gave this some thought. I think the proper way is to rework the HTTPProtocol class such that it holds either a "regular HTTP protocol" or a "websocket protocol" implementation, but only one at a time. This way we can keep memory usage to a minimum, while still allowing an upgrade transition at runtime. It also plays nicely with the templatization.

[PerMalmberg]

@squonk11 As you can see, I've begun an implementation of websockets, currently working on the connection upgrade. Do you happen to have a good debugging tool for it, Firefox doesn't give any info on why it doesn't accept the websocket connection from /websocket.html, generated here?

[squonk11]

Sorry for not answering yesterday - Saturdays and/or Sundays I have to take care for my father.
Thank you for already taking care of this topic. I think today I will have some time to take a look at the work you have done so far.

[PerMalmberg]

No worries, @squonk11. IRL is important :)

[squonk11]

Sorry, but I did not get much time to check your sourcecode. What I have seen so far is that you already provided the skeleton for websockets and the detection of the upgrade. But reading the RegularHTTPProtocol.cpp (in order to make a similar WebsocketProtocol.cpp) is still a bit difficult for me as I am not so familiar with templates. I think I need to read first a little bit about using templates.
Additionally I am doing such work during my free time (which currently is quite small) - I would like to support/co-work with you on this but it will take time...

[PerMalmberg]

@squonk11 Its OK. I wanted this functionality myself so don't feel you owe me anything. I'm currently working out how to decode the incoming frames and how to re-use/transform the HTTPPacket class to act as a carrier for both regular HTTP and Websocket data.

If you have any gotchas from when you implemented the protocol feel free to share them.

FYI: I'm not going to support data frames with sizes given by 64 bits, it just doesn't make sense for an embedded device.

[squonk11]

When I implemented the websocket server I implemented a WebsocketHandler class with virtual functions for onMessage, onClose and onError which the end-user has to override. For reading the data I was using a std::streambuf.
I encountered two pitfalls during the implementation:

1. The extended payload length in the websocket frame is a 16 bit (or 64 bit) value in network byte order; so, that ntohs() (or ntohl() ) have to be used.
2. The received data has to be de-masked only if the mask bit in the websocket frame is set.

Your approach not to implement frame sizes with 64bit size probably makes sense - although in my application I have to transmit a file with a size of approx. 170kB. Here 64k (16bit) is too small. In order to handle that amount of data I will be using a Wroover module with 8MB PSRAM. But I think I can also cut the file into pieces of <64kB.

[PerMalmberg]

I realized that handling 65k or larger is no additional work compared to handling <65k so I implemented it.

[PerMalmberg]

@squonk11 So the code is now in a state where its possible to send messages of three different sizes from the /websocket.html page, and for each message the server will respond with a hard-coded message. Next step is to add a way to easily receive the data and send data on the same channel from the application. I'm not yet sure how to do that yet.

[squonk11]

I see you are extremely fast. For me this step would have taken much longer.
From the application point of view it would be nice to have a class WSHandler with virtual functions like onMessage, onError, onClose, etc. which the user has to override. The parameter passed to onMessage could be a std::streambuf from which the application can read. Maybe you can have a look at this repository in the classes WebSocketInputStreambuf and WebsocketHandler. In WebsocketHandler there are also functions like send() or close(). I remember a "peculiarity" in the closing of the websocket via the WebsocketHandler has the disadvantage that here memory has to be freed which was malloced in another class/function. This is not a nice style of coding. But I did not find a better solution for this. Feel free to ask if you need details on the solution there.
But other implementations are also o.k for me if you have better ideas or special requirements.

[PerMalmberg]

@squonk11 Well, I know my way around the system.

Just tried running it on the ESP32, it crashes when pressing the "Send 65k" button, with the following message, "Send short" button works so not sure what is going on.

Guru Meditation Error: Core  0 panic'ed (LoadProhibited). Exception was unhandled.
Core 0 register dump:
PC      : 0x40091cd0  PS      : 0x00060533  A0      : 0x800922c6  A1      : 0x3ffd8a50  
0x40091cd0: is_free at /home/permal/esp/esp-idf/components/heap/multi_heap.c:131
 (inlined by) multi_heap_malloc_impl at /home/permal/esp/esp-idf/components/heap/multi_heap.c:432

A2      : 0x3f8046c8  A3      : 0x000006a8  A4      : 0x3f82c284  A5      : 0x000000fe  
A6      : 0x00000001  A7      : 0x00000000  A8      : 0x00000000  A9      : 0xba465551  
A10     : 0x3f82fd80  A11     : 0x0000081c  A12     : 0x3f82fd84  A13     : 0x3f82bc90  
A14     : 0x00000001  A15     : 0x00000000  SAR     : 0x00000019  EXCCAUSE: 0x0000001c  
EXCVADDR: 0xba465551  LBEG    : 0x400914fc  LEND    : 0x40091512  LCOUNT  : 0xffffffff  
0x400914fc: memcpy at /builds/idf/crosstool-NG/.build/xtensa-esp32-elf/src/newlib/newlib/libc/machine/xtensa/memcpy.S:146

0x40091512: memcpy at /builds/idf/crosstool-NG/.build/xtensa-esp32-elf/src/newlib/newlib/libc/machine/xtensa/memcpy.S:161


ELF file SHA256: b128111c039163747f16df4696be3f089a81025c511728f6ae254b877c797ceb

Backtrace: 0x40091ccd:0x3ffd8a50 0x400922c3:0x3ffd8a80 0x40086a99:0x3ffd8ab0 0x40086ba5:0x3ffd8ae0 0x40081d7d:0x3ffd8b30 0x401aa093:0x3ffd8b60 0x401b476b:0x3ffd8ba0 0x4008c9c5:0x3ffd8c20 0x4009847e:0x3ffd8c60
0x40091ccd: multi_heap_assert at /home/permal/esp/esp-idf/components/heap/multi_heap_platform.h:54
 (inlined by) multi_heap_malloc_impl at /home/permal/esp/esp-idf/components/heap/multi_heap.c:431

0x400922c3: multi_heap_malloc at /home/permal/esp/esp-idf/components/heap/multi_heap_poisoning.c:191

0x40086a99: heap_caps_malloc at /home/permal/esp/esp-idf/components/heap/heap_caps.c:111

0x40086ba5: heap_caps_malloc_prefer at /home/permal/esp/esp-idf/components/heap/heap_caps.c:189

0x40081d7d: wifi_malloc at /home/permal/esp/esp-idf/components/esp32/esp_adapter.c:66

0x401aa093: esf_buf_alloc at ??:?

0x401b476b: wdevProcessRxSucDataAll at ??:?

0x4008c9c5: ppTask at ??:?

0x4009847e: vPortTaskWrapper at /home/permal/esp/esp-idf/components/freertos/port.c:143

[PerMalmberg]

Tried to find what caused the error....three hours later....updated ESP-IDF, recompiled and it no longer crashes. I have no idea what the issue was. I ran the https_server/websocket example on my Linux machine with ASAN enabled without any issues..... The update contained new wifi blobs, could have been part of that..... 😕

[squonk11]

Since there are many functions concerning memory allocation involved I think there was a problem with the higher amount of data for the >65k. I agree that it is not really satisfying if an error suddenly disappears - it's better to know what happened and what fixed it. But this will be difficult to figure out...
Another thing I encountered in the past in recent ESP32 projects was that the stack was selected too small and thus the ESP crashed resulting in stange error messages which were not really related to the real problem.

[PerMalmberg]

Yeah, problems just disappearing are not fun, but I'm going to but it down the the outstanding memory cache issue logged in ESP-IDF, the same one that likely prevents TLS from functioning on the ESP when compiling with gcc8, which I am.

I'll be pushing an update in about an hour containing a near complete implementation.

[PerMalmberg]

Think I've found the cause of the crash, fixing that before pushing.

[PerMalmberg]

@squonk11 Alright, fixed the issue, all my fault. 😮

This line shows how to tell the framework what websocket server type to use for the specified API path. You asked for things like on_connect/on_disconnect events, but they aren't needed - when the WebsocketServer (and derived class) is created, that's when the connection has been established, likewise on disconnect, the instance is destructed so the d'ctor is where to do cleanup.

This line shows how to send a response to the client.

[squonk11]

The first "This line shows" in your last post has no link. I guess it is: this?
As I said I am not so familiar with templates - I need to study your code and play with it a while in order to understand everything.

[PerMalmberg]

Ops, yes sorry, I missed to add the link.

Here's how it works:

WSEchoServer inherits from the WebsocketServer, which allows the framework to treat it as a WebsocketServer, regardless of what the application programmer puts into it, i.e. regular base classe with polymorphism.

The line you rightly point to tells the framework that any upgrade request made to the /chat path will result in the provided WebsocketServer, which in this case actually is a WSEchoServer to be used to handle that specific connection.

[PerMalmberg]

@squonk11 With the last few adjustment it feels pretty stable. Please give it a try and let me know what you think/find.

[squonk11]

o.k. I will have a try during the next days.

[PerMalmberg]

o.k. I will have a try during the next days.

Cool. I'll be mostly offline for a week, starting Thursday so replies may be slow or none at all during that time.

[PerMalmberg]

Funny how things can feel stable for so long and then something pops up that totally pulls the rug from under your feet. I started running the websocket stuff over TLS and experienced SIGSEVs. Turns out SecureSocket had an issue. Also, When ASAN was enabled, performance was horrible without enabling optimizations. It's all fixed in the last push.

[PerMalmberg]

Last commit for a week. I implemented a working Ping/Pong, but destruction of the WSEchoServer crashes (as expected) since queues are not designed to be destructed once they have been created. There's two solution to this: 1) Use a pool for of the WS servers, or 2) redesign the queues. The first one gives the benefit of being memory static, the seconds one is likely a lot of work but also something I've been wanting to do for a while. I'll have to think on what's the best cause of action.

[PerMalmberg]

Note to myself. If QueueNotification holds std::weak_ptrinstead ofITaskEventQueue* in itsqueuesmember and allTaskEventQueueare created on the heap via a factory method (same as the timers are), thenQueueNotification::wait_for_notification()can check if the queue still is valid viastd::shared_ptr::lock()` which would prevent further use of that queue once it has been destructed.

This looks to be a fairly small change. It breaks all uses of TaskEventQueue and ISRTaskEventQueue, but I think that it'll be worth it.

[squonk11]

I am currently starting to test your WSEchoServer. But before actually using WSEchoServer code I am already running into problems a soon as I include the feature/49-websocket branch into the esp-idf components: when I try to compile any existing projects I get the error message
Syntax error in cmake code at C:/Users/springob/esp/test/build/build_properties.temp.cmake:2 when parsing string C:\Python27\python.exe Invalid escape sequence \P
Obviously there is a problem with setting the path to Python.exe. The cmake files contains the Python path with "" (windows style) all other paths are using "/" (unix style).
Do you have a hint what is going wrong?
Just for information: I am using CMake under windows to compile.

[PerMalmberg]

No idea, I'm on Linux and have never tried using this on Windows. For reference, here's how I build and run the test projects:

cd path/to/root_of_smooth
(select test project in test/CMakeLists.txt)
mkdir build
cd build
cmake .. -G Ninja -DESP_PLATFORM=1 -DCMAKE_TOOLCHAIN_FILE=$IDF_PATH/tools/cmake/toolchain-esp32.cmake -DTARGET=esp32 && ninja clean && ninja -j 8
~/esp/esp-idf/tools/idf.py -C .. -p /dev/ttyUSB1 flash monitor

[squonk11]

Unfortunately I have no performant Linux machine available. I could try with WSL under Windows - but this is just some kind of Linux-emulator and compiling is really slow. I will at least try in order to see if it works this way.

[squonk11]

at least menuconfig and compilation of existing projects works under WSL. But in WSL I additionally have the problem with accessing the serial interfaces and the esp-idf provided monitor program can not be stopped by Ctrl-] because the key is not recognized.
I think I have to install a real Linux on an external SSD in order to continue. That will take me some time in order to set everything up correctly (especially my three monitors under Linux might be a problem...).

[PerMalmberg]

Or install it in a VM under Hyper-V, or Virtual Box?

[squonk11]

I tried with VirtualBox but the serial interface does not work properly - only one out of five flash attempts work. That is not really satisfying. I also tried to fix the problem under WSL - but also without success. Now I setup your sources inside a "real" esp-idf component using the cmake build system provided by espressif under windows. This now compiles without error.
When flashing it I try to access via http and https. Both have its problems:

1. When I try to access the web_root on the SD-card using HTTP I get the error message 404 and in the monitor I see: I (137995) ServerSocket: Connection accepted I (138046) HTTPServer: Request: GET: '/' I (138049) HTTPServer: Reply: Not Found W (143063) SocketDispatcher: Receive timeout on socket 0x3fff1c10 (5000 ms) I (143064) Socket: [, 0, 56, 0x3fff1c10]: Socket stopping I (143079) Socket: [, 0, -1, 0x3fff1c10]: Disconnected
   Adding "/index.html" to the URL also does not help.

2. When I try to access web_root using HTTPS I get the error message Secure Connection Failed. An error occurred during a connection to 192.168.1.31:8443. SSL peer reports incorrect Message Authentication Code. Error code: SSL_ERROR_BAD_MAC_ALERT in the browser and in the monitor I see I (80137) ServerSocket: Connection accepted E (82301) SecureSocket: mbedtls_ssl_handshake_step returned -29056: SSL - Verification of the message MAC failed I (82302) Socket: [, 0, 56, 0x3fff1f64]: Socket stopping I (82308) Socket: [, 0, -1, 0x3fff1f64]: Disconnected

Do you have any hints?

[PerMalmberg]

1. Did you copy anything to the Web root folder on the SD card? 2. Yes, that's an outstanding issue in IDF, should be linked in another issue in this repo.

…

On Tue, 23 Jul 2019, 22:10 Lothar, ***@***.***> wrote: I tried with VirtualBox but the serial interface does not work properly - only one out of five flash attempts work. That is not really satisfying. I also tried to fix the problem under WSL - but also without success. Now I setup your sources inside a "real" esp-idf component using the cmake build system provided by espressif under windows. This now compiles without error. When flashing it I try to access via http and https. Both have its problems: 1. When I try to access the web_root on the SD-card using HTTP I get the error message 404 and in the monitor I see: I (137995) ServerSocket: Connection accepted I (138046) HTTPServer: Request: GET: '/' I (138049) HTTPServer: Reply: Not Found W (143063) SocketDispatcher: Receive timeout on socket 0x3fff1c10 (5000 ms) I (143064) Socket: [, 0, 56, 0x3fff1c10]: Socket stopping I (143079) Socket: [, 0, -1, 0x3fff1c10]: Disconnected Adding "/index.html" to the URL also does not help. 2. When I try to access web_root using HTTPS I get the error message Secure Connection Failed. An error occurred during a connection to 192.168.1.31:8443. SSL peer reports incorrect Message Authentication Code. Error code: SSL_ERROR_BAD_MAC_ALERT in the browser and in the monitor I see I (80137) ServerSocket: Connection accepted E (82301) SecureSocket: mbedtls_ssl_handshake_step returned -29056: SSL - Verification of the message MAC failed I (82302) Socket: [, 0, 56, 0x3fff1f64]: Socket stopping I (82308) Socket: [, 0, -1, 0x3fff1f64]: Disconnected Do you have any hints? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#49?email_source=notifications&email_token=AAU2LLC5URTCWVQR4IONNUDQA5JSVA5CNFSM4ICLCPR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2UEGCI#issuecomment-514343689>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAU2LLEKVGIC76CWSPP7H33QA5JSVANCNFSM4ICLCPRQ> .

[squonk11]

ad 1: yes, I copied the contents of this into the folder "web_root" on the SD card. I also tried to copy this contents into the root folder and into "sdcard/web_root" - but nothing helps.

[squonk11]

I see that you activate the following handlers in the end of main:
insecure_server->on(HTTPMethod::GET, "/api/blob", blob); insecure_server->on(HTTPMethod::POST, "/upload", upload); insecure_server->enable_websocket_on<WSEchoServer>("/echo");
maybe I need to add a handler to "/" ?

[PerMalmberg]

Is the SD card getting mounted properly? It should work as-is, no need to add handler for "/".

…

On Tue, 23 Jul 2019, 22:45 Lothar, ***@***.***> wrote: ad 1: yes, I copied the contents of this <https://github.com/PerMalmberg/Smooth/tree/master/test/http_server_test/web_root> into the folder "web_root" on the SD card. I also tried to copy this contents into the root folder and into "sdcard/web_root" - but nothing helps. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#49?email_source=notifications&email_token=AAU2LLHLBDOMN4QU5PFGMLLQA5NWDA5CNFSM4ICLCPR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2UHIFQ#issuecomment-514356246>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAU2LLHABPDTDTZLORAZ4PLQA5NWDANCNFSM4ICLCPRQ> .

[squonk11]

yes, the SD card gets detected properly. During the boot there is a message saying that it works with 20MHz and it gives the correct size.
Maybe this evening I could try to read and write the SD card using the functions provided by your framework. If this works the problem is somewhere else.

[PerMalmberg]

How is the SD card formated? FAT32? Also make sure you have long filenames enabled in menuconfig.

…

On Wed, 24 Jul 2019, 10:51 Lothar, ***@***.***> wrote: yes, the SD card gets detected properly. During the boot there is a message saying that it works with 20MHz and it gives the correct size. Maybe this evening I could try to read and write the SD card using the functions provided by your framework. If this works the problem is somewhere else. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#49?email_source=notifications&email_token=AAU2LLGVDA3YKWSCKIHQ2XDQBACZRA5CNFSM4ICLCPR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2VPVLY#issuecomment-514521775>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAU2LLDBPVSSJH3426MWQ33QBACZRANCNFSM4ICLCPRQ> .

[squonk11]

yes, long filenames could be the issue. I remember that 8.3 filenames are configured by default and I did not change this setting in menuconfig. I will check this this evening but probably this is the issue.

[PerMalmberg]

If you're using the sdkconfig that comes with smooth it should have it enabled....

…

On Wed, 24 Jul 2019, 12:56 Lothar, ***@***.***> wrote: yes, long filenames could be the issue. I remember that 8.3 filenames are configured by default and I did not change this setting in menuconfig. I will check this this evening but probably this is the issue. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#49?email_source=notifications&email_token=AAU2LLGEK6NOM2LFS6IGSCDQBAROFA5CNFSM4ICLCPR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2V2ONY#issuecomment-514565943>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAU2LLB7CKCDDKVEFFRXSULQBAROFANCNFSM4ICLCPRQ> .

[squonk11]

unfortunately I was not using the sdkconfig of smooth because I was not sure if it is 100% compatible with esp-idf. So I setup a new one. During this process I also had to set some configurations concerning stack of main() and memory for the TLS data (using PSRAM) because I encountered some errors due to that.
Do you know if there are any other settings I should take care of?

[squonk11]

The long filenames were the problem. But I then had problems loading the complete page - the ESP32 crashed while loading the icons. Then I compared the sdkconfig of Smooth to mine and made some corrections. Now the page loads without problems.
When testing the Websockets I encounter the problem that e.g. for 65k not the total amount of data is transferred and the websocket connection gets disconnected. I will check the sdkconfig again.

[PerMalmberg]

So what happens when not all data is loaded? I've not seen any such problems...doesn't mean there are none though...

…

On Wed, 24 Jul 2019, 21:10 Lothar, ***@***.***> wrote: The long filenames were the problem. But I then had problems loading the complete page - the ESP32 crashed while loading the icons. Then I compared the sdkconfig of Smooth to mine and made some corrections. Now the page loads without problems. When testing the Websockets I encounter the problem that e.g. for 65k not the total amount of data is transferred and the websocket connection gets disconnected. I will check the sdkconfig again. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#49?email_source=notifications&email_token=AAU2LLEPIBCODNGG5RYFRSDQBCLILA5CNFSM4ICLCPR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2XFAVI#issuecomment-514740309>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAU2LLC34T6XWID56CJRKOTQBCLILANCNFSM4ICLCPRQ> .

[squonk11]

I must correct myself: I checked with wireshark and I see that the data fully arrives. After the transmission the websocket connection is being closed - this is what I did not expect. I see that the connection is not being closed by the client (browser); so, the framework closes it. I would expect that the connection persists until a timeout elapses or until I click the "disconnect" button.
If I send 1M the ESP32 crashes. Is that correct?

[PerMalmberg]

I'd expect that too, and that is what I have experienced. I've been clicking those buttons until they wore out, trying to break things... What's in the log output?

…

On Wed, 24 Jul 2019, 22:41 Lothar, ***@***.***> wrote: I must correct myself: I checked with wireshark and I see that the data fully arrives. After the transmission the websocket connection is being closed - this is what I did not expect. I see that the connection is not being closed by the client (browser); so, the framework closes it. I would expect that the connection persists until a timeout elapses or until I click the "disconnect" button. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#49?email_source=notifications&email_token=AAU2LLBMIMDJN5U4ADD6UATQBCV5BA5CNFSM4ICLCPR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2XMWOQ#issuecomment-514771770>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAU2LLDPVRVFQLIY7JWPM53QBCV5BANCNFSM4ICLCPRQ> .

[squonk11]

Now I checked the console log. There is the error message: The connection to ws://192.168.1.31:8080/echo was interrupted while the page was loading. This happens after 4096 bytes ("a") were received. Then the connection is disconnected. Is this behavior correct?

[PerMalmberg]

That's the client log, what does the esp log?

…

On Wed, 24 Jul 2019, 23:18 Lothar, ***@***.***> wrote: Now I checked the console log. There is the error message: The connection to ws://192.168.1.31:8080/echo was interrupted while the page was loading. This happens after 4096 bytes ("a") were received. Then the connection is disconnected. Is this behavior correct? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#49?email_source=notifications&email_token=AAU2LLC7ZDAQVB24BSX5ELTQBC2JVA5CNFSM4ICLCPR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2XPXHA#issuecomment-514784156>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAU2LLB7TC5UEGYFUSEBGKLQBC2JVANCNFSM4ICLCPRQ> .

[squonk11]

The ESP log is:
I (2785006) ServerSocket: Connection accepted
I (2787370) Socket: [, 0, 56, 0x3ffed474]: Socket closed
I (2787371) Socket: [, 0, 56, 0x3ffed474]: Socket stopping
I (2787375) Socket: [, 0, -1, 0x3ffed474]: Disconnected
no error message.

[PerMalmberg]

Hm, will see if I can reproduce it when I get home this evening or tomorrow.

…

On Thu, 25 Jul 2019, 00:03 Lothar, ***@***.***> wrote: The ESP log is: I (2785006) ServerSocket: Connection accepted I (2787370) Socket: [, 0, 56, 0x3ffed474]: Socket closed I (2787371) Socket: [, 0, 56, 0x3ffed474]: Socket stopping I (2787375) Socket: [, 0, -1, 0x3ffed474]: Disconnected no error message. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#49?email_source=notifications&email_token=AAU2LLBNKVMFC2RBDJAKLOLQBC7Q7A5CNFSM4ICLCPR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2XTL6I#issuecomment-514799097>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAU2LLE5VZ3LGLGIHHUBYQTQBC7Q7ANCNFSM4ICLCPRQ> .

[squonk11]

o.k. for me there is not hurry - currently it is a private project.
One additional information: I am using Firefox. Shall I provide a wireshark file?

[PerMalmberg]

Attach it, might be useful.

…

On Thu, 25 Jul 2019, 13:40 Lothar, ***@***.***> wrote: o.k. for me there is not hurry - currently it is a private project. One additional information: I am using Firefox. Shall I provide a wireshark file? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#49?email_source=notifications&email_token=AAU2LLHGW5WL7LP6R7SCPSTQBGGMDA5CNFSM4ICLCPR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2ZHAHQ#issuecomment-515010590>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAU2LLB3AJSNNYGXNSSTHBLQBGGMDANCNFSM4ICLCPRQ> .

[PerMalmberg]

Not sure I can reproduce the issue you specify, but I do see the following:

W (49802) wifi: alloc eb len=24 type=3 fail, heap:4021356

W (49803) wifi: m f null

W (49813) wifi: alloc eb len=24 type=3 fail, heap:4019276

W (49814) wifi: m f null

This means the the wifi driver can't allocate memory as per #3592. When this happens, I do get disconnects (and the device is no longer reachable).

[PerMalmberg]

@squonk11 I think I get the same error you get now, but only when running on the ESP:

The connection to ws://192.168.10.94:8080/echo was interrupted while the page was loading.

In #53 I added a reason message for all calls to ISocket::close() and it now prints Underlying socket closed (recv returned 0) which leads me to believe it is initiated on the client side. This has to be verified, but nothing in Smooth is actively closing the socket. I guess Wireshark can tell us who sends the RST signal?

[PerMalmberg]

@squonk11 I ofc meant who sends the first TCP FIN flag. It seems to me it is sent from the browser (my ESP IP is 192.168.10.94)

What I I don't understand is why it seemingly only happens when sending large >65k messages?

I've attached the Wireshark capture. Thoughts?

socket_close.zip

[PerMalmberg]

This is a capture of a 125 byte message. It doesn't close the socket.
socket_close_125.zip

[squonk11]

When I check your wireshark capture "socket_close" I see that you first sent three times 125 byte packages which are sent back correctly. After that you probably switched to the 64k packet and try to send that. But here I see that these are not sent as "Websocket" protocol but as "TCP". It looks as if the client makes the mistake... ?
But I see that you are using the same function for sending smaller and larger packets. hmm... strange.

[squonk11]

I think I need to correct myself: probably the websocket transmission of lager size over several packets is done with TCP packets and only the last packet is marked as "Websocket". Unfortunately I am not sure about that because in my previous project I was only using packets smaller that 1400 bytes and so all packets were marked as "Websocket" in wireshark.

[squonk11]

Checking the Wireshark capture I see the following peculiarities:

1. The client (browser) is sending as websocket frame: 0x81,0xfe,0xff,0xff. That means it is setting the FIN bit and the MASK bit and opcode 0x01. Then it sends 64k data.
2. The server is sending the data in chunks of 4096 byte, probably due to the fact that the ESP has not enough memory to store the complete 64k data.
3. The first data chunk is started with a websocket frame of 0x81,0x7e,0x10,0x00. That means it is setting the FIN bit and opcode 0x01. Then it sends data: all "a".
4. The second data chunk is started with the same websocket frame as in 3. and send data: all "a"
5. The third data chunk is started with a websocket frame of 0x01,0x7e,0x10,0x00. That means this time the FIN bit is not set and the opcode is 0x01. The data again is all "a"
6. The fourth data chunk is started with 0x00,0x7e,0x10,0x00. This time FIN not set and the opcode is 0x00 (continue) this time. This time the data is not all "a" but some other data. Maybe the pointer to the data is corrupted on the ESP side?
7. The sixth up to the 16th data chunk are sent with a websocket frame of 0x00,0x7e,0x10,0x00 and the data still is not all "a"
8. The 17th chunk starts with 0x80,0x7e,0x10,0x00, so FIN bit set and opcode 0x00 (continue).

I think the handling of the FIN bit and the opcodes is not properly selected. In addition to that the data which is being sent by the server is not the data which it should be.

[PerMalmberg]

Hm, I'll have a look at the handling of FIN and control codes.

The server is sending the data in chunks of 4096 byte, probably due to the fact that the ESP has not enough memory to store the complete 64k data.

The example server is configured to handle all data in 4096 blocks to keep memory usage constant and deterministic. This can be set on a per-server basis.

If your analysis is correct - why doesn't it happen when running the same server-side application on Linux?

[squonk11]

good point!
I will have a look at the sources - but as already said: I have some problems understanding it since you are using lots of C++17 features. My knowledge up to now is mostly limited to c++98 with some features from C++11. So, I need to read and learn about it.

[PerMalmberg]

If you want we can have chat via Discord, happy to teach you some C++ in return for you helping out with this implementation. Most of this is C++11/14, barely any C++17 used so far.

[squonk11]

Are you able to provide a wireshark trace using Linux as the server? I would like to compare the communication in order to see if there are differences.

[PerMalmberg]

I've found a thing I'm fixing, hold on.

[PerMalmberg]

Ok, it now seems to work much better. FF doesn't want to accept messages larger than 16384 bytes (16k) though, so setting the "block size" to anything larger than that results in a disconnect.

One thing to keep in mind when testing this is that the client side (web browser) doesn't send "Amount to send" in one big message, it create multiple messages of "block size" size. This is different from message fragment as per the WS protocol specification. The server side (ESP/WSEchoServer example) will however echo the incoming data as fragmented messages as per the set chunk size when creating the webserver instances via ContentChunkSize.

[squonk11]

I see you made several modifications. I will check this tomorrow.

[squonk11]

Now I tested again - but it still doesn't work for me. Sending websocket packets of 125 bytes is o.k. but 64k still does not work. I attach a wireshark trace showing this; my ESP32 is 192.168.1.31. This trace shows:

1. two sucessfull transmissions and echo of 125 bytes (packets 37, 38 and 60,61)
2. transmission of 1st chunk (4096 bytes) of 64k in reassembled packet 75
3. transmission of 2nd chunk of 64k in reassembled packet 82
4. transmission of 3rd chunk of 64k in reassembled packet 88
5. echo of 1st chunk in reassembled packet 93
6. transmission of 4th chunk of 64k in reassembled packet 98
7. echo of 2nd chunk in reassembled packet 104
8. transmission of 5th chunk of 64k in reassembled packet 108
9. start of echo of 3rd chunk in packet 109
10. in packet 110: reset of TCP connection by the PC as a "reply/acknowledge" to packet 109

This is what I also noticed the last days: the connection is not being closed on the websocket level - it is closed on the TCP level. But I currently cannot detect why.
Do you have any idea?

websocket_disconnect.zip

[squonk11]

Now I tested with static constexpr int ContentChunkSize = 1024; instead of 4096. That works much better - the connection does not get disconnected for several transmissions. But then after several transmissions the connections gets disconnected during a transmission.

[PerMalmberg]

This is what I also noticed the last days: the connection is not being closed on the websocket level - it is closed on the TCP level. But I currently cannot detect why.

Two possibilities:

• That the browser simply closes the socket when it receives unexpected data/data not according to the protocol - there's not much else it can do.
• Network issues.

So, I guess the question is this: Is my implementation actually sending data that doesn't adhere to the protocol specification? I spent quite some time yesterday trying to detect any way that could happen, but didn't find any. I did notice that when running the code on the ESP, sending larger amounts of data often results in issues as outlined in this issue. Do you seen any log messages starting with "wifi", as per examples in that issue?

[squonk11]

My log looks like this:

I (2040436) ServerSocket: Connection accepted
I (2043041) SocketDispatcher: Active sockets: 3
I (2045656) MemStat: [INTERNAL]
I (2045659) MemStat: 8-bit F:101680 LB:67204 M:99228 | 32-bit: F:115648 LB:67204 M:113176
I (2045659) MemStat: [INTERNAL | DMA]
I (2045661) MemStat: 8-bit F:101680 LB:67204 M:99228 | 32-bit: F:101680 LB:67204 M:99228
I (2045662) MemStat: [SPIRAM]
I (2045664) MemStat: 8-bit F:3971616 LB:3926000 M:3934432 | 32-bit: F:3971616 LB:3926000 M:3934432
I (2045664) MemStat: Name       Stack   Min free stack
I (2045665) MemStat: MainTask   25600   20552
I (2045666) MemStat: SocketDispatcher   20480   13840
I (2050424) Socket: [, 0, 56, 0x3ffed1b4]: Failure during send
I (2050425) Socket: [, 0, 56, 0x3ffed1b4]: Socket stopping
I (2050429) Socket: [, 0, -1, 0x3ffed1b4]: Disconnected

So, I don't see any wifi warning or error - just the message "Failure during send".

[PerMalmberg]

Ok, then the socket is probably being closed from the client side. But why?!

[squonk11]

when I set ContentChunkSize=1024 I see that sometimes the FIN bit in the reply is set and sometimes not. I would expect that FIN is always set when the full data of one chunk fits in one TCP-packet (which is the case here).
When FIN bis is uncorectly(?) set false the next response has FIN=true and opcode:Continuation. Opcode:Continuation should not be necessary because all data of one chunk is sent in one TCP-packet.
I think somewhere here is the problem.

[PerMalmberg]

Just to be clear - ContentChunkSize has nothing to do with how the TCP-layer sends data. It only determine how small websocket fragments a websocket message is split into when sent and received.

Assuming this is what you meant, then yes, FIN should be set when the entire message fit into a single fragment of ContentChunkSize.

Do you see logic error in https://github.com/PerMalmberg/Smooth/blob/feature/49-websocket/lib/smooth/application/network/http/websocket/responses/WSResponse.cpp ?

[squonk11]

yes, the meaning of ContentChunkSize is clear for me. I was talking about TCP packets because I see in the wireshark traces, that in the case of ContentChunkSize=1024 a chunk always fits into one TCP packet; of course it could be splited in several TCP packets - but it is not.
The logic in WSResponse.cpp when setting up the header of the websocket frame looks o.k. to me. But maybe "first_fragment" and "last_fragment" are not set correctly? This I am not really able to analyze because I still do have problems understanding your code and how classes work together.
If I take a look at the result (see attached wireshark trace) I see the following problem:

1. In packet 159 the first response of 1024 bytes is sent: FIN=false; opcode=text --> could be o.k.
2. In packet 169 the second response with 1024 bytes is sent: FIN=true; opcode=continuation (but the data is not all "a" but "0x61". --> why FIN=true, there is still 2048 bytes missing. why data=0x61 and not "a"?
3. In packet 173 the third response is sent: FIN=false; opcode=Unknown (14)

This all looks wired. Maybe the problem is somewhere in the more global logic; e.g the order in which things are processed. Maybe some data structures are getting corrupted?

websocket_disconnect_2.zip

[PerMalmberg]

I just pushed a major update for #52 in another branch which will enable event queues to be destructed at run time. This in turn will allow #50 to be implemented. I have however stumbled upon an issue regarding std::weak_ptr that needs to be sorted first.

I'll have a look at your capture tomorrow, I need a break for the evening.

[PerMalmberg]

Had a look at your trace now, and I don't see the same thing you're reporting.

Packet 169 shows 1024 byte of "a", no 0x61.

[PerMalmberg]

Btw, 0x61 is 'a' ..not sure what else you're expecting/seeing.

[PerMalmberg]

I changed the webpage to send JSON-formatted data instead of just strings to make it easier to see where data starts and ends when looking in Wireshark. I still see no problems with the data formatting. I do however have issues with the Wifi, but thats down to my network.

Running this on Linux, it works just fine as far as I can tell.

[squonk11]

ups, that's a bit stupid from my side not to recognize that 0x61 is equal to "a". Here probably my brain was distorted because wireshark displays the data in the packets before as "a" and starting with packet #169 it displays the data as 0x61.
But nevertheless: in packet #173 the websocket header seems to be destroyed: the reserved bits are "1" (normally they are "0") and the opcode is "14". Probably due to this in packet #175 the browser resets the TCP connection and by this also terminates the websocket connection.
Are you able to see similar scenarios in your wireshark traces?
Since it works on Linux I could imagine that the problem might be related to stack or heap issues because in Linux there are less memory limitations...

[PerMalmberg]

Well, there is definitely a difference between running on the ESP and Linux. One of the ESP devs said today that they've found an issue with Wifi and PSRAM, I'm wondering of that also affects this....

[PerMalmberg]

Right, so after further digging, adding debug outputs and switching to Chrome from Firefox to for better WS debugging tools, I do see problems. It appears that I'm sometimes sending bad data, which results in the browser closing the connection. Now its just a matter of determining why this happens, and where.

[PerMalmberg]

And it is thankfully reproducible on Linux.

[squonk11]

Good to know that it also happens on Linux. I will try to make some investigation this evening (when I am back home) using debug outputs on the ESP32. Maybe the root cause can be isolated by this.

[PerMalmberg]

I know now that something goes wrong when receiving data in WebSocketProtocol.cpp. Sometimes, the data in the buffer looks like garbage. I'm wondering if the de-masking is to blame?

[squonk11]

I now detected that sometimes the data sent from ESP to the client gets partially corrupted. I can see in the wireshark traces that some of the "a" are damaged. Everytime when this happens the client (browser) closes the TCP connection.
In the attached wireshark trace you can see this in packet #672; bytes 0x246 up to 0x248 are not "a" but 0xcf, 0xdf and 0xe9. This specific data sent from client to the ESP32 is o.k. - no corruption every byte is "a".
Maybe the client closes the TCP connection because the TCP checksum is not o.k (although wireshark does not indicate this... hmmm.).
websocket_disconnect_5.zip

[PerMalmberg]

Yep - it was the de-masking of the data that were wrong, so when the data was echoed back, it was just garbage. I tried to do it on the fly, but the indexing got wrong. Now it is done on a per-fragment basis instead. I'm still not sure if this covers all cases with different chunk sizes, but at at least I can't force the error any longer

I did have a look at this earlier today, but didn't see the issue then... oh well, got to keep oneself busy, right? :)

[PerMalmberg]

Ah, found a way to force the issue - set a chink size of 53 and send data in blocks of 8092. At least I now know what the problem is.

[PerMalmberg]

Argh, dammit. Its still possible to break it.

[PerMalmberg]

Hm, this is annoying. All of a sudden, a single message can't be properly demasked, but those before and after can. Is the masking key somehow corrupted?

I)WP: {"message_id":89,"data":"abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abc
(I)WP: defghijklmonpqrstuvwxyz12","length":4096}
(I)WP: {"message_id":90,"data":"abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abc
(I)WP: defghijklmonpqrstuvwxyz12","length":4096}
(I)WP: ��B>�N<��F?���j��K:�
a��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8��I<��E0��@5�](�Y,�Uj���n���b��M8
(I)WP: defghijklmonpqrstuvwxyz12","length":4096}
(I)WP: {"message_id":92,"data":"abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abc
(I)WP: defghijklmonpqrstuvwxyz12","length":4096}
(I)WP: {"message_id":93,"data":"abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abcdefghijklmonpqrstuvwxyz1234567890abc
(I)WP: defghijklmonpqrstuvwxyz12","length":4096}

[PerMalmberg]

Observation: It's always the first fragment of the message. The second part is OK.

(I)WP: �h�`�9�b���a�p�7�h�d�+�?�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f�/�b�#�n�'�k�;�v�?�r�3�4�y�0�}�<�+�f
(I)WP: defghijklmonpqrstuvwxyz12","length":4096}

[PerMalmberg]

Has to be a logic error in WebsocketProtocol::data_received. Taking a break until tomorrow evening.

[PerMalmberg]

Made the code as simple as I can and while it is much more stable now, sometimes the de-masked data is still garbage in the first fragment. The de-mask key is intact between the different fragments so that's not the problem.

It only seems to happen when FF or Chrome sends lots of data in a short amount of time. For example, I can send 10M multiple times, then all of a sudden there is demasked garbage.

[squonk11]

I could imagine that the problem is related to concurrency of incoming packets. E.g. I see that the demasking is done at the very end of WebsocketProtocol::data_received and that you do the demasking depending on the call to is_data_masked() just before. If in the meantime already the next packet arrived then is_data_masked() will deliver false because frame_data.header[1] & 0x80 for the letter "a" will be 0x0.

[PerMalmberg]

The mask key is held for all fragments so it is still valid (and the code for demaksing is running)

https://github.com/PerMalmberg/Smooth/blob/feature/49-websocket/lib/smooth/application/network/http/websocket/WebsocketProtocol.cpp#L165

[PerMalmberg]

I think I've found the actual root cause. I've forgotten that there is no guarantee that data_received receives the amount requested in get_wanted_amount, only that it never receives more than the requested amount. As such the the protocol gets out of sync with the data and things stop working. I'm going to push a work-in-progress commit that logs some data that shows this is the case. To fix it, I'll probably rewrite the data_received, get_wanted_amount and get_write_pos. as can be seen in the coming commit, it just gets messy when trying to patch it.

[squonk11]

When I implemented websocket communication in the previous project I used std::streambuf for this. That worked quite painless. But I must confess that I didn't test it with large data objects.

[PerMalmberg]

With the last changes I'm now able to push gigabytes through a single websock connection without errors on a Linux system. However, when run on the ESP and really stressing the system, the connection breaks occationally, but I cannot see that it has anything to do with the implementation.

As such, I consider the implementation correct until someone proves otherwise, and it is repeatable on a Linux system.

[PerMalmberg]

@squonk11 I've merge websocket support into master. Closing this issue. Please open a new one if needed.

[squonk11]

I just wanted to inform you about the following observation:
I still had issues with large websocket transmissions: I found out that there are sometimes just single bytes destroyed in the packets received by the browser and due to this the browser stopped the TCP connection.
Now I found your conversation here. In this thread @igrr published a fix to the psram library. I also downloaded this fix and installed it.
Now the problem dissapeard! Your library seems to work perfectly now.

One other topic: I will need the ESP32 wifi to be working in access point mode. I will make a class "wifiAP" for this (similar to your wifi class). Are you interested in a pull request?

[PerMalmberg]

@squonk11 That's a good observation, and one I was hoping for. Memory issue are a pain; I'm glad I can develop on Linux before running things on the ESP, it allows me to be more sure of my own code.

Yes, I will accept PRs; expect constructive criticism. I've long thought of adding AP-support to the Wifi class, but never gotten around to it. Give some thought on if you can add it to the existing one, possibly renaming set_ap_credentialsto just set_credentials etc.

Please open a new issue for further discussions.