ESP8266 crashes when is receiving data from websocketserver about arduinowebsockets

[gilmaimon]

Hi Jose,

Sounds weird, let's try to solve it. Can you write what software you use on the server? I'll try to setup a similar server and see if I have issues with long messages as well. If I can access the same server you are using it will be best (if it's a public service).

Also, can you turn on the esp8266 debug logs (from the boards menu) and attach those.

Thanks,
Gil.

[adelin-mcbsoft]

Hi Jose,
Let me say few words, perhaps will help you:

1. First of all, you told me you are using PHP backend, which makes me think you are using some websocket implementation found on the internet;
Most of those I found in the beginning, do not really read the headers properly and cannot handle long messages, they just truncate it or throw an error.
Been there, done that, ended up creating my own correct implementation by reading the RFCs, works like a charm now, HOWEVER

2. I did some testing on few Arduino modules, though I work with ESP32. The maximum amount I could safely send was 16kb, working with it as a client. I sometimes reached more (about 24kb), but from what I've tested, this was the safest amount of data to send. Anyway, should be a lot. I'm talking here about one single message, you could send as much as you want in multiple messages, by encoding & chunking it.

I don't think its a limitation of the library (and @gilmaimon confirmed so), but rather of the ESP32's Arduino libraries that handle the strings. Didn't investigate further, as 16kb were enough for me. Even the browsers limit the amount of data you can send/receive (and that's 128kb); seems a bit more, but browsers have much more memory than an micro controller.

Hope it helps a bit,
All the best,
A.

[elC0mpa]

Ok @gilmaimon, let me tell you something. First of all, I used the ESPAsyncWebServer to simulate the same behavior as the php websocketserver. By doing this i can identify if the problem is related to the server or to the client. Because of the problem remains, I was able to detect that the problem is not related to the server. The file size is about 4 kB, reason why we can descart that the esp is running out of memory.
I modified the server code to send Just a part of the file, not entirely. Because i send the json file as a string, I just decided to send 500 chars, and it worked. But after that i decided to send 600 chars and the Esp01 which is acting as the websocketclient crashed. Tomorrow i will take a look to EspExceptionDecoder, I hope it could Help me

[elC0mpa]

Thanks @gilmaimon, I will follow your advice and later i will tell you how it was. Thanks for your answer

[gilmaimon]

I'm having some troubles with my esp8266, but I think I solved it.
Can you try again with the changes commited to master?

Gil.

[gilmaimon]

Sure,

ESP8266 has a watchdog that makes sure that TCP buffers and network operations are done within some set intervals. Those operations occur when calling delay or yield (which is called by delay).

I've added yield in random places in the library and it seems to fix the issue. So I moved the yields from those random places to the classes in the library that wraps WiFiClient and other esp-networking stuff. Every operation now calls yield so I can be certain it is fairly regularly.

With yield called regularly, even when a big message is being read, the watchdog seems to not alert and reboot the program. However as I've said, I'm having some troubles with my esp8266. So let me know if it works for you.

By the way, on esp32 your sketch works perfectly.

Gil.

[elC0mpa]

Thanks @gilmaimon, I am going to work right now, but when i finsih, I will try it. 👌🏻

[elC0mpa]

Thanks for your answer @gilmaimon, I can not publish the server code because i don't have it and it is written in php. I need to comunicate with that server wich will be running on a Raspberry pi. Let me ask you something, which is the maxium lenght of received data in this library, because i am pretty sure that this is the problem. Remember it worked with the same code when i changed the size of the json file.

[gilmaimon]

Hi Jose,

There is no limitation.

How big is the file you are getting, could the esp be out of memory? Do you have an esp32 around?

[elC0mpa]

Thanks for your answer @gilmaimon and @adelin-mcbsoft. First of all, I have no an Esp32 to test if the problem is related to RAM, but i am pretty sure that the file is not so big. I will check it later.

1. First of all, you told me you are using PHP backend, which makes me think you are using some websocket implementation found on the internet;
Most of those I found in the beginning, do not really read the headers properly and cannot handle long messages, they just truncate it or throw an error.
How can i know if this php implementation is correct? Remember that using the same code, I changed only the json file and everything worked as expected. Maybe it is something related to frames, but i am not sure.
Thanks for your answer

[gilmaimon]

Cool, let us know when you know more about the file size.

Regarding the correctness of the implementation. I assume it is correct, but might require the use of some extension I'm not supporting. Anyway, it shouldn't crash the esp.

If you can use EspExceptionDecoder to decode the crash stacktrace (those hex bytes) it will help identify what exactly is failing.

Also I'll point out again those ESP8266 debug logs, which might help.

Thanks,
Gil.

[gilmaimon]

Hi Jose,

I don't think this is a library limitation or bug. There is no difference between message of size 500 and of size 600 (considering the code which runs inside the library in each case). Also, the same logic running on my PC can handle 4K messages with no problem - which is also why I don't think this is a logic bug.

Best thing to do is to decode the stack trace, this will let us know if the crash is related to memory or something else. I wouldn't discard running out of memory, the memory usage can always be improved and there are other stuff using your esp RAM. Also, some of the memory is used for the internal TCP buffers (I think around 8K?).

Best steps to take are:

• Share the code you are using exactly (if you don't mind sharing)
• Decode the crash and print the decoded stack trace (make sure to censor private information like paths on your PC)

I'm sure we will sort this out.
Gil.

[elC0mpa]

Hello @gilmaimon, I already used the espExceptionDecoder, and it wasn't succesfull. The thing is, that it looks like my problem is not an exception. When it happens, the serial monitor shows : soft WDT reset and after that, it shows the stack trace, so it doesn't decodify any exception. I will send you the json file, and maybe you could Help me By receiving it using your library. I will be waiting for your answer, thanks in advance

[gilmaimon]

Jose, look how deep we are into this discussion and this is the first time you mention that the exception is a soft WDT reset...

1. Please copy and paste the crash log here
2. Please share the code you are using
3. Please try again to decode the stack trace
4. The error you mention is called because esp8266 has a thing called software watchdog. The watchdog will reset your sketch if it is stuck for too long. It is possible that receiving a 4k file just takes too long. It is also possible that you have some other code that is causing the issue.

Helpful resources:

• https://arduino-esp8266.readthedocs.io/en/latest/faq/a02-my-esp-crashes.html
• https://forum.arduino.cc/index.php?topic=442570.0

Gil.

[elC0mpa]

Thanks for sharing these links @gilmaimon, I learned some things By reading them, I decided to share my code and my json file with you. I really hope you could Help me. I followed the steps of how to use the espExceptionDecoder, but it doesn't worked. Despiste this, I am pretty sure that the soft WDT overflows in the line number 22, which is the line where i read the incoming msg from the websocketserver. Thanks in advance
Json file

{
   "clientes": {
     "1": {
       "id": 1,
       "nombre": "Cliente 1",
       "edad": 29,
       "sexo": "M",
       "preferencia_tv": "Deportes",
       "color": "#fff",
       "temperatura_agua": "fria",
       "mascota": 1,
       "carro": 0,
       "created_at": "2019-09-24 19:58:55",
       "updated_at": "2019-09-24 19:58:55",
       "tiempo_en_area_simulacion": 4,
       "tiempo_en_area": 4,
       "tiempo_restante_en_area": 0,
       "area": {
         "id": 6,
         "nombre": "Piscina",
         "icono": "fa-support",
         "tiempo_en_area": 4,
         "created_at": "2019-09-24 19:58:55",
         "updated_at": "2019-09-25 15:06:35"
       }
     },
     "2": {
       "id": 2,
       "nombre": "Cliente 2",
       "edad": 29,
       "sexo": "M",
       "preferencia_tv": "Deportes",
       "color": "#fff",
       "temperatura_agua": "fria",
       "mascota": 1,
       "carro": 0,
       "created_at": "2019-09-24 19:58:55",
       "updated_at": "2019-09-24 19:58:55",
       "tiempo_en_area_simulacion": 2,
       "tiempo_en_area": 4,
       "tiempo_restante_en_area": 2,
       "area": {
         "id": 6,
         "nombre": "Piscina",
         "icono": "fa-support",
         "tiempo_en_area": 4,
         "created_at": "2019-09-24 19:58:55",
         "updated_at": "2019-09-25 15:06:35"
       }
     },
     "3": {
       "id": 3,
       "nombre": "Cliente 3",
       "edad": 29,
       "sexo": "M",
       "preferencia_tv": "Deportes",
       "color": "#fff",
       "temperatura_agua": "fria",
       "mascota": 1,
       "carro": 0,
       "created_at": "2019-09-24 19:58:55",
       "updated_at": "2019-09-24 19:58:55",
       "tiempo_en_area_simulacion": 1,
       "tiempo_en_area": 1,
       "tiempo_restante_en_area": 0,
       "area": {
         "id": 1,
         "nombre": "Habitaci?n Ba?o",
         "icono": "fa-bath",
         "tiempo_en_area": 1,
         "created_at": "2019-09-24 19:58:55",
         "updated_at": "2019-09-24 19:58:55"
       }
     },
     "4": {
       "id": 4,
       "nombre": "Cliente 4",
       "edad": 29,
       "sexo": "M",
       "preferencia_tv": "Deportes",
       "color": "#fff",
       "temperatura_agua": "fria",
       "mascota": 1,
       "carro": 0,
       "created_at": "2019-09-24 19:58:55",
       "updated_at": "2019-09-24 19:58:55",
       "tiempo_en_area_simulacion": 1,
       "tiempo_en_area": 1,
       "tiempo_restante_en_area": 0,
       "area": {
         "id": 4,
         "nombre": "Parqueo - Mascota",
         "icono": "fa-github-alt",
         "tiempo_en_area": 1,
         "created_at": "2019-09-24 19:58:55",
         "updated_at": "2019-09-24 19:58:55"
       }
     }
   },
   "preferencias": {
     "id": 1,
     "limite_de_basura": 100,
     "coeficiente_de_basura": 2,
     "tiempo_de_simulacion": 1,
     "dia": "07:00 AM",
     "noche": "07:00 PM",
     "cant_actuadores_energia": 5,
     "cant_actuadores_agua": 5,
     "created_at": "2019-09-24 19:58:55",
     "updated_at": "2019-10-02 16:07:54"
   },
   "simulacion": {
     "basura_acumulada": 100,
     "dia": 176,
     "hora": 18,
     "cantidad_clientes": 4,
     "sesion": "dia"
   }
}

platformio code

#include <Arduino.h>
#include <ArduinoWebsockets.h>

#define SSID "AsyncWebSocketTest"
#define PSWD ""
#define PORT 82
#define IPADDRESS "192.168.0.1"
#define ROUTE "/"
#define TEST true
#define BAUDRATE 115200

#define USE_SERIAL Serial

using namespace websockets;

WebsocketsClient client;

#pragma region  Events

void onMessageCallback(WebsocketsMessage msg)
{
  auto data = msg.data();
  Serial.println(data);
}

void onEventsCallback(WebsocketsEvent event, String data)
{
  if (event == WebsocketsEvent::ConnectionOpened)
  {
    Serial.println("Connection Opened");
  }
  else if (event == WebsocketsEvent::ConnectionClosed)
  {
    Serial.print("Connection Closed: ");
    auto code = client.getCloseReason();
    Serial.println(code);
  }
  else if (event == WebsocketsEvent::GotPing)
  {
    Serial.println("Got a ping");
  }
  else if (event == WebsocketsEvent::GotPong)
  {
    Serial.println("Got a pong");
  }
}
#pragma endregion

void setup() {
  // put your setup code here, to run once:
  USE_SERIAL.begin(BAUDRATE);

	USE_SERIAL.setDebugOutput(true);

	USE_SERIAL.println();
	USE_SERIAL.println();
	USE_SERIAL.println();

	for(uint8_t t = 4; t > 0; t--) {
		USE_SERIAL.printf("[SETUP] BOOT WAIT %d...\n", t);
		USE_SERIAL.flush();
		delay(1000);
	}

  WiFi.mode(WIFI_STA);
	Serial.println("Trying to connect...");
	WiFi.begin(SSID, PSWD);
	Serial.println("Connected to Wifi");
	delay(1000);
	Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
	  
	client.onMessage(onMessageCallback);
  client.onEvent(onEventsCallback);
  
  #if TEST
  client.connect(IPADDRESS, PORT, ROUTE);
  #else
  client.connect("ws://192.168.1.11:6001");
  #endif
  client.ping();
}

void loop() {
  // put your main code here, to run repeatedly:
  client.poll();
}

EDIT: I've allowed myself to edit your comment just so the code is easier to read (@gilmaimon)

[elC0mpa]

Ok, will try later and tomorrow i will give you the feedback. Could you tell me which was the problem? I would like to know, because despite i have been working with Microcontrollers since a long time, this topic is out of my confort zone. Thanks again

[elC0mpa]

Hello @gilmaimon, let me tell you that now it is worst, when the esp starts executing the code, it restarts and prints the Next message:
ets Jan 8 2013, rst cause:1, boot mode :(3, 6).
I hope you could fix this. Please remember that i want to keep using your library 🤝

[gilmaimon]

I don't have an esp8266 around me until the weekend, but I can guess what is causing the issue. Can you try again with the changes commited to master?

[elC0mpa]

Ok @gilmaimon, I will try later and i will update you with the results 👌🏻

[elC0mpa]

Hello @gilmaimon, I realized that you removed the yield() function from the constructor of the class. The thing is that the problem is still there, Just when the esp8266 starts executing the code, it restarts. Have you another idea of what could be causing this?

[gilmaimon]

Does it crash immediately or when connecting? I'll keep working on it this weekend.

[gilmaimon]

I believe the crash is happening when the client c'tor is called. I'm not sure why it crashes, the same exact code runs on esp32 without crashing. There is probably some bug in the esp8266 specific code in my library or in the esp8266 libs themselves.

Any ways, the issue happens because the c'tor is called before setup. If you would change the code such that it delays initialization until setup, things should work. This is not a fix, but a temporary workaround.

Try:

WebsocketsClient* client;

void setup() {
   // stuf...
   client = new WebsocketsClient();
   client->connect(.......);
}

You will need to replace you accessors with -> instead of a . because client is now a pointer.
Let me know if it worked, it should. I will look into it again this weekend and hopefully figure out whats causing the issue.

Gil.

[elC0mpa]

Does it crash immediately or when connecting? I'll keep working on it this weekend.

It crashes inmediately @gilmaimon

[elC0mpa]

I believe the crash is happening when the client c'tor is called. I'm not sure why it crashes, the same exact code runs on esp32 without crashing. There is probably some bug in the esp8266 specific code in my library or in the esp8266 libs themselves.

Any ways, the issue happens because the c'tor is called before setup. If you would change the code such that it delays initialization until setup, things should work. This is not a fix, but a temporary workaround.

Try:

WebsocketsClient* client;

void setup() {
   // stuf...
   client = new WebsocketsClient();
   client->connect(.......);
}

You will need to replace you accessors with -> instead of a . because client is now a pointer.
Of course
Let me know if it worked, it should. I will look into it again this weekend and hopefully figure out whats causing the issue.
I will do it and i will comment you about the results
Gil.

[elC0mpa]

Perfect @gilmaimon, now the esp8266 is able to receive the entire json file without beign restarted because of the software WDT. I will be waiting for you to fix the problem of the constructor of the class. Please notify me when it will be able to download as a New version of the library. If you need me to test it before the release, let me know. 👌🏻

[gilmaimon]

Hi Jose
I believe I fixed the issue. Seems like there was another function that was called in the c'tor which had a yield in it (which I recently added) and it caused the issue.

Overall, I think it is a bug that calling yield before setup crashes the esp8266 (esp32 can handle it), but I worked around it anyways.

You should be able to revert your code to it's previous state and everything should work fine. Please let me know if it did.

Best wishes,
Gil.

[elC0mpa]

Hi @gilmaimon, I will download the repo and revert my code to its previous state later, now i am at work. I will let you know when i do it. Thanks in advance 👏🏼

[elC0mpa]

Bug fixed @gilmaimon. Now it works as expected. You can now publish a New release of the library 👌🏻. Let me ask you a final question : now there is no limit of data to be readed by using this library, is it? 🤓

[gilmaimon]

Thank you! I've posted the new version and it should be available via the library manager soon.
There should be no limitation beyond the websockets protocol itself (and physical limitations, of course).

Gil.