Firmata STRING_DATA bug? about arduino

[mgcrea]

@soundanalogous Thanks a lot for your answer, it looks like using:

    var bytes = new Buffer(string + '\0', 'utf8');

Solves the string corruption issue!

---

One last question, it looks like the length of the string is limited from JS -> Arduino but not the other way around.

Firmata.sendString("10-bit addressing mode is not yet supported");

Does work on the arduino side, the same thing on the JS side:

sendString("10-bit addressing mode is not yet supported");

Won't return anything. It looks like I can't send strings > 30bytes using the configurable branch.

  firmata sending a 29-sized string="XXXXXXXXXXXXXXXXXXXXXXXXXXXXX" +974ms
  firmata receiving a 58-sized string="XXXXXXXXXXXXXXXXXXXXXXXXXXXXX" +25ms
  firmata sending a 30-sized string="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" +976ms
  firmata receiving a 60-sized string="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" +23ms
  firmata sending a 31-sized string="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" +979ms
  firmata sending a 32-sized string="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" +1s

Tested on master branch, It won't report anything over >= 15 bytes.

  firmata sending a 13-sized string="XXXXXXXXXXXXX" +991ms
  firmata receiving a 26-sized string="XXXXXXXXXXXXX" +12ms
  firmata sending a 14-sized string="XXXXXXXXXXXXXX" +989ms
  firmata receiving a 28-sized string="XXXXXXXXXXXXXX" +14ms
  firmata sending a 15-sized string="XXXXXXXXXXXXXXX" +988ms
  firmata sending a 16-sized string="XXXXXXXXXXXXXXXX" +1s

And, I'm not sure it is normal that I receive a string where every character is separated by NULL \0 (hence the double size)?

Thanks again.

[soundanalogous]

This issue may be that c strings are null terminated, but I don't think JavaScript is sending a null-terminated string. The EchoString example passes a pointer to a character array, but if JavaScript didn't append a zero (as in null, not ASCII value for zero) you'll get an odd message as your test shows. The solution may be within the processSysexMessage method of Firmata.cpp for the STRING_DATA case, check the last character in the buffer and if it is not 0, append 0 before the callback (so just after the while block).

[soundanalogous]

Firmata.sendString will send each character in 2 7-bit chunks (because the protocol is midi). However, most ASCII characters have a decimal value of < 128 so that is why you see the char value followed by a zero. I suspect your JavaScript method for receiving Firmata strings is not processing incoming strings correctly. See my implementation on line 471 for an example: https://github.com/soundanalogous/Breakout/blob/master/src/core/IOBoard.js. Also see the sendString method on line 1202 of the same file.

[mgcrea]

Thanks @soundanalogous, indeed the javascript implementation was wrong.

But it looks like your's may be broken too! I did not manage to handle exotic utf-8 chars properly (like Ä). Finally found something that seems to work : mgcrea/node-firmata@1f5db9f

Do you have any idea about the 30-bytes limitation? Is it something expected?

Thanks!

[soundanalogous]

The issue with the 30-byte limitation is that you're overrunning the serial buffer. For a standard Arduino (Uno, etc) the buffer is 64 bytes: https://github.com/arduino/Arduino/blob/master/hardware/arduino/cores/arduino/HardwareSerial.cpp#L59.

Because Firmata is sending 2 bytes per character you're going to overrun the serial buffer if you send a string larger than 30 characters (because START_SYSEX and STRING_DATA each are each 1 byte commands so that leaves you with 30 characters in your string message and the END_SYSEX is likely not even received since it's the 1st byte to overrun the buffer). I'm not sure what the buffer size is for the Due or a Teensy 3.0. Theoretically they could have larger buffers since they are much faster, but if you're seeing a 30 char limitation on a Teensy 3.0 then it also has a 64-bit buffer.

Regarding utf-8 characters, I never really had a use case for them (but can see the point of including some accented characters as you illustrate). The Firmat::sendString method sends an array of 8-bit characters so you could only send utf-8 characters that are in the 0 - 255 range so you couldn't fully support utf-8 (which has characters up to 32 bits). However many of the accented characters are in the 128 - 255 range so that is why your implementation works and it makes sense to support it to that extent (up to 8-bits per character). Users just have to understand that they can only send characters in the 0-255 range of the unicode charset.

If you wanted to support strings longer than 30 characters you'd have to send them in chunks. I'm not sure what the use case is for such large string though. It only makes sense if your application only sends string and nothing else and even then a binary protocol may be more efficient although more cryptic). It would be better in most cases to implement a binary protocol using sendSysex rather than sending large strings. If you need a large string in that case, send a key from the Arduino and look up the value in your client application (this is how the error messages should have been sent). Sending strings from an 8-bit microcontroller is expensive, this is why Firmata is a binary protocol. When Firmata was originally written Arduinos where using the ATMega 168. That's only 16k (with 2k occupied by the bootloader). However with new boards like Due sending strings is more practical and sending and receiving strings may even be within reason. The primary use case I see with the Due is sending and receiving JSON strings.

[soundanalogous]

To answer your 15 vs 32 byte string question... The reason for the 15 byte limitation in master and 30 byte limitation is related to the buffer size issue, but it's enforced in Firmata. In the master branch, the input buffer is limited to 32 bytes and in the configurable branch the buffer is limited to 64 bytes. I'm assuming the 32 byte limitation in master goes back to the older ATMega168 based arduinos. I can increase the size in master to 64 bytes but not any larger.

[hansamann]

Hi all, I am trying to use node firmata with the EchoString example of Firmata on Arduino - so far I cannot make it work. Has this been solved and could someone post a node example that shoudl work with the Arduino Firmata library?

Thx
Sven

[soundanalogous]

You'll need to pull the dev branch. It has not yet been merged into master because the change is not backwards compatible with the existing version of Firmata.

[hansamann]

OK, downloaded the dev branch as zip and then renamed the folder to Firmata_Dev - I put that folder under der libraries folder for Arduino and then included that dev Firmata library. That should work, right? I then chose from the Fromata_Dev examples the EchoString example and flashed that onto a Leonardo.

Now, my node code is still this:

'use strict'

//which port? ls /dev | grep usb

var firmata = require('firmata');
var board = new firmata.Board('/dev/tty.usbmodemfa131',function(){
console.log("READY.");
setTimeout(function() {
board.sendString("Hello");
}, 1000);
});

board.on("string", function(data) {
console.log(data);
});

In short, it does not work right now... When I hit the reset button on the Leonardo, then the code continously shows "Ready." - which is also strange but definitely a node issue. Any idea or example I coudl try out in node? Is the dev library of Firmata for Arduino correctly included, never done this... ? Thx!