Very large messages getting clipped? about scuttlebutt

[mbrevoort]

OK, so a clarification. The messages are not coming concatenated. Rather data is being emitted in for the form of a partial message (clipped stringified JSON) without a line ending and then the next data event emitted is an entirely different update that ends with a line break and then that entire thing is parsed and errors.

[mbrevoort]

Also, not sure if this is s Scuttlebutt problem or with CRDT.

[juliangruber]

could this have to do with max net packet sizes?

[mbrevoort]

@juliangruber it must to be something like that but even so with TCP shouldn't the order should be preserved? I'm currently digging through the duplex models and the streams module itself to see if there's anyway for things to get out of order there.

[mbrevoort]

Unless it's not an order problem at all but rather truncating and never sending the remainder.

[juliangruber]

I had this case once where

res.write('foobar');

didn't have the same effect as

res.write('foo');
res.write('bar');

but maybe that has been addressed in a newer node version?

[dominictarr]

If anywhere, this should be handled in https://github.com/dominictarr/stream-serializer/blob/master/index.js
stream-serializer.

Messages are appended together until a \n is found,
and then that item is json parsed and passed to the write stream that expected a regular object.

https://github.com/dominictarr/stream-serializer/blob/master/index.js#L32-L36

[dominictarr]

okay, I hate to say this but... "this should never happen"

what are you streaming this over?
is this over tcp or websockets?

this is inside fleet, right?

[dominictarr]

if something was writing out a rogue \n then this could happen. but I'm not sure what could be doing this...

[mbrevoort]

I did a tcpdump and I can in fact see the data being split between two packets (sent by client). In the tcp stream I can see in between these packets the server happens to send a scuttlebutt digest to the client. It is the combination of the partial message from the client and the unrelated digest the server send to the client that get's JSON.parsed and errors.

So it seems that the data sent from the server is being emitted into the stream from the client?

[mbrevoort]

@dominictarr this particular case is with seaport over tcp.

[dominictarr]

can you get on irc? this is obviously urgent (#stackvm on freenode)

[dominictarr]

oh, @juliangruber you shouldn't be writing to the stream directly

[dominictarr]

this issue was fixed in seaport. https://github.com/substack/seaport/pull/41

[juliangruber]

@dominictarr: that was in a http stream

[dominictarr]

@juliangruber with scuttlebutt?

[juliangruber]

nope, totally unrelated :D