xdenser / node-struct

Handy work with buffers in NodeJS

Struct

NodeJS module to work with buffers as structures (or records) of various fields (like c struct declaration, or pascal record).

Installation

To install with npm:

npm install struct

Example

Define some structure:

var Struct = require('struct');

var Person = Struct()
	.chars('firstName',10)
	.chars('lastName',10)
	.array('items',3,'chars',10)
	.word16Sle('balance'),
People = Struct()
	.word8('presentCount')
	.array('list',2,Person);

Now allocate buffer for it

People.allocate();
var buf = People.buffer();

Clear buffer to see how it will change later:

var buf = People.buffer();
for (var i = 0; i < buf.length ; i++) {
	buf[i] = 0;
}
console.log(buf);

Output: <Buffer 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00>

Now you can access memory as defined binary structure with fields property in a handy manner.

var proxy = People.fields;
proxy.presentCount = 2;
console.log(buf);

Output:

<Buffer 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00>

And so on

proxy.list[0].firstName = 'John';
console.log(buf);

Output: <Buffer 02 4a 6f 68 6e 00 00 00 00 00 00 ...

Reference

##Struct()

creates struct object, you may define your data structure with chain calls to following methods:

###word8(name) defines one byte unsigned field, name - always defines name of field

word8Sle(name),word8Sbe(name)

define one byte signed field

word16Sle(name),word16Sbe(name)

define 16 bit signed field with little-endian and big-endian byte order

word16Ule(name),word16Ube(name)

define 16 bit unsigned field with little-endian and big-endian byte order

word32Sle(name),word32Sbe(name),word32Ule(name),word32Ube(name),word64Sle(name),word64Sbe(name),word64Ule(name),word64Ube(name)

same for 32 and 64 bit fields

floatle(name),floatbe(name)

define one float field

doublele(name),doublebe(name)

define one double field

chars(name,length[,encoding])

defines array of chars with encoding ('ascii' by default) encoding, name - name of the field, length - length of array

charsnt(name,length[,encoding])

same as chars but ensure that string is null terminated and buffer remaining space is fill with \0

array(name, length, type, ...)

defines array of fields (internally it is Struct() object with field names set to 0,1,2,... ).

• name - name of array field;
• length - length of array;
• type - string||Struct(), string is interpreted as name of Struct() method to call for each array element.

For example array('numbers',5,'word16Ule') will define array of 2 byte unsigned words (x86 byte order) with 5 elements. Any parameters that follow type will be passed to definition function. So array('someName',3,'chars',20) defines 3 element array of 20 chars. You also may pass Struct() object to make array of structures.

struct(name, Struct() )

defines field that itself is a structure.

Other methods

get(fieldName)

allows access to field (reads value from it's offset in buffer)

set(fieldName, value)

allows access to field (write value at it's offset in buffer)

allocate()

allocates buffer for defined structure with proper size. This is used when you need to format data in buffer and send or write it out.

_setBuff(buffer)

sets buffer reference of object to other buffer. This may be used to parse or adjust some binary data received or read from somewhere.

buffer()

returns currently used buffer. Before you may read or write to structure you have to call either allocate() or _setBuff(buffer). This is not true only for Struct() objects that are fields themselves, as they are allocated automatically by parent Struct object.

fields property

Struct().fields is a Proxy object allowing to access structure fields in a handy manner - as any other javascript object.

node-struct's Issues

Field offset and size not accessible

There's no way to retrieve the offset where each field is located. The offset is just used implicitly when constructing the structure, but I think it would be nice if each field get an 'offset' entry to expose this information.

Probably, a 'length' and storing this in an array, to make it possible to retrieve the list in a static order would be nice to have.

Make a minor or major version bump

Yesterday, #16 was merged and broke lots of modules using tilde or caret versions. Can you do a minor or major version bump rather than a patch version bump? The exported API has changed, so I would expect at least a minor version, if not a major. Should probably be 1.0.0.

Cannot allocate multiple structs

The following code:

var Person = Struct()
    .chars('firstName',10)
    .chars('lastName',10)
    .array('items',3,'chars',10)
    .word16Sle('balance'),
People = Struct()
    .word8('presentCount')
    .array('list',2,Person);


Person.allocate();
People.allocate();

console.log(People.buffer());
console.log(Person.buffer());

should log two (pretty large) Buffers, however today it will log:
Buffer < 36 00 ....lots of data ...>
Buffer <>

With the second buffer always being empty.

Invalid integer value

Using this hex value: 0x100000000000002 should print this integer to the console: 72057594037927938

What I am seeing is this: 72057594037927940

I have a few test cases, but this is the easiest to paste:
var idStruct = Struct().word64Ule('id');
idStruct.allocate();
idStruct.fields.id = 72057594037927938;

console.log( "id: " + idStruct.get('id') );

Array size read from buffer

Usually array in buffer are defined with array size contained in a field stored above, that can be done to read non defined array length ?

Typescript declarations

Hi,
currently I'm using your package in a Typescript project that I have inherited from a collegue. I really enjoy working with it, so far. What do you think about generating some Typescript declaration files?

I've never done it by hand so far, but Microsoft provides a generator that might be helpful dts-gen (https://github.com/Microsoft/dts-gen).

Calling dts-gen -m struct is not very helpful:

/** Declaration file generated by dts-gen */

export = struct;

declare function struct(...args: any[]): any;

declare namespace struct {
    // Circular reference from struct
    const Struct: any;

    const prototype: {
    };

}

Clone function does not properly copy properties

There is an issue with node-struct in how the clone function is implemented, the closures array is copied using the assignment operator rather than the slice, so modifying the fields for the cloned struct results in modifying the original struct as well.

Original Code

this.clone = function () { var c = new Struct; var p = c._getPriv(); p.closures = priv.closures; return c; }

Modified With Fix

this.clone = function () { var c = new Struct; var p = c._getPriv(); p.closures = priv.closures.slice(0); return c; }

cannot find Key error

Below is code which is leading error ,

var Struct = require('struct');
/*************************/
var Person = Struct()
.chars('username',60)
.chars('id',100);

Person.allocate();
Person.set('username',"sravani");
Person.set('id',"123456889994");

console.log("username: "+ Person.get('username') +"id:" +Person.get('sid'));

/**************/
[root@sravani nodejs]# node struct.js
home/wsadmin/src/sravani/nodejs/node_modules/struct/index.js:398
throw new Error('Can not find field ' + key);
^

Error: Can not find field sid
at Struct.get (/home/wsadmin/src/sravani/nodejs/node_modules/struct/index.js:398:19)
at Object. (/home/wsadmin/src/sravani/nodejs/struct.js:11:65)
at Module._compile (module.js:571:32)
at Object.Module._extensions..js (module.js:580:10)
at Module.load (module.js:488:32)
at tryModuleLoad (module.js:447:12)
at Function.Module._load (module.js:439:3)
at Module.runMain (module.js:605:10)
at run (bootstrap_node.js:423:7)
at startup (bootstrap_node.js:147:9)

time_t variables

I'm trying to use this library with a C struct that contains a few time_t variables. Looking on my system they're 8 byte variables. However, when I read with time_t variable any sort of word64 variable in your library the data after this read gets corrupted (I'm assuming this means I'm reading too much or too little data from the buffer). I've also tried word32 and word16 variables with no luck (really just shooting in the dark at that point).

Do you have any suggestions for reading time_t variables? (I know it's a unique request)

Pass an array length from a field.

Hello,

So I wanted to get an object (fields) from a buffer. I was wondering how do you pass already read field as the length of an array?

This is what I've tried but It doesn't work...

  var PacketRecordsBodyOneBSegment = Struct()
      .word8('number_of_io'); // length of the following array ...
  var length = PacketRecordsBodyOneBSegment.fields.number_of_io;
  console.log('OneB length:', length); // undefined
  PacketRecordsBodyOneBSegment = PacketRecordsBodyOneBSegment
      .array('events', length, PacketRecordsBodyOneBio);

Ideally, something like this would be awesome, but I don't know how to implement it:

  var PacketRecordsBodyOneBSegment = Struct()
      .word8('number_of_io'); // length of the following array ...
      .array('events', 'number_of_io', PacketRecordsBodyOneBio);

Thanks.

What about long?

I'm using a 64-bit Linux. My problem is that long has 64 bits on my system while it has 32 bits on 32 bit machines. Is there a way to automatically detect on which system this lib is running and to automagically adjust the length of longs?

PS: I want it to run cross platform. Therefore, just using word64Sle is not sufficient.

Dose the API Serport "float" and "double" type fields,my project require it to work

I am in a Chinese Company，in my project,I want exchange data from node.js to C++ Application.Can I Use this library to help me?

Usage question.

Hi. I'm a little unsure if I'm using this correctly. My code:

var struct = require('struct').Struct().word64Ube('connection_id').word32Ube('action').word32Ube('transaction_id');

struct.allocate();

struct.set("connection_id", 0x41727101980);
struct.set("action", 0);
struct.set("transaction_id", 84548454);

var buf = struct.buffer();

and then passing the buffer on to a dgram client. Is this correct usage?

Struct.chars('hex') not work

Hello, I use struct 0.0.6, but found Struct.chars('hex') has a problem:

var Struct = require('struct').Struct;
var Test = Struct()
    .chars("flag", 2, 'hex');

var buf = new Buffer(2);

var s = Test.clone();

buf[0] = 97;
buf[1] = 98;
console.log(buf.toString('hex', 0, 2));
// outputs "6162", it's right

s._setBuff(buf);
console.log(s.fields.flag);
// outputs "ab", but hex value "6162" expected

And Struct parse to buffer also have this problem:

var expected = new Buffer('aa', 'hex');
console.log(expected);
// outputs: <Buffer aa>

var Struct = require('struct').Struct;
var buf = new Buffer(1);
var ThisPacket = Struct()
    .chars("cmd", 1, 'hex');
var s = ThisPacket.clone();
s._setBuff(buf);
var fields = s.fields;
fields.cmd = 'aa';
console.log(buf);
// outputs: <Buffer 61>

It seems Struct.chars() only support Node's default encoding utf8.

If the length of the array is unknown

Hey. What about if the structure is an array and its length is not defined? I see an option to use an array with a larger advance. Is it going to be?

`word8` function removal broke `semver`

This change: 9c46c8a#diff-168726dbe96b3ce427e7fedce31bb0bcL190 is a breaking change, and only the patch version was upped.

This should have been a major version change.

Null-terminated string

No Null-terminating string support?

Add 'encoding' as 3rd param in Structs.chars

Hi Denys

Instead of hardcoding 'ascii' as the encoding of chars(name,length), if you can make it as an optional param that will be great.
something like
chars(name, length, encoding)

-sanjeev

Support for `chars` reversed byte order

Some buffer streams are returned in an internal byte order and are thus reversed. Could offer an option to support this or new functions.

Why '_' prefix for _setBuff

By looking at the API i dont see the reason why setting a buffer is prefixed with '_',i think that there should be a "public" method for this. Which is the reason for that decision?

How do I use the same template to read different buffers?

I am using struct to write the header of binary files. If the struct template is Header, is it ok to do:

Header.allocate();
var data1=Header.buffer();

fill data1 with my settings, and then do again

Header.allocate();
var data2=Header.buffer();

to create file number 2?