pabigot / buffer-layout Goto Github PK

Union.encode will only recognize the discriminator and content of a source object if the property names are variant and content respectively.

In some applications structures may be extended over time in a forwards-compatible way, with the decoder accepting any buffer that is complete through a contiguous initial sequence of the structure members. Add support to optionally decode the prefix in this situation.

Hi,

I encounter a float problem. Is this normal ?

Code :

var lo = require("buffer-layout");
var ds = lo.f32();
var b = Buffer.alloc(ds.span);
ds.encode(3.2, b);
var c = ds.decode(b);
console.log("b:", b);
console.log("3.2:", c);

Output:

b: <Buffer cd cc 4c 40>
3.2: 3.200000047683716

Env :

• Node : 6.9.1
• OS : Ubuntu 16.04 x86_64

The discriminator for unions may be assumed to be integral, but need not be present within the union structure itself. Provide a way to obtain it from a layout element elsewhere in the buffer or by an oracle.

The code is strict, having to remember to use var instead of let/const in the tests is annoying.

Hey @pabigot, thanks for this library! We have been using it extensively for manual deserialization of blockchain data at Solana. I would like to propose that this library transition to be more browser friendly by using the Uint8Array type over the Node.js specific Buffer type. Currently we rely on browser polyfills but the latest bundler tools are dropping support for those so it'd be great to remove the need for polyfills entirely.

Are you open to this change being made to the next major release of buffer-layout? I'd be happy to own the transition since if we don't upstream this change, I'll be forking this package instead.

I needed to decode C strings from inside a char[4][40], eventually I created my own Layout class (which works) but feel I must have missed or misunderstood something. How can the below be achieved without the subclass?

class SubBufferLayout extends Layout {

  constructor( elementLayout, span ) {
    super( span )
    this.elementLayout = elementLayout
  }

  decode( b, offset ) {
    let sub = b.slice( 0, offset + this.span )
    return this.elementLayout.decode( sub, offset )
  }

}

And usage:

lo.seq( new SubBufferLayout( lo.cstr(), 40 ), 4, 'blahBlah' )

Base layout using UInt/UIntBE with a contained sequence of bit fields. MSB or LSB ordering should be allowed.

Nice library! It might be helpful to add examples or at least a quick note on the big-endian types. I wasn't able to find them without diving into the source.

Don't force it to be content; same for the default layout.

This would support object constructors that use the same order and names as the layout.

Support a customized single-bit bitfield wrapper which expresses the value as a bool rather than a number.

Use this to slice out pieces that can't reasonably be represented in a standard JavaScript type, such as unsigned 64-bit integers.

Only makes sense for Structure instances but this avoids the need to use lodash extend to set properties in a constructor.

As with UnionDiscriminator.

Hello I have an array of structs coming from a Rust app that I am trying to deserialize using buffer-layout. It sort of works but the values are coming in a strange sort of offset. Basically my struct has two fields: archive_id and created_on but the created_on is being filled into the next item in the array instead of together with the matching archive_id.

// this is the rust struct that is coming into my client js code
#[derive(BorshSerialize, BorshDeserialize, Debug)] pub struct ChatMessage { pub archive_id: String, pub created_on: String }

// this is the deserialization code in javascript
const archive_id = lo.cstr("archive_id"); const created_on = lo.cstr("created_on"); const dataStruct = lo.struct([archive_id, created_on], "ChatMessage"); const ds = lo.seq(dataStruct, CHAT_MESSAGE_ELEMENTS_COUNT); const messages = ds.decode(sentAccount.data);

// this is the how the results come in. note on rust side the logs look normal

The existing one fails to address the fact that the demonstration C struct would normally have internal padding that isn't accounted for in the Structure. Also an example for Union would be useful, since that's one of the main motivating use cases.

Problem

There are no type definitions available for TypeScript users

Like CString, but allow the length to be determined by the length of the provided buffer or value, rather than the offset of a NUL.

When a layout includes an anonymous Structure the layouts within that structure should be added directly to the object created using the parent layout.

For Structure, Union, BitStructure, and anything else that decodes into an object add an optional constructor property which is used to generate the empty object into which layout components are added as properties.

There appears to be a use case for a layout for undecoded data that, like UTF8, is variable length. An idiom of using a null layout handles top-level cases (the value is the buffer itself), but explicit expression would support variable length unions.

While JavaScript can't handle the full range, there should be an option to encode/decode these values to the closest Number that supports them.

This code:

const bs = lo.bits(lo.u8(), 'flags');
const str = lo.struct([bs]);

will result in failure to process the structure flags field because the intended property name is interpreted as the rarely-used second parameter boolean specifying the bit order, leaving the field without a property name and so with no way to represent the field content.

If the second parameter to lo.bits is a string and there is no third parameter, treat the second parameter as property and set msb to undefined.

The following should be permitted:

  const un = lo.union(lo.u8('mode'), lo.blob(4));
  un.addVariant(0, 'off');

so that off provides only the mode property with value zero.

TypeError, RangeError, and plain Error are used: make sure that use is consistent with best practices.

Now that buffer-layout is more browser friendly, it would be great to keep it that way and have tests to prevent future breakage.