chainsafe / ssz-js

Simple Serialize in TypeScript

This repo has moved to https://github.com/ChainSafe/lodestar/tree/master/packages/ssz

ssz-js's Issues

Cut a new release

Preferably after #31, #32, #33

Needed to flesh out ssz stubs in lodestar.

Add bytesN, remove hashN and address types

ethereum/consensus-specs#455 got merged

Remove/refactor hash and address encoding into bytesN encoding.

hashN is equivalent to bytesN.
address is equivalent to bytes20.

Update container field serialization order

Currently, the library lexographically sorts the container fields and serialized the values in that order.

The ssz spec has changed to require containers (js objects) to be serialized in the order which the container's fields are defined. (see the PR here and the spec here)

We should assume that type.fields is a list and use that ordering here

Add usage examples

What you think is missing

Examples on how to use this ssz library.

Proposed documentation to be added

We should add examples on how to serialize and deserialize data and be explicit on how have conforming inputs into these functions.

How would your proposed documentation help?

Adding usage examples will help people wanting to use this library in their projects.

Explicit uint and int

What functionality is being implemented

The reference implementation now supports uint's and int's explicitly. The serialize and deserialize functionality should follow.

Where is it referenced in the reference python implementation

##Serialize
https://github.com/ethereum/beacon_chain/blob/master/ssz/ssz.py#L10
https://github.com/ethereum/beacon_chain/blob/master/ssz/ssz.py#L14

##Deserialize
https://github.com/ethereum/beacon_chain/blob/master/ssz/ssz.py#L38
https://github.com/ethereum/beacon_chain/blob/master/ssz/ssz.py#L43

Add boolean serialisation

What functionality is being implemented

Boolean serialisation and deserialisation.

Where is it referenced in the reference python implementation

It is not in the Python implementation yet but is in the spec.
https://github.com/ethereum/eth2.0-specs/pull/68/files

Add typescript type declaration file

Is your feature request related to a problem? Please describe.
It would be nice to have typings available so lodestar development could take advantage of it.

Describe the solution you'd like
Add type declaration file according to https://www.typescriptlang.org/docs/handbook/declaration-files/introduction.html

License not correctly filled out

Please make sure that the license does not contain right or left arrows. If you search the license for it you'll find something like the below: <some_content_that_needs_to_be_replaced>

Fix BN imports

Fix BN imports so that users do not need non-standard typescript settings.

Complete serialize()

I have an initial PoC for the serialize function in index.js.
The only portion that is missing is the following:

elif isinstance(typ, type):
        length = int.from_bytes(data[start:start+4], 'big')
        values = {}
        pos = start + 4
        for k in sorted(typ.fields.keys()):
            values[k], pos = _deserialize(data, pos, typ.fields[k])
        assert pos == start + 4 + length
return typ(**values), pos

The above portion needs to be implemented in JS and can be found here: https://github.com/ethereum/beacon_chain/blob/master/ssz/ssz.py.

Test utility functions

Add tests for utilities in src/util/hash and src/util/types

• pack
• merkleize
• mixInLength
• copyType
• parseType
• isSSZType
• isBasicType
• isCompositeType

Add support for the tree hashing algorithm

What functionality is being implemented

Add support for the tree hashing algorithm.

Where is it referenced in the reference python implementation or the ssz specification

https://github.com/ethereum/eth2.0-specs/blob/master/specs/simple-serialize.md#tree_hash

Export assertValidValue

assertValidValue is a useful function that should be exported for use in eg: lodestar test utilities

Make this library work in the browser

Is your feature request related to a problem? Please describe.

This feature request is not related to a problem.

Describe the solution you'd like

Currently, this library works only in node.js. It makes use of node.js features that don't necessarily work in the browser. In order to get people using this library in their projects, it needs to be able to work in the browser.

Describe alternatives you've considered

Not applicable.

Additional context

Not applicable.

Return the deserialized value directly from `deserialize`

Is your feature request related to a problem? Please describe.
Its annoying to have to unwrap a deserialized value from an object whenever deserializing.

Describe the solution you'd like
It would be nice to have an option to deserialize directly to a value, rather than deserializing to an object containing the value.
Bonus points for still allowing the "extended form", { deserializedValue, offset }

eg:

const myValue: uint64 = deserialize(buf, uint64);

vs.

const myValue: uint64 = deserialize(buf, uint64).deserializedValue;

We'll want to do something like
add an optional parameter to deserialize to return the "extended form" of the deserialized output, (the extended form being whats returned currently)
eg:

const myValue: uint64 = deserialize(buf, uint64 /*, 0, false */ );
const extended: uint64 = deserialize(buf, uint64, 0, true); // <- note the optional param set to true
extended.deserializedValue == myValue

BigInt

Describe the solution you'd like
Are there any thoughts about the usage of BigInt instead of BN.js?

Additional context
Open discussion in the Web3.js repository: (web3/web3.js#2171)

Bundle for Web

Is your feature request related to a problem? Please describe.

From @wemeetagain (#35):

After doing some digging, it appears that, contrary to the exact language in #33, the standard way of dealing with Buffer (allowing it to work in the browser) is to let the 'last step' of the build process do the polyfill. This makes more sense than I originally thought, rationale below.

What that means, is that most libraries relying on Buffer (eg: our dependency @polkadot/util included) use Buffer assuming its availability, and then later consumers of these libraries (for web) would bundle in a polyfill as a step of the bundling process.

The reason for this approach is that it simplifies larger projects from unintended hackiness surrounding Buffer, specifically when running on node.
Imagine library B uses library A. If library A hardcodes the feross/buffer dependency, library B, (and even all consumers of library B), are bound to use that implementation, which is really meant as a browser polyfill.

It makes more sense for the bundler, which is already collating the tree of a project's dependencies, to have more information about the run-time environment, and make the appropriate choice at the end.
That way, all references to Buffer can either be linked at once, or all of them left alone.

Describe the solution you'd like
Configured bundler that includes a Buffer polyfill

Describe alternatives you've considered
See #33 #16

Additional context
N/A

deserialize function

What functionality is being implemented

Implementation of the deserialize functionality of SSZ. Does the reverse of the serialize function for all supported types.

Where is it referenced in the reference python implementation

https://github.com/ethereum/beacon_chain/blob/master/ssz/ssz.py#L33
https://github.com/ethereum/beacon_chain/blob/master/ssz/ssz.py#L72

Test hashTreeRoot

Add tests for hashTreeRoot

Add tests for uint24 and int24

What functionality is being implemented

We currently support unsigned/signed integers up to 256-bits. However, we forgot (u)int24 and its supporting tests.

Where is it referenced in the reference python implementation

It is specified in the official ssz specification here: https://github.com/ethereum/eth2.0-specs/blob/master/specs/simple-serialize.md#uint-816243264256-1
https://github.com/ethereum/beacon_chain/blob/ccf75cc679601b53e03fb5a410fc3262c503ace1/ssz/ssz.py#L10

Consider returning Uint8Array from serialize

Based on discussion here we settled on representing byte arrays as Uint8Arrays rather than Buffers.

We may want to align this library in that direction or revisit the discussion in the context of a Chainsafe-wide consistent approach.

Fix deprecated buffer usage

• in ssz bn is using deprecated buffer method and causing performance issue
• fix bn.js usage to use non deprecated buffer methods

Test signedRoot

Add tests for signedRoot

Explicitly import buffer library

As mentioned in gitter:
Import buffer library to allow for Buffer use in the browser.

Support fixed-size bignumber de/serialization

Currently, the library assumes the value is a javascript number if the type is uintX.

It looks like we'll need bignum support for signatures, etc. (see ProposerSlashing data structure for example of uint384)

Error in intByteLength

This should ensure bitSize > 0. Should probably also be called byteSize.

https://github.com/ChainSafeSystems/ssz-js/blob/0d93bf660323f771e632baa70f728dedd75dc35e/src/intBytes.js#L3-L9

Fix readme documentation

readme still refers to 0.4.x calling convention, should be updated for 0.5.x

Implementing to_dict()

In the python reference implementation of ssz, there is a function to_dict() that converts a class or a list to a python dictionary.

I think we don't need to implement all of the details of that function because Js classes are objects. Thus, we would only need to convert an array to an object.

Any thoughts?

Consider reordering deserialize parameters

Is your feature request related to a problem? Please describe.
The parameter order of deserialize differs minorly from serialize and treeHash, in that type is the third parameter, instead of the second.

Describe the solution you'd like
Swap the start and type parameters. This would maintain greater consistency between our function signatures and allow us to more practically make start an optional parameter, with a default of 0 (since in most cases you can assume wanting to deserialize from the beginning).

now:

let x: BeaconBlock = deserialize(data, 0, BeaconBlock);

proposed:

let x: BeaconBlock = deserialize(data, BeaconBlock);

Describe alternatives you've considered

• Leave as is
• Swap start and type, but keep start a required parameter

Additional context
noticed this during #31

remove -js

as it is in TS we should consider renaming or dropping -js entirely from it.

currently there is more shell than JS in the project according to github lol.

Add hash96/7 serialisation

What functionality is being implemented

Should serialise hash96/7 for BLS pubkeys

Where is it referenced in the reference python implementation

It is referenced in the spec:
https://github.com/ethereum/eth2.0-specs/blob/master/specs/simple-serialize.md#hash96
https://github.com/ethereum/eth2.0-specs/blob/master/specs/simple-serialize.md#hash97

Support Big-endian

@Mikerah: A little food for thought: Would it be valuable to have the option to encode integers in big endian? SSZ has some really nice properties that others might want to leverage in other projects that may or may not be related to Ethereum. They might require integers encoded in big endian. This could be a future feature and will not be a priority within the next few months.

After closing #25 i valid point was made by Mikerah to add support for big-endian. It would require a bit of a re-write of logic, but testing for it would be fairly straight forward. We could cherry-pick the previous test suite for big-endian and then create two test directories:

1. Supporting big-endian
2. support little-endian

Switch to integer encoding to little-endian

ethereum/consensus-specs#139 got merged.
Integers should now be encoded in little-endian format.
This includes prefixed lengths and encoded integers.

Comparing Beacon types - optimisation

What is your question?
Currently when checking if some ssz type is equal we serialize it and compare serialized buffer, which is time and memory consuming. To avoid it we could add method equals to ssz library which would compare every field of type natively (exmp. BN.equals(BN), buffer.equals(buffer))

Serialization of 64, 128, 256 integer types

What functionality is being implemented

The beacon chain state includes data types currently not supported by ssz-js, in particular uint64 and uint256. These should be implemented to be able to serialize the beacon chain state.

Where is it referenced in the reference python implementation

https://github.com/ethereum/beacon_chain/blob/master/beacon_chain/state/attestation_record.py#L13-L20

Error on incorrect type

What functionality is being implemented

Currently we simply return null if the type is not recognized. Since we are using strings for types mistakes should be expected and thus we should at least warn the user that the type is not valid.

Where is it referenced in the reference python implementation or the ssz specification

No specific mention afaik. Other implementations seem to throw errors on invalid types (eg. https://github.com/prysmaticlabs/prysm/blob/c5b196006d939ac2392a3b131a2b35a8a21cf45c/shared/ssz/encode.go#L101)

Write a test for serialize()

As mentioned in #2, serialize() is almost completed. We will need to write a test for it to make sure it coincides with the python implementation. Several core developers such as Danny Ryan are working on interoperability tests so that we can all check our implementations.