piotr-oles / as-proto

Protobuf implementation in AssemblyScript

as-proto

Protobuf implementation in AssemblyScript

Features

• Encodes and decodes protobuf messages
• Generates AssemblyScript files using protoc plugin
• Produces relatively small .wasm files
• Relatively fast, especially for messages that contains only primitive types

Installation

This package requires Node 10.4+ or modern browser with WebAssembly support. Requires protoc installed for code generation.

# with npm
npm install --save as-proto
npm install --save-dev as-proto-gen

# with yarn
yarn add as-proto
yarn add --dev as-proto-gen

Code generation

To generate AssemblyScript file from .proto file, use following command:

protoc --plugin=protoc-gen-as=./node_modules/.bin/as-proto-gen --as_out=. ./file.proto

This command will create ./file.ts file from ./file.proto file.

Generated code example:

// star-repo-message.proto
syntax = "proto3";

message StarRepoMessage {
  string author = 1;
  string repo   = 2;
}

// star-repo-message.ts

// Code generated by protoc-gen-as. DO NOT EDIT.
// Versions:
//   protoc-gen-as v0.2.5
//   protoc        v3.21.4

import { Writer, Reader } from "as-proto/assembly";

export class StarRepoMessage {
  static encode(message: StarRepoMessage, writer: Writer): void {
    writer.uint32(10);
    writer.string(message.author);

    writer.uint32(18);
    writer.string(message.repo);
  }

  static decode(reader: Reader, length: i32): StarRepoMessage {
    const end: usize = length < 0 ? reader.end : reader.ptr + length;
    const message = new StarRepoMessage();

    while (reader.ptr < end) {
      const tag = reader.uint32();
      switch (tag >>> 3) {
        case 1:
          message.author = reader.string();
          break;

        case 2:
          message.repo = reader.string();
          break;

        default:
          reader.skipType(tag & 7);
          break;
      }
    }

    return message;
  }

  author: string;
  repo: string;

  constructor(author: string = "", repo: string = "") {
    this.author = author;
    this.repo = repo;
  }
}

Helper methods

In order to generate helper methods for encoding and decoding a message, pass the gen-helper-methods option with the as_opt parameter:

protoc --plugin=protoc-gen-as=./node_modules/.bin/as-proto-gen --as_opt=gen-helper-methods --as_out=. ./file.proto

This will add the following methods in a generated file:

export function encodeStarRepoMessage(message: StarRepoMessage): Uint8Array {
  return Protobuf.encode(message, StarRepoMessage.encode);
}

export function decodeStarRepoMessage(buffer: Uint8Array): StarRepoMessage {
  return Protobuf.decode<StarRepoMessage>(buffer, StarRepoMessage.decode);
}

Dependencies

This package will generate messages for all proto dependencies (for example import "google/protobuf/timestamp.proto";) unless you pass --as_opt=no-gen-dependencies option.

Usage

To encode and decode protobuf messages, all you need is Protobuf class and generated message class:

import { Protobuf } from 'as-proto/assembly';
import { StarRepoMessage } from './star-repo-message'; // generated file

const message = new StarRepoMessage('piotr-oles', 'as-proto');

// encode
const encoded = Protobuf.encode(message, StarRepoMessage.encode);
assert(encoded instanceof Uint8Array);

// decode
const decoded = Protobuf.decode(encoded, StarRepoMessage.decode);
assert(decoded instanceof StarRepoMessage);
assert(decoded.author === 'piotr-oles');
assert(decoded.repo === 'as-proto');

If the helper methods were generated, they can be used to reduce boilerplate code:

import {
  StarRepoMessage,
  encodeStarRepoMessage,
  decodeStarRepoMessage
} from './star-repo-message'; // generated file

const message = new StarRepoMessage('piotr-oles', 'as-proto');

const encoded = encodeStarRepoMessage(message);
const decoded = decodeStarRepoMessage(encoded);

Currently, the package doesn't support GRPC definitions - only basic Protobuf messages.

Performance

I used performance benchmark from ts-proto library and added case for as-proto. The results on Intel Core i7 2.2 Ghz (MacBook Pro 2015):

benchmarking encoding performance ...

as-proto x 1,295,297 ops/sec ±0.30% (92 runs sampled)
protobuf.js (reflect) x 589,073 ops/sec ±0.27% (88 runs sampled)
protobuf.js (static) x 589,866 ops/sec ±1.66% (89 runs sampled)
JSON (string) x 379,723 ops/sec ±0.30% (95 runs sampled)
JSON (buffer) x 295,340 ops/sec ±0.26% (93 runs sampled)
google-protobuf x 338,984 ops/sec ±1.25% (84 runs sampled)

               as-proto was fastest
  protobuf.js (reflect) was 54.5% ops/sec slower (factor 2.2)
   protobuf.js (static) was 55.1% ops/sec slower (factor 2.2)
          JSON (string) was 70.7% ops/sec slower (factor 3.4)
        google-protobuf was 74.1% ops/sec slower (factor 3.9)
          JSON (buffer) was 77.2% ops/sec slower (factor 4.4)

benchmarking decoding performance ...

as-proto x 889,283 ops/sec ±0.51% (94 runs sampled)
protobuf.js (reflect) x 1,308,310 ops/sec ±0.24% (95 runs sampled)
protobuf.js (static) x 1,375,425 ops/sec ±2.86% (92 runs sampled)
JSON (string) x 387,722 ops/sec ±0.56% (95 runs sampled)
JSON (buffer) x 345,785 ops/sec ±0.33% (94 runs sampled)
google-protobuf x 359,038 ops/sec ±0.32% (94 runs sampled)

   protobuf.js (static) was fastest
  protobuf.js (reflect) was 2.4% ops/sec slower (factor 1.0)
               as-proto was 33.8% ops/sec slower (factor 1.5)
          JSON (string) was 71.2% ops/sec slower (factor 3.5)
        google-protobuf was 73.2% ops/sec slower (factor 3.7)
          JSON (buffer) was 74.2% ops/sec slower (factor 3.9)

benchmarking combined performance ...

as-proto x 548,291 ops/sec ±0.41% (96 runs sampled)
protobuf.js (reflect) x 421,963 ops/sec ±1.41% (89 runs sampled)
protobuf.js (static) x 439,242 ops/sec ±0.85% (96 runs sampled)
JSON (string) x 186,513 ops/sec ±0.25% (94 runs sampled)
JSON (buffer) x 153,775 ops/sec ±0.54% (94 runs sampled)
google-protobuf x 160,281 ops/sec ±0.46% (91 runs sampled)

               as-proto was fastest
   protobuf.js (static) was 20.2% ops/sec slower (factor 1.3)
  protobuf.js (reflect) was 23.8% ops/sec slower (factor 1.3)
          JSON (string) was 65.9% ops/sec slower (factor 2.9)
        google-protobuf was 70.8% ops/sec slower (factor 3.4)
          JSON (buffer) was 72.0% ops/sec slower (factor 3.6)

The library is slower on decoding mostly because of GC - AssemblyScript provides very simple (and small) GC which is not as good as V8 GC. The as-proto beats JavaScript on decoding when messages contain only primitive values or other messages (no strings and arrays).

License

MIT

as-proto's Issues

Importing protos from the same namespace results in "Duplicate identifier" errors

a.proto

namespace foo;

message A {}

b.proto

namespace foo;

import "a.proto";

message B {
  optional A a = 1;
}

This results in assemblyscript like:

import { foo } from "a";

export namespace foo {
  // ...
}

This results in an error:

ERROR TS2300: Duplicate identifier 'foo'.

Similar errors also occur if you import multiple protos from the same namespace.

change import path in generated .ts files

Hi.
according to https://discord.com/channels/721472913886281818/721472913886281821/1002570366016819250 with as 2.0 the import of the as dependencies need to point explicitly to the assembly folder.
Unfortunately it seems to me that the .ts files generated by as-proto-gen starts with import { Writer, Reader } from "as-proto"; while now they need import { Writer, Reader } from "as-proto/assembly";.

I don't know if there is some way to autodetect the as version and/or some parameter to pass to protoc, but now it seems the tool is not fully compatible with as 2.0

Use of `unmanaged` on generated classes causing memory leaks

Hi,
We ran into an issue where the @unmanaged decorator that is added to certain generated classes causes a memory leak when decoding objects.

Specifically, we have a .proto file like

message NullableDouble {
  double value = 1;
  bool isNull = 2;
  string dummy = 3;
}

which results in Assemblyscript code like

// Code generated by protoc-gen-as. DO NOT EDIT.
// Versions:
//   protoc-gen-as v1.2.0
//   protoc        v3.21.12

import { Writer, Reader, Protobuf } from "as-proto/assembly";

@unmanaged
export class NullableDouble {
  static encode(message: NullableDouble, writer: Writer): void {
    writer.uint32(9);
    writer.double(message.value);

    writer.uint32(16);
    writer.bool(message.isNull);
  }

  static decode(reader: Reader, length: i32): NullableDouble {
    const end: usize = length < 0 ? reader.end : reader.ptr + length;
    const message = new NullableDouble();

    while (reader.ptr < end) {
      const tag = reader.uint32();
      switch (tag >>> 3) {
        case 1:
          message.value = reader.double();
          break;

        case 2:
          message.isNull = reader.bool();
          break;

        default:
          reader.skipType(tag & 7);
          break;
      }
    }

    return message;
  }

  value: f64;
  isNull: bool;

  constructor(value: f64 = 0.0, isNull: bool = false, dummy: string = "") {
    this.value = value;
    this.isNull = isNull;
  }
}

export function encodeNullableDouble(message: NullableDouble): Uint8Array {
  return Protobuf.encode(message, NullableDouble.encode);
}

export function decodeNullableDouble(buffer: Uint8Array): NullableDouble {
  return Protobuf.decode<NullableDouble>(buffer, NullableDouble.decode);
}

Since the decode function creates a new instance of NullableDouble each time (which is an unmanaged class) it appears to lead to constant growth of memory where these instances are not garbage collected. In ours tests, Assemblyscript would keep expanding the memory it had reserved, even while the actual "known" size of the managed memory heap was not increasing.

Fail to compile with assemblyscript >0.22

It compiles fine with assemblyscript <=0.22, but fails with 0.23 and 0.24

ERROR TS2564: Property '~lib/as-proto/assembly/internal/FixedWriter/FixedWriter#buf' has no initializer and is not assigned in the constructor before 'this' is used or returned.
    :
 24 │ private buf: Uint8Array;
    │         ~~~
    └─ in ~lib/as-proto/assembly/internal/FixedWriter.ts(24,11)
    :
 34 │ this.buf = new Uint8Array(this.sizer.len);
    │                           ~~~~~~~~~~
    └─ in ~lib/as-proto/assembly/internal/FixedWriter.ts(34,31)

All imports are treated as relative

If you have a proto in a subdirectory

a/b.proto

import "c/d.proto";

This generates assemblyscript like:

a/b.ts

import {...} from "./c/d";

This seems wrong because the compiler tries to look for a/c/d.ts, when in fact the file is at c/d.ts.

(Possibly something is weird/abnormal about my setup or I'm doing something wrong -- apologies if this is the case).

error with codification

This is my proto file

syntax = "proto3";

package mytest;

message ab {
   uint64 amount = 1 [jstype = JS_STRING];
}

message cd {
   ab value = 1;
   uint64 num = 2 [jstype = JS_STRING];
}

Then I'm trying to encode and decode this object:

const x = new mytest.ab(30000000000);
const y = new mytest.cd(x, 1);
const encoded = Protobuf.encode(y, mytest.cd.encode);
// encoded: 0x0a070880d88ee16f100100
Protobuf.decode<mytest.cd>(encoded, mytest.cd.decode);

and it fails with the error RuntimeError: unreachable

I just tried the same codification with protobufjs and I get a slightly different result:

0x0a070880d88ee16f100100 - as-proto
0x0a060880d88ee16f1001 - protobufjs

If I use the encoded value generated by protobufjs then as-proto is able to decode it. So the issue is with the codification. In particular the extra 00 that is added at the end.

I'm using [email protected]

Import fails with as-proto 0.5 and asc 0.22

Starting with this commit dfda224 the import generated by as-proto-gen changed from as-proto to as-proto/assembly. AssemblyScript's compiler is trying to resolve this path to as-proto/assembly.ts instead of as-proto/assembly/index.ts as specified in the package.json main option. The compiler fails with the following output

$ asb build
ERROR TS6054: File '~lib/as-proto/assembly.ts' not found.

 import { Writer, Reader } from "as-proto/assembly";
                                ~~~~~~~~~~~~~~~~~~~
 in assembly/proto/contract.ts(6,32)

/Users/Francis.Fortier/.nvm/versions/node/v16.17.0/lib/node_modules/asbuild/node_modules/yargs/build/lib/yargs.js:1132
                throw err;
                ^

[1 parse error(s)]

`null` return values do not follow proto3 spec

string and bytes proto3 types do not have correct default value semantics.

According to the Protobuf Documentation, string and bytes types should be default initialized to an empty string and empty byte array, respectively.

For reference, my proto type:

message event_arguments {
   string name = 1;
   bytes data = 2;
   repeated bytes impacted = 3;
}

Generates a class with the following fields and constructor:

  name: string | null;
  data: Uint8Array | null;
  impacted: Array<Uint8Array>;

  constructor(
    name: string | null = null,
    data: Uint8Array | null = null,
    impacted: Array<Uint8Array> = []
  ) {
    this.name = name;
    this.data = data;
    this.impacted = impacted;
  }

I believe removing the | null from Uint8Array and string generated fields would fix the problem.
Note: impacted appears to be correct and | null should probably remain on all fields marked optional.

Example with import from google/protobuf types

E.g. if you want to use timestamps, it makes sense to

import "google/protobuf/timestamp.proto";

When as-proto processes this definition it produces AssemblyScript like this:

import { google } from "./google/protobuf/timestamp";

but there is no such source file to import from, at least that I have been able to find. The fact that the import is processed to something that looks sensible makes me think it is intentional, so I need an example to see how it is supposed to be used.

cannot export class

It seems that the class generated by as-proto-gen is not compatible with as 2.0:

> asc assembly/index.ts --target debug

WARNING AS235: Only variables, functions and enums become WebAssembly module exports.
   :
 3 │ export class Person {
   │              ~~~~~~
   └─ in assembly/proto/person.ts(3,14)

using

$ npm list       
[email protected] /Users/a206719281/eclipse-workspace/_prova_as
+-- [email protected]
+-- [email protected]
+-- [email protected]
`-- [email protected]

Oneof/union types don't work very well

It's kind of a problem with AssemblyScript, not as-proto per se, that it doesn't support union types or undefined, but protobufs often contain "oneof" fields and as-proto currently doesn't do a great job of transpiling them.

Given this:

syntax = "proto3";

message Simple {
    oneof payload {
        bool flag = 1;
        int32 value = 2;
        bytes data = 3;
        string text = 4;
    }
}

as-proto will generate this:

// Code generated by protoc-gen-as. DO NOT EDIT.
// Versions:
//   protoc-gen-as v0.9.1
//   protoc        v3.19.6

import { Writer, Reader } from "as-proto/assembly";

export class Simple {
  static encode(message: Simple, writer: Writer): void {
    writer.uint32(8);
    writer.bool(message.flag);

    writer.uint32(16);
    writer.int32(message.value);

    writer.uint32(26);
    writer.bytes(message.data);

    writer.uint32(34);
    writer.string(message.text);
  }

  static decode(reader: Reader, length: i32): Simple {
    const end: usize = length < 0 ? reader.end : reader.ptr + length;
    const message = new Simple();

    while (reader.ptr < end) {
      const tag = reader.uint32();
      switch (tag >>> 3) {
        case 1:
          message.flag = reader.bool();
          break;

        case 2:
          message.value = reader.int32();
          break;

        case 3:
          message.data = reader.bytes();
          break;

        case 4:
          message.text = reader.string();
          break;

        default:
          reader.skipType(tag & 7);
          break;
      }
    }

    return message;
  }

  flag: bool;
  value: i32;
  data: Uint8Array;
  text: string;

  constructor(
    flag: bool = false,
    value: i32 = 0,
    data: Uint8Array = new Uint8Array(0),
    text: string = ""
  ) {
    this.flag = flag;
    this.value = value;
    this.data = data;
    this.text = text;
  }
}

The "payload" field isn't even referenced, and the constructor forces the Simple type to have all of the "oneof" properties (not one of them).

If you try it with ts-proto instead it can generate a union type, e.g.

export interface Simple {
  payload?:
    | { $case: "flag"; flag: boolean }
    | { $case: "value"; value: number }
    | { $case: "data"; data: Uint8Array }
    | { $case: "text"; text: string };
}

but that won't work in AssemblyScript. It would need to be something like this, which I think is what is meant by the very cursory comment on union types in the language guide (https://www.assemblyscript.org/status.html#language-features):

import { Reader, Writer, Protobuf } from "as-proto/assembly/index";

export const protobufPackage = "";

export class SimplePayload {
	$case: string = "text";
	flag: bool = false;
	value: i32 = 0;
	data: Uint8Array = new Uint8Array(0);
	text: string = '';
	static text(text: string) : SimplePayload {
		var instance = new SimplePayload();
		instance.$case = "text";
		instance.text = text;
		return instance;
	}
	static data(data: Uint8Array) : SimplePayload {
		var instance = new SimplePayload();
		instance.$case = "data";
		instance.data = data;
		return instance;
	}
	static value(value: i32) : SimplePayload {
		var instance = new SimplePayload();
		instance.$case = "value";
		instance.value = value;
		return instance;
	}
	static flag(flag: bool) : SimplePayload {
		var instance = new SimplePayload();
		instance.$case = "flag";
		instance.flag = flag;
		return instance;
	}
}

export class Simple {

	payload: SimplePayload = new SimplePayload();

	static encode(message: Simple, writer: Writer): void {
		if (message.payload) {
			if (message.payload.$case === "flag") {
				writer.uint32(8);
				writer.bool(message.payload.flag);
			}
			if (message.payload.$case === "value") {
				writer.uint32(16);
				writer.int32(message.payload.value);
			}
			if (message.payload.$case === "data") {
				writer.uint32(26);
				writer.bytes(message.payload.data);
			}
			if (message.payload.$case === "text") {
				writer.uint32(34);
				writer.string(message.payload.text);
			}
		}
	}

	static decode(reader: Reader, length: i32): Simple {
		const end: usize = length < 0 ? reader.end : reader.ptr + length;
		const message = new Simple();
		while (reader.ptr < end) {
			const tag = reader.uint32();
			switch (tag >>> 3) {
				case 1:
					message.payload = SimplePayload.flag(reader.bool());
					break;
				case 2:
					message.payload = SimplePayload.value(reader.int32());
					break;
				case 3:
					message.payload = SimplePayload.data(reader.bytes());
					break;
				case 4:
					message.payload = SimplePayload.text(reader.string());
					break;
				default:
					reader.skipType(tag & 7);
					break;
			}
		}
		return message;
	}
}

Maps create compilation errors in generated code because entry type is not defined

Input:

syntax = "proto3";

message TestMessage {
    bytes payload = 1;
    map<string, string> headers = 2;
}

Output:

// Code generated by protoc-gen-as. DO NOT EDIT.
// Versions:
//   protoc-gen-as v0.4.0
//   protoc        v3.19.4

import { Writer, Reader } from "as-proto";

export class TestMessage {
  static encode(message: TestMessage, writer: Writer): void {
    writer.uint32(10);
    writer.bytes(message.payload);

    const headers = message.headers;
    for (let i = 0; i < headers.length; ++i) {
      writer.uint32(18);
      writer.fork();
      TestMessage.HeadersEntry.encode(headers[i], writer);
      writer.ldelim();
    }
  }

  static decode(reader: Reader, length: i32): TestMessage {
    const end: usize = length < 0 ? reader.end : reader.ptr + length;
    const message = new TestMessage();

    while (reader.ptr < end) {
      const tag = reader.uint32();
      switch (tag >>> 3) {
        case 1:
          message.payload = reader.bytes();
          break;

        case 2:
          message.headers.push(
            TestMessage.HeadersEntry.decode(reader, reader.uint32())
          );
          break;

        default:
          reader.skipType(tag & 7);
          break;
      }
    }

    return message;
  }

  payload: Uint8Array;
  headers: Array<TestMessage.HeadersEntry>;

  constructor(
    payload: Uint8Array = new Uint8Array(0),
    headers: Array<TestMessage.HeadersEntry> = []
  ) {
    this.payload = payload;
    this.headers = headers;
  }
}

export namespace TestMessage {}

The output kind of looks OK, but the HeadersEntry type is never defined.

Relative imports to google's type

Hello. I have the following protobuf definition

~/proto/contract/contract.proto

syntax = "proto3";

import "google/protobuf/timestamp.proto";

message Metadata {
  google.protobuf.Timestamp creation_time = 1;
}

when generating the assembly script, it creates the following import

import {Timestamp} from '/some/absolute/path/where/my/code/is/google/protobuf/Timestamp.ts`

class Metadata {
  //...
}

The reason for this is this code:

as-proto/packages/as-proto-gen/src/generate/file.ts

Lines 80 to 95 in 9a242cf

	export function getOutputFilePath(
	fileDescriptor: FileDescriptorProto,
	messageOrEnumDescriptor: DescriptorProto | EnumDescriptorProto,
	parentMessageDescriptors: DescriptorProto[] = []
	) {
	const messageName = messageOrEnumDescriptor.getName();
	assert.ok(messageName !== undefined);
	const outputFileName = sanitizeFileName(`${messageName}.ts`);
	return [
	getFilePrefix(fileDescriptor),
	getNestedMessagePrefix(parentMessageDescriptors),
	outputFileName,
	].join("/");
	}

The file prefix and the nested message prefix will both be undefined because it's not a nested message and there's no package defined. Which means taht the output file path will be //Metadata.ts so when this code will be executed

as-proto/packages/as-proto-gen/src/generate/ref.ts

Lines 46 to 49 in 9a242cf

	const relativeMessageOrEnumFilePath = path.relative(
	path.dirname(fileContext.getFilePath()),
	messageOrEnumFilePath
	);

It will try to find the relative path of //Metadata.ts with google/protobuf/Timestamp.ts which will result in an absolute path containing the current working directory.

The workaround is to always set a package name to get proto/contract//Metadata.ts and this seems to work correctly.

Proto extensions are not supported

(Please correct me if I'm missed something).

as-proto should export JS files, and not TS

Right now https://github.com/piotr-oles/as-proto/blob/main/packages/as-proto/package.json#L4 uses the index.ts as a main. When should probably be a js file

Meaning consumer needs to compile it, and if using Typescript already may have some issues working in that package's own config