eriktim / elm-protocol-buffers

An Elm implementation of the Protocol Buffers specification

Protocol Buffers

This package lets you create encoders and decoders for (de)serializing data according to the Protobuf specification. Typically, you should not use this package directly as the required encoders and decoders can be generated directly from .proto specification files, e.g. by using this protoc plugin.

Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.

Motivation

As Evan explained in his vision on data interchange, there are many ways to send information between clients and servers, like JSON, XML, GraphQL and the one this package provides: Protocol Buffers.

Where Elm spoils us with an excellent type system, we lose this safety at the boundaries of our application, e.g. when talking to servers. Protobuf forces you to specify the interchange format explicity up front, making it type-safe. It also:

• makes you decouple code for data exchange from your business logic, enabling your application to evolve over time;
• makes you send fewer bytes over the wire;
• allows you to generate code using the same data interchange format for your backend as well;
• is Google's lingua franca for data.

This package provides an API to help converting between Elm values and bytes by implementing Protocol Buffers. elm-protocol-buffers also opens the door to add support for the communication protocol gRPC, which could be an interesting next step.

Extensive Example

Given a Protobuf .proto file

message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;

  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

  message PhoneNumber {
    required string number = 1;
    optional PhoneType type = 2 [default = HOME];
  }

  repeated PhoneNumber phone = 4;
}

this package handles converting between Person and Bytes values:

import Protobuf.Decode as Decode
import Protobuf.Encode as Encode



-- MODEL


type alias Person =
    { name : String
    , id : Int
    , email : String
    , phone : List PhoneNumber
    }


type alias PhoneNumber =
    { number : String
    , type_ : PhoneType
    }


type PhoneType
    = Mobile
    | Home
    | Work



-- ENCODE


toPersonEncoder : Person -> Encode.Encoder
toPersonEncoder person =
    Encode.message
        [ ( 1, Encode.string person.name )
        , ( 2, Encode.int32 person.id )
        , ( 3, Encode.string person.email )
        , ( 4, Encode.list toPhoneNumberEncoder person.phone )
        ]


toPhoneNumberEncoder : PhoneNumber -> Encode.Encoder
toPhoneNumberEncoder phoneNumber =
    Encode.message
        [ ( 1, Encode.string phoneNumber.number )
        , ( 2, toPhoneTypeEncoder phoneNumber.type_ )
        ]


toPhoneTypeEncoder : PhoneType -> Encode.Encoder
toPhoneTypeEncoder phoneType =
    case phoneType of
        Mobile ->
            Encode.int32 0

        Home ->
            Encode.int32 1

        Work ->
            Encode.int32 2



-- DECODE


personDecoder : Decode.Decoder Person
personDecoder =
    Decode.message (Person "" 0 "" [])
        [ Decode.required 1 Decode.string setName
        , Decode.required 2 Decode.int32 setId
        , Decode.optional 3 Decode.string setEmail
        , Decode.repeated 4 phoneNumberDecoder .phone setPhone
        ]


phoneNumberDecoder : Decode.Decoder PhoneNumber
phoneNumberDecoder =
    Decode.message (PhoneNumber "" Home)
        [ Decode.required 1 Decode.string setNumber
        , Decode.optional 2 phoneTypeDecoder setType
        ]


phoneTypeDecoder : Decode.Decoder PhoneType
phoneTypeDecoder =
    Decode.int32
        |> Decode.map
            (\value ->
                case value of
                    0 ->
                        Mobile

                    1 ->
                        Home

                    2 ->
                        Work

                    _ ->
                        Home
            )



-- SETTERS


setName : a -> { b | name : a } -> { b | name : a }
setName value model =
    { model | name = value }


setId : a -> { b | id : a } -> { b | id : a }
setId value model =
    { model | id = value }


setEmail : a -> { b | email : a } -> { b | email : a }
setEmail value model =
    { model | email = value }


setPhone : a -> { b | phone : a } -> { b | phone : a }
setPhone value model =
    { model | phone = value }


setNumber : a -> { b | number : a } -> { b | number : a }
setNumber value model =
    { model | number = value }


setType : a -> { b | type_ : a } -> { b | type_ : a }
setType value model =
    { model | type_ = value }

Known Limitations

This packages aims to support both proto2 and proto3. However, there are some limitations:

• JavaScript uses 64-bit floating point numbers for both integers and floats. Integers are only considered safe to use up to about 54 bits. For 64-bit integers, make sure to use Int64 and its related functions.

• Unknown fields are well-formed protocol buffer serialized data representing fields that the decoder does not recognize (for example when the field is removed from the .proto file but the server is not yet updated). Currently, these fields are ignored and hence are lost when re-serializing messages.

• extensions are currently not supported.

elm-protocol-buffers's Issues

Support for JSON encoders/decoders.

Proto3 supports canonical encoding in JSON, implementing JSON-based encoders and decoders would allow to easily plug elm in front of backends that use proto3 for API description. tiziano88/elm-protobuf that was implementing this, but unfortunately it doesn't seem to be maintained anymore and their protoc-gen-elm plugin doesn't generate valid code anymore.

gRPC-web support

Now that gRPC-Web seems to be becoming the standard to access gRPC backend
services from web app (https://grpc.io/blog/grpc-web-ga/), it would be great to support gRPC.

Any plans?

Relevant links:

• gRPC-Web "spec": https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-WEB.md#protocol-differences-vs-grpc-over-http2
• gRPC-Web js library & protoc plugin: https://github.com/grpc/grpc-web
• gRPC-Web Go wrapper & proxy and Typescript library & protoc plugin by Improbable - https://github.com/improbable-eng/grpc-web
• gRPC-Web envoy filter - https://www.envoyproxy.io/docs/envoy/latest/configuration/http_filters/grpc_web_filter

Doesn't handle multiple messages having the same `oneof` key

If you make

message Type1 {
  oneof my_options {
    string s = 1;
    int64 i = 2;
  }

message Type2 {
  oneof my_options {
    string s1 = 1;
    string s2 = 2;
  }
}

You end up with two declarations of:

type MyOptions
    = ...