go-edn / edn

Go implementation of EDN (Extensible Data Notation)

Go implementation of EDN, extensible data notation

go-edn is a Golang library to read and write EDN (extensible data notation), a subset of Clojure used for transferring data between applications, much like JSON or XML. EDN is also a very good language for configuration files, much like a JSON-like version of YAML.

This library is heavily influenced by the JSON library that ships with Go, and people familiar with that package should know the basics of how this library works. In fact, this should be close to a drop-in replacement for the encoding/json package if you only use basic functionality.

This implementation is complete, stable, and presumably also bug free. This is why you don't see any changes in the repository.

If you wonder why you should (or should not) use EDN, you can have a look at the why document.

Installation and Usage

The import path for the package is olympos.io/encoding/edn

To install it, run:

go get olympos.io/encoding/edn

To use it in your project, you import olympos.io/encoding/edn and refer to it as edn like this:

import "olympos.io/encoding/edn"

//...

edn.DoStuff()

The previous import path of this library was gopkg.in/edn.v1, which is still permanently supported.

Quickstart

You can follow http://blog.golang.org/json-and-go and replace every occurence of JSON with EDN (and the JSON data with EDN data), and the text makes almost perfect sense. The only caveat is that, since EDN is more general than JSON, go-edn stores arbitrary maps on the form map[interface{}]interface{}.

go-edn also ships with keywords, symbols and tags as types.

For a longer introduction on how to use the library, see introduction.md. If you're familiar with the JSON package, then the API Documentation might be the only thing you need.

Example Usage

Say you want to describe your pet forum's users as EDN. They have the following types:

type Animal struct {
	Name string
	Type string `edn:"kind"`
}

type Person struct {
	Name      string
	Birthyear int `edn:"born"`
	Pets      []Animal
}

With go-edn, we can do as follows to read and write these types:

import "olympos.io/encoding/edn"

//...


func ReturnData() (Person, error) {
	data := `{:name "Hans",
              :born 1970,
              :pets [{:name "Cap'n Jack" :kind "Sparrow"}
                     {:name "Freddy" :kind "Cockatiel"}]}`
	var user Person
	err := edn.Unmarshal([]byte(data), &user)
	// user '==' Person{"Hans", 1970,
	//             []Animal{{"Cap'n Jack", "Sparrow"}, {"Freddy", "Cockatiel"}}}
	return user, err
}

If you want to write that user again, just Marshal it:

bs, err := edn.Marshal(user)

Dependencies

go-edn has no external dependencies, except the default Go library. However, as it depends on math/big.Float, go-edn requires Go 1.5 or higher.

License

Copyright © 2015-2019 Jean Niklas L'orange and contributors

Distributed under the BSD 3-clause license, which is available in the file LICENSE.

edn's Issues

What is the recommended way to init "globalTags"?

What is an idiomatic way to "create" go-edn with custom conversion functions (that are added to globalTags via edn.AddTagFn)?

i.e. does not seem reasonable to call "edn.AddTagFn("uuid", uuid.FromString)" every time before unmarshalling the message, hence edn needs to be initialized beforehand.

It could be a Go question rather than go-edn. I can certainly do my own init() function that I will call somewhere in main() but that feels leaky.

Better handling for missing or badly formatted tag functions

The format of a tag function is:

func foo(input T) (U, error) {
  // ...
}

More than once I've forgotten to include the error in the tags return type (making it just func(T) U). This package seems to silently fail when this happens, in my case returning the tag itself as a value in the unmarshald output. I kept thinking it was because I misspelled the tag, or something to that affect. Especially because this is the same behaviour it has for missing tags. A better default would be to consider missing tags an error or badly formulated tag functions as an error.

Pretty print edn when marshalling

Is it possible to marshal edn data into nicely formatted string? Currently MarshelIndent produces this:

{
  :deps {
    io.netty/netty-buffer {
      :mvn/version "4.1.30.Final"
    },
    potemkin {
      :mvn/version "0.4.5"
    }
  }
}

when I want this:

{:deps    {potemkin              {:mvn/version "0.4.5"}
           io.netty/netty-buffer {:mvn/version "4.1.30.Final"}}}

Use JSON struct tags as fallback for missing EDN tags?

I have a library which contains some generated code which uses json tags only for struct fields, and I'd like to be able to use edn to marshal/unmarshal the data types, but unfortunately I need to change each struct field and add edn tags in order to have the same tag names as the JSON ones.

Is there a way to use go-edn in a way which fallbacks to JSON tags if there is no EDN tag for the struct field?

Thanks!

Improve decode performance further

The performance of the implementation should be good for decoding, but it could be better.

Improve error messages

single quote is a valid rune for symbols and keywords

Currently symbols and keywords that have single quotes (i.e. :info-didn't-change) will fail to parse since "'" is not a valid rune.

Since okSymbol function in lexer.go is used for more than just symbols, I am not sure where you think is best to add it.

For now I added it here: https://github.com/tolitius/go-edn/blob/v1/lexer.go#L251

That is of course if you agree that "'" is a valid rune

Rewrite rules

Add rewrite rules to the implementation.

Although it's possible to rewrite with decoding then encoding, we lose discard tokens, comments and map/set ordering. That's not an option if you want to modify user configurations.

Decoder decodes wrong value for structs in certain cases

The code

package main

import (
    "fmt"

    "gopkg.in/edn.v1"
)

type Struct struct {
    Foo string
}

func main() {
    var val Struct
    err := edn.UnmarshalString(`{:foo "1", :bar "2"}`, &val)
    fmt.Println(err)
    fmt.Printf("%#v\n", val)

    err = edn.UnmarshalString(`{:foo "1", :bar 2}`, &val)
    fmt.Println(err)
    fmt.Printf("%#v\n", val)
}

will print out

<nil>
main.Struct{Foo:"2"}
edn: cannot unmarshal int into Go value of type string
main.Struct{Foo:"1"}

However, it should print out

<nil>
main.Struct{Foo:"1"}
<nil>
main.Struct{Foo:"1"}

Note that order is important, i.e. {:bar "2", :foo "1"} and {:bar 2, :foo "1"} works as expected.

Set to []string converter

What would be a good way to have go-edn convert a set #{:a :b} to []string{"a" "b"}? I understand that the reverse is tricky, but in this case I just need "EDN => Go".

Would it be edn.AddTagFn + using some internal parse tools from go-edn?

An alternative would be to parse EDN sets to map[string]bool and just know I need to convert them "manually" before I each use.

Partial Unmarshalling

Is there a way to partially unmarshall EDN. For example:

var data = `{:name "Hans",
             :born 1970,
             :pets [{:name "Cap'n Jack" :kind "Sparrow"}
                    {:name "Freddy" :kind "Cockatiel"}]}`

to leave pets as EDN via something like:

type Person struct {
	Name      string
	Birthyear int    `edn:"born"`
	Pets      []byte `edn:"*"`
}

This is to later persist Person structs to a schema based (i.e. SQL/Cassandra/etc) store, with having an ability to persist Pets "as EDN" bytes, so Clojure programs would be able to edn/read-string these parts of Person after reading them back from a store.

Of course in the real life scenario Pets may be a quite nested and complex data structure that just would not need a corresponding schema: would need to be saved "as is" to a store.

struct encoding and whitespace

was decoding a struct and the resulting map does not have whitespace between the keyword and value and between the next value?

in this case I was expecting {:name \"Hans\" :born 1990}

func TestMapEncoding(t *testing.T) {

	type Person struct {
		Name      string `edn:"name"`
		Birthyear int    `edn:"born"`
	}
	user := Person{Name: "Hans", Birthyear: 1990}

	dat, _ := Marshal(user)
	if string(dat) != "{:name \"Hans\" :born 1990}" {
		t.Error("fails", string(dat)) 
	}
}

Crash with nil in set

Hi,

It looks like there is a problem with nils in sets. For example, using the edn_pp program, I get a panic with #{ 1 2 nil 3}:

~/projects/go/edn/examples/edn_pp: ./edn_pp 
1
1
"hello"
"hello"
\c
99
[1 2 3]
[1 2 3]
#{1 2 3}
#{1 2 3}
#{1 2 nil 3}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0xa0 pc=0x80ef6]

goroutine 1 [running]:
gopkg.in/edn%2ev1.(*Decoder).Decode.func1(0x8202ffe60)
    /Users/russolsen/projects/go/play/src/gopkg.in/edn.v1/decode.go:215 +0x88
gopkg.in/edn%2ev1.(*Decoder).setInterface(0x8202da0e0, 0x0, 0x0)
    /Users/russolsen/projects/go/play/src/gopkg.in/edn.v1/decode.go:844 +0x256
gopkg.in/edn%2ev1.(*Decoder).set(0x8202da0e0, 0x127320, 0x8202ccd30, 0xd4)
    /Users/russolsen/projects/go/play/src/gopkg.in/edn.v1/decode.go:783 +0x2f8
gopkg.in/edn%2ev1.(*Decoder).value(0x8202da0e0, 0x11a300, 0x8202ccd30, 0x16)
    /Users/russolsen/projects/go/play/src/gopkg.in/edn.v1/decode.go:417 +0x245
gopkg.in/edn%2ev1.(*Decoder).Decode(0x8202da0e0, 0x11a300, 0x8202ccd30, 0x0, 0x0)
    /Users/russolsen/projects/go/play/src/gopkg.in/edn.v1/decode.go:231 +0x177
main.main()
    /Users/russolsen/projects/go/edn/examples/edn_pp/edn_pp.go:21 +0x136

Treat EDN nil as Go zero value

some systems that I don't control could send:

{:a nil :b 34}

which would result in:

edn: cannot unmarshal nil into Go value of type string.

it would make sense to convert EDN nils to a Go type zero values, or simply drop them. Then we can either not worry about it or use ",omitempty" to ignore them.

What do you think?

Add EncodeConvert to implementation

We have edn.AddTagFn and edn.AddTagStruct which makes it possible to convert any tagged EDN value into any untagged value.

edn.EncodeConvert would complement edn.AddTagFn, taking in a function of one argument that returns two arguments: The value to actually encode, and an error if something went wrong.

edn.EncodeAsTag would complement edn.AddTagStruct in the same fashion.

In this way, people can use other people's libraries without having to make wrapper types for every type they export. For example, attaching a definition for time.Time would look like this:

edn.EncodeConvert(func(t time.Time) (edn.Tag, error) {
    return edn.Tag{"inst", t.Format(time.RFC3339Nano)}, nil
})