vindaar / flatbuffers

Another library for reading/writing Nim objects to binary files

flatBuffers

Note: You probably want to use https://github.com/treeform/flatty, unless you want a library that works on a Buffer type using manual allocation instead of stdlib string.

This is a helper package to make it easier to copy (potentially nested) Nim objects via procedures that only work on flat datatypes.

Typically, whenever Nim objects are used that either contain string or seq types, anything involving copyMem or otherwise procedures that require the data to be flat in memory are out.

By using asFlat (possibly combined with toString) you can convert a Nim object into a Buffer (or string). Using flatTo[T] you convert a Buffer back to a T. fromString can be used to reconstruct a Buffer from a string created via toString.

Why yet another flat binary library?

This is based on a bespoke library used in nimhdf5 that deals with strings and seqs (any variable length data) in a special way related to how HDF5 stores its data. When I needed the functionality for flatBuffers I just quickly turned that nimhdf5 submodule into this library. At the time I had forgotten flatty is even at thing.

flatbuffers's Issues

Add support for built-in sets?

I'm using this library for a small personal project and it serializes everything I want to except for built-in sets. Flatty also seemed to break (not compile) with built-in sets even though its code seemingly tries to do something with them. For example something like this won't compile:

import std/[strformat, os]
import pkg/flatBuffers

type
  FileWithPerms* = object
    path*: string
    permissions*: set[FilePermission]

const testFile = FileWithPerms(
  path: "/a/file",
  permissions: {fpUserRead, fpUserWrite},
)

proc flatbufRead[T](path: string): T =
  let buf = readFile(path).fromString
  result = flatTo[T](buf)

asFlat(testFile).writeBuffer("flatbuf-file.bin")
let flatbufFile = flatbufRead[FileWithPerms]("flatbuf-meta.bin")
echo fmt"File is {flatbufFile} after write/read"

Built-in sets should be easy to handle as they are basically just a big array of bytes under the hood. They also support copyMem. However my quick attempt at adding them to this library didn't work out, because I didn't find a way to get their base type. Maybe I should try adding them again, because copying them is easy and I found out some new things.

nim> var b: set[uint16]
nim> b.incl 65535
nim> b.incl 20000
nim> b.incl 20001
nim> b.incl 42184
nim> b.incl 12330
nim> b
{12330, 20000, 20001, 42184, 65535} == type set[uint16]
nim> sizeof b
8192 == type int
nim> var c: set[uint16]
nim> copyMem(c.addr, b.addr, 8192)
nim> c
{122, 12330, 20000, 20001, 42184, 65535} == type set[uint16]

Package naming conflict

Hi, I have an awkward request to make due to an issue with package names...

I'm working on a FlatBuffers package which implements a proper Nim package for Google's FlatBuffers serialization format (I think you can see the naming conflict coming).

Normally, I'd be more than happy to simply rename my package to something else (like flatbuffers-runtime), however part of Google's FlatBuffers toolset includes flatc, which can be used to generate Nim code to perform part of the serialization task. The main issue is that flatc --nim generates Nim code which expects a package named flatbuffers, but Google doesn't provide a proper nimble package for it (and also their implementation has bugs, but that's unrelated).

Because of their code generator, my package needs to be called flatbuffers to be compatible out-of-the-box, which unfortunately creates a naming conflict for nimble because this package already has the name.

So I hate to ask this, but would you consider possibly renaming your package? If not, I understand and will file an issue with Google to find a solution on their end.