Force Defaults option for Serialization about flatsharp

[jamescourtney]

Interesting -- I wasn't familiar with that one. I'm really curious what your use case for this is, if you don't mind expanding. It's something I can definitely look at adding.

I see the Google folks have it as part of their Buffer Builder and not part of the schema. That hints to me that this is a behavior people may turn on or off selectively. Is that what you had in mind? Or did you have a table or two that you always want to write defaults?

The other thing FlatSharp can't give you is the IsValuePresent from the FlatBuffer page. That's just not something the model here allows me to bubble up.

Of course, this would only apply to non-nullable scalars. Vectors, Unions, Strings, Structs, etc would all still need to default to null and not be written.

[jamescourtney]

As an aside, do optional scalars not do what you want?

table Foo
{
    Bar:int = null;
}

FlatSharp treats this as an int? that you can test for null.

[Jabotical]

Yes exactly, we have a table where we always want to write defaults. This is because we're relying on the ability to mutate the fields in C++ before passing the serialized data along.

So it really doesn't have anything to do with testing whether or not the field is present -- quite the opposite really, we want its presence guaranteed.

I recognize this goes against what is arguably a fundamental design element of Flatbuffers. Though the existence of force_defaults (ForceDefaults() in the C# FlatBufferBuilder, as you noted) indicates we're not alone in desiring this particular kludge.

Most of these fields are enums. We could create an unused default 0 assignment for all of those, but that seems even kludgier and harder to maintain.

The official schema writing guide also brings up the possibility of wrapping each field in a struct. We could certainly do that, but again it's rather ungainly -- and we're trying to use this table as part of a public API, so we're hoping to avoid anything too cumbersome in there.

We don't really need particularly fine-grained control -- even if we could turn this behavior on globally, that'd be fine by us at present (though as you observed, the setting might most naturally apply at the Builder/Serializer level. Not that we'd mind if it could be applied via a table property). I've even started trying to figure out how to hard-code it in your Compiler library, though if there's a possibility of you potentially adding it more officially we certainly wouldn't complain!

[jamescourtney]

I see; this makes sense.

If the ask is to add a way to suppress the default checks on a scalar field, then I can definitely help with that, and it's not a big change at all. The test cases will be harder than the code.

My mental model right now is to:

1. Extend the FlatBufferItemAttribute to include a ForceScalarDefaults option, which will bomb on non-scalar / enum fields.
2. Add a new FBS metadata extension that applies to table fields that invokes this behavior. Something like fs_forceDefaults.

So, the current state of the repo is that I'm working on the next major version of FlatSharp (5.X). There are some breaking changes in version 5, but they are relatively minor and are documented in #102. The quality is what I'd consider a high-quality beta (perhaps even a release candidate), with the biggest outstanding item as documentation. My preference is to add this in master and not try to backport it. Are you comfortable testing with a prerelease version?

[Jabotical]

Yes, that's the ask all right! We want to suppress the heck out of the default checks on scalar/enum fields.
Great to hear it doesn't sound too daunting, and that you might be up for making that happen.

So we'd tag each table field individually? Totally acceptable if so. Though if there were a way to apply it to all scalars/enums in a table, that would be even slicker.

Yes, we'd definitely be game to try the prerelease version, especially considering your confidence in it. None of those breaking changes look like they should affect us (we use Unions, but don't clone 'em).

[jamescourtney]

I have this checked in, and some preliminary documentation. The nuget packages are still on version 4.3, but this will be revved to 5 pretty quickly. You'll need to drop them in a local feed or just try using the compiler from the command line (the syntax is just FlatSharp.Compiler.exe -i "input file" -o "output directory")

The PR is here: #118
The prelease nuget packages are in this appveyor build under "artifacts": https://ci.appveyor.com/project/jamescourtney/flatsharp/builds/38075204/artifacts
The docs for the new FBS annotation are here: https://github.com/jamescourtney/FlatSharp/wiki/FBS-Annotations-(FlatSharp-5.X)

As far as the naming of ForceWrite and fs_forceWrite, I'm still a bit on the fence. Part of me prefers something more descriptive like fs_forceWriteScalars, but I also like keeping the metadata attributes short. Do you have any suggestions?

[Jabotical]

Many thanks, that appears to work great so far! Will be nice when the official nuget packages include it, but we can certainly use your appveyor build for now.

Hmm, tough call on the name. I like fs_forceWrite because it exactly mirrors what appears to be the standard name for that Flatbuffers feature. On the other hand, if one is not already familiar with said feature, fs_forceWriteScalars makes it more clear what the property does.

I suppose your preference for keeping the metadata attributes short would tip the scales towards fs_forceWrite. Especially since it can be used to decorate individual fields, the "scalar" part maybe seems a little redundant there? And it's not like you haven't clearly documented what it does (in addition to more general Flatbuffers documentation talking about the same feature). I say keep it as is (though either would be fine by me -- I'm also a fan of descriptiveness)

[jamescourtney]

I'm glad it works! I think I'll stick with the fs_forceWrite naming, even though I'm not thrilled with it.

As far as nuget goes, I'm finishing up a few odds and ends relating to struct vectors, updating the samples, and documentation. FlatSharp is in a little bit of an odd spot with the current size of the project. It's big enough that people use it (and yell at me if I break something!), but not big enough that I can put out a release candidate and expect to get any feedback on it, so releases for me are about test cases, code coverage, and my gut feeling about the quality of the bits.

I'm hopeful that I can publish version 5 in the next week or so, but I need to spend some time trying to break it and looking for testing gaps. It is nice to know that everything is working for you, though!

[jamescourtney]

ETA for 5.0 to be released is in the next 48 hours. I'm just running the last set of performance validations to ensure there haven't been any regressions in that area, and that testing takes quite a bit of time.

[Jabotical]

Very cool! Looking forward to seeing how this milestone release treats us (though I imagine it should be pretty close to the 4.3 we're currently using).

Oh, and congrats on having your project grow big enough to have people yell at you if you break something. Yeah, having a good set of automatic test cases should provide pretty solid release confidence for a project like this, since you can so deterministically cover the preponderance of ways it would be used (and when you get yelled at, you can add another test case).