percyqaz / percyqaz.data

Utility library for SQLite database storage, Markdown parsing and JSON mapping to and from F# types

Percyqaz.Data

Has expanded from its original purpose from a JSON mapping library, so I've updated the name to Percyqaz.Data
I am too lazy to update documentation right now (given that only my projects depend on this), but below is the README from a couple years ago

---

Percyqaz.Json

Yet another JSON library, specificially for my F# JSON needs which were not met by JSON.Net or Chiron

How it works:

1. Parsing and writing:
   • JSON text is parsed using FParsec into an F# discriminated union/tree.
   • The tree can be formatted back as valid JSON text.
2. Encoding and decoding:
   • Each type has a hand-made codec that maps between it and the JSON tree.
3. Ease of use:
   • Many important codecs are ready out-of-the-box.
   • It is simple to hand-write your own codecs for types.
   • Records and DUs can be tagged with an attribute to auto-generate a codec.

Key design principles

• No care for how the outputted JSON text is formatted (validity is the only concern)
• Null-safety: You can't deserialise JSON to get null for non-nullable F# objects etc
• Support for all simple F# types out-of-box
• Simple code: Exceptions are thrown internally instead of railroading/result types
• Easily extensible: You can add codecs for types as you need

Getting started

You can get this package on NuGet

DOCUMENTATION IS CURRENTLY OUTDATED FOR VERSION 3.0

IM LAZY AND ALSO NOBODY IS ACTIVELY USING THIS SO NOT A PRIORITY

You should read the documentation here (it's not long)

percyqaz.data's Issues

Support for Value Options

Typeshape doesn't have a FSharpValueOption recogniser
will look into making an issue/writing my own solution here/writing my solution as a PR for typeshape

for now Value Options use the default Struct Union codecs

Extreme performance issue on records without a Default implementation

We're talking minutes to deserialise a list of about 1000 records what on earth is going on here

Multithreaded access

Tests: Basic round trip for every built-in type

v3 of Json

Codec<'T> has To and From method

Use reflection to find an appropriate codec

Cache the codec members as delegates for performance

Benchmark

Support for F# Sets

currently unable to reuse the 'T list or 'T array codecs due to compiler not knowing if 'T : comparison

JsonParseResult -> the result or throw the error

very simple but useful feature to add

Bigger test coverage

Here's what needs to be covered

• Basic round trips for all the default rules (including float edge cases)

• Performance benchmarks once codecs are cached for small, medium, large objects

• Intended decode-time fail cases (null strings, null arrays, missing required properties in records, etc)

• Intended cache-time fail cases (no codec rule, invalid JsonCodec member)

• Round trips when implementing custom rules

• Basic API stuff (IO exceptions, etc)

Support for timespans and datetimes

Require attribute indicating no default record provided?

Most recent change removes the need for a Default implementation of records but now

• Records will now magically serialise in unplanned ways, for example you may make a typo in Default or otherwise believe you have given default values when you have in fact not.
• This assumes all members of the record do not need an original value when deserialising. While true for most things this may not be true for user defined deserialisation.
  • This also sparks an issue of marking user defined serialisation with "Expecting a non-null instance" but an idea on how to neatly design this hasn't come to me yet

Suggested fix: Make not having a default an attribute. Then at pickler-creation-time mistakes are caught instead of later down the line

Support for enums

Complete Record auto codec behaviour

• Flag "RequireAll" true means all props are non-optional
• Flag "RequireAll" false means anything missing in json data can be filled with defaults
• Default property used to provide defaults (otherwise use normal defaults)

Support for Guids

Recursive type support

Recursive types hang because there needs to be indirection when constructing the codecs

Update Readme and Documentation

v3 works differently to v1, documentation is accurate to latest release but not to latest code in repo

Support for struct tuples

Easy way to create auto codecs for external records/DUs

Add WithAutoCodec<'T> method to API that adds that type to a list
If in the list it uses auto codec even if there is no attribute

Percyqaz.Data

Features:

• CSV
• BSON
• SQLite drivers?

Support for DU auto codecs

Tests: Custom codecs (including generics)

turn Percyqaz.Json.Json namespace into object-oriented system instead

this is so you can create different json pickler setups with different rules rather than one global system (that can be side-effected in unexpected ways)

also will reduce some of the mess of globally adding rules as it will all be tied to an instance

Shorter qualified access to Percyqaz.Json.Json.JsonParseResult.X

preferable if this could be reached by Percyqaz.Json.JsonParseResult.X because then opening Percyqaz.Json allows the abbreviation X

Thread safety

Access to API needs to be concurrent safe - Maybe just use a concurrent dictionary as it's the only shared state

String formatter should write directly to stream

Codec combinator system

a handful of combinators for easy building of hand-made codecs for records would be nice

can probably take inspiration from chiron/fleece which are entirely this approach

Rewrite JSON to use less memory

• Replace intermediary JSON AST with a structure that reads through a stream and enumerates JSON nodes
• Replace serialisation with a structure that writes JSON nodes to a stream
• Replace all existing reader/writer logic

In short instead of parsing entire 100mb files of JSON into X mb of JSON-typed structures in memory, then transforming that into X mb of F# objects, scan the 100mb in as a stream and read objects out as you go

Error handling

Error midway through serialising -> What happens?

• Way to just end/cut short any open JSON nodes so the file is well formed
• Tell users not to throw exceptions in their serialisers

Whats the point of doing it if this is literally how System.Text/Newtonsoft works

Fun