mbraceproject / fspickler

A fast multi-format message serializer for .NET

FsPickler

FsPickler is a serialization library that facilitates the distribution of .NET objects. The implementation focuses on performance and completeness in supported types, including F# types. It supports multiple, pluggable serialization formats including Binary, Xml, JSON and BSON. The library is based on the functional programming concept of pickler combinators which has been adapted to accommodate the object oriented nature of the .NET framework.

Packages of the library are available on Nuget [1,2].

Documentation

• Tutorial A short introduction to FsPickler.
• Technical Overview A walkthrough of the library's implementation details.
• New .NET Core Benchmarks.
• API Reference Auto-generated library documentation.

All documentation and related material can be found here.

Build Status

Head (branch master), Build & Unit tests

• Windows
• Linux

fspickler's Issues

minor release 1.3.0 was not backwards compatible

I just upgraded from 1.2.21 to 1.3.5 and apparently ran into the renaming in core API that occurred in 1.3.0. According to semantic versioning this should have been a major (2.0.0) release because of the breaking change

FsPickler.CreateBinary() ==> FsPickler.CreateBinarySerializer()

All well after I changed my code.

Questions about "_flags"

Hey there, just had two quick questions about how certain things in my Pickled JSON end up as an object that looks like:

{ "_flags": "cached", "id": 45 }

Is that some form of JSON compression?
If so, is there any way to return the uncompressed JSON without attempting to convert it to the type that was used to make it in the first place?

Thanks!

NonSerializableTypeException with ISerializable types

I'm trying to serialize types (e.g. ZonedDateTime) from the NodaTime library which are marked as ISerializable (source), though also contain non-serializable fields. When generating a pickler using:

let p : Nessos.FsPickler.Pickler<NodaTime.OffsetDateTime> = Nessos.FsPickler.FsPickler.GeneratePickler()

it throws NonSerializableTypeException: Type 'NodaTime.OffsetDateTime' contains non-serializable field of type 'NodaTime.CalendarSystem'.

Should FsPickler prefer the ISerializable implementation over field based serialization? Alternatively, should we encourage NodaTime to support DataContract based serialization instead?

Do not change binary format all the time

It's very painful, especially considering that other packages like Suave depend on FsPickler and updating them causes FsPickler update, which in turn causes huge problems on production, when we have serialized data which should be readable by next release of our system.

Versioning

Protobuf supports versioning by assigning number to each field. As long as type and number matches, serialized data can be deserialized correctly. I can rename the actual field, move it around, add new fields (which would have default value if serialized data didn't have the field value).

Is it possible to achieve the same with FsPickler? Or as soon as I change schema things would break?

Also, have you tried to test serialized size comparison of FsPickler to other serializers?

Types serialized by 32bit process cause 'stream error' when deserialized by 64bit process in v 0.51

Hi - I have tested this fairly thoroughly (v 0.5.1) and it appears to be consistently failing in the above scenario. I tested it against a build of the current master and the issue appears to be gone. Would you mind releasing a new nuget package?

Xamarin Support

Right now, FsPickler isn't a PCL library and I don't know if it can be. It it can, it will be easy to have it work for Xamarin iOS + Android.

I have a prototype branch of FsPickler on Xamarin here: https://github.com/TIHan/FsPickler/tree/xamarin. It will only work if you have OSX and Xamarin Studio + iOS/Android SDKs installed. These are platform specific projects. Just open FsPickler.Mac.sln and look at the src directory. FsPickler.Touch + FsPicker.Droid.

I do not know what else is necessary for this to happen, but I figure, this is a good start.

Samples to use with C@

I heard FsPickler can be used with C# but didn't find any examples. Is it possible to use it in C#?

FSharp.Core nuget 4.0.0.1

hello, the latest FSharp.Core nuget is 4.0.0.1 is there a particular reason why you hardcoded the 3.1.2.5 version in the paket.dependencies file?

https://www.nuget.org/packages/FSharp.Core

Pickling something through an FSX session and unpickler in an app

If I try to do the above, I get an error such as: -

Nessos.FsPickler.FsPicklerException was unhandled by user code
Message: An exception of type 'Nessos.FsPickler.FsPicklerException' occurred in FsPickler.dll but was not handled in user code
Additional information: Error deserializing object of type 'Microsoft.FSharp.Collections.FSharpList`1[MyNamespace.MyType]'.

Expected pickle of type 'Microsoft.FSharp.Collections.FSharpList`1[MyNamespace.MyType]' but was 'Microsoft.FSharp.Collections.FSharpList`1[FSI_0002.MyNamespace.MyType]'.

Should the FSI_0002 really be there? Is there any way to stop it happening?

.Net 3.5 support

FsPickler has great potential for usage in Unity3D, as the serialization is fast enough to offer some performance benefits to games.

Unfortunately, it doesn't compile for .Net 3.5 at the moment, which is the version of the framework that Unity3D works with. (although it's actually some heavily modified mono version)

If there's any chance at supporting this version, I'm sure FsPickler would see some not-insignificant adoption among Unity3D users, especially if we could put out a Nuget package for it.

I'm currently doing a cursory look at what's going to be involved in porting this to 3.5, as I'd really like to use it.

I'll update this issue with my findings, but I'd appreciate input on this from the maintainers.

CustomPickler fallback on default generated pickler

I want to serialize a subset of a graph. When I deserialize later, I will reconnect with the external objects. To do that, I use my own tag-to-object table. When I first encounter an external object, I can write out the tag followed by the object. If I encounter it again, I can just write out the tag. I can use the StreamingContext to pass this state through the pickling, so that part is fine.

I can use the [<CustomPickler>] attribute and supply a pickler that does all this, but having written the tag, I then have to manually pickle the object. What I'd like to do is have some way for the custom pickler to obtain the pickler that would have been generated (as if the [<CustomPickler>] attribute had not been applied), so that I can use that to write out the object. Likewise when reading.

Is there some way to do this? The only way I could think of is for all my references to use a Wrapper<'T> struct, which holds the reference, so that the wrapper can have the custom pickler, but the referenced type 'T inside it does not, but I don't really want to complicate all my types and code just for this.

Thanks,

David.

Unable to Deserialize basic examples

Sorry if this is a silly question, I am very new to FsPickler, but when trying to run any of the examples from the ReadMe I am getting exception "Unable to read beyond the stream" when attempting to Deserialize. I am using MemoryStream as the underlying stream. Could you provide a more complete example that includes how to properly read from the underlying stream?

Here is the code I am attempting to run from my script file:

let fsp = new FsPickler()

// typed serialization
let stream = new System.IO.MemoryStream()
fsp.Serialize(stream, Some [1; 2; 3])
let byteArray = stream.ToArray()

fsp.Deserialize(stream)

API breakage in minor version?

It seems a minor version has broken its API contract!

SuaveIO/suave#302

Could you please revert the above mentioned change?

Serializing singletons using ISerializable/IObjectReference fails

The exception message is "is ISerializable but does not implement constructor with parameters (SerializationInfo, StreamingContext)." But in the case of a singleton this is expected. SetType is used on SerializationInfo to set a IObjectReference helper type which contains the required constructor.

For example:
https://msdn.microsoft.com/en-us/library/system.runtime.serialization.iobjectreference%28v=vs.110%29.aspx

Crashing on Uri

  Nessos.FsPickler.FsPicklerException: Error deserializing object of type 'ns.A+B+ReactorModel'. ---> System.InvalidCastException: Cannot cast from source type to destination type.
at (wrapper castclass) object.__castclass_with_cache (object,intptr,intptr) <IL 0x0002c, 0x0005b>
at Nessos.FsPickler.CompositePickler`1<Nessos.FsPickler.AssemblyInfo>.Read (Nessos.FsPickler.ReadState,string) <0x002b3>
at Nessos.FsPickler.ReflectionPicklers/[email protected] (Nessos.FsPickler.ReadState,string) <IL 0x00076, 0x000c0>
at Microsoft.FSharp.Core.FSharpFunc`2<Nessos.FsPickler.ReadState, string>.InvokeFast<System.Reflection.MemberInfo> (Microsoft.FSharp.Core.FSharpFunc`2<Nessos.FsPickler.ReadState, Microsoft.FSharp.Core.FSharpFunc`2<string, System.Reflection.MemberInfo>>,Nessos.FsPickler.ReadState,string) <0x0004e>
at <StartupCode$FsPickler>.$CompositePickler/reader@112<System.Type, System.Reflection.MemberInfo>.Invoke (Nessos.FsPickler.ReadState,string) <0x00037>
at Microsoft.FSharp.Core.FSharpFunc`2<Nessos.FsPickler.ReadState, string>.InvokeFast<System.Type> (Microsoft.FSharp.Core.FSharpFunc`2<Nessos.FsPickler.ReadState, Microsoft.FSharp.Core.FSharpFunc`2<string, System.Type>>,Nessos.FsPickler.ReadState,string) <0x0004e>
at Nessos.FsPickler.CompositePickler`1<System.Type>.Read (Nessos.FsPickler.ReadState,string) <0x00337>
at <StartupCode$FsPickler>.$ISerializablePickler/reader@118-36<System.Uri>.Invoke (Nessos.FsPickler.ReadState,string) <0x002d3>
at Microsoft.FSharp.Core.FSharpFunc`2<Nessos.FsPickler.ReadState, string>.InvokeFast<System.Uri> (Microsoft.FSharp.Core.FSharpFunc`2<Nessos.FsPickler.ReadState, Microsoft.FSharp.Core.FSharpFunc`2<string, System.Uri>>,Nessos.FsPickler.ReadState,string) <0x0004e>
// etc

Unfortunately I don't have a better repro right now as it's pretty deep inside my own code.

dependency on the older version of the Newtonsoft.Json

Hi,
Is it possible to upgrade to a new version of a Newtonsoft.Json or just modify nuget package that it can accompdete a new Json.net?
I'm getting an error:
Attempting to resolve dependency 'Newtonsoft.Json (≥ 6.0.5 && < 6.1.0)'.
Installing 'FSharp.Core 3.1.2.1'.
Successfully installed 'FSharp.Core 3.1.2.1'.
Installing 'Newtonsoft.Json 6.0.5'.
Successfully installed 'Newtonsoft.Json 6.0.5'.
Installing 'FsPickler.Json 1.2.9'.
Successfully installed 'FsPickler.Json 1.2.9'.
Installing 'FsPickler.CSharp 1.2.9'.
Successfully installed 'FsPickler.CSharp 1.2.9'.
Install failed. Rolling back...
Install-Package : Already referencing a newer version of 'Newtonsoft.Json'.

• but I have to use the "Newtonsoft.Json" version="7.0.1"

Thanks!

json rpc 2.0

Json rpc 2 packs returned data into result parameter, for instance:

{"jsonrpc": "2.0", "result": 19, "id": 3}

Any error is reported back in error parameter:

{"jsonrpc": "2.0", "error": {"code": -32601, "message": "Method not found"}, "id": "1"}

We could say that result and error are optional parameter because if request is successful the error parameter is not required, and if error is returned no result parameter is returned.

I tested FsPicker json deseralization on the following data:

type ErrorData =
    {
        code: int
        message: string
    }

type JsonRpcResponse<'T> =
    {
        result: 'T
        error: ErrorData
    }

type AccountDetailsResponse = 
    {
        currencyCode: string
        firstName: string
        lastName: string
        localeCode: string
        region: string
        timezone: string
        discountRate: double
        pointsBalance: int
    }

let runTest() = 
    let json = """{"jsonrpc":"2.0","result":{"currencyCode":"EUR","firstName":"Stefan","lastName":"Belopotocan","localeCode":"en","region":"GBR","timezone":"CET","discountRate":0.0,"pointsBalance":0},"id":1}"""
    let json2 = """{"result":{"currencyCode":"EUR","firstName":"Stefan","lastName":"Belopotocan","localeCode":"en","region":"GBR","timezone":"CET","discountRate":0.0,"pointsBalance":0},"error":{"code":-32700,"message":"DSC-0008"}}"""

    let jsp = FsPickler.CreateJson(indent = true, omitHeader = true)

    let data = jsp.UnPickleOfString<JsonRpcResponse<AccountDetailsResponse>>(json2)

    printfn "%A" data

For “json” data this code throws exeception:

{"expected property 'result' but was 'jsonrpc'."}

Because json data starts with the parameter: jsonrpc, what is just syntactical sugar for json rpc 2 response message. If I remove parts which are not used by my mapping model: JsonRpcResponse and use sample defined by json2 value, the data are deserialized correctly, but that is not the way how real json api responses are used in my application, as the same api method could returned response without some optional parameters.

What was my first naïve approach was to declare error parameter as optional but it does not work. Even if you change order of properties, for instance in the record
AccountDetailsResponse you declare currencyCode as third parameter and not as the first one as it is returned by api, then deserialization fails. In real api applications you do not need to map all api models to your domain models as well.

It would be nice if your library would map only selected properties skipping not used ones.

Deserializing a map gives an empty map

I have a Map<string, Record>, and serialize it using:

let fsp = FsPickler()
fsp.Serialize(stream, map)

I checked the file and it looks fine, but when I deserialize it again with fsp.Deserialize<_>(stream) I get an empty map

What happens at deserialization-time if the FsPickler version is different?

Let's say I am using FsPickler version 100. I pickle to Json text and the version 100 is written into the pickled output.

I deserialize with FsPickler version 100. Everything is fine.

But what happens when I deserialize with FsPickler 200? Will that happen successfully even though serialization was conducted by a different version of FsPickler?

Thanks,

Brody

What approach could be taken to improve the performance for a large object graph?

I have a relatively large object graph, potential for ~ 500mb+ in size and the deserialization is dominating the performance of the application.

FsPickler binary serialization is already performing admirably well compared to a few other alternatives (so well done for that) but I was wondering what avenues may be open to squeezing any more performance out of FsPickler?

Extenally marking types as excluded/nonserialized/serializable

Hi,
I'm trying to dump my current serialization setup for a a large object graph of types I have.

The way I've started to work with FsPickler is to start sprinkling Serializable attributes all over my code.
This it perfectly acceptable, but the problem I'm experiencing is with non Serializeable types.

I would like to be able to somehow inform FsPickler that a type is not serializable (rather than marking each field in other classes that is of that type).

I cannot seem to find this ability in the current code in FieldPicklers.fs or anywhere else in FsPickler.

I would like to work on this, but I thought that it would be better to start some sort of a discussion on how these things should be done.

I can see two approaches:

1. Add something along the lines of a NotSerializableAttribute class that would come from FsPickler...
   This sounds like a very BAD idea, since it would mean that user code now actually depends on FsPickler, and would generally suck, so I don't think this is even worth pursuing
2. Provide a Set/List of excluded types, so that when FsPickler encounters these types, it will automatically treat the fields that points to these types as-if they were marked with the NonSerialized attribute

In addition to what I've outline in (2), the same idea could be applied for the use of SerializableAttribute, i.e., another Set/List could be supplied that would contain types that SHOULD be treated as Serializable even if they are not marked as such through the Serializable attribute. This will have the additional advantages:

• No changes are needed for existing code, only the code performing the serialization needs to care about who/what gets serialized
• Support for "adding" serialization for binary/compiled code that doesn't have the Serializable attribute but could be serialized otherwise can be done post-fact

Any thoughts?
Is the direction of adding Sets of Types to the (de)serialization api-points acceptable?

CustomPickler on class with an unpickleable base class

I have a (derived) class I'd like to pickle. So I've implemented CustomPickler. However, my class derives from a dynamic class which FsPickler cannot pickle. My CustomPickler on the derived class doesn't ask FsPickler to pickle the unpickleable base class. However, FsPickler won't pickle my derived class.

Expressed in terms of a failing test case:
iainnicol@6d63baa

IYHO, is this a valid use case?

Regards.

Why are F# records not cached by ref?

Why does FsRecordPickler.Create create a CompositePickler with cacheByRef = false?

An F# record uses structural comparison by default, but if you pass them around, you are passing references. You might store several references to the same record, so why not cache by reference when pickling? It can do no harm, and might do some good.

In my case, I am using [<ReferenceEquality; NoComparison>], because I am using records in a DAG. But even with structural equality, there's no harm in caching by reference, surely?

David.

Who uses FsPickler?

I have created a new section in the mains docs page. If you have used and enjoyed FsPickler, why not list your own project and/or testimonial? You can either directly submit a pull request here or reply here and I will add it myself.

Support for serializing large datasets

I was trying to use FsPickler to pull a largish (~27 million records) to disk and found that about 30 minutes in it failed with:

Nessos.FsPicker.FsPicklerException: Error serializing instance of type System.String[] ---> System.Runtime.Serialization.SerializationException: The internal array cannot expand to greater than Int32.MaxValue elements.

This occurs in a call to ObjectIDGenerator.Rehash() from ObjectIDGenerator.GetID(Object obj, Boolean& firstTime) which is called in FsPickler in CompositePickler`1.Write(WriteState state, String tag, T value) in CompositePickler.fs on line 189

It turns out this is a common problem in .NET, due to the internal use of ObjectIDGenerator to look for cycles. However, as this is a straight pull of a SQL database I can guarantee there's no cycles in this case.

On a side note, the behavior of ObjectIDGenerator is poor in the failure case. It does not release the memory from its table thus leaving a big chunk of ram in use.

Extending 3rd party lib custom picklers

I vaguely remember a possibility to resolve custom picklers externally, i.e. it was possible to define picklers for an existing type outside of the type/library.
Unfortunately this seems to be gone [1] :(

One solution at user level would be to allow extension methods to be considered for pickler resolving:
Lib 1: type Test = Test of ()
Lib 2:

[<CustomPickler>]
type Test with
    static member resolve (p : IPicklerResolver) : Pickler<Urdar> =`....

The solution would be neat but of course possibly impossible to implement due to the compilation technique of extensions in F#.

What is the recommended way to accomplish this type of extensibility?

[1] #26

System.MissingMethodException

When UnPickling with JsonPickler, it throws a MissingMethodException:

System.MissingMethodException: Method not found: 'Void Newtonsoft.Json.JsonReader.set_SupportMultipleContent(Boolean)

Repro:

#r @"..\packages\Newtonsoft.Json.6.0.3\lib\net45\Newtonsoft.Json.dll"
#r @"..\packages\FsPickler.0.9.4-alpha\lib\net45\FsPickler.dll"
#r @"..\packages\FsPickler.Json.0.9.4-alpha\lib\net45\FsPickler.Json.dll"

type Cases =
    | This of content:string
    | That of value:int * content:string

open Nessos.FsPickler
let pick = JsonPickler()
let v = pick.PickleToString (This "hello")


let e = pick.UnPickleOfString<Cases> v

Introduce CustomPickler imperatively?

Introduction

I have learned a lot reading the codebase and a few relevant issues. In particular the admonishment that POCOs are unsupported by design (edit: custom pickling semantics could only be introduced by customizing the target type directly) and the assumed homogeneity of producers/consumers implies a rigidity to the serialization that make it unsuitable as vehicle for data persistence or heterogeneous interchange.

Nevertheless, there are two particular strengths of FsPickler that compel me to see if it might be extended to slightly different ends:

• Implementing a new wire format is distinct from defining the pickler composition, making it easier to support different interchange formats like Cognitect's Transit
• It is already perfect for the aforementioned homogenous communication

Motivation

There are two situations that arise frequently:

type Customer = {
  Name : CustomerName
  Email : EmailAddress
}
and CustomerName = CustomerName of string
and EmailAddress = EmailAddress of string

Ideally these would make use of phantom types, but lacking that facility single-case discriminated unions work well enough.

type Widget = { Kind: Foo }
type Foo = Bar | Baz of Baz
type Baz = Zap | Zazz

Basically, this is modelling a hierarchical taxonomy, treating the DUs as true tags in pattern matching business rules. I also think the fact that units of measure (UoM) are type erased is relevant. I think integer coding would be a relevant use case as well.

I would like to be able to arbitrarily define "bottom" for a type. So e.g. in JSON:
{"Name":"Don";"Email":"dsyme@..."} and {"Kind": "Baz Zap"}

Put another way, I need the option to encode some portion of the object graph using stable, private semantics. Moreover, I would like to opt-in to that regime only when creating a pickler for e.g. persistence or interchange, leaving the default global cache unchanged for node to node communication.

Discussion

FsPickler is a very sophisticated library, and I'm reticent to suggest any concrete changes, but please indulge this code example as a sketch:

let eP : Pickler<EmailAddress> = //Pickler.alt implementation?
let cP : Pickler<CustomerName> = //...
let storagePickler = FsPickler.generatePickler<Customer> [ eP; cP ]

I think #29 is very relevant, though with slightly different motivations.

In summary, I am under the impression that FsPickler is nearly ideal for implementing the backbone of a multi-targeted serialization strategy. I recognize this use is not its raison d'être, but its power and compositionality leave me optimistic that it could be put effectively to this use with relatively small changes.

Serializing union and record types declared in F# PCLs

It seems that F# union and record types can not be serialized when they are declared in an F# portable class library. I am getting NonSerializableTypeExceptions even when I want to serialize simple instances of type Test = | Test for example.

The F# compiler does not attach the [Serializable] attribute to these types, because it's unsupported in PCLs.

Is there any way to enforce the generation of the Picklers that are required for these types?

FsPickler.Json: Version of Newtonsoft.Json

Hi Eirik

I want to upgrade to 1.2.5 (for the excellent new custom pickler registration feature), but am unable to because the version of Newtonsoft.Json in FsPickler.Json is now pinned to 6.0.5 since the Paket upgrade:

In version 1.2.4 it was (perhaps not intentionally) unrestricted above 6.0.5:

Another of my dependencies depends on the latest stable: 6.0.8, and since I'm using Paket it resolves to the latest version with compatible version constraints: in this case 1.2.4

Just checking that this is intentional, since I know that Newtonsoft.Json is not exactly famous for confirming to SemVer between patch versions. However it could just be a consequence of the Paket conversion.

If so I suggest relaxing the version constraint in paket.dependencies to ~> 6.0.5, which I believe will resolve to >= 6.0.5 && < 6.1 in the nuspec. This is slightly tighter than the original >= 6.0.5 constraint.

Cheers,

Andrew

Please rename assemblyInfo.fs to AssemblyInfo.fs

Error when building on linux:

FSC: error FS0225: Source file '/tmp/nix-build-Nessos.FsPickler-1.2.0.drv-0/FsPickler-v1.2.0-sr
c/src/FsPickler/AssemblyInfo.fs' could not be found

Not all filesystems are case insensitive :)

Example doesn't work - Code shown below

Hi

I've been trying to use this library and had a general issue with the stream being unreadable. So I've had a go at using the simplest case example from this tutorial instead.

The code below is what I've tried (I've omitted the mandatory open statements).

let binearySerializer = FsPickler.CreateBinarySerializer()
let xmlSerializer = FsPickler.CreateXmlSerializer(indent = true)

let stream = Unchecked.defaultof<System.IO.Stream>

xmlSerializer.Serialize(stream, [0. .. 0.1 .. 1.])
xmlSerializer.Deserialize<float list>(stream)

The error I get is the following:

System.ArgumentNullException: Value cannot be null.
Parameter name: stream

at System.IO.StreamWriter..ctor(Stream stream, Encoding encoding, Int32 bufferSize, Boolean leaveOpen)
at Nessos.FsPickler.XmlPickleFormatProvider.Nessos-FsPickler-IPickleFormatProvider-CreateWriter(Stream stream, Encoding encoding, Boolean _arg1, Boolean leaveOpen) in C:\Users\eirik\Development\nessos\FsPickler\src\FsPickler\Format\XmlFormat.fs:line 382
at Nessos.FsPickler.FsPicklerSerializer.Serialize[T](Stream stream, T value, FSharpOption1 pickler, FSharpOption1 streamingContext, FSharpOption1 encoding, FSharpOption1 leaveOpen) in C:\Users\eirik\Development\nessos\FsPickler\src\FsPickler\FsPickler\Serializer.fs:line 49
at <StartupCode$FSI_0057>.$FSI_0057.main@() in c:\users\richard\documents\visual studio 14\Projects\ConsoleApplication12\ConsoleApplication12\Program.fs:line 31
Stopped due to error

I'm using Visual Studio 2015 Community Edition and F#4

Publish a NuGet package

Link to MBrace framework on your front page is broken

That's all.

FsPicker.Serialize() cannot serialize Nullable<_>

type My(id: Nullable<int>) = member x.Id = id

let pickler = FsPickler()
let ms = new MemoryStream()
pickler.Serialize (ms, My(Nullable 3))

which results the following error:

System.ArgumentException: GenericArguments[0], 'System.Nullable`1[System.Int32]', on 'FsPickler.Pickler`1[T] Create[T](FsPickler.IPicklerResolver)' violates the constraint of type 'T'. ---> System.Security.VerificationException: Method FsPickler.DotNetPicklers+StructPickler.Create: type argument 'System.Nullable`1[System.Int32]' violates the constraint of type parameter 'T'.
   at System.RuntimeMethodHandle.GetStubIfNeeded(RuntimeMethodHandleInternal method, RuntimeType declaringType, RuntimeType[] methodInstantiation)
   at System.Reflection.RuntimeMethodInfo.MakeGenericMethod(Type[] methodInstantiation)
   --- End of inner exception stack trace ---
   at System.RuntimeType.ValidateGenericArguments(MemberInfo definition, RuntimeType[] genericArguments, Exception e)
   at System.Reflection.RuntimeMethodInfo.MakeGenericMethod(Type[] methodInstantiation)
   at FsPickler.DotNetPicklers.StructPickler.CreateUntyped(Type t, IPicklerResolver resolver)
   at [email protected](Type t)
   at FsPickler.DotNetPicklers.ClassPickler.Create[T](IPicklerResolver resolver)
   at FsPickler.Utils.MethodInfo.GuardedInvoke(MethodInfo m, Object instance, Object[] parameters)
   at FsPickler.DotNetPicklers.ClassPickler.CreateUntyped(Type t, IPicklerResolver resolver)
   at [email protected](Type t)
   at FsPickler.PicklerCache.FsPickler-IPicklerResolver-Resolve[T]()
   at FsPickler.FsPickler.Serialize[T](Stream stream, T value, FSharpOption`1 streamingContext, FSharpOption`1 encoding, FSharpOption`1 leaveOpen)
   at <StartupCode$FSI_0011>.$FSI_0011.main@()

System.MissingMethodException with XUnit

I have the following code:

module flibble

open Xunit
open Nessos.FsPickler
open Nessos.FsPickler.Json

[<Fact>]
let ``Should blah``() =
    let jsonPickler = FsPickler.CreateJson()
    ()

When I run this test I get:

System.MissingMethodExceptionMethod not found: 'Nessos.FsPickler.Json.JsonSerializer Nessos.FsPickler.Json.FsPickler.CreateJson(Microsoft.FSharp.Core.FSharpOption`1<Boolean>, Microsoft.FSharp.Core.FSharpOption`1<Boolean>, Microsoft.FSharp.Core.FSharpOption`1<Nessos.FsPickler.ITypeNameConverter>)'.
   at flibble.Should blah()

I am using .NET Framework 4.5 and F# 3.1. I have tried both the console runner and the Resharper Visual Studio runner, both have the same effect.

Any ideas as to how I can fix this?

Thanks!

C# friendly API

Since it is a general-purpose serializer, it would be nice to avoid FSharp.Core references in C# projects just because of method signatures with optional parameters

CustomPickler forces other projects to reference FsPickler

I have an F# record and one of its fields is the result of an expensive calculation derived from the other fields. Currently this is not lazy, although it might be in the future, but even so the idea is to avoid performing the calculation more than once.

The field is quite a large data structure, so I don't want to save it. Instead I want to recalculate it when I restore the object. Since this is an immutable F# record, ISerializable and DataContract are unsuitable, so the CustomPickler attribute seems like the only way to go.

The problem is that even if the static CreatePickler method is made private, calling projects need to include FsPickler due to the IPicklerResolver method argument. The compiler error is:

error FS0074:
The type referenced through 'Nessos.FsPickler.IPicklerResolver' is defined in an assembly that is not referenced.
You must add a reference to assembly 'FsPickler'.

My calling projects are not concerned with data serialization, so it doesn't seem right that they should have to include FsPickler. Can you think of a way around this problem?

David.

Lists are not serilized as Json array

for instance

type Rec = { value: int list }
pick.PickleToString { value = [1;2;3] }

outputs

{"value":{"length":3,"list":[1,2,3]}}

It would be better to have something like:

{"value": [1,2,3] }

Pickling to Code Quotations for Deserialization in a Different Runtime

I'm trying to use FsPickler to send code quotations over a named pipe from a server running inside of FSI to a client running inside of the Unity3d engine. The client has a quotation evaluator that it uses to execute code against the engine's internal API. The issue comes from the reflection based loading of the FSharp.Core and mscorlib at different version numbers

Here's an example of the different JSON produced

There are also some differences in the instance and id numbers, and the FSI version having a third Item = null in a few places, but I'm not sure how relevant those are to block being deserialized effectively.

Is there a way to specify the target runtime and target version of FSharp.Core for serialization? and if not could this be incorporated as a future feature?

Build: tests failing on mono 3.6, CentOS 6.5, F# 3.1

Hi again =)

https://tc-oss.intelliplan.net/viewLog.html?buildId=456&buildTypeId=FSharp_FsPickler_Build&tab=buildLog

Snip:

[20:51:05][Step 1/1] 1) Building /opt/teamcity-agent/work/16689743ff57dcb9/FsPickler.sln failed with exitcode 1.
[20:51:05][Step 1/1] 2) : /usr/lib/mono/xbuild/12.0/bin/Microsoft.Common.targets(0,0): Cannot copy /opt/teamcity-agent/work/16689743ff57dcb9/tests/FsPickler.Tests/app.config to /opt/teamcity-agent/work/16689743ff57dcb9/bin/FsPickler.Tests.dll.config, as the source file doesn't exist.
[20:51:05][Step 1/1] 3) CS1705: CSC(0,0): Assembly `FsPickler.Json, Version=0.9.11.0, Culture=neutral, PublicKeyToken=null' references `FSharp.Core, Version=4.3.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' which has a higher version number than imported assembly `FSharp.Core, Version=4.3.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
[20:51:06][Step 1/1] Process exited with code 42

I've set this up free of charge because I use this software at work and I have an interest in it being stable. If you sign up with a user and e-mail I can make you the administrator of your project which will let you get e-mail when something breaks.

Relevant to fsharp/fsharp#317

Discriminated unions use integer for cases

It could be more readable to use case name instead.

thoughts ?

Default encoding add BOM to output which create invalid Json string

when using json.Pickle v with default encoding, the result is a byte[] starting with UTF8 byte order mark.

This first caracter is not valid in json and is not ignored by others deserializers (try with jsonlint).

A workaround is to use System.Text.UTF8Encoding(false) for encoding, but it could be better to use it as default.

Allow for assembly version independent object

Not an issue - more of a feature request.

The ObjHeader type hash uses the AssemblyQualifiedName and thus encodes the assembly version into the hash. Would it be possibly to have a more lenient option where it will still attempt to deserialize the object independently of the version it was serialized against?

Cheers
Karl

Newtonsoft.Json dependency trouble

I have an issue when I want to stick Newtonsoft.Json to ~6.0 version

An unhandled exception of type 'System.IO.FileLoadException' occurred in FsPickler.dll

Additional information: Could not load file or assembly 'Newtonsoft.Json, Version=7.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)

https://github.com/pavelhritonenko/fspickler-newtonsoft - repository for reproduce issue

Replacement for member f.UnPickleOfString (valueType : Type?

In 1.2 the following got removed:

   /// <summary>
    ///     Unpickle value to given type.
    /// </summary>
    /// <param name="valueType">type of pickled value.</param>
    /// <param name="pickle">Pickle.</param>
    /// <param name="streamingContext">streaming context.</param>
    member f.UnPickleOfString (valueType : Type, pickle : string, [<O;D(null)>]?streamingContext) : obj =
        unpickleString (fun m -> f.Deserialize(valueType, m, ?streamingContext = streamingContext)) pickle

I was using this in the following way:

    let deserialize<'a> (event: ResolvedEvent) =
        let serializedString = Encoding.UTF8.GetString event.Event.Data
        let serializedType = typedefof<'a>
        let event = json.UnPickleOfString(valueType = serializedType, pickle = serializedString)
        event :?> 'a

What is the intended usage with the rewritten API to accomplish the same goal?

Cannot serialize an instance of type marked with [<DataContract>] attribute

It works OK for a plane type:

#r @"..\..\packages\FsPickler.1.0.4\lib\net45\FsPickler.dll"

open System.IO
open Nessos.FsPickler

type Hash(value: byte[]) =
    member __.Value = value
    override x.ToString() = sprintf "%A" x.Value

let pickler = FsPickler.CreateBinary()
let stream = new MemoryStream()
let h = Hash "1234567890ABCDEF"B
pickler.Serialize(stream, h, leaveOpen = true) 
stream.Position <- 0L
let h': Hash = pickler.Deserialize stream
h'.Value
// val it : byte [] =
//  [|49uy; 50uy; 51uy; 52uy; 53uy; 54uy; 55uy; 56uy; 57uy; 48uy; 65uy; 66uy;
//    67uy; 68uy; 69uy; 70uy|]

Now add [<DataContract>]:

#r @"..\..\packages\FsPickler.1.0.4\lib\net45\FsPickler.dll"
#r "System.Runtime.Serialization"

open System.IO
open Nessos.FsPickler
open System.Runtime.Serialization

[<DataContract>] // <===== here
type Hash(value: byte[]) =
    member __.Value = value
    override x.ToString() = sprintf "%A" x.Value

let pickler = FsPickler.CreateBinary()
let stream = new MemoryStream()
let h = Hash "1234567890ABCDEF"B
pickler.Serialize(stream, h, leaveOpen = true) 
stream.Position <- 0L
let h': Hash = pickler.Deserialize stream
h'.Value
// val it : byte [] = null 

FsPickler cannot serialize C# auto properties from base class

First, try to serialize-deserialize F# types:

[<AbstractClass>]
type My() = 
    member val BoolProp = false with get, set

type My1() = 
    inherit My()
    member val Str = "" with get, set

let pickler = FsPickler()
let ms = new MemoryStream()
pickler.Serialize (ms, My1(BoolProp = true, Str = "str") :> My)
ms.Position <- 0L
let my: My = pickler.Deserialize ms

result:

val it : My = FSI_0030+My1 {BoolProp = true; Str = "str";}

which is OK.

Now serialize an object from a C# assembly:

namespace ClassLibrary1
{
    [Serializable]
    public abstract class Class1
    {
        public bool BoolProp { get; set; }
    }

    [Serializable]
    public class Class2 : Class1
    {
        public string StrProp { get; set; }
    }
}

let pickler = FsPickler()
let ms = new MemoryStream()
pickler.Serialize (ms, ClassLibrary1.Class2 (BoolProp = true, StrProp = "str") :> ClassLibrary1.Class1)
ms.Position <- 0L
let my: ClassLibrary1.Class1 = pickler.Deserialize ms

result:

val it : ClassLibrary1.Class1 = ClassLibrary1.Class2 {BoolProp = false; StrProp = "str";}

The BoolProp contains false however true is expected.

Cannot deserialize record with two fields of same CLI type

type T =
    { F1: IList<int>
      F2: IList<int> }

let t = 
    { F1 = [||]
      F2 = [||] }

let p = FsPickler()
let ms = new MemoryStream()
p.Serialize(ms, t :> obj)
ms.Position <- 0L
let p1: obj = p.Deserialize ms

result:

System.ArgumentException: An item with the same key has already been added.
   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
   at [email protected](Reader r)
   at FsPickler.Pickler`1.UntypedRead(Reader r, Boolean managed)
   at FsPickler.Reader.Read[T](Pickler`1 pickler)
   at recordDeserializer(Pickler[] , Reader )
   at FsPickler.Pickler`1.UntypedRead(Reader r, Boolean managed)
   at FsPickler.Reader.Read[T](Pickler`1 pickler)
   at FsPickler.FsPickler.Deserialize[T](Stream stream, FSharpOption`1 streamingContext, FSharpOption`1 encoding, FSharpOption`1 leaveOpen)
   at <StartupCode$FSI_0158>.$FSI_0158.main@() in FsPicklerTest.fsx:line 76