jankotek / elsa Goto Github PK

MapDB should have some level of compatibility with Kryo 3.0. It should be possible to:

1. plug kryo serializer into Elsa serialization

2. Use Elsa as Kryo serializer, so frameworks which already use Kryo could also use Elsa.

Either this will be added into separate project. Or distributed in Elsa Jar and Kryo would become compile time dependency.

Some well known JVM classes are not handled by Elsa. There are some problems with their deserializaiton, some fields are private, object stack can not be handled... etc

That includes:

• Calendar
• StringBuillder and StringBuffer
• Currency
• EnumSet
• TimeZone

Following classes are private, and there is problem with object stack on deserialization

• Collections.synchronized*()
• Collections.unmodifiable*()

Following singletons are registered in MapDB, but not in plain Elsa

• Collections.emptyList
• Collections.singleton
• Collections.emptyMap
• Collections.emptySet
• Collections.singletonList
• Collections.singletonMap

Elsa should also handle Externalizable from other frameworks, namely:

• KryoSerializable

Investigate Objenesis for easy object instantiation https://github.com/easymock/objenesis

Also check if Serializable interface is required right now

From email at Kryo mailing list:

My use case would be a large-ish (several GB) in-memory state that needs
to be saved periodically (every five minutes or so). No in-place
updates, new state is generated from scratch (but a large amount of
substructures are supposed to be shared with previous state).

So the object graph has new root on each update and data are immutable. Now it makes sense.

In theory it could be handled the same way as circular references.
Serializer would save its state (IdentityHashMap) between updates.
On next update serializer would just continue with original Map and
replace objects from first version, with their pointers.

There are two problems:

• memory overhead of keeping IdentHashMap 30% extra references?
• IdentHashMap would prevent GC, so weak refs..

Support class field rename in SerializerPojo.

• add ElsaMaker.registerClassFieldRename() to register alias
• handle matrix:
  • class catalog in binary data contains (old/new) field name
  • existing class has (old/new) field
• handle getXXX() getter in deserialization as alias for non-existing renamed field
• handle setXXX() setter in serialization as alias for non-existing renamed field

Support class rename in SerializerPojo. Unregistered class names are stored in binary data.

• add ElsaMaker.registerClassRename() to register alternative class name
• alternative name will be used if the new class name is not available
• handle following combinations: Old class available (yes/no); binary data have (old/new) class

Check serialization of Scala collections (maps, lists) is efficient. It should use Elsa graph traversal for entries, rather than fall-back into ObjectOutputStream for entire collection.

I'd like to be able to store an Instant.

Good developers don't use java.util.Date anymore.

There may be other types in the package which other people would find useful to serialise as well.

In the Elsa README the Maven dependency XML is as follows:

<dependency>
    <groupId>org.mapdb</groupId>
    <artifactId>mapdb</artifactId>
    <version>VERSION</version>
</dependency>

should the artifactId be 'elsa' instead of 'mapdb'? The paragraph just above the XML says Elsa is no longer included in MapDB.

Elsa uses recursion to dive into object graph. That could generate long call stack and cause StackOverflow exception. Compared to Java serialization Elsa uses less stack frames and behaves better.

It should be possible to eliminate recursion and dive into object graph with simple loop. Some data structure needs to replace call stack (trie?). It should store path traversed on graph dive. Special care needs to be taken to preserve order of objects

Getting java.lang.OutOfMemoryError when initilize on android:

            ElsaSerializerBase.Deserializer<Bitmap> deser = new ElsaSerializerBase.Deserializer<Bitmap>() {
                @Override
                public Bitmap deserialize(DataInput in, ElsaStack objectStack) throws IOException {

                    int bufferLength = in.readInt();

                    byte[] byteArray = new byte[bufferLength];
                    in.readFully(byteArray, 0, bufferLength);
                    return BitmapFactory.decodeByteArray(byteArray, 0, bufferLength);
                }
            };



            ElsaSerializerBase.Serializer<Bitmap> ser = new ElsaSerializerBase.Serializer<Bitmap>() {
                @Override
                public void serialize(DataOutput out, Bitmap bitmap, ElsaStack objectStack) throws IOException {
                    ByteArrayOutputStream stream = new ByteArrayOutputStream();
                    bitmap.compress(Bitmap.CompressFormat.JPEG, 80, stream);

                    byte[] byteArray = stream.toByteArray();

                    out.writeInt(byteArray.length);
                    out.write(byteArray);
                }
            };

            serializer = new ElsaMaker()
                    .registerSerializer(BITMAP_HEADER, Bitmap.class, ser)
                    .registerDeserializer(BITMAP_HEADER, deser)
                    .make();

java.lang.OutOfMemoryError: Failed to allocate a 953893208 byte allocation with 6291456 free bytes and 507MB until OOM, max allowed footprint 10774144, growth limit 536870912