[Java] NPE when serializing with withMetaShare about incubator-fury

[LiangliangSui]

MetaContext will be shared across different Fury objects, that's what it is used for

Ok, got it, if it is reused in different Fury objects, then we do need setMetaContext.

Thank you for helping me solve this problem. Thanks again.

[LiangliangSui]

Could we instantiate a MetaContext object in the SerializationContext constructor with withMetaShare enabled, without requiring the user to manually set a MetaContext instance through the SerializationContext#setMetaContext interface?

For example, the following implementation.

package org.apache.fury.resolver;

import java.util.IdentityHashMap;
import org.apache.fury.config.Config;
import org.apache.fury.config.FuryBuilder;

/**
 * A context is used to add some context-related information, so that the serializers can set up
 * relation between serializing different objects. The context will be reset after finished
 * serializing/deserializing the object tree.
 */
public final class SerializationContext {
  private final IdentityHashMap<Object, Object> objects = new IdentityHashMap<>();
  private final boolean scopedMetaShareEnabled;
  private final boolean withMetaShare;
  private MetaContext metaContext;

  public SerializationContext(Config config) {
    scopedMetaShareEnabled = config.isScopedMetaShareEnabled();
    withMetaShare = config.isMetaShareEnabled();
    if (withMetaShare) {
      metaContext = new MetaContext();
    }
  }

  /** Return the previous value associated with <tt>key</tt>, or <tt>null</tt>. */
  public Object add(Object key, Object value) {
    return objects.put(key, value);
  }

  public boolean containsKey(Object key) {
    return objects.containsKey(key);
  }

  public Object get(Object key) {
    return objects.get(key);
  }

  public MetaContext getMetaContext() {
    return metaContext;
  }

  /**
   * Set meta context, which can be used to share data across multiple serialization call. Note that
   * {@code metaContext} will be cleared after the serialization is finished. Please set the context
   * before every serialization if metaShare is enabled by {@link FuryBuilder#withMetaShare(boolean)}
   */
  public void setMetaContext(MetaContext metaContext) {
    this.metaContext = metaContext;
  }

  public void resetWrite() {
    if (!objects.isEmpty()) {
      objects.clear();
    }
    if (scopedMetaShareEnabled) {
      metaContext.classMap.clear();
      metaContext.writingClassDefs.clear();
    } else if (withMetaShare) {
      metaContext = new MetaContext();
    } else {
      metaContext = null;
    }
  }

  public void resetRead() {
    if (!objects.isEmpty()) {
      objects.clear();
    }
    if (scopedMetaShareEnabled) {
      metaContext.readClassInfos.clear();
      metaContext.readClassDefs.clear();
    } else if (withMetaShare) {
      metaContext = new MetaContext();
    } else {
      metaContext = null;
    }
  }

  public void reset() {
    if (!objects.isEmpty()) {
      objects.clear();
    }
    if (scopedMetaShareEnabled) {
      metaContext.classMap.clear();
      metaContext.writingClassDefs.clear();
      metaContext.readClassInfos.clear();
      metaContext.readClassDefs.clear();
    } else if (withMetaShare) {
      metaContext = new MetaContext();
    } else {
      metaContext = null;
    }
  }
}

[chaokunyang]

No, we can't. That's what scopedMetaShare option do. We will make it as the default option in the future. But the meta share itself, will always need to set MetaContext. As you can see, the NPE here is throw by fury on purpose

[LiangliangSui]

Users will not use the created MetaContext directly. What is the purpose of letting users set MetaContext through the setMetaContext interface? Is it just to let users know that MetaContext is shared between multiple serializations?

[LiangliangSui]

If we just want to let users know that MetaContext is shared between multiple serializations, can we explain this problem in the withMetaShare option?

Our goal should be to let users know that MetaContext is shared between multiple serializations, and it can be set optionally through the setMetaContext interface, rather than forcing users to set it when enabling withMetaShare.

I don't know if I understand it correctly

[chaokunyang]

I think adding a precheck in org.apache.fury.resolver.ClassResolver#writeClassDefs is enough. We can provide a detailed error message to indicate users need to set MetaContext here.

[chaokunyang]

If we just want to let users know that MetaContext is shared between multiple serializations, can we explain this problem in the withMetaShare option?

Our goal should be to let users know that MetaContext is shared between multiple serializations, and it can be set optionally through the setMetaContext interface, rather than forcing users to set it when enabling withMetaShare.

I don't know if I understand it correctly

Not exactly, users will use this API directly, as you can see from the use case:

  public void testSimpleRecordMetaShared(boolean codegen) {
    Fury fury =
        Fury.builder()
            .requireClassRegistration(false)
            .withCodegen(codegen)
            .withMetaShare(true)
            .build();
    Foo foo = new Foo(10, "abc", new ArrayList<>(Arrays.asList("a", "b")), 'x');
    MetaContext context = new MetaContext();
    fury.getSerializationContext().setMetaContext(context);
    byte[] bytes = fury.serialize(foo);
    fury.getSerializationContext().setMetaContext(context);
    Assert.assertEquals(fury.deserialize(bytes), foo);
  }

This will be used by rpc framework too reuse meta across multiple serializaition. Or used by bigdata frameworks to serialize data by batch.

[LiangliangSui]

Not exactly, users will use this API directly, as you can see from the use case:

I mean that users will not directly use any API and data in MetaContext, but it is exposed to users.

This will be used by rpc framework too reuse meta across multiple serializaition. Or used by bigdata frameworks to serialize data by batch.

I have another question, will MetaContext be shared across different Fury objects?

[chaokunyang]

Users will use this API:

  @Test(dataProvider = "codegen")
  public void testBatch(boolean codegen) {
    Fury fury =
        Fury.builder()
            .requireClassRegistration(false)
            .withCodegen(codegen)
            .withMetaShare(true)
            .build();
    MetaContext context = new MetaContext();
    for (int i = 0; i < 10000; i++) {
      Foo foo = new Foo(10, "abc", new ArrayList<>(Arrays.asList("a", "b")), 'x');
      fury.getSerializationContext().setMetaContext(context);
      byte[] bytes = fury.serialize(foo);
      fury.getSerializationContext().setMetaContext(context);
      Assert.assertEquals(fury.deserialize(bytes), foo);
    }
  }

[chaokunyang]

MetaContext will be shared across different Fury objects, that's what it is used for