openhft / chronicle-wire Goto Github PK

This is not a big issue, but the exception handling is a bit confusing.

In a lot of the low-level logic there are methods declaring unchecked exceptions. For example in ValueIn:

@Nullable
Object marshallable(Object object, SerializationStrategy strategy)
        throws **BufferUnderflowException**, **IORuntimeException**;

However, these unchecked exceptions are not propagated consistently in method signatures of higher-level logic.

From this mix of declared and undeclared unchecked exceptions it's very hard to figure out what to expect. Not propagating the exceptions makes it look like they have been handled. However, this is not usually the case.

I guess the easiest improvement would be to add catch clauses to the Chronicle Queue code examples (in the Chronicle-Queue project).

But for the Chronicle-Wire project, it would be nice to add some documentation around the API. For example, is there a specific set of API calls where we can expect unchecked exceptions (e.g. due to corrupt data)? Also, some general documentation around exception throwing strategy (i.e. letting the user take care of any exceptions) would be much appreciated.

Thanks for making great software!

Scenario where it's difficult to understand what to expect

Let's say we do the following (using Chronicle-Queue):

        try (DocumentContext dc = tailer.readingDocument()) {
            if (dc.isPresent()) {
                ValueIn valueIn = dc.wire().getValueIn();
                next = valueIn.object();
        }

In this case we might get a RuntimeException thrown (e.g. if data is corrupt) by the call to valueIn.object() -- which does not declare any exceptions in it's signature.

For example:

Caught throwable: net.openhft.chronicle.core.io.IORuntimeException: Unknown_0x0
at net.openhft.chronicle.wire.BinaryWire$BinaryValueIn.bool(BinaryWire.java:2975)
at net.openhft.chronicle.wire.WireMarshaller$BooleanFieldAccess.setValue(WireMarshaller.java:752)
at net.openhft.chronicle.wire.WireMarshaller$FieldAccess.read(WireMarshaller.java:313)
at net.openhft.chronicle.wire.WireMarshaller.readMarshallable(WireMarshaller.java:143)
at net.openhft.chronicle.wire.Wires.readMarshallable(Wires.java:253)
at net.openhft.chronicle.wire.SerializationStrategies$5.readUsing(SerializationStrategies.java:121)
at net.openhft.chronicle.wire.BinaryWire$BinaryValueIn.marshallable(BinaryWire.java:2897)
at net.openhft.chronicle.wire.ValueIn.object(ValueIn.java:482)
at net.openhft.chronicle.wire.BinaryWire$BinaryValueIn.objectWithInferredType(BinaryWire.java:3158)
at net.openhft.chronicle.wire.ValueIn.object(ValueIn.java:435)

Using version 4.5.27, but as far as I can see it's the same thing in master.

I am using Chronicle-Queue(CQ) and Chronicle Map(CM) which pulls in the Chronicle Wire v1.7.6 . I am able to the write the object into CQ without any issue. This object model has fields which are of array type. So, when CQ tries to reading off the object, it fails to populate the array field and throws exception IlegalArgumentException: Argument is not an array.

I tried debugging through the code and control goes through BinaryWire.java where it is attempting to populate the object with the values. In this method, in the accept invocation of the passed lambda, the object itself is passed. I feel that the actual array instance should have been created and stored in the passed object and this array instance should have been passed. What happens in WireMarshaller.java, when it checks Array.getLength(array), it results in IllegalArgumentException.

LMK, if you need any further details. Just for the sake of completeness, it is a scala application and the Arrays component type are Java enums with their name as the values in the array.

See net.openhft.chronicle.wire.CopyTest

Will just add a javadoc for now, but we need to rename it later when we are ready for breaking change

This work should be committed to its own closed source repo.

The functionality of what is required is still to be clarified.

This task is related to OpenHFT/Chronicle-Queue#473

The expectation is that this would NOT require a C++ version of CORE or BYTES.
Wire would have to add support for reading and writing string and the data types that wire supports

Added a FieldNumberParselet to allow the methodReader() to work with customer marshable methods that start with a field number.

The requirement is to skip the entry, rather than not do anything.

see net.openhft.chronicle.wire.fieldnumber.FieldNumberTest

It appears that ref-counters are broken after a wire is applied to a mapped bytes:

java.lang.IllegalStateException: Released

    at net.openhft.chronicle.core.ReferenceCounter.release(ReferenceCounter.java:47)
    at net.openhft.chronicle.bytes.AbstractBytes.release(AbstractBytes.java:329)
    at net.openhft.chronicle.wire.WireResourcesTest.testMappedBytesWireRelease(WireResourcesTest.java:47)

Test case here: #21

see net.openhft.chronicle.wire.WiresTest#test

 public static class Base {
        Bytes base = Bytes.elasticByteBuffer();
    }

    @Test
    public void test() {

        Base base1 = new Base();
        base1.base.clear().append("value1");

        Base base2 = new Base();
        base2.base.clear().append("value2");

        Wires.reset(base1);
        Wires.reset(base2);

        base1.base.clear().append("value1");

        Assert.assertEquals("", base2.base.toString());

    }

MethodIds are now encoded as field numbers.

Could you improvement the doc on the project main page, there are some typo and ambiguous descriptions, which is hard to understand. And could you provide some design descriptions to help people to evaluate if it is a fit.

Meta-issue for tracking work on Java9 compatibility

This is showing up as expensive in a flame graph.

The following code outputs obj2.equals(obj3): false for "\uD83E\uDDC0 extra" with WireType.BINARY.

With "\uD83E\uDDC0", it throws a runtime exception.

Both example strings work successfully with WireType.TEXT and WireType.JSON.

import com.google.common.base.Preconditions;
import net.openhft.chronicle.bytes.Bytes;
import net.openhft.chronicle.core.io.IORuntimeException;
import net.openhft.chronicle.wire.Marshallable;
import net.openhft.chronicle.wire.WireIn;
import net.openhft.chronicle.wire.WireOut;
import net.openhft.chronicle.wire.WireType;

import java.nio.ByteBuffer;

/**
 * Created by dsmith on 11/20/16.
 */
public class MarshallableObj implements Marshallable {
    private final StringBuilder builder = new StringBuilder();

    public void clear() {
        builder.setLength(0);
    }

    public void append(CharSequence cs) {
        builder.append(cs);
    }

    @Override
    public void readMarshallable(WireIn wire) throws IORuntimeException {
        builder.setLength(0);
        Preconditions.checkState(wire.getValueIn().textTo(builder) != null);
    }

    @Override
    public void writeMarshallable(WireOut wire) {
        wire.getValueOut().text(builder);
    }

    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;

        MarshallableObj that = (MarshallableObj) o;

        return builder.toString().equals(that.builder.toString());
    }

    @Override
    public int hashCode() {
        return builder.toString().hashCode();
    }

    @Override
    public String toString() {
        return builder.toString();
    }

    public static void main(String[] args) {
        final WireType wireType = WireType.BINARY;
        final String exampleString = "\uD83E\uDDC0 extra";

        final MarshallableObj obj1 = new MarshallableObj();
        final MarshallableObj obj2 = new MarshallableObj();
        final MarshallableObj obj3 = new MarshallableObj();

        obj1.append(exampleString);
        obj2.append(exampleString);

        System.out.println("obj1.equals(obj2): " + obj1.equals(obj2));

        final Bytes<ByteBuffer> bytes = Bytes.elasticByteBuffer();
        obj2.writeMarshallable(wireType.apply(bytes));

        final String output = bytes.toString();
        System.out.println("output: [" + output + "]");

        obj3.readMarshallable(wireType.apply(Bytes.from(output)));

        System.out.println("obj2.equals(obj3): " + obj2.equals(obj3));
    }
}

Somewhere between Wire 1.3.6 and present versions compatibility was broken, so e. g. JSON produced by Wire 1.3.6 is not readable by newer versions. This makes impossible to use Chronicle Map 3 together with present (if any) versions of Chronicle Queue: http://stackoverflow.com/questions/37954548/exception-while-reading-from-the-chronicle-map-file#comment63402779_37957959

Hi there ,
Any suggestions how to deal with BigDecimal, since our existing code base are highly rely on it to represent financial data

I have been using Chronicle-Queue and was investigating the compression offered by COMPRESSED_BINARY in WireType.

Unfortunately while I was eventually able to get compression to apply (as not all interactions with an ExcerptAppender seemed to trigger compression), decompression did not.

I can see that the tests for compression/decompression are @ignored, which is frustrating given the availability of COMPRESSED_BINARY. From the Chronicle-Queue project you can therefore create a queue with a WireType that doesn't work correctly. It might be that it works OK for write/read of text, but I was trying to write/read bytes.

Having debugged this (as it was not clear why it wasn't working, no comments/warnings etc.), I can offer what I think is the issue.

There's a couple of places that look like they're all suffering from the same class of issue. e.g. in WireIn#bytes, the first thing that is done is a call to readLength(). Unfortunately it seems that if our code is TYPE_PREFIX (and some of the other codes), there is a side-effect that the read pointer is increment past the code, stop bit and the compression details.

When we return to bytes(...) and do a readCode(), the code we get is U8_ARRAY, which is the type code that is written for the underlying compressed bytes when we perform compression. What should happen is that readCode gives us TYPE_PREFIX and then we decompress the bytes.

Having looked at the code I think a fix might be to only call getLength() after we have peeked and determined the code or only calling it when we have U8_ARRAY (as the length of the compressed data is unlikely to be very useful to us, we're about to decompress it).

I'm able to write bytes with chronicle queue. Using a TextWire, it looks like things get pushed out base64 encoded and uncompressed.

    @Override
    public void writeMarshallable(@NotNull WireOut wire) {
        wire.getValueOut().bytes(bytes);
    }

I'm trying to implement the read side of things:

    @Override
    public void readMarshallable(@NotNull WireIn wire) throws IORuntimeException {
        System.arraycopy(wire.getValueIn().bytes(), 0, bytes, 0, bytes.length);
    }

Unfortunately, it can't load up Compressions:

java.lang.NoClassDefFoundError: org/xerial/snappy/SnappyInputStream
    at net.openhft.chronicle.bytes.util.Compressions.<clinit>(Compressions.java:101)
    at net.openhft.chronicle.bytes.util.Compression.uncompress(Compression.java:99)

Am I using the interface incorrectly?

I could import the snappy library, but it's a bit inefficient since we are only using byte[]s for small ids that don't need any sort of snappy compression.

Perhaps I'm misusing marshallable within sequence? However, it doesn't print out the element separators for multiple objects within a sequence. The result is invalid yaml.

The output can, however, be read back in by Chronicle-wire. Interesting to note that the JsonWire extension overrides pushState() to set leaf() in this case as well - unsure if that would resolve.

gist of the test case attached below.

Cheers

Ryan

Gist: https://gist.github.com/ryanlea/5547d80f04f65b2d506528effc0f4330

Make a change to chronicle queue so that it uses an event name when writing the TimedStoreRecovery#recoverAndWriteHeader message. ( currently it does not do this ), then in the methodReader skip over this message, as the current code will crash if it encounters the TimedStoreRecovery.

OpenHFT/Chronicle-Queue#415

ERROR when loading in wire bundle

Can sun.misc be declared as optional?
What are the exact dependencies I have to load to make this work?

ERROR: Bundle net.openhft.chronicle-wire [80] Error starting file:bundles/net.openhft.chronicle-wire_1.3.4.jar (org.osgi.framework.BundleException: Unresolved constraint in bundle net.openhft.chronicle-wire [80]: Unable to resolve 80.0: missing requirement [80.0] osgi.wiring.package; (&(osgi.wiring.package=net.openhft.chronicle.bytes)(version>=1.2.0)(!(version>=2.0.0))) [caused by: Unable to resolve 81.0: missing requirement [81.0] osgi.wiring.package; (&(osgi.wiring.package=net.openhft.chronicle.core)(version>=1.3.0)(!(version>=2.0.0))) [caused by: Unable to resolve 82.0: missing requirement [82.0] osgi.wiring.package; (osgi.wiring.package=sun.misc)]])

Wires.ENCODE_TID_IN_HEADER needs to be set for this test, however it's value can be set to false before the test is run.

The test runs by itself but fails if Wires is initialised first.

I have experienced a stackoverflow exception when writing a object to a ValueOut while appending to a Chronicle Queue. I think the issues lies in the Wire library. The class of the object I am trying to write implements Serializable, and does not implement Marshallable.

The use of valueOut.object() works on some serializable classes, but not on one. I think the reason is that this class references itself in a List.

class A implements Serializable
{
    private String text;
    private List<A> list;
}

A call to ValueOut.object(myA) reaches the WireMarshaller.of() which recursively walks the fields of all members, but does not spot that it has already analysed a class before when it hits a reference to itself. This results in a stackoverflow exception.

I can post an example if this explanation isn't clear, but was wondering whether you were aware of this issue as a limitation or whether there is a known way to support writing objects like this without making them Marshallable (not always possible on legacy code).

public class WireModel implements Marshallable {

private long id;
private int revision;
private String key;

public WireModel(long id, int revision, String key) {
    this.id = id;
    this.revision = revision;
    this.key = key;
}

@Override
public void readMarshallable(@NotNull WireIn wire) throws IllegalStateException {
    this.id = wire.read(() -> "id").int64();
    this.revision = wire.read(() -> "revision").int32();
    this.key = wire.read(() -> "ket").text();
}

@Override
public void writeMarshallable(WireOut wire) {
    wire       
            .write(() -> "id").int64(id)
            .write(() -> "revision").int32(revision)
            .write(() -> "key").text(key);
}

what an I doing wrong here?
keep getting same error regardless of read or write I tried.

testMultipleReads0: Unordered fields not supported yet, Expected=id was: ''
testMultipleReads1: Unsupported type '3'

the "fast path" is no longer being invoked and so we can't take advantage of this optimisation any more

net.openhft.chronicle.wire.AbstractMarshallable.hashCode() line:32
  net.openhft.chronicle.wire.Marshallable.$hashCode(WriteMarshallable) line:43
    net.openhft.chronicle.wire.HashWire.hash32(WriteMarshallable) line:66
      net.openhft.chronicle.wire.HashWire.hash32(Object) line:70

new HashWire()

Unfortunately this happens for most primitives however long is the one which is causing a problem for a client.

Hi,
don't know if that's expected, but gradle complains when I add: compile 'net.openhft:chronicle-wire:1.+'. My graddle project as I rely only maven central repo, not your snapshot repo.
thanks,
B

The following code results in:

output: [obj: {
  "{"
}
]
Exception in thread "main" net.openhft.chronicle.core.io.IORuntimeException: Unterminated { while reading marshallable {,code='�', bytes=
	at net.openhft.chronicle.wire.TextWire$TextValueIn.marshallable(TextWire.java:2240)
	at net.openhft.chronicle.wire.ValueIn.marshallable(ValueIn.java:319)
	at com.redcurrent.backend.push.gcm.marshallable.MarshallableObj$Outer.readMarshallable(MarshallableObj.java:68)
	at com.redcurrent.backend.push.gcm.marshallable.MarshallableObj.main(MarshallableObj.java:108)

import com.google.common.base.Preconditions;
import net.openhft.chronicle.bytes.Bytes;
import net.openhft.chronicle.core.io.IORuntimeException;
import net.openhft.chronicle.wire.Marshallable;
import net.openhft.chronicle.wire.WireIn;
import net.openhft.chronicle.wire.WireOut;
import net.openhft.chronicle.wire.WireType;
import org.jetbrains.annotations.NotNull;

import java.nio.ByteBuffer;

/**
 * Created by dsmith on 11/11/16.
 */
public class MarshallableObj implements Marshallable {
    private final StringBuilder builder = new StringBuilder();

    public void clear() {
        builder.setLength(0);
    }

    public void append(CharSequence cs) {
        builder.append(cs);
    }

    @Override
    public void readMarshallable(WireIn wire) throws IORuntimeException {
        builder.setLength(0);
        Preconditions.checkState(wire.getValueIn().textTo(builder) != null);
    }

    @Override
    public void writeMarshallable(WireOut wire) {
        wire.getValueOut().text(builder);
    }

    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;

        MarshallableObj that = (MarshallableObj) o;

        return builder.toString().equals(that.builder.toString());
    }

    @Override
    public int hashCode() {
        return builder.toString().hashCode();
    }

    @Override
    public String toString() {
        return builder.toString();
    }

    public static class Outer implements Marshallable {
        private final MarshallableObj obj = new MarshallableObj();

        public MarshallableObj getObj() {
            return obj;
        }

        @Override
        public void readMarshallable(@NotNull WireIn wire) throws IORuntimeException {
            wire.read(() -> "obj").marshallable(obj);
        }

        @Override
        public void writeMarshallable(@NotNull WireOut wire) {
            wire.write(() -> "obj").marshallable(obj);
        }

        @Override
        public boolean equals(Object o) {
            if (this == o) return true;
            if (o == null || getClass() != o.getClass()) return false;

            Outer outer = (Outer) o;

            return obj.equals(outer.obj);

        }

        @Override
        public int hashCode() {
            return obj.hashCode();
        }
    }

    public static void main(String[] args) {
        final WireType wireType = WireType.TEXT;
        final String exampleString = "{";

        final Outer obj1 = new Outer();
        final Outer obj2 = new Outer();

        obj1.getObj().append(exampleString);

        final Bytes<ByteBuffer> bytes = Bytes.elasticByteBuffer();
        obj1.writeMarshallable(wireType.apply(bytes));

        final String output = bytes.toString();
        System.out.println("output: [" + output + "]");

        obj2.readMarshallable(wireType.apply(Bytes.from(output)));

        System.out.println("obj1.equals(obj2): " + obj1.equals(obj2));
    }
}

Changing to exampleString="}" executes with the equals being false.

Both examples successfully execute w/ WireType.BINARY.

Given this, and #37, it seems like TEXT isn't really ready for prime time?

AbstractMarshallable and net.openhft.chronicle.wire.WireMarshaller#readMarshallable etc. read and write direct from fields and don't use properties.
This issue a placeholder to investigate the performance impact etc. of allowing properties to be used in preference to fields.

Hi - I was thinking of writing an extension to TcpWireHandler to receive serialised objects which are then persisted to a Chronicle Queue.

Is there any recommended way to persist the message to the queue in this case ? I am keen to avoid serialising the object again as presumably the WireIn object has access to the underlying bytes.

Thanks,

David

VanillaWireParser was not recognising method names when encoded as MethodId integers. This resulted in debug messages like:

Unknown method-name='1' from 25 at ...

Changed history message to use writeEventName, to bring it inline with other places that the history message is written, as this makes it easier to detect if the history message is written using a custom bytes marshaller or wire. If its written with an event name, when we come to read it, we can safely assume, that it was written with wire.

e.g.

enumField: !UnknownEnum VALUE

reports, Unexpected character V. It could instead report, unknown class UnknownEnum

The following code outputs

obj1.equals(obj2): true
output: [hello\nworld
]
obj2.equals(obj3): false

I expect that obj2.equals(obj3) should be true.

import com.google.common.base.Preconditions;
import net.openhft.chronicle.bytes.Bytes;
import net.openhft.chronicle.core.io.IORuntimeException;
import net.openhft.chronicle.wire.Marshallable;
import net.openhft.chronicle.wire.WireIn;
import net.openhft.chronicle.wire.WireOut;
import net.openhft.chronicle.wire.WireType;

import java.nio.ByteBuffer;

/**
 * Created by dsmith on 11/11/16.
 */
public class MarshallableObj implements Marshallable {
    private final StringBuilder builder = new StringBuilder();

    public void clear() {
        builder.setLength(0);
    }

    public void append(CharSequence cs) {
        builder.append(cs);
    }

    @Override
    public void readMarshallable(WireIn wire) throws IORuntimeException {
        builder.setLength(0);
        Preconditions.checkState(wire.getValueIn().textTo(builder) != null);
    }

    @Override
    public void writeMarshallable(WireOut wire) {
        wire.getValueOut().text(builder);
    }

    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;

        MarshallableObj that = (MarshallableObj) o;

        return builder.toString().equals(that.builder.toString());
    }

    @Override
    public int hashCode() {
        return builder.toString().hashCode();
    }

    @Override
    public String toString() {
        return builder.toString();
    }

    public static void main(String[] args) {
        final WireType wireType = WireType.TEXT;
        final String exampleString = "hello\nworld";

        final MarshallableObj obj1 = new MarshallableObj();
        final MarshallableObj obj2 = new MarshallableObj();
        final MarshallableObj obj3 = new MarshallableObj();

        obj1.append(exampleString);
        obj2.append(exampleString);

        System.out.println("obj1.equals(obj2): " + obj1.equals(obj2));

        final Bytes<ByteBuffer> bytes = Bytes.elasticByteBuffer();
        obj2.writeMarshallable(wireType.apply(bytes));

        final String output = bytes.toString();
        System.out.println("output: [" + output + "]");

        obj3.readMarshallable(wireType.apply(Bytes.from(output)));

        System.out.println("obj2.equals(obj3): " + obj2.equals(obj3));
    }
}

Strangely enough, if you use "hello\\ncruel\nworld" or ""hello\ncruel\\nworld" it outputs true. Also, WireType.BINARY works in all cases. I'm assuming that I'm using the interface correctly since it works w/ BINARY.

testMappedBytesClose and testMappedBytesWireRelease2 now fail with latest Bytes.

as well as fields in a DTO.

This applies to text/yaml not binary.

please ignore the following event names when they are meta data {"header", "index", "index2index", "roll"}

Based on [OpenHFT/Chronicle-Queue] TreeMap encoding Issue (#321)