WritableBytesConverter always falls back to JdkBytesConverter about elasticsearch-hadoop

[masseyke]

Thanks for the report, and that looks to me like a bug in the commit you linked, too.

[masseyke]

@jasperiq what are you doing to hit this (I want to test it as much like your use as possible)? Thanks.

[jasperiq]

I think this sums it up:

MultipleOutputs<NullWritable, Text> outputs = new Multiple ;
String namedOutput = "some named output";
String documentJson = "any kind of json";
outputs.write(namedOutput, NullWritable.get(), new Text(documentJson));

It goes wrong when wrapping the documentJson in a Text writable. Also when I wrap byte[] in Text it goes wrong.

This actual issue I have is that because of it does the toString() in the JdkBytesConverter it becomes inefficient. I start with String, in Text this is translated into a new byte[], JdkBytesConverter creates another String, and because of this String, BytesArray needs to make sure it's in UTF-8 and creates a byte[] again. This all makes me run into heap when writing larger documents.

I worked around this by directly write byte[] to the MultipleOutputs for now. However, looking at the code, I think the intention was to support Text writables in an efficient way as well.

I hope this explains my issue enough. Let me know if you have more questions.

[masseyke]

OK thanks. I think my change in #2075 will fix that.

[jasperiq]

Indeed!