Import to pig fails when JSON contains an array about elasticsearch-hadoop

[costin]

Can you upload a small snippet of your data set so I can replicate the problem on my side? Just upload it somewhere since I'm not sure Github can handle attachments and posting here might cause whitespace problems.

[bwmeier]

I added a short (1 result) json document that shows the result from an elasticsearch query on my system. The result set contains an array in lines 19, 22, and 23, any of which will cause the problem.

https://github.com/bwmeier/elasticsearch-hadoop/blob/master/elasticsearch_array_result.json

If you have any questions about the result, let me know.

[costin]

@bwmeier the problem should be fixed in master. As @tzolov pointed out, there was a bug in parsing arrays into proper Writables causing the default, byte array serialization, to occur.
This should be now fix. Please try it out and let me know if that's not the case.

[bwmeier]

The error's changed slightly, but not significantly. I now get the following exception trace, which again looks like the array processing, just with the ArrayList type instead of the Byte[]. I suspect that pig needs a translation from ArrayList to DataBag, or something of the sort, since the message is related to "standard Pig type".

2013-04-19 12:12:04,057 WARN org.apache.hadoop.mapred.Child: Error running child
java.lang.RuntimeException: Unexpected data type java.util.ArrayList found in stream. Note only standard Pig type is supported when you output from UDF/LoadFunc
    at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:559)
    at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)
    at org.apache.pig.data.BinInterSedes.writeMap(BinInterSedes.java:581)
    at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:451)
    at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)
    at org.apache.pig.data.BinInterSedes.writeMap(BinInterSedes.java:581)
    at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:451)
    at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)
    at org.apache.pig.data.utils.SedesHelper.writeGenericTuple(SedesHelper.java:135)
    at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:613)
    at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:443)
    at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)
    at org.apache.pig.impl.io.InterRecordWriter.write(InterRecordWriter.java:73)
    at org.apache.pig.impl.io.InterStorage.putNext(InterStorage.java:87)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:98)
    at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:559)
    at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
    at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:106)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:285)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:278)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:673)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)

[costin]

Understood - I've reopened the issue and will try to address it by next week.

[bwmeier]

Thanks @costin, I appreciate the work. I'll test this out when I get the chance, but it won't be for a few days :-)

[costin]

Hi Boyd,

I think I've found (and fixed) the issue. Pig complex types were not handled properly when reading them back from Pig. Note that the conversion (not just for Pig but Hive, M/R etc...) will be overhauled however it should be working just fine now - the only issue is that bags are converted to tuples when are deserialized (and that's because we don't hold yet any extra information to allow us to differentiate between tuple and bags).
However this is something we plan to fix in the future.

It would be great if you could try out the latest master - see the readme on how to get the latest nightly build (I've just pushed one right now: http://build.elasticsearch.org/browse/ESHADOOP-NIGHTLY-40

[bwmeier]

Hi Costin, I've been on vacation since the 3rd, so I'm just now getting the
chance to look at this. I'll test it out and get back to you :-)

On Sat, May 4, 2013 at 3:19 AM, Costin Leau [email protected]:

Hi Boyd,

I think I've found (and fixed) the issue. Pig complex types were not
handled properly when reading them back from Pig. Note that the conversion
(not just for Pig but Hive, M/R etc...) will be overhauled however it
should be working just fine now - the only issue is that bags are converted
to tuples when are deserialized (and that's because we don't hold yet any
extra information to allow us to differentiate between tuple and bags).
However this is something we plan to fix in the future.

It would be great if you could try out the latest master - see the readme
on how to get the latest nightly build (I've just pushed one right now:
http://build.elasticsearch.org/browse/ESHADOOP-NIGHTLY-40

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/36#issuecomment-17429716
.

[bwmeier]

Costin, the fix works. However, the performance is pretty poor - I was able to process 37500 records and write them to JsonStorage, but it took about 45 minutes, I'm not sure if it's related to the fix or not.

I tested the version of the fix pointed to by that commit, I have not tested anything later than that.

[costin]

Note that there's an upcoming serialization improvement for writing to ES coming hopefully in the next few days.
It sounds though as this is not what you're doing - can you share your test (through a gist/email/pm maybe)?
You mention JsonStorage but I'm not sure why you would want to use since it adds the overhead of converting to JSON which we already handle.
Can you explain your use case more?