alexholmes / json-mapreduce Goto Github PK

@OverRide
protected boolean isSplitable(JobContext context, Path file) {
CompressionCodec codec =
new CompressionCodecFactory(context.getConfiguration()).getCodec(file);
return codec == null;
}

error:

xception in thread "main" java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
at com.alexholmes.json.mapreduce.MultiLineJsonInputFormat.isSplitable(MultiLineJsonInputFormat.java:41)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:352)

This issue is not related to your code but more around creating runnable jar from eclipse and then using the command to run your ExampleJob code.

hadoop jar json-mapreduce-runnable.jar com.alexholmes.json.mapreduce.ExampleJob in out

I have added my own example and that requires some external jars. So in order to export the project I had to choose runnable jar instead of jar. Now when I create a runnable jar (from eclipse) I get error for my program. So I tried creating runnable jar for ExampleJob (launch config) and test and I see that it fails with following stack

Please note that both in and out directories are not present.

Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.io.IOException: Input directory 'com.alexholmes.json.mapreduce.ExampleJob' exists - please remove and rerun this example
at com.alexholmes.json.mapreduce.ExampleJob.writeInput(ExampleJob.java:98)
at com.alexholmes.json.mapreduce.ExampleJob.run(ExampleJob.java:120)

However when I create normal jar (not a runnable jar) then

hadoop jar json-mapreduce.jar com.alexholmes.json.mapreduce.ExampleJob in out

behaves as expected. Let me know if you know anything here.

json-mapreduce does not support UTF-8. The problem is in src/main/java/com/alexholmes/json/parser/PartitionedJsonParser.java that does not use a UTF-8 supported InputStream.