Cascading example has an exception at the end, and some incompatibily with Cascalog about elasticsearch-hadoop

[costin]

Thanks for the report. The errors are somewhat conspicuous. The first one
seems to be caused by the fact that we merge hadoop old and new api into
the same class (which migt be exacerbated by clojure) while the latter
looks like a bug. I recall quite a number of changes between cascading
2.0.x and 2.1.x - not sure whether these are to blame or not...
Have you tried upgrading cascading with cascalog?

On Thursday, May 9, 2013, Jeroen van Dijk wrote:

Thanks for this repo. I've tried the Cascading code and it seems to work.
The input example gives an error at the end though, see
https://github.com/jeroenvandijk/elasticsearch-hadoop-trial/blob/master/src/trial/input.clj

I also tried to get the ES Cascading code to work with Cascalog. The input
code worked perfectly, the output code failed at the Cascading level
though, see
https://github.com/jeroenvandijk/elasticsearch-hadoop-trial/blob/master/src/trial/cascalog/output.clj.
I don't why this would happens. Cascalog does depend on an older version of
Cascading (2.0.8), but I'm not sure if that's the issue.

The complete trial repo can be found here:
https://github.com/jeroenvandijk/elasticsearch-hadoop-trial

I hope you don't mind the Clojure code. I hope it translates easily to the
original Java code.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/44
.

Costin

[jeroenvandijk]

Thanks for you reply. I haven't tried changing the internals of Cascalog if that is what you mean. This project already uses Cascading 2.15 so I guess Cascalog is fine with that for the sink part, but might not be for the source part. I did try downgrading the Cascading version to 2.0.8 (the one Cascalog wants), but that broke the sink version and gave the same error for the source version if I recall correctly.

[costin]

That's good to know as it excludes the version incompatibilities in
cascading.

On Thursday, May 9, 2013, Jeroen van Dijk wrote:

Thanks for you reply. I haven't tried changing the internals of Cascalog
if that is what you mean. This project already uses Cascading 2.15 so I
guess Cascalog is fine with that for the sink part, but might not be for
the source part. I did try downgrading the Cascading version to 2.0.8 (the
one Cascalog wants), but that broke the sink version and gave the same
error for the source version if I recall correctly.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/44#issuecomment-17683196
.

Costin

[costin]

The latest nightly build (#50) fixes the AbstractMethodError you saw during the input ingestion.
I'll take a look at the output code tomorrow (and hope to fix it the same day).

[costin]

I have eliminated the NPE from the source part in Cascalog. It looks like Cascalog is not fully upgraded to Cascading 2.1.x since it breaks the init contract (Tap#flowConfInit is not called, which the NPE to occur).
I've bumped into a different issue related to parsing which I need to fix (see #49).
These being said, unless Cascalog is properly upgraded to Cascading 2.1 (and Hadoop 1.x which is marked as stable and has been released for quite some time now), I foresee other issues down the road.

[jeroenvandijk]

I just had some time to look into this again. Thanks for taking the time to fix this. Should the nullpointer issue be fixed now? I still got that issue locally. Maybe I have an old version?

Cascalog needs to use a newer version Indeed. Hopefully this will happen soon.

[costin]

The NPE should not occur - at least I thought I fixed that. Make sure you use the latest nightly build (you can double check the jar manifest to get the build number) and if it still occurs, please post the stacktrace in a gist or something.

Regarding cascalog I'm not sure why the upgrade hasn't happened yet - Hadoop 1.1.2 is considered stable (as oppose to 0.20) while Cascading is already at 2.1.6 (as opposed to 2.0.8)

[costin]

@jeroenvandijk I've tested the latest 1.10 cascalog release and all the tests pass fine. I have forked your project, applied some minor updates and published the results here.

I'm closing the issue for now but would be great to hear whether things are working on your end.

Cheers!