Blocker: Kafka Connect HDFS compile issue: blocker to getting started about kafka-connect-hdfs

[ewencp]

There are a few dependencies besides Kafka that may need snapshot builds: https://github.com/confluentinc/kafka-connect-hdfs/wiki/FAQ#when-i-try-to-compile-i-get-an-error-about-a-missing-snapshot-dependency

[rekhajoshm]

Sorry for getting late, got busy. However the steps you mentioned still do not work for me @ewencp
I am on Apache Maven 3.3.9 .The file is present in my local repo, installed it directly, but jar has some issue..

On building kafka connect hdfs, it threw the same error,
[WARNING] The POM for io.confluent:kafka-connect-avro-converter:jar:3.1.0 is missing, no dependency information available
Downloading: http://packages.confluent.io/maven/io/confluent/kafka-connect-avro-converter/3.1.0/kafka-connect-avro-converter-3.1.0.jar
Downloading: https://repo.maven.apache.org/maven2/io/confluent/kafka-connect-avro-converter/3.1.0/kafka-connect-avro-converter-3.1.0.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.084 s

so I installed it from my local repo
mvn install:install-file -DgroupId=io.confluent -DartifactId=kafka-connect-avro-converter -Dversion=3.1.0-SNAPSHOT -Dpackaging=jar -Dfile=/.m2/repository/io/confluent/kafka-connect-avro-converter/3.1.0-SNAPSHOT/kafka-connect-avro-converter-3.1.0-SNAPSHOT.jar.lastUpdated

but the build throws error in opening zip file.
../.m2/repository/io/confluent/kafka-connect-avro-converter/3.1.0-SNAPSHOT/kafka-connect-avro-converter-3.1.0-SNAPSHOT.jar; error in opening zip file
[INFO] 1 error
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE

Please let me know if I am missing something.thanks.

[ewencp]

@rekhajoshm You should not need to manually execute any install-file commands. It also looks like you are installing the wrong file -- a lastUpdated file is not the jar itself. It is an internal file used by Maven to track when it last tried to download a file.

Did the build and install of the confluentinc/schema-registry repository complete without errors? That is the missing dependency but it is included in the instructions I linked to.

[rekhajoshm]

Thanks for the quick reply @ewencp .Yes the mvn install on schema-registry had finished successfully.And I have done these steps multiple times now.

[INFO] kafka-serde-tools-package .......................... SUCCESS [ 0.797 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS

I undid the install of .lastUpdated jar, but the avro connector issue still persists for connect-hdfs.

Downloading: http://packages.confluent.io/maven/io/confluent/kafka-connect-avro-converter/3.1.0-SNAPSHOT/maven-metadata.xml
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.516 s
[INFO] Finished at: 2016-06-13T11:24:57-07:00
[INFO] Final Memory: 17M/309M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project kafka-connect-hdfs: Could not resolve dependencies for project io.confluent:kafka-connect-hdfs:jar:3.1.0-SNAPSHOT: Could not find artifact io.confluent:kafka-connect-avro-converter:jar:3.1.0-SNAPSHOT

[ewencp]

@rekhajoshm Can you paste a few more lines from above the info you pasted? That'll show whether the version info is as expected and the path that at least a few of the jars got installed to.

Are you on the master branch and pulled from all the repositories to make sure you're on matching versions? I just ran this all from scratch, having removed my .m2/repository entirely and moving my settings.xml (since that picks up our internal maven) and it seems to make it through everything fine. There was one dependency issue on our external Maven repository on rest-utils, but I'm fixing that up in confluentinc/rest-utils#41 and that doesn't look related to your issue.

[rekhajoshm]

Thanks @ewencp for following up.
I am on master branch on schema-registry and it pulls kafka-connect-avro-converter to local maven but version is 2.0.0-SNAPSHOT.

When I try building up kafka-connect using 2.0.0-SNAPSHOT version for avro, connect-hdfs works.But I suspect this might mean loss of functionality(which I will discover later) and also if instead of 2.0.0-SNAPSHOT, why 3.0.0-SNAPSHOT is not installed correctly.
Please let me know if I am missing something.Thanks!

updated connect-hdfs snippet:

io.confluent
kafka-connect-avro-converter
2.0.0-SNAPSHOT

schema-registry install log:
[INFO]
[INFO] --- maven-jar-plugin:2.6:jar (default-jar) @ kafka-connect-avro-converter ---
[INFO] Building jar: LOCAL_PATH/Documents/code/confluent-kafka/schema-registry/schema-registry/avro-converter/target/kafka-connect-avro-converter-2.0.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-jar-plugin:2.6:test-jar (default) @ kafka-connect-avro-converter ---
[INFO] Building jar: LOCAL_PATH/Documents/code/confluent-kafka/schema-registry/schema-registry/avro-converter/target/kafka-connect-avro-converter-2.0.0-SNAPSHOT-tests.jar
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ kafka-connect-avro-converter ---
[INFO] Installing LOCAL_PATH/Documents/code/confluent-kafka/schema-registry/schema-registry/avro-converter/target/kafka-connect-avro-converter-2.0.0-SNAPSHOT.jar to LOCAL_PATH/.m2/repository/io/confluent/kafka-connect-avro-converter/2.0.0-SNAPSHOT/kafka-connect-avro-converter-2.0.0-SNAPSHOT.jar
[INFO] Installing LOCAL_PATH/Documents/code/confluent-kafka/schema-registry/schema-registry/avro-converter/pom.xml to LOCAL_PATH/.m2/repository/io/confluent/kafka-connect-avro-converter/2.0.0-SNAPSHOT/kafka-connect-avro-converter-2.0.0-SNAPSHOT.pom
[INFO] Installing LOCAL_PATH/Documents/code/confluent-kafka/schema-registry/schema-registry/avro-converter/target/kafka-connect-avro-converter-2.0.0-SNAPSHOT-tests.jar to LOCAL_PATH/.m2/repository/io/confluent/kafka-connect-avro-converter/2.0.0-SNAPSHOT/kafka-connect-avro-converter-2.0.0-SNAPSHOT-tests.jar

[INFO] BUILD SUCCESS

[ewencp]

@rekhajoshm Can you run git pull? The master branch of schema-registry definitely isn't 2.0.0 anymore and hasn't been for awhile -- see https://github.com/confluentinc/schema-registry/blob/master/pom.xml#L10

[rekhajoshm]

Thanks @ewencp .my bad. thought i had the latest in these checkouts.Anyhow did a refresh of all repos @ewencp - Kafka, Avro converter, common, rest-utils now and get an issue with kafka-clients on rest-utils build.

[INFO] ------------------------------------------------------------------------
[WARNING] The POM for org.apache.kafka:kafka_2.11:jar:test:0.10.0.0-cp1 is missing, no dependency information available
[WARNING] The POM for org.apache.kafka:kafka-clients:jar:test:0.10.0.0-cp1 is missing, no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] rest-utils-parent .................................. SUCCESS [ 0.430 s]
[INFO] rest-utils ......................................... FAILURE [ 0.187 s]
[INFO] rest-utils-test .................................... SKIPPED
[INFO] rest-utils-example ................................. SKIPPED
[INFO] rest-utils-package ................................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.729 s
[INFO] Finished at: 2016-06-13T13:07:44-07:00
[INFO] Final Memory: 12M/309M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project rest-utils: Could not resolve dependencies for project io.confluent:rest-utils:jar:3.1.0-SNAPSHOT: The following artifacts could not be resolved: org.apache.kafka:kafka_2.11:jar:test:0.10.0.0-cp1, org.apache.kafka:kafka-clients:jar:test:0.10.0.0-cp1: Failure to find org.apache.kafka:kafka_2.11:jar:test:0.10.0.0-cp1 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced

[ewencp]

That's the issue being addressed in confluentinc/rest-utils#41

[rekhajoshm]

Have locally taken your change, and woohoo, hdfs connect is working now!
Thanks for your patience @ewencp in working through this issue.