error while executing HADOOP_DATASET_METADATA_ETL about datahub

[SunZhaonan]

wherehows.db.username is the connection info that used to connect to wherehows mysql database. It's in the wh_etl_job table. https://github.com/linkedin/WhereHows/wiki/Set-Up-New-Metadata-ETL-Jobs#2-fill-in-all-the-configurations-needed-for-the-job-type

You can also refer to set up in the VM.

Just found we lack of an API to add that configuration. Will add that later.

[shuklab]

It seems the correct table is wh_property and not wh_etl_job.

[SunZhaonan]

Oh yes, let me correct that

[carnalim]

I am getting this same error but when I do a select * from wh_property,
wherehows.db.username wherehows N NULL
I have that property already inserted into my DB. Just not sure why it would be still having that error. Is it because I did a manual insert statements?

[SunZhaonan]

Manual insert should be fine, that's what I did here. Do you also see the error of 'KeyError: wherehows.db.username' ?

[carnalim]

I am getting this: The file below is iostream its blank in the comment.
`
2016-04-06 15:21:54 ERROR Job Launcher:83 - Traceback (most recent call last):
File "", line 340, in
KeyError: wherehows.db.username

at org.python.core.Py.KeyError(Py.java:249)
at org.python.core.PyObject.__getitem__(PyObject.java:738)
at org.python.pycode._pyx1.f$0(<iostream>:356)
at org.python.pycode._pyx1.call_function(<iostream>)
at org.python.core.PyTableCode.call(PyTableCode.java:167)
at org.python.core.PyCode.call(PyCode.java:18)
at org.python.core.Py.runCode(Py.java:1386)
at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:296)
at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:291)
at metadata.etl.dataset.hdfs.HdfsMetadataEtl.load(HdfsMetadataEtl.java:160)
at metadata.etl.EtlJob.run(EtlJob.java:181)
at metadata.etl.Launcher.main(Launcher.java:77)`

[carnalim]

Has there been any traction on this issue?

[SunZhaonan]

Follow the steps here : https://github.com/linkedin/WhereHows/wiki/Set-Up-New-Metadata-ETL-Jobs#2-fill-in-all-the-configurations-needed-for-the-job-type

Insert the wherehows.db.username (the mysql connection username) into wh_property can solve this problem.

[carnalim]

As I mentioned in the above comment I have already inserted those parameters yet still receiving that error. jdbc.url, password, username, etc all filled in.

[SunZhaonan]

Looks like your error happened when java pass the parameters to jython.

Are you running on your local machine or in the VM we provided?
Are you executing it as a single job testing mode or executing it as production mode by start the play service?

Could you do these :
1 at this line : https://github.com/linkedin/WhereHows/blob/master/metadata-etl/src/main/resources/jython/HdfsLoad.py#L335
add print args

2 at this line : https://github.com/linkedin/WhereHows/blob/master/metadata-etl/src/main/java/metadata/etl/dataset/hdfs/HdfsMetadataEtl.java#L170
add System.out.println(interpreter.getSystemState().argv);

This will print out the parameter before and after call jython interpreter. So we can check whether the parameter have passed in jython successfully.

[carnalim]

So pulled the latest code, added those lines and deployed. When I run the test mode job something different is happening:

[root@ip-206-40-95 backend-service-1.0-SNAPSHOT]# java -Djob=HADOOP_DATASET_METADATA_ETL -Dconfig=sample.properties -cp "lib/*" metadata.etl.Launcher
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/space/wherehows/backend-service-1.0-SNAPSHOT/lib/slf4j-log4j12-1.7.12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/space/wherehows/backend-service-1.0-SNAPSHOT/lib/ch.qos.logback.logback-classic-1.0.13.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

This is what my config looks like:

# HDFS properties
hdfs.cluster=cluster1
hdfs.private_key_location=/home/jmblnvr/.ssh/id_rsa
hdfs.remote.raw_metadata=.wherehows/hdfs_metadata_file
hdfs.remote.sample=.wherehows/hdfs_sample.dat
hdfs.remote.user=jmblnvr
hdfs.remote.machine=localhost
hdfs.remote.jar=.wherehows/schemaFetch.jar
hdfs.local.raw_metadata=/tmp/hdfs_metadata_file
hdfs.local.metadata=/tmp/hdfs_metadata.dat
hdfs.local.field_metadata=/tmp/hdfs_field_metadata
hdfs.local.sample=/tmp/hdfs_sample.dat
hdfs.white_list=/user,/src
hdfs.num_of_thread=2
hdfs.file_path_regex_source_map=[{".*":"Hdfs"}]

Last time it executed but this time it just shows those log4j things but nothing else.

[SunZhaonan]

This looks like a slf4j binding problem. It may caused by classpath configuration. Will check it later.

[SunZhaonan]

The output is in the /var/tmp/wherehows/wherehows.log file .
The log4j output was changed in this commit : a0b7cb9

You can add 'stdout' back so it will show up in the console.