Engine fails on UAST when run spark-shell with --packages argument in cluster mode about jgit-spark-connector

[r0mainK]

Was having the same problem, solved by doing apt-get update && apt-get install libxml2 libxml2-dev make gcc g++ curl python3-dev && pip3 install --upgrade bblfsh on pod and workers

[ajnavarro]

the --packages command is downloading the file from here: http://repo1.maven.org/maven2/tech/sourced/engine/0.6.1/engine-0.6.1.jar

Can you try downloading this jar file and trying it using --jars?

[r0mainK]

Actually I ran into this problems afterwards, but was able to solve by cleaning up my workers after installing new bblfsh, e.g.

kubectl exec -it l-spark-spark-worker-<id>

rm -r /opt/spark/jars/*
ls -l /usr/local/lib/python3.4/dist-packages/pyspark/jars | wc -l   
# 209 jars
cp /usr/local/lib/python3.4/dist-packages/pyspark/jars/*.jar /opt/spark/jars/
cp /opt/spark-2.2.0-bin-hadoop2.7/jars/
ls -l /opt/spark-2.2.0-bin-hadoop2.7/work/app-20180429202823-0200/10/*.jar | wc -l
# 51 jars (this folder was created after reproducing this issue)
cp /opt/spark-2.2.0-bin-hadoop2.7/work/app-20180429202823-0200/10/*.jar /opt/spark/jars/

It seemed to fix the problem for me, I was able to run extract UAST afterwords with no problem. You may want to delete the ivy2cache before running the shell just in case, too.

[smacker]

@ajnavarro I have tried it already. Look at the second case from the description:

# wget http://repo1.maven.org/maven2/tech/sourced/engine/0.6.1/engine-0.6.1.jar
# spark-shell --jars /engine-0.6.1.jar --master=spark://p-spark-master:7077

@r0mainK thanks for insights! Interesting!

[ajnavarro]

@smacker, did the insights given by @r0mainK work for you?

[smacker]

just cleaning up /opt/spark/jars/* helped