Comments (5)
from griffin.
hi bhlx3lyx7
thank you,I have reslove this problem.but when i use livy to submit sparkjob,the errors are
as below:
Warning: Skip remote jar hdfs://wecloud-cluster/project/pgxl/griffin/griffin-measure.jar.
Warning: Skip remote jar hdfs://wecloud-cluster/project/pgxl/griffin/datanucleus-api-jdo-3.2.6.jar.
Warning: Skip remote jar hdfs://wecloud-cluster/project/pgxl/griffin/datanucleus-core-3.2.10.jar.
Warning: Skip remote jar hdfs://wecloud-cluster/project/pgxl/griffin/datanucleus-rdbms-3.2.9.jar.
java.lang.ClassNotFoundException: org.apache.griffin.measure.Application
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.spark.util.Utils$.classForName(Utils.scala:175)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:689)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
from griffin.
It seems like to be some version issue, the griffin-measure.jar is using scala 2.10 by default, which means the spark version should be matched with scala 2.10.
Griffin measure works on spark 1.6, but the scala version could be modified, you can package your own jar for scala 2.11, just modify scala.version
and scala.binary.version
in the pom.xml of measure module, run mvn clean package
in the measure directory.
from griffin.
Thanks for your response,i have resloved the problem by change the config.but now the job can be submitted to the yarn cluser,other problems comes.
According to the confs in the sparkJob.properties,sparkJob.spark.jars.packages is "com.databricks:spark-avro_2.10:2.0.1".
When I submit jobs by using "--packages com.databricks:spark-avro_2.10:2.0.1 ",it works well.But when use livy to submit jobs,the errors comes:
Application application_1512628890181_92576 failed 2 times due to AM Container for appattempt_1512628890181_92576_000002 exited with exitCode: -1000
For more detailed output, check application tracking page:http://xy180-wecloud-198:8088/proxy/application_1512628890181_92576/Then, click on links to logs of each attempt.
Diagnostics: File does not exist: hdfs://wecloud-cluster/user/pgxl/.sparkStaging/application_1512628890181_92576/com.databricks_spark-avro_2.10-2.0.1.jar
java.io.FileNotFoundException: File does not exist: hdfs://wecloud-cluster/user/pgxl/.sparkStaging/application_1512628890181_92576/com.databricks_spark-avro_2.10-2.0.1.jar
I have delete the code about spark.jars.packages in the class SparkSubmitJob,it also don't works.
from griffin.
It's a tricky issue, the measure module supports avro file by this package, but we don't want to package it in measure.jar, so we need to tell spark where it is.
sparkJob.spark.jars.packages=com.databricks:spark-avro_2.10:2.0.1
will let spark application download this package when it starts, it exits if download fails.
You can download this package manually, put it in hdfs, and set the config in sparkJob.properties, like sparkJob.jars_4=hdfs:///path/to/spark-avro_2.10.jar
, then modify some code in service module to enable this property when submit to livy, that will resolve this issue.
In the next version, we'll consider to modify sparkJob.properties, to fit livy parameters friendly.
from griffin.
Related Issues (20)
- Name is still "Bark" on API doc page
- Model Stage is always initial status, never be executed HOT 8
- Asset type only contains hivetable, but how to connect with my hive table? HOT 1
- Cannot create container with more than 127 parents HOT 5
- Mongo Error when starting griffin-env docker container HOT 5
- how to register partitioned hive table asset
- run in docker problem HOT 2
- build docker images DataStreamer Exception HOT 1
- Run docker image griffin-env JDOFatalDataStoreException HOT 4
- Docker failed to launch spark Worker HOT 1
- Multiple datastore HOT 1
- spark-submit error with the compute of TotalCount_griffin.users_info_src HOT 1
- I want to ask, did I create a DQMetrics to display the graphics failure?
- Hello,I had a problem, I created a metric, but what about healthy graphics, not showing?
- 400 Missing Required Header for CSRF protection. HOT 1
- Issue, unable to connect to the web UI HOT 5
- jobs fail when use yarn-cluster HOT 1
- when post2Livy ,the griffin service exception :[ERROR 5251 --- [ryBean_Worker-5] o.a.g.c.j.SparkSubmitJob : Post to livy error. 500 Internal Server Erro]
- use incubator-griffin-master submit job the livy get exception : [spark-submit exited with code 254 ] HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from griffin.