azavea / ansible-spark

An Ansible role for installing Apache Spark.

ansible-spark

An Ansible role for installing Apache Spark.

Role Variables

• spark_version - Spark version.
• spark_install_java - flag toggling the JDK installation using the builtin azavea.java role dependency (default: yes)
• spark_cloudera_distribution - Cloudera distribution version (default: cdh5.4)
• spark_symlinks_enabled (default yes) - if yes deploy 2 symlinks (<spark_home>/conf -> /etc/spark/conf ; <spark_home> -> spark_usr_dir )
• spark_shims_enabled (default yes) - if yes deploy the shims (like /usr/bin/spark-shell, /usr/bin/spark-submit)
• spark_env_extras - An optional dictionary with key and value attributes to add to spark-env.sh (e.g. MESOS_NATIVE_LIBRARY: "/usr/local/lib/libmesos.so")
• spark_defaults_extras - An optional dictionary with key and value attributes to add to spark-defaults.conf (e.g. "spark.eventLog.enabled": true)
• spark_user_groups - an optional list of (OS)groups the spark user should belong to
• spark_user_shell - the spark user's default shell (default: /bin/false)
• spark_log4j_loggers - A list of dictionaries configuring the spark log4j loggers (default: logger config from spark/conf/log4j.properties.template)

Example Playbook

See the examples directory.

ansible-spark's Issues

Give example of a small Spark cluster

It'd be nice to see how to use the role to create a small Spark cluster.

tests with molecule 2.10 on Ubuntu and centos

I am happy to announce I got a first working version of molecule v2 tests ready for another PR (once my other PRs are merged 👍 )

A quick Roadmap of my plans:

• test on various distros (already implemented on Ubuntu 16.04, centos7)
• basic tests verifying the spark deployment incl. the supported config files
• tests for deployed spark-services (needs a PR first of spark-services)
• tests of a small spark cluster (1. using standalone spark master/agents, tough we ran spark on mesos)

Feel free to comment with ideas

Add export of SPARK_DIST_CLASSPATH to spark-env.sh

Using Spark 1.5.0 requires that this line be present in the spark env:

export SPARK_DIST_CLASSPATH=$(hadoop classpath)

creating of the spark user fails, with an existing kerberos/ldap managed user that has different home or shell attributes

I provided a fix in our repos (based on latest 'develop' branch, see #10 ) : https://github.com/teralytics/ansible-spark/pull/7

Can we get ansible-spark for CentOs

Hello All,

Can we get an ansible-spark playbook for centos? Or how we can modify this playbook to work on centos?

Thanks & regards,
Prashantha As

Add support for Debian Jessie

With the following diff applied against 8a51ef5, I was able to get Spark installed on the debian/jessie64 Vagrant box:

diff --git a/examples/Vagrantfile b/examples/Vagrantfile
index c309b84..d42cf9a 100644
--- a/examples/Vagrantfile
+++ b/examples/Vagrantfile
@@ -15,7 +15,9 @@ if [ "up", "provision" ].include?(ARGV.first) &&
 end

 Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
-  config.vm.box = "ubuntu/trusty64"
+  config.vm.box = "debian/jessie64"
+
+  config.vm.synced_folder ".", "/vagrant", disabled: true

   config.vm.network "forwarded_port", guest: 4040, host: 4040

diff --git a/examples/site.yml b/examples/site.yml
index 0016d20..a8340b7 100644
--- a/examples/site.yml
+++ b/examples/site.yml
@@ -2,7 +2,8 @@
 - hosts: all

   vars:
-    java_version: "7u51-2.4.*"
+    spark_version: "1.6.0-bin-hadoop2.6"
+    java_version: "7u95-2.6.*"
     spark_env_extras:
       TEST_B: "b"
       TEST_A: "a"

Output of running spark-shell:

vagrant@debian-jessie:~$ mkdir /tmp/spark-events && spark-shell
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's repl log4j profile: org/apache/spark/log4j-defaults-repl.properties
To adjust logging level use sc.setLogLevel("INFO")
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.6.0
      /_/

Using Scala version 2.10.5 (OpenJDK 64-Bit Server VM, Java 1.7.0_95)
Type in expressions to have them evaluated.
Type :help for more information.
16/03/24 19:23:15 WARN Utils: Your hostname, debian-jessie resolves to a loopback address: 127.0.1.1; using 10.0.2.15 instead (on interface eth0)
16/03/24 19:23:15 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Spark context available as sc.
16/03/24 19:23:20 WARN General: Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/usr/lib/spark-1.6.0-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/usr/lib/spark/lib/datanucleus-api-jdo-3.2.6.jar."
16/03/24 19:23:20 WARN General: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/usr/lib/spark/lib/datanucleus-core-3.2.10.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/usr/lib/spark-1.6.0-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar."
16/03/24 19:23:20 WARN General: Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/usr/lib/spark-1.6.0-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/usr/lib/spark/lib/datanucleus-rdbms-3.2.9.jar."
16/03/24 19:23:20 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/03/24 19:23:21 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/03/24 19:23:26 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/03/24 19:23:27 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
16/03/24 19:23:30 WARN General: Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/usr/lib/spark-1.6.0-bin-hadoop2.6/lib/datanucleus-api-jdo-3.2.6.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/usr/lib/spark/lib/datanucleus-api-jdo-3.2.6.jar."
16/03/24 19:23:30 WARN General: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/usr/lib/spark/lib/datanucleus-core-3.2.10.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/usr/lib/spark-1.6.0-bin-hadoop2.6/lib/datanucleus-core-3.2.10.jar."
16/03/24 19:23:30 WARN General: Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/usr/lib/spark-1.6.0-bin-hadoop2.6/lib/datanucleus-rdbms-3.2.9.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/usr/lib/spark/lib/datanucleus-rdbms-3.2.9.jar."
16/03/24 19:23:30 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/03/24 19:23:31 WARN Connection: BoneCP specified but not present in CLASSPATH (or one of dependencies)
16/03/24 19:23:37 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
16/03/24 19:23:37 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
SQL context available as sqlContext.

scala> 

Remove temporary utils.sh fix

Determine if utils.sh is properly installed by 1.1.0+cdh5.2.1+63-1.cdh5.2.1.p0.9~trusty-cdh5.2.1.

See here for details.