alaurentinoofficial / basic-spark-scala

Requirements

• Install Java JDK 11 Install here
• Install Scala 2 Install here
• Install SBT Install here
• Download Spark (3.1.2) Download here

Configure spark

Mac OS

1. Extract the spark folder spark-3.1.2-bin-hadoop3.2.tgz

tar -xfz ./spark-3.1.2-bin-hadoop3.2.tgz

3. Create folder ~/spark

mkdir ~/spark

4. Move the spark folder

$ mv ./spark-3.1.2-bin-hadoop3.2 ~/spark/spark-3.1.2

5. Add export variables and update the path using ZSH profile (Replace "{Your User}" to your user in OS)

$ echo '\nexport JAVA_HOME=$(/usr/libexec/java_home)' >> ~/.zshrc
$ echo 'export SPARK_HOME="/Users/{Your User}/spark/spark-3.1.2"' >> ~/.zshrc
$ echo 'export PATH="$SPARK_HOME/bin:$PATH"\n' >> ~/.zshrc

Environments

Visual Studio Code

Link to download and Install

1. Install the Scala metals extension