This Dockerfile sets up a complete streaming environment for experimenting with Kafka and Spark streaming (PySpark). It installs
- Kafka 2.4.0;
- Spark 2.4.4.
It additionnally installs
- Anaconda distribution for Python 3.7;
- Jupyter notebook for Python;
- Geopandas python package for geospatial data (http://geopandas.org/index.html).
Docker is a software container platform, which allows to isolate OS environments. Its main benefits over virtual machines are a reduced footprint (starting from 60MB for a Linux distribution - for example ubuntu), and an easy way reconfigure an OS environment. Note: The Docker container for this course is 7.7GB, mostly due to the Anaconda (>1GB) and Spark (>1GB) distributions.
For more information on Docker, see:
To download and install, see: (NB: install the Community Edition)
Get started Docker tutorial:
We strongly encourage you to follow the tutorial, and learn how to build, run, pull and push a container.
In order to avoid building the image from scratch, a prebuilt image is made available from DockerHub, see below.
The image is called kafkasparkgio
and is available from DockerHub (Note: image is 7.7GB, make sure you have a reasonably good Internet conection).
To install the image, use the standard docker pull
command
docker pull giobbu/kafkasparkgio
Create a folder and cd into it.
mkdir Folder_Geospatial_KafkaSpark
cd Folder_Geospatial_KafkaSpark
Git clone the repository for the course
git clone https://github.com/giobbu/Geospatial_KafkaSpark
Cd to the Geospatial_KafkaSpark
folder
cd Geospatial_KafkaSpark
Finally, give recursive permission to all for writing to it (ease the sharing with Docker container)
chmod -R a+rwx .
The Docker container should now be able to read/write to your host Geospatial_KafkaSpark
folder.
From the Geospatial_KafkaSpark
folder, start the container with
docker run -v `pwd`:/home/guest/host -p 8888:8888 -p 4040:4040 -p 23:22 -it giobbu/kafkasparkgio bash
Notes
- -v is used to share folder (right permissions given above will allow your changes to be saved on your computer). The "-v pwd:/home/guest/host" shares the local folder (i.e. folder containing Dockerfile, ipynb files, etc...) on your computer;
- -it starts the Docker container in interactive mode, so you can use the console and Bash;
- -p is for sharing ports between the container and the host. 8888 is the notebook port, and 4040 the Spark UI port.
SSH allows to get a onnection to the container
ssh -p 23 guest@containerIP
where 'containerIP' is the IP of th container (127.0.0.1 on Linux). Password is 'guest'.
Once run, you are logged in as root in the container. Run the startup_script.sh (in /usr/bin) to start
- SSH server. You can connect to the container using user 'guest' and password 'guest'
- Zookeeper server
- Kafka server
startup_script.sh
Connect as user 'guest' and go to 'host' folder (shared with the host)
su guest
Start Jupyter notebook
notebook
and connect from your browser at port host:8888 (where 'host' is the IP for your host. If run locally on your computer, this should be 127.0.0.1 or 192.168.99.100, check Docker documentation)
Open OBUcsvToKafka.ipynb and run all cells.
Open sparkStreamingPredictionsOBU.ipynb and run cells up to start streaming.
It is available in your browser at port 4040
The container is based on CentOS 6 Linux distribution. The main steps of the building process are
- Install some common Linux tools (wget, unzip, tar, ssh tools, ...), and Java (1.8)
- Create a guest user (UID important for sharing folders with host!, see below), and install Spark and sbt, Kafka, Anaconda and Jupyter notbooks for the guest user
- Go back to root user, and install startup script (for starting SSH), sentenv.sh script to set up environment variables (JAVA, Kafka, Spark, ...)and spark-default.conf
In the Dockerfile, the line
RUN useradd guest -u 1000
creates the user under which the container will be run as a guest user. The username is 'guest', with password 'guest', and the '-u' parameter sets the linux UID for that user.
In order to make sharing of folders easier between the container and your host, make sure this UID matches your user UID on the host. You can see what your host UID is with
echo $UID
On some versions of Mac OSX and Windows, the Docker Toolbox is used instead of the native Docker Engine on the system.
In this case, instead of connecting to the IP address of the localhost (i.e. 127.0.0.1), one should connect to the IP address of the virtual machine where the Docker Engine is running (192.168.99.100, by default).
For example, to connect to the Jupyter Notebook, you should open your browser at 192.168.99.100:8888
The Docker Toolbox on Windows 10 Home shares by default only the primary hard drive (i.e. the one where Windows is installed -> C:).
In case your machine has more than one hard disk two solutions exists:
- Either the Geospatial_KafkaSpark folder has to be located on the main hard drive (C:)
- Or, a manual mountpoint to the second hard drive must be created in the configuration of VirtualBox (the provisioner running the virtual machine containing Docker Engine) as described here.