sabman / pysparkgeoanalysis

:globe_with_meridians: Interactive Workshop on GeoAnalysis using PySpark

Docker Image Test Status:

A Small Course on Big Data - GeoAnalysis using PySpark

House Keeping

Who's Here?

I love staying in touch here's a link to a form where you can add your details for me to stay in touch with you. I also love feedback good and bad! I love to get better at my job. So as we go though this course I want you to keep in mind that I will ask you to provide some feedback afterwards. You can keep it anonymous of choose to tell me who you are. See feedback form here: Feedback Form

• Who is using Spark in Production?
• Who is doing Geospatial Analysis using Spark?
• Who is a programmer?
• Who is a Data Janitor... err I mean Scientist 😄
• Who is a hedge fund manager? ... here's my number 181821113 (bank account number, that is!)
• Who is doing something else? I have missed?

Introduction

This workshop will introduce you to Apache Spark via the exciting domain of Geospatial Analysis.

Setup

Dependencies:

See: docker/README.md

Data

If you use docker the data will automatically downloaded into the work-flow folder. See docker/README.md

pysparkgeoanalysis's Issues

Add Geo related sides

At PyData people didn't know about Geo. So it's good to have a set of slides introducing Geo to Python people. While at at FOSS4G and GeoPython people would know more about Geo but not Spark hence i have created #1

• Fiona
• Shapely
• GeoPandas
• PySAL

Add Spark related sections to slides:

• Introduction to Spark @sabman started this
• What is PySpark and how it works
• Using Postgresql and other databases with PySpark
• Introduction DataFrames
• PySpark MLLib example
• Writing a stand alone PySpark Application
• Deploying a PySpark Application

Restructure the workshop

Initial

• Quick Intro to Docker
• Get people set up

Workshop

• Spark Overview
• Workshop Scenario
• Line Count Example with RDDs
• Line Count Example with DataFrames
• Introduce Geo Ecosystem
• Data Exploration
• Spatial Join using SparkSQL and UDF
• SpatialSpark
• Full Analysis Pipeline
• Visualization

please add binding port 4040:4040 in docker doc

seen in geopython ;)

Add automated build to ensure Docker setup is correct

Fix SpatialSpark to work with Python

• improve Python API
• support other spatial joins beside broadcast join

refs https://github.com/sabman/SpatialSpark

How can I reproduce the sixth section?

Hello, @sabman I'm trying to reproduce your sixth section 006-spatial-spark.ipynb. But I get this error:

Traceback (most recent call last):
  File "/data1/alge/yugang/roadnet/map_matching/detest_track/generate_image/spark_test.py", line 78, in <module>
    joinRDD = BroadcastSpatialJoin.apply(sc._jsc, dfABC._jdf, dfAB._jdf, SpatialOperator.Intersects(), 0.0)
TypeError: 'JavaPackage' object is not callable

I guess it's because I didn't download the spatial-spark_2.10-1.1.1-beta-SNAPSHOT.jar package. But I can't access your download link https://dl.dropboxusercontent.com/u/96303065/spark-workshop/spatial-spark_2.10-1.1.1-beta-SNAPSHOT.jar , which no longer working.

Can you help me? Thanks in advance.