tomaugspurger / dask-tutorial-pycon-2018

Parallel Data Analysis with Dask

Materials for the Dask tutorial at PyCon 2018.

First Time Setup

If you don't have git installed, you can download a ZIP copy of the repository using the green button above. Note that the file will be called dask-tutorial-pycon-2018-master, instead of dask-tutorial-pycon-2018. Adjust the commands below accordingly.

Install Miniconda or ensure you have Python 3.6 installed on your system.

# Update conda
conda update conda

# Clone the repository. Or download the ZIP and add `-master` to the name.
git clone https://github.com/TomAugspurger/dask-tutorial-pycon-2018

# Enter the repository
cd dask-tutorial-pycon-2018

# Create the environment
conda env create

# Activate the environment
conda activate dask-pycon

# Download data
python prep_data.py

# Start jupyterlab
jupyter lab

If you aren't using conda

# Clone the repository. Or download the ZIP and add `-master` to the name.
git clone https://github.com/TomAugspurger/dask-tutorial-pycon-2018

# Enter the repository
cd dsak-tutorial-pycon-2018

# Create a virtualenv
python3 -m venv .env

# Activate the env
# See https://docs.python.org/3/library/venv.html#creating-virtual-environments
# For bash it's
source .env/bin/activate

# Install the dependencies
python -m pip install -r requirements.txt

# Download data
python prep_data.py

# Start jupyterlab
jupyter lab

Connect to the Cluster

We have a pangeo deployment running that'll provide everyone with their own cluster to try out Dask on some larger problems. You can log into the cluster by going to:

dask-tutorial-pycon-2018's Issues

Dockerfile updates

• install graphviz
• debug labextension
• figure out why juptyerlab can't reach the gcr.io/ version

Data Files

What do we want to include? @jcrist's tutorial from PyData Seattle used nycflights, which is 42M compressed. Do we want to include that in the repo, or throw it on S3 / GCSFS and download it in a prep script?

CommandNotFound

After following the instructions I got this problem both on my laptop and on a VM on GCP

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
If your shell is Bash or a Bourne variant, enable conda for the current user with
    $ echo ". /opt/conda/etc/profile.d/conda.sh" >> ~/.bashrc
or, for all users, enable conda with
    $ sudo ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh
The options above will permanently enable the 'conda' command, but they do NOT
put conda's base (root) environment on PATH.  To do so, run
    $ conda activate
in your terminal, or to put the base environment on PATH permanently, run
    $ echo "conda activate" >> ~/.bashrc
Previous to conda 4.4, the recommended way to activate conda was to modify PATH in
your ~/.bashrc file.  You should manually remove the line that looks like
    export PATH="/opt/conda/bin:$PATH"
^^^ The above line should NO LONGER be in your ~/.bashrc file! ^^^

Notebook order

1. Delayed
2. dataframe
3. schedulers
4. arrays?
5. distributed intro
6. distributed dataframes
7. distributed advanced (concurrent.futures)
8. ML?

cc @jcrist @martindurant