umayrh / sparktuner

Autotuner for Spark applications

Spark Tuner

Objectives:

• Tune Spark configuration parameters in a hands-off manner
• Learn from tuning experiences over time to:
  • Tune more efficiently over time,
  • Answer counterfactual questions about application performance, and
  • Suggest interventions to improve application performance (potentially even code changes or environment updates apart from configuration setting)

Setup, build, and usage

Setup

This package assumes that Apache Spark is installed, and the following environment variables have already been set: SPARK_HOME, and, optionally, HADOOP_CONF_DIR. See dev-README.md for details.

All Python dependencies are listed in:

• requirements.txt
• build.gradle
• setup.py

Build

• ./gradlew clean build to download and install all dependencies from scratch, and run tests.
• ./gradlew flake8 to lint for style issues.
• ./gradlew pytest to run tests.
• ./gradlew build -x getRequirements to install all dependencies (assumes they've already been downloaded).

Some interesting build artifacts are:

• build/deployable/bin/sparktuner
• build/deployable/bin/sparktuner.pex
• build/distributions/sparktuner-0.1.0.tar.gz
• build/wheel-cache/sparktuner-0.1.0-py2-none-any.whl

The Python virtual environment resides in build/venv, and can be activated using source build/venv/bin/activate and deactivated using deactivate.

Usage

To see usage information, ./build/deployable/bin/sparktuner --help

Sample commands:

• build/deployable/bin/sparktuner --no-dups --name sartre_spark_sortre --path ../../sparkScala/sort/build/libs/sort-0.1-all.jar --deploy_mode client --master "local[*]" --class com.umayrh.sort.Main --spark_parallelism "1,10" --program_conf "10000 /tmp/sparktuner_sort"

• build/deployable/bin/sparktuner --no-dups --name sartre_spark_sortre --path ../../sparkScala/sort/build/libs/sort-0.1-all.jar --deploy_mode client --master "local[*]" --class com.umayrh.sort.Main --executor_memory "50mb,1gb" --program_conf "10000 /tmp/sparktuner_sort"

• build/deployable/bin/sparktuner --no-dups --name sartre_spark_sortre --path ../../sparkScala/sort/build/libs/sort-0.1-all.jar --deploy_mode client --master "local[*]" --class com.umayrh.sort.Main --driver_memory "1GB,6GB" --program_conf "1000000 /tmp/sparktuner_sort"

sparktuner's Issues

Remove dependency on the package 'humanfriendly'

Is your feature request related to a problem? Please describe.
Generally, it's desirable to keep package dependencies as lean as possible. We use humanfriendly to parse a strings like "50b", "100kb", and "250mb" and convert them to integer values. We should be able to remove this dependency by implementing the parse_size() function in util.py (see here).

Describe the solution you'd like
An implementation of parse_size in util.py and corresponding tests in test_util.py.

Integrate CodeClimate's test coverage

Is your feature request related to a problem? Please describe.
We use Coveralls for analyzing test coverage but it would be nice to also enable CodeClimate.

Here are some useful links:

• https://github.com/codeclimate/test-reporter/blob/master/examples/python_examples.md
• https://docs.codeclimate.com/docs/travis-ci-test-coverage#section-travis-ci-single-test-suite-non-parallel-builds

Afterwards, you can add CodeClimate's test coverage badge by adding the following line to README.md:

[![Test Coverage](https://api.codeclimate.com/v1/badges/1b9ae406a6e8b922405a/test_coverage)](https://codeclimate.com/github/umayrh/sparktuner/test_coverage)

Improve code coverage to >85%

Is your feature request related to a problem? Please describe.
The current test coverage can easily be improved on. See this for Coverall's current coverage analysis.

Describe the solution you'd like
Coveralls reports an overall 85% percent coverage. It'd be nice if the coverage is also at least 50% for each module.