rsuderman / sharktank

SHARK Inference Modeling and Serving

SHARK Modeling and Serving Libraries

WARNING: This is an early preview that is in progress. It is not ready for general use.

Development Getting Started

Use this as a guide to get started developing the project using pinned, pre-release dependencies. You are welcome to deviate as you see fit, but these canonical directions mirror what the CI does.

Setup a venv

We recommend setting up a virtual environment (venv). The project is configured to ignore .venv directories, and editors like VSCode pick them up by default.

python -m venv --prompt sharktank .venv
source .venv/bin/activate

Install PyTorch for Your System

If no explicit action is taken, the default PyTorch version will be installed. This will give you a current CUDA-based version. Install a different variant by doing so explicitly first:

CPU:

pip install -r pytorch-cpu-requirements.txt

ROCM:

pip install -r pytorch-rocm-requirements.txt

Install Development Packages

This assumes you have SHARK-Turbine checked out adjacent (note that for the moment we rely on pre-release versions, so installation is a bit harder).

# Clone and install editable SHARK-Turbine dep in deps/
pip install -f https://iree.dev/pip-release-links.html --src deps \
  -e "git+https://github.com/nod-ai/SHARK-Turbine.git#egg=SHARK-Turbine&subdirectory=core"

# Install editable local projects.
pip install -r requirements -e sharktank/ shortfin/

Running Tests

pytest sharktank
pytest shortfin

Optional: Pre-commits and developer settings

This project is set up to use the pre-commit tooling. To install it in your local repo, run: pre-commit install. After this point, when making commits locally, hooks will run. See https://pre-commit.com/