internetarchive / arklet

ARK minter, binder, resolver

Arklet - A basic ARK resolver

What is an ARK?

See https://arks.org/

What is Arklet?

Arklet is a Python Django application for minting, binding, and resolving ARKs. It is intended to follow best practices set out by https://arks.org/.

Technical design notes:

• Django is the only required dependency.
• Supports each Django and Python version that is itself supported by the maintainers (Python 3.7-3.10, Django 3.2.x-4.0.x).
  • Tests are run against the upcoming versions of Python and Django as well.
• This repo can be run as a standalone service
• ...or the ark package can be installed as a reusable app in other Django projects.
  • If using the included arklet/settings.py file django-environ is also required.
• Arklet is database agnostic.

Arklet is developed with poetry, pytest, black, tox, and more.

Running

Locally with Postgres

Create the default .env file in the project's root directory

# /!\ Set your own secret key /!\
ARKLET_DJANGO_SECRET_KEY=[YOUR_SECRET]

# For local development, set to True
ARKLET_DEBUG=True

The following steps walk through running Postgres, installing with poetry, and starting the app. You can omit any of the extras listed in the poetry install step if they are not used in your deployment. The included arklet/settings.py file does require environ. You can skip installing the development dependencies by passing --no-dev to poetry. Django is the only required dependency.

cd path/to/project
mkdir postgres-data
docker run --name arklet-postgres -v postgres-data:/var/lib/postgresql/data \
    -p 5432:5432 \
    -e POSTGRES_USER=arklet -e POSTGRES_PASSWORD=arklet \
    -d postgres
poetry install --extras "postgres sentry environ"
poetry run python manage.py migrate
poetry run python manage.py createsuperuser
poetry run python manage.py runserver

Separate dockers

Using docker, we can use a this provided config file.

See above for running PostgreSQL, and run the Arklet docker as follows (in bash):

docker build \
    --target dev \
    -t "arklet" -f ./Dockerfile . \
    --build-arg ENV=DEV \
&& docker run --rm -it \
    -p 8000:8000 \
    --env-file=./docker/env.docker.local \
    -e ARKLETDEBUG="true" \
    --name arklet \
    -v `pwd`/ark:/app/ark \
    -v `pwd`/ark_import:/app/ark_import \
    -v `pwd`/arklet:/app/arklet \
    arklet

With docker-compose

Using the provided docker-compose.yml with default settings in the docker configuration directory :

docker-compose up

By default, the folders ark, ark_import and arklet are mounted in the container. Should you wish to attach a console to the arklet container (needed to create the django superuser) :

# In another shell
docker exec -it arklet_django /bin/bash
# You're now in the docker container
./manage.py createsuperuser

First steps

Create your first NAAN, Key, and Shoulder in the admin: 127.0.0.1:8000/admin

And by the way, you now host a working ARK resolver! You can already try the following ones :

• http://127.0.0.1:8000/ark:/13960/t5n960f7n
• http://127.0.0.1:8000/ark:/67375/C0X-SPWFRSGR-N
• http://127.0.0.1:8000/ark:/12148/bpt6k65358454
• ...

Happy minting, binding, and resolving!

Configuration Options

See arklet/settings.py for the full list of options to put in your config file.

Deploying

With docker

Using the provided Dockerfile (is you wish to set a build target, use prod, but being the default target you can skip this), provide the following values in your environment:

• ARKLET_DJANGO_SECRET_KEY=[YOUR_SECRET]
• ARKLET_DEBUG=False
• ARKLET_HOST=0.0.0.0
• ARKLET_PORT=[Port of choice]
• ARKLET_POSTGRES_NAME=[DB NAME]
• ARKLET_POSTGRES_USER=[DB USER]
• ARKLET_POSTGRES_PASSWORD=[DB PASS]
• ARKLET_POSTGRES_HOST=[DB HOST]
• ARKLET_POSTGRES_PORT=[DB PORT]

arklet's Issues

add #arks repo topic?

https://github.com/topics/arks

to improve SNR for archival resource key stuff versus some blockchain stuff on github.

Auth Keys should be encrypted like other passwords

Storing the Key.key as plaintext is a bad idea. There are implementations of token auth in Django Rest Framework, but one of the design goals for arklet is to have only standard lib and Django as production dependencies. We want it to be really inexpensive to stay with the latest versions of Python and Django.

Maybe we could use the built-in Django password functionality? https://docs.djangoproject.com/en/4.0/topics/auth/passwords/#module-django.contrib.auth.hashers

I don't want to use a User model for API authentication because it makes fleet secret management a little harder (have to coordinate password changes with the secrets getting updated across the fleet). The access key model works a little bit better for secret rotation. We can create a new access key, start distributing it, wait for everything to update, and deactivate the old key.

The Key model should not use the key UUIDField as the primary key on the table. We should add a regular integer primary key, and then create a separate charfield to hold the hash.

https://docs.djangoproject.com/en/4.0/topics/auth/passwords/#django.contrib.auth.hashers.check_password
https://docs.djangoproject.com/en/4.0/topics/auth/passwords/#django.contrib.auth.hashers.make_password

check_password can be used against the authorization header
make_password can be used with a UUID4 input to create the hash text to store in the Key.key field.

This would also necessitate a new view for one-time display of newly created keys.

Packaging as reusable Django app

Arklet was made as a standalone service, but it might make sense to package it as a reusable Django app too. We'll be creating some "about" content for Internet Archive. This content doesn't belong in the public repo; it wouldn't be relevant to other groups.

I'm considering creating a separate repo for Internet Archive's deployment which pulls in a package built from this repo as a reusable app. Even if we do that, I think it's useful to keep this repo as something that runs as-is. Developers should have a choice between running this repo as is, or adding the ark app to their new or existing Django site.

https://docs.djangoproject.com/en/4.0/intro/reusable-apps/

Automated testing framework

Hello,

It would be great to add a testing framework to the project. Beyond Django unittests I suggest a behaviour-driven (BDD) component : Behave. There's an available Django integration, and has as main advantage that tests can be easily written, understood, and enhances code-reuse (which is a way to avoid test code rot).

What a simple test might look like :

Feature : Minting ark identifiers
  Scenario: Simple minting request
    Given we have a running arklet service
      When we mint an id for url "http://example.com/this_test_is_ok"
        Then we obtain an ARK id
        And it forwards to "http://example.com/this_test_is_ok"

(see behave documentation for more information)

If the suggestion is validated, I can work on a PR that implements testing (be it behave or unittests).

Poetry & tooling

Having used pip, poetry and pipenv & even contributed to some, I'd like to suggest to move to Poetry for packaging & version management.

1. It uses pyproject.toml, as recommended by PEP518, along with it's own lockfile (poetry.lock) which doesn't get in the way
2. The tool evolves pretty well, simple to use, and appreciated by the community. Having used it for the pas 2 years, I have yet to find a problem with it
3. Easy local venv management
4. Integrates well with docker : RUN poetry config virtualenvs.create false && poetry install -no-dev --no-interaction --no-ansi

Along with poetry can be added code cleanup, formatting & linting tools : isort, flake8, pylint, black, ... we just have to agree on the basic set. Most use pyproject.toml for settings, except for flake8 which needs his own file (or pyproject.toml with some monkey patching).

Is Poetry OK ? And which tools could be added to the repo (I'd take all of them, but that may be overkill) ?