kprestel / pytest-docker-db

A pytest plugin to use docker databases when running pytest

pytest-docker-db

A plugin to use docker databases for pytests

What is this?

This is a pytest plugin that hooks into the host's :code:docker daemon to create and teardown containers within the pytest life cycle.

Features

• Use custom :code:Dockerfiles or base images from Dockerhub
• Use volumes to persist test data in between testing sessions
• Specify what port you want the container to be listening on the host machine and the container
• Optionally persist the container between test sessions
• Name the containers created

Requirements

• docker-py>=6
• pytest>=7
• docker

Installation

You can install pytest-docker-db via pip from PyPI

pip install pytest-docker-db

Configuration

• db-volume-args

  • Provide the "-v" arguments that you would pass to the "docker run" command if you were using the cli. If you need multiple volumes mounted separate them with commas.
  • The basic syntax is :code:/host/vol/path:/path/in/container:rw. If using a named volume, the syntax would be :code:vol-name:/path/in/container:rw 'For more information please visit the docker documentation'

• db-image

  • Specify the name of the image to use as the DB.

    • Must be in the form of "image_name":"tag".

• db-name

  • Specify the name of the container. If this is not specified a random container name will be used with the prefix docker-db

• db-host-port

  • Specify the port that the db should be listening to on the host machine.

• db-port

  • Specify the port that the db should be listening to in the container. This is often the default port used by your database.

• db-persist-container

  • If set, the container created will not be torn down after the test suite has ran. By default any image created will be torn down and removed after the test suite has finished.

• db-dockerfile

  • Specify the name of the Dockerfile within the directory set as the :code:db-build-context

    • If a path is given as well as an image name, the Dockerfile will be used.

• db-docker-context

  • The directory to use as the docker build context.

• db-docker-env-vars

  • A comma separated list of environment variables to pass to docker run
    • --db-docker-env-vars=FOO=BAR,PASSWORD=BAZ

Usage

Plugin contains one fixture: docker_db - it's a session scoped fixture that returns a docker-py container object. For almost all use cases the user will not care about this object.

The recommended way to use this fixture is to create an :code:autouse=True fixture in your conftest.py file to automatically invoke the setup of the containers.

    @pytest.fixture(scope='session', autouse=True)
    def my_docker_db(docker_db):
        pass

Can be configured via the :code:pytest CLI or the :code:pytest.ini file.

pytest.ini:

    [pytest]
    db-volume-args=/home/kp/vol:/var/lib/postgresql/data:rw
    db-image=postgres:latest
    db-name=test-postgres
    db-port=5432
    db-host-port=5434

pytest CLI using a Dockerhub

    pytest --db-image=postgrest:latest --db-name=test-postgres --db-port=5432 --db-host-port=5434

pytest CLI using a custom image

    pytest --db-dockerfile=Dockerfile --db-name=test-postgres --db-port=5432 --db-host-port=5434

pytest CLI using a custom image and passing environment variables to it

    pytest --db-dockerfile=Dockerfile --db-name=test-postgres --db-port=5432 --db-host-port=5434 --db-docker-env-vars=POSTGRES_PASSWORD=FOO,POSTGRES_USER=BAR

Contributing

Contributions are very welcome. Tests can be run with tox, please ensure the coverage at least stays the same before you submit a pull request.

License

Distributed under the terms of the MIT license, "pytest-docker-db" is free and open source software

Issues

If you encounter any problems, please file an issue along with a detailed description.

MIT: http://opensource.org/licenses/MIT file an issue: https://github.com/kprestel/pytest-docker-db/issues pytest: https://github.com/pytest-dev/pytest pip: https://pypi.python.org/pypi/pip/ PyPI: https://pypi.python.org/pypi docker-py container: http://docker-py.readthedocs.io/en/stable/containers.html Dockerhub: https://hub.docker.com/ docker documentation: https://docs.docker.com/storage/volumes/#start-a-container-with-a-volume

pytest-docker-db's Issues

Support for Multiple Containers

Hi, thanks for this library. Looks super useful for me. My api relies on two db (postgres and neo4j). Do you plan to support multiple db creation? I will take a look at the source and see if it is something I could assist with.

Thanks,
Zach

Testing Issue

When running unit tests, a couple error out due to the tests using this mount location:
/home/kp/vol which doesn't exist on my computer. I switched these to /tmp and they worked fine.

Plugin seems to not wait until the db is ready

Hi,

First of all, I want to thank you for the effort that you put into the plugin. I am pretty new to Python and Django,, and when trying to make the plugin work in different ways but I am always facing a road block. I created another issue but this one is with other configs.

setup.cfg:

addopts = --reuse-db --import-mode=importlib -vv -x --lf -p no:warnings -n1 --cov --cov-config=./setup.cfg
db-volume-args=/tmp/docker/vol:/var/lib/postgresql/data:rw
db-image = postgres:12.8-alpine
db-name = api_tests_db
db-port = 5432
db-host-port = 30432
db-persist-container = false
db-docker-env-vars = POSTGRES_USER=some,POSTGRES_PASSWORD=some,POSTGRES_DB=test_gringaming

conftest.py (needs some clean up):

import pytest
import os
from rest_framework.test import APIClient
from django.utils.translation import activate
from django.db import connections

pytest_plugins = ["pytest_docker_db"]

os.environ['DJANGO_SETTINGS_MODULE'] = 'grin_app.settings.local'

database = {
    'ENGINE': 'django.db.backends.postgresql',
    'NAME': 'test_gringaming',
    'USER': 'some',
    'PASSWORD': 'some',
    'HOST': 'localhost',
    'PORT': '30432',
}


def pytest_configure():
    from django.conf import settings

    #settings.DATABASES['default'] = database
    

def clear_caches():
    """Invalidate the connection caches."""
    import threading
    connections._connections = threading.local()

    print(connections.databases)

    # this will clear the cached property
    del connections.databases
    

#Make a fixture for DRF's APIClient object and name it like api_client to start testing directly from the endpoint:
# conftest.py
@pytest.fixture
def api_client():
    return APIClient

@pytest.fixture(autouse=True)
def use_dummy_cache_backend(settings):
    settings.CACHES = {
        "default": {
            "BACKEND": "django.core.cache.backends.dummy.DummyCache",
        }
    }

    #settings.DATABASES['default'] = database
    

@pytest.fixture(autouse=True)
def set_default_language():
    activate('en')

from django.test.utils import override_settings

@pytest.fixture(scope='session')
@override_settings()
def django_db_modify_db_settings():
    pass
    #print('@@@@@@@ django_db_modify_db_settings')
    #clear_caches()

    #from django.conf import settings

    #settings.DATABASES['default'] = database
    

    #clear_caches()

@pytest.fixture(scope='session')
def django_db_setup(
    #request,
    #django_db_setup,
    #django_test_environment,
    #django_db_blocker,
    #django_db_use_migrations,
    #django_db_keepdb,
    #django_db_createdb,
    django_db_modify_db_settings,
):
    print('@@@@@@@ django_db_setup') 
    clear_caches()

    from django.conf import settings

    settings.DATABASES['default'] = database
    

    clear_caches()

@pytest.fixture(scope='session', autouse=True)
def my_docker_db(docker_db):
    from django.conf import settings

    settings.DATABASES['default'] = database
    

And with this configuration I got two errors, one the very first time that I ran the test with the configs and the last one happens after the first run:

django.db.utils.OperationalError: FATAL: the database system is starting up

django.db.utils.OperationalError: FATAL: no pg_hba.conf entry for host "172.17.0.1", user "some", database "test_gringaming", SSL off

Tried to see if was a security thing so added 'OPTIONS': {'sslmode': 'require'}, to the default database config but it turns into this error:

server does not support SSL, but SSL was required

Any ideas on what could be going wrong?

Thanks in advance

Docker container conflict when db-persist-container = true

Hi,

Thank you for your contribution to the community of pytest. I am new to Python world in general. I am facing different issues with the plugin bu the one that I would like to mention here is when I use: db-persist-container = true

This is what I have in setup.cfg:

addopts = --reuse-db --import-mode=importlib -vv -x --lf -p no:warnings -n1 --cov --cov-config=./setup.cfg
db-volume-args=/tmp/docker/vol:/var/lib/postgresql/data:rw
db-image = postgres:12.8-alpine
db-name = api_tests_db
db-port = 5432
db-host-port = 30432
db-persist-container = true
db-docker-env-vars = POSTGRES_USER=some,POSTGRES_PASSWORD=some,POSTGRES_DB=test_gringaming

When I run the test I get:

Failed: Unable to start container: postgres:12.8-alpine, Error: 409 Client Error for http+docker://localhost/v1.41/containers/create?name=api_tests_db: Conflict ("Conflict. The container name "/api_tests_db" is already in use by container "2645deedf391fe1e995eb25023ddfd8992e6dc7c7db93a487a19b13474e24ec1". You have to remove (or rename) that container to be able to reuse that name.")

conftest.py (needs some clean up):

import pytest
import os
from rest_framework.test import APIClient
from django.utils.translation import activate
from django.db import connections

pytest_plugins = ["pytest_docker_db"]

os.environ['DJANGO_SETTINGS_MODULE'] = 'grin_app.settings.local'

database = {
    'ENGINE': 'django.db.backends.postgresql',
    'NAME': 'test_gringaming',
    'USER': 'some',
    'PASSWORD': 'some',
    'HOST': 'localhost',
    'PORT': '30432',
}


def pytest_configure():
    from django.conf import settings

    #settings.DATABASES['default'] = database
    

def clear_caches():
    """Invalidate the connection caches."""
    import threading
    connections._connections = threading.local()

    print(connections.databases)

    # this will clear the cached property
    del connections.databases
    

#Make a fixture for DRF's APIClient object and name it like api_client to start testing directly from the endpoint:
# conftest.py
@pytest.fixture
def api_client():
    return APIClient

@pytest.fixture(autouse=True)
def use_dummy_cache_backend(settings):
    settings.CACHES = {
        "default": {
            "BACKEND": "django.core.cache.backends.dummy.DummyCache",
        }
    }

    #settings.DATABASES['default'] = database
    

@pytest.fixture(autouse=True)
def set_default_language():
    activate('en')

from django.test.utils import override_settings

@pytest.fixture(scope='session')
@override_settings()
def django_db_modify_db_settings():
    pass
    #print('@@@@@@@ django_db_modify_db_settings')
    #clear_caches()

    #from django.conf import settings

    #settings.DATABASES['default'] = database
    

    #clear_caches()

@pytest.fixture(scope='session')
def django_db_setup(
    #request,
    #django_db_setup,
    #django_test_environment,
    #django_db_blocker,
    #django_db_use_migrations,
    #django_db_keepdb,
    #django_db_createdb,
    django_db_modify_db_settings,
):
    print('@@@@@@@ django_db_setup') 
    clear_caches()

    from django.conf import settings

    settings.DATABASES['default'] = database
    

    clear_caches()

@pytest.fixture(scope='session', autouse=True)
def my_docker_db(docker_db):
    from django.conf import settings

    settings.DATABASES['default'] = database

What I was expecting is the plugin to keep track of the container id that was created before or get it by name, and just use that one instead of creating a new one with the same container name.

find_unused_port needs to return the right value from getsockname() call

In:

    @staticmethod
    def _find_unused_port():
        with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
            s.bind(("", 0))
            return s.getsockname()[0]

The tuple returned by getsockname() contains the following:
('0.0.0.0', 51200)
And so this function should be returning the second element rather than the first.

Add pytest-plugin topic on github

You may want to make your project discoverable by adding pytest-plugin topic to this repository. See https://github.com/topics/pytest-plugin

How to pass the postgres user and password ?

I want the postgres database created with a user and password. How would I do that without a custom docker image ?