Alembic error when adding this extension to ckan-docker about ckanext-datajson

[dcrendon]

Thank you for the response @FuhuXia! This clears up a lot, it makes sense that installing duplicate/older requirements could cause issues. Currently I'm able to get away with not installing the requirements for the datajson extension, I ran a test harvest on https://open.gsa.gov/data.json and everything seems to have ran correctly.

[dcrendon]

I was able to get it working by not installing the requirements after installing the extension, not sure why that fixed it though.

Updated dokcerfile.dev:

FROM ckan/ckan-dev:2.10

### Harvester ###
RUN pip3 install -e 'git+https://github.com/ckan/ckanext-harvest.git@master#egg=ckanext-harvest'
RUN pip3 install -r ${APP_DIR}/src/ckanext-harvest/pip-requirements.txt
# will also require gather_consumer and fetch_consumer processes running (please see https://github.com/ckan/ckanext-harvest)

## DCAT ###
RUN  pip3 install -e 'git+https://github.com/ckan/ckanext-dcat.git#egg=ckanext-dcat'
RUN pip3 install -r ${APP_DIR}/src/ckanext-dcat/requirements.txt

### S3FileStore ###
RUN  pip3 install -e 'git+https://github.com/keitaroinc/ckanext-s3filestore#egg=ckanext-s3filestore'
COPY plugins/s3filestore/uploader.py ${APP_DIR}/src/ckanext-s3filestore/ckanext/s3filestore/uploader.py
COPY plugins/s3filestore/click_commands.py ${APP_DIR}/src/ckanext-s3filestore/ckanext/s3filestore/click_commands.py
RUN pip3 install -r ${APP_DIR}/src/ckanext-s3filestore/requirements.txt

### DataJson ###
RUN pip3 install -e 'git+https://github.com/GSA/ckanext-datajson#egg=ckanext-datajson'

# Clone the extension(s) your are writing for your own project in the `src` folder
# to get them mounted in this image at runtime

# Copy custom initialization scripts
COPY docker-entrypoint.d/* /docker-entrypoint.d/

# Apply any patches needed to CKAN core or any of the built extensions (not the
# runtime mounted ones)
COPY patches ${APP_DIR}/patches

RUN for d in $APP_DIR/patches/*; do \
        if [ -d $d ]; then \
            for f in `ls $d/*.patch | sort -g`; do \
                cd $SRC_DIR/`basename "$d"` && echo "$0: Applying patch $f to $SRC_DIR/`basename $d`"; patch -p1 < "$f" ; \
            done ; \
        fi ; \
    done

[FuhuXia]

@dcrendon

• Installing packages/versions defined in ckanext-datajson/requirements.txt is recommended. But since those packages are pretty general, most likely there are installed from other extensions or ckan core, so you might be able to get away without it.
• Installing ckanext-datajson/dev-requirements.txt is not needed, unless you are doing datajson extension development and need to test your code change.
• cd into ckanext-datajson/ and run setup is not needed since it is already installed by previous command pip3 install.
• Not sure about the Alembic error message. Using your Dockerfile steps I could not replicated it. If you can remove docker/db containers/networks/volumes before each docker run, it might help to stop Alembic glitches.
• Do pay attention to the first line in the https://github.com/GSA/ckanext-datajson/blob/main/requirements.txt#L1-L2. I just noticed it defines a old version of harvest extention. I will have it updated to the latest stable release version of harvest, or you can grab the change from this PR to see if it helps to eliminate the Alembic error.