opendata-swiss / ckanext-switzerland-ng Goto Github PK

View Code? Open in Web Editor NEW

6.0 6.0 4.0 17.14 MB

License: GNU Affero General Public License v3.0

Python 56.19% HTML 35.91% CSS 0.60% JavaScript 7.30%

ckanext-switzerland-ng's People

Contributors

Stargazers

Watchers

Forkers

shubham-mahajan apteksdi mikanebu inderps

ckanext-switzerland-ng's Issues

Make extension CKAN 2.9.5 and Python 3 compatible

In order to make this extension to be compatible with CKAN 2.9.x and Python 3 some changes have to be done. Here is the list of changes:

ckanext-dcatapchharvest

https://github.com/opendata-swiss/ckanext-dcatapchharvest/blob/master/requirements.txt#L1 - change iribaker==0.1.2 to iribaker==0.2
https://github.com/opendata-swiss/ckanext-dcatapchharvest/blob/master/ckanext/dcatapchharvest/dcat_helpers.py#L3 - replace

from urlparse import urlparse

with

try:
    from urlparse import urlparse
except:
    from urllib.parse import urlparse

to be compatible with Py2 and Py3, both

ckanext-switzerland-ng

https://github.com/opendata-swiss/ckanext-switzerland-ng/blob/master/ckanext/switzerland/plugins.py#L3, replace from ckan.common import OrderedDict with from collections import OrderedDict
https://github.com/opendata-swiss/ckanext-switzerland-ng/blob/master/ckanext/switzerland/helpers/backend_helpers.py#L10, replace

from urlparse import urlparse

with

try:
    from urlparse import urlparse
except:
    from urllib.parse import urlparse

https://github.com/opendata-swiss/ckanext-switzerland-ng/blob/master/ckanext/switzerland/helpers/format_utils.py#L7 - the same as above
In https://github.com/opendata-swiss/ckanext-switzerland-ng/blob/master/ckanext/switzerland/helpers/format_utils.py#L26 replace iteritems() with items()
In addition, using webassets have to be implemented in templates

Here is the custom Dockerfile I'm using for the setup:

FROM debian:10

# Setting the locale   
RUN apt-get update
RUN DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y locales
RUN DEBIAN_FRONTEND=noninteractive locale-gen de_DE.UTF-8

# Internals, you probably don't need to change these
ENV APP_DIR=/srv/app
ENV SRC_DIR=/srv/app/src
ENV CKAN_INI=${APP_DIR}/production.ini
ENV PIP_SRC=${SRC_DIR}
ENV CKAN_STORAGE_PATH=/var/lib/ckan
ENV GIT_URL=https://github.com/ckan/ckan.git
# CKAN version to build
ENV GIT_BRANCH=ckan-2.9.4
# Customize these on the .env file if needed
ENV CKAN_SITE_URL=http://localhost:5000

WORKDIR ${APP_DIR}

# Install required system packages
RUN apt-get -q -y update \
    && DEBIAN_FRONTEND=noninteractive apt-get -q -y upgrade

RUN apt-get -q -y install \
    python3 \
    python3-dev \
    python3-pip \
    virtualenv \
    python3-wheel \
    libpq-dev \
    libxml2-dev \
    libxslt-dev \
    libgeos-dev \
    libssl-dev \
    libffi-dev \
    libsasl2-dev  \
    libldap2-dev \
    libssl-dev \
    postgresql-client \
    build-essential \
    git-core \
    wget \
    curl \
    zlib1g-dev \
    libpcre3 \
    libpcre3-dev \
    sudo \
    uwsgi \
    uwsgi-dev \
    uwsgi-plugin-python3 \
    uwsgi-extra

RUN apt-get -q -y install \
    python3-venv

RUN apt-get -q clean \
    && rm -rf /var/lib/apt/lists/*

# Create SRC_DIR
RUN mkdir -p ${SRC_DIR} && \
    # Install pip and supervisord
    curl -o ${SRC_DIR}/get-pip.py https://bootstrap.pypa.io/get-pip.py && \
    python3 ${SRC_DIR}/get-pip.py && \
    pip3 install supervisor && \
    mkdir /etc/supervisord.d && \
    #pip wheel --wheel-dir=/wheels uwsgi gevent && \
    rm -rf ${SRC_DIR}/get-pip.py

# set python to point to python3
RUN update-alternatives --install /usr/bin/python python /usr/bin/python3 10

COPY common/supervisord.conf /etc

# Define environment variables
ENV CKAN_HOME /usr/lib/ckan
ENV CKAN_VENV $CKAN_HOME/venv
ENV CKAN_CONFIG /etc/ckan

# Setup virtual environment for CKAN
RUN mkdir -p $CKAN_VENV $CKAN_CONFIG $CKAN_STORAGE_PATH && \
    python3 -m venv $CKAN_VENV && \
    ln -s $CKAN_VENV/bin/pip3 /usr/local/bin/ckan-pip3 &&\
    ln -s $CKAN_VENV/bin/ckan /usr/local/bin/ckan

ENV PATH=${CKAN_VENV}/bin:${PATH}

# Clone CKAN
RUN git clone -b ${GIT_BRANCH} ${GIT_URL} $CKAN_VENV/src/ckan
RUN cp -r $CKAN_VENV/src/ckan ${SRC_DIR}/ckan

# Create a local user and group to run the app
RUN groupadd -g 92 -r ckan
RUN adduser --uid 92 --home /srv/app --no-create-home --disabled-password --system --ingroup ckan --ingroup sudo ckan

# Install CKAN in virtual environment
RUN ckan-pip3 install -U pip && \
    ckan-pip3 install --upgrade --no-cache-dir -r $CKAN_VENV/src/ckan/requirement-setuptools.txt && \
    ckan-pip3 install --upgrade --no-cache-dir -r $CKAN_VENV/src/ckan/requirements.txt && \
    ckan-pip3 install -e $CKAN_VENV/src/ckan/ && \
    ln -s $CKAN_VENV/src/ckan/ckan/config/who.ini $CKAN_CONFIG/who.ini && \
    cp -v $CKAN_VENV/src/ckan/contrib/docker/ckan-entrypoint.sh /ckan-entrypoint.sh && \
    chmod +x /ckan-entrypoint.sh && \
    chown -R ckan:ckan $CKAN_HOME $CKAN_VENV $CKAN_CONFIG $CKAN_STORAGE_PATH

# Install ckanext-envvars
RUN ckan-pip3 install -e git+https://github.com/okfn/ckanext-envvars.git#egg=ckanext-envvars

RUN ckan generate config ${CKAN_INI} && \
    ckan config-tool ${CKAN_INI} "ckan.plugins = ${CKAN__PLUGINS}" && \
    ckan config-tool ${CKAN_INI} "ckan.site_url = ${CKAN__SITE_URL}" && \
    ckan config-tool ${CKAN_INI} -s logger_ckanext -e level=INFO

# Create local storage folder
RUN chown -R ckan:ckan $CKAN_STORAGE_PATH

# Create entrypoint directory for children image scripts
RUN mkdir /docker-entrypoint.d

RUN chown ckan -R ${APP_DIR}

# Set timezone
ENV TZ=UTC
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

#### Install Extensions ####

RUN ckan-pip3 install --no-cache-dir -e git+https://github.com/datopian/ckanext-auth.git#egg=ckanext-auth && \
    ckan-pip3 install -e git+https://github.com/ckan/ckanext-scheming.git#egg=ckanext-scheming && \
    ckan-pip3 install -r https://raw.githubusercontent.com/ckan/ckanext-fluent/master/requirements.txt && \
    ckan-pip3 install -e git+https://github.com/ckan/ckanext-fluent.git#egg=ckanext-fluent && \
    ckan-pip3 install -r https://raw.githubusercontent.com/ckan/ckanext-showcase/master/requirements.txt && \
    ckan-pip3 install -e git+https://github.com/ckan/ckanext-showcase.git#egg=ckanext-showcase && \
    ckan-pip3 install -e git+https://github.com/davidread/ckanext-hierarchy.git#egg=ckanext-hierarchy && \
    ckan-pip3 install -r https://raw.githubusercontent.com/ckan/ckanext-hierarchy/master/requirements.txt && \
    ckan-pip3 install -e git+https://github.com/ckan/ckanext-harvest.git#egg=ckanext-harvest && \
    ckan-pip3 install -r https://raw.githubusercontent.com/ckan/ckanext-harvest/master/requirements.txt && \
    ckan-pip3 install -e git+https://github.com/ckan/ckanext-dcat.git#egg=ckanext-dcat && \
    ckan-pip3 install -r https://raw.githubusercontent.com/ckan/ckanext-dcat/master/requirements.txt && \
    # ckan-pip3 install -e git+https://github.com/opendata-swiss/ckanext-dcatapchharvest.git#egg=ckanext-dcataapchharvest && \
    # ckan-pip3 install -r https://raw.githubusercontent.com/opendata-swiss/ckanext-dcatapchharvest/master/requirements.txt && \
    ckan-pip3 install -e git+https://github.com/datopian/ckanext-dataexplorer-react.git@webassets#egg=ckanext-dataexplorer-react && \
    ckan-pip3 install -r https://raw.githubusercontent.com/datopian/ckanext-s3filestore/ckan-2.9/requirements.txt && \
    ckan-pip3 install -e git+https://github.com/datopian/[email protected]#egg=ckanext-s3filestore && \
    ckan-pip3 install -e git+https://github.com/keitaroinc/ckanext-issues@ef694bdfbc223f833a9ba114a8a9ac50cc4520d1#egg=ckanext-issues
# Install ckanext-pages
RUN ckan-pip3 install -e git+https://github.com/ckan/[email protected]#egg=ckanext-pages && \
    ckan-pip3 install --no-cache-dir -r ${SRC_DIR}/ckanext-pages/requirements.txt
# Install ckanext-pdfview
RUN ckan-pip3 install ckanext-pdfview

# Install Xloader
RUN ckan-pip3 install ckanext-xloader && \
    ckan-pip3 install -r https://raw.githubusercontent.com/ckan/ckanext-xloader/master/requirements.txt && \
    ckan-pip3 install -U requests[security]

# RUN ckan-pip3 install -r https://raw.githubusercontent.com/opendata-swiss/ckanext-switzerland-ng/master/requirements.txt && \
#     ckan-pip3 install -e git+https://github.com/opendata-swiss/ckanext-switzerland-ng.git#egg=ckanext-switzerland

# Install ckanext-switzerland for local development
COPY src/ckanext-switzerland /srv/app/src/ckanext-switzerland
RUN pip install -r file:///srv/app/src/ckanext-switzerland/requirements.txt && \
    pip install -e file:///srv/app/src/ckanext-switzerland#egg=ckanext-switzerland

##### Local development #####
# Clone the extensions in ckan/src folder, before you start your dev work
# Install ckanext-dcataapchharvest for local development
COPY src/ckanext-dcataapchharvest /srv/app/src/ckanext-dcataapchharvest
RUN pip install -r file:///srv/app/src/ckanext-dcataapchharvest/requirements.txt && \
    pip install -e file:///srv/app/src/ckanext-dcataapchharvest#egg=ckanext-dcataapchharvest

COPY setup/prerun.py ${CKAN_VENV}
COPY setup/supervisor.worker.conf /etc/supervisord.d/worker.conf
COPY setup/start_ckan.sh ${APP_DIR}
ADD https://raw.githubusercontent.com/ckan/ckan/${GIT_BRANCH}/wsgi.py ${CKAN_VENV}
RUN cp $CKAN_VENV/src/ckan/ckan/config/who.ini ${APP_DIR}/who.ini

ENV CKAN__PLUGINS envvars image_view text_view webpage_view pdf_view hierarchy_display hierarchy_form datastore xloader resource_proxy auth dataexplorer_view dataexplorer_table_view dataexplorer_chart_view dataexplorer_map_view fluent dcat_ch_rdf_harvester ogdch_dcat ogdch ogdch_pkg ogdch_res ogdch_group ogdch_org ogdch_showcase ogdch_archive

RUN ckan config-tool ${CKAN_INI} "ckan.plugins = ${CKAN__PLUGINS}" && \
    ckan config-tool ${CKAN_INI} "ckan.site_url = ${CKAN__SITE_URL}"

# Apply Patches
# RUN if [ -d "$APP_DIR/patches/ckan" ] ; then rm -rf $APP_DIR/patches/ckan ; fi
# COPY patches ${APP_DIR}/patches
# RUN for d in $APP_DIR/patches/*; do \
#     for f in `ls $d/*.patch | sort -g`; do \
#     cd $SRC_DIR/`basename "$d"` && echo "$0: Applying patch $f to $SRC_DIR/`basename $d`"; patch -p1 < "$f" ; \
#     done ; \
#     done

EXPOSE 5000

HEALTHCHECK --interval=60s --timeout=5s --retries=5 CMD curl --fail http://localhost:5000/api/3/action/status_show || exit 1

RUN chmod +x /srv/app/start_ckan.sh

# RUN ckan config-tool ${CKAN_INI} "ckanext.dcat.rdf.profiles = euro_dcat_ap dcatap_de" && \
#     ckan config-tool ${CKAN_INI} "ckanext.dcat.clean_tags = true" && \
#     ckan config-tool ${CKAN_INI} "ckan.harvest.log_level = info"

USER ckan

CMD ["/srv/app/start_ckan.sh"]

What CKAN and Python version required for this theme?

Hi,

We are trying to implement CKAN with ckanext-switzerland-ng theme.

But on CKAN 2.9.5 with python 3.6, we are getting error during install.

Can you please tell which version of CKAN + Python is required to use ckanext-switzerland-ng theme.

Thanks so much for your fast help!

Installation of ckanext-switzerland not working

Hello,

I recently followed the readme on the installation of the ckanext-switzerland extension, and the url that is suggested to be cloned (https://github.com/ogdch/ckanext-switzerland.git )is throwing an error message:

fatal: could not create work tree dir 'ckanext-switzerland': Permission denied

Thank you for your help

Archived repo

Please add a link to this repo into the GitHub Settings > web link, and/or README of the old one, so people (like me :) don't get confused about why it's been left hanging: https://github.com/opendata-swiss/ckanext-switzerland
Thanks!

Preview is not working

The preview does not render inside of a dataset resource, instead an error message is shown. Seems to be a cross-site-frame-protection issue. Tested with Chrome and Firefox on this CSV resource being served from ckan.opendata.swiss. Opening the frame in a new window works fine. We might in any case want to add a "Open in full screen" link underneath the frame, as in for example, dribdat projects (via Whitepaper).

Changing Language in GUI not possible

Hello together,
I've built a CKAN docker image with CKAN 2.8.6 and python 2.7. I have also installed this extension with all his dependencies.
everything is working, except that the whole GUI is only in english available and i cannot change the language (there is only english in the dropdown and when manually setting/adjusting the path in the URL e.g. /de/datasets it crashes and in the logs python says: UndefinedError: 'dict object' has no attribute 'de').

Looks for me, that ckan doesn't load the languages in his "config".
i have also set the ckan.localeXXX configs and also compiled the .pot and .po files to .mo files

any suggestion what else i could try ?
Do you maybe have a Dockerfile for the OpenData Swiss CKAN Instance or snippets how to correctly build the CKAN Image with this extension ?

P.S. the creation of datasets with the 4 different Languages (de,fr,it,en) works. Only changing the GUI language doesn't work.

Thank you for your help.

Cannot install pandas requirement on CKAN 2.9.5

We are using CKAN 2.9.5 which uses Python 3.8.10. The base docker-image is from keitaroinc.

We are doing a multi-stage build, where in the first stage a Ubuntu 20.04 is used from keitaroinc.
When pandas is getting installed (version in requirements.txt is pandas==0.24.2), it doesn't find any pre-built wheel for Python 3.8.10 and therefore a version gets build locally. This is a slow build and after some time following error occures :

         17 | #warning "Using deprecated NumPy API, disable it with " \
            |  ^~~~~~~
      pandas/_libs/src/ujson/python/objToJSON.c: In function â??initObjToJSONâ??:
      pandas/_libs/src/ujson/python/objToJSON.c:194:12: error: â??NUMPY_IMPORT_ARRAY_RETVALâ?? undeclared (first use in this function)
        194 |     return NUMPY_IMPORT_ARRAY_RETVAL;
            |            ^~~~~~~~~~~~~~~~~~~~~~~~~
      pandas/_libs/src/ujson/python/objToJSON.c:194:12: note: each undeclared identifier is reported only once for each function it app
ears in
      pandas/_libs/src/ujson/python/objToJSON.c:195:1: warning: control reaches end of non-void function [-Wreturn-type]
        195 | }
            | ^
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pandas

It cannot build pandas : error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1

We would like to test this extension on a newer version of ckan 2.9+ out.
Would it be possible, that you can upgrade the pandas dependency, so that a pandas will be used, where allready a pre-built wheel exists ? (in the hopes that not building it locally this would solve the problem).

P.S.
this is the dockerfile with the packages and extensions that get first installed in the first stage of the docker build :

FROM keitaro/ckan:2.9.5-focal as extbuild

USER root

RUN apt-get update && \
    apt-get -qq -y install \
    wget \
    build-essential \
    python3-dev \
    gcc \
    g++ \
    libffi-dev \
    openssl \
    libssl-dev \
    rustc \
    cargo

RUN pip install --upgrade pip
RUN pip install --upgrade setuptools

#ckanext-scheming
###
ENV SCHEMING_GIT_URL=https://github.com/ckan/ckanext-scheming.git
ENV SCHEMING_RAW_GIT_URL=https://raw.githubusercontent.com/ckan/ckanext-scheming
ENV SCHEMING_GIT_BRANCH=release-2.1.0

RUN pip wheel --wheel-dir=/wheels git+${SCHEMING_GIT_URL}@${SCHEMING_GIT_BRANCH}#egg=ckanext-scheming
#no dependencies/requirements.txt needed
###

#ckanext-fluent
###
ENV FLUENT_GIT_URL=https://github.com/ckan/ckanext-fluent.git
ENV FLUENT_RAW_GIT_URL=https://raw.githubusercontent.com/ckan/ckanext-fluent
ENV FLUENT_GIT_BRANCH=c735d32eddb34be7fae3177e181d9938d7fc303d

RUN pip wheel --wheel-dir=/wheels git+${FLUENT_GIT_URL}@${FLUENT_GIT_BRANCH}#egg=ckanext-fluent
RUN pip wheel --wheel-dir=/wheels -r ${FLUENT_RAW_GIT_URL}/${FLUENT_GIT_BRANCH}/requirements.txt
RUN curl -o /wheels/fluent.txt ${FLUENT_RAW_GIT_URL}/${FLUENT_GIT_BRANCH}/requirements.txt
###

#ckanext-switzerland
#https://github.com/opendata-swiss/ckanext-switzerland-ng/tree/8ba558d19a74b0029af5cd8bb02bbbeddb5a586a
ENV SWITZERLAND_GIT_URL=https://github.com/opendata-swiss/ckanext-switzerland-ng.git
ENV SWITZERLAND_RAW_GIT_URL=https://raw.githubusercontent.com/opendata-swiss/ckanext-switzerland-ng
ENV SWITZERLAND_GIT_BRANCH=8ba558d19a74b0029af5cd8bb02bbbeddb5a586a

##### => build crashes here on installing switzerland extensions on installing/building pandas
RUN pip wheel --wheel-dir=/wheels git+${SWITZERLAND_GIT_URL}@${SWITZERLAND_GIT_BRANCH}#egg=ckanext-switzerland
RUN pip wheel --wheel-dir=/wheels -r ${SWITZERLAND_RAW_GIT_URL}/${SWITZERLAND_GIT_BRANCH}/requirements.txt
RUN curl -o /wheels/switzerland.txt ${SWITZERLAND_RAW_GIT_URL}/${SWITZERLAND_GIT_BRANCH}/requirements.txt
####

USER ckan

....
#second stage of docker build 
#copy over build wheels and install them

Meta header tags are missing

The HTML header of the page is missing descriptive <meta> tags, or support for OpenGraph etc. that allows rich embedding into blogs and forums. This was working in the past, and I suspect it was somewhere at this point that the tags were still being injected. Please bring back the META love.

And yes, I realize that rich embedding is provided by ckan-embed and that, as one might say, "Vor der eigenen Türe wischen" - the component could use a refresh and more usage as well.

Screenshot of the View Source from a random dataset, showing a bare HTML header:

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.