plone / plone-backend

Plone backend Docker images using Python 3 and pip.

plone/plone-backend

Plone backend Docker images using Python 3 and pip.

Note: These are the official images for the Plone 6 release, together with plone-frontend. These images are not Buildout based!

Tags

Supported tags and respective Dockerfile links

Plone Version	Tags	Dockerfile
6	6.0.11.1, 6.0, 6, latest	(6.0.x/Dockerfile)
6 (nightly)	nightly	(Dockerfile.nightly)

Unsupported tags

Note: These images for Plone 5 are not officially supported by the Plone community.

Plone Version	Tags	Dockerfile
5.2	5, 5.2, 5.2.14	(5.2.x/Dockerfile)

See also the official Buildout-based Plone 5 images.

Usage

Please refer to the Official Plone Documentation for further documentation and examples.

Contribute

• Issue Tracker
• Source Code
• Documentation

Please DO NOT commit to version branches directly. Even for the smallest and most trivial fix. ALWAYS open a pull request and ask somebody else to merge your code. NEVER merge it yourself.

Credits

This project is supported by:

License

The project is licensed under GPLv2.

plone-backend's Issues

Not possible to set multiple ADDONS in environment variable

It fails with

❯ docker run -i --rm -e ZSERVER_HOST=0.0.0.0 -e ZSERVER_PORT=55001 -p 55001:55001 -e ADDONS="plone.app.robotframework plone.restapi" -e PROFILES="plone.app.contenttypes:plone-content plone.volto:default-homepage" plone-backend:6 ./bin/robot-server plone.app.robotframework.testing.PLONE_ROBOT_TESTING
=======================================================================================
Installing ADDONS plone.app.robotframework plone.restapi
THIS IS NOT MEANT TO BE USED IN PRODUCTION
Read about it: https://github.com/plone/plone-backend/#extending-from-this-image
=======================================================================================
ERROR: Invalid requirement: 'plone.app.robotframework plone.restapi'
WARNING: You are using pip version 21.3; however, version 21.3.1 is available.
You should consider upgrading via the '/app/bin/python -m pip install --upgrade pip' command.

Support for RelStorage cache-local-dir?

Can be useful this configuration in relstorage.conf?
This is optional, so i don't know what's the best way to add it into the relstorage.conf file.

Implement nightly builds of buildout.coredev

Cant figure out connecting the apache solr via plone-backend .

i have been working connecting apache solr via the plone-backend. i still couldnot figure it out. most of the examples are using the the buildout.cfg for connecting the solr. thus, how can we make use of the plone-backed image to integrate apache solr.

vulnerability in future

Not sure this is the right place.
Our jenkins ppelines perform automatic securitiny vulnerability checks and when I tried to build a docker image based on plone-backend 6.0.2, it gives the following warning:

09:32:14  [2023-03-08T08:32:14.876Z] CVE-2022-40899 (HIGH) for future:0.18.2:/app/lib/python3.11/site-packages/future-0.18.2.dist-info/METADATA - python-future: remote attackers can cause denial of service via crafted Set-Cookie header from malicious web server: (https://avd.aquasec.com/nvd/cve-2022-40899)
09:32:14  [2023-03-08T08:32:14.876Z]     An issue discovered in Python Charmers Future 0.18.2 and earlier allows remote attackers to cause a denial of service via crafted Set-Cookie header from malicious web server.
09:32:14  [2023-03-08T08:32:14.876Z] 
09:32:14  [2023-03-08T08:32:14.876Z] Fixed in 0.18.3
09:32:14  [2023-03-08T08:32:14.876Z] -----

I guess we should update future in https://dist.plone.org/release/6.0.2/constraints.txt
If anyone directs me to where I can do that, I'll make a PR

Some add-ons fail to compile their mo files

My container has a bunch of add-ons which I have added using the sanctioned method (not the ADDONS variable although I suspect this affects them too).

When these add-ons start, I see permission denied errors in the log about their compilation.

What could it be and how can we fix this structurally? Do we expect the Zope server to write to directories that contain code?

ZEO absent from 6.0.4

Hello!

Our Plone servers use ZEO to run multiple backends within the same Kubernetes pod.

We noticed upon upgrade that the ZEO client libraries are missing from the latest stable published container image (we derive our container images from this one). The traceback reported it cannot import the ZEO module.

Is there a reason why this is missing?

The workaround I have implemented in our container file goes like this:

# Add ZEO
RUN /app/bin/pip install -U ZEO -c /app/constraints.txt

Tests for the containers using the ZEO library should be added so this does not happen in the future again.

use cookiecutter-zope-instance to configure zope.

I already use this approach in my customer projects and it makes the configuration way more flexible, read: allows any configuration option supported by cookiecutter-zope-instance and this are almost all.

Permission denied when using external volume drivers

Using NFS or Amazon EBS driver for volumes will end-up with a permission denied error:

11/5/2021 1:56:28 PMmkdir: cannot create directory ‘/data/filestorage’: Permission denied
11/5/2021 1:56:28 PMmkdir: cannot create directory ‘/data/blobstorage’: Permission denied
11/5/2021 1:56:28 PMmkdir: cannot create directory ‘/data/cache’: Permission denied
11/5/2021 1:56:28 PMmkdir: cannot create directory ‘/data/log’: Permission denied
11/5/2021 1:56:44 PMmkdir: cannot create directory ‘/data/filestorage’: Permission denied
11/5/2021 1:56:44 PMmkdir: cannot create directory ‘/data/blobstorage’: Permission denied
11/5/2021 1:56:44 PMmkdir: cannot create directory ‘/data/cache’: Permission denied
11/5/2021 1:56:44 PMmkdir: cannot create directory ‘/data/log’: Permission denied

Thus, I propose to remove USER plone from Dockerfile and gosu via docker-entrypoint.sh.

I'll prepare a PR.

check/remove superflous apt dependencies

see https://community.plone.org/t/plone-6-backend-with-classic-ui-apt-requirements/15821

(check for x86 and ARM if there are always wheels).

why using python venv inside container?

inside the docker files (standard and nightly) the project is setup as you would in a local dev-environment.

why? the container already gives that level of encapsulation we need.

am I missing a point here?

Python Version as ARG

Pass Python version in as ARG. see #85
cc @davisagli

How to start instance in fg?

Is there a way to run this image in fg to debug addons?

I'm developing an add-on with plone-backend image and i need to put some pdb into its code but i don't know how to start the instance in fg.

I tried to bypass the entrypoint with command=bash but after that i don't know which script run and how.

Am i missing something or this isn't available yet? If not, how do you develop with plone6/pip?

Do not create Volume `/data` unless needed.

There is no need for any volume if i.e. run with Relstorage and Postgresql.
Caches can live non-persistent.
Log should go to console. Accesslog configured different.
Same with VOLUME /data - this is not needed.

Do not use own skeletons here, but use `cookiecutter-zope-instance`

This cookie-cutter were made for this task, here we have everything twice in several skeleton files.
I think this could be much more streamlined using the cookiecutter-zope-instance - and while at it improve it (I can move it to plone or collective).

The cookie-cutter would need to be installed in the build step together with cookiecutter script.
At startup it can generate the configuration (it takes only some millis).
It is documented (vs. the current READMe has many bugs and is consfusing)

This could probably also help using the image for development.

Use GHCR for test image

at the moment its docker hub and GHCR is supposed to be faster with GH

Refactor repository to use branches and tags

Branches:

• main (6.0.x)
• 5.2.x

Tags for every release

Update README.md on Docker hub on changes to main

No way to add an emergency user except by going to debug shell

There's no script to add an emergency user.

Look into using chiselled Ubuntu containers

https://canonical.com/blog/chiselled-ubuntu-ga

Should we update extra Python ecosystem packages installed on Plone 5.2.13 container

@mauritsvanrees @davisagli @pbauer

I want to make a new release of our plone-backend images for Plone 5.2 now that Maurits has released 5.2.13. But I notice in the Dockerfile we install/add some extra packages that are really outdated.

Also pip is still pinned ad 22.0.4 when the constraints.txt for Plone 5.2.13 now has pip 23.2.

The image is 'officially' not support, as in: we started promoting the container images for Plone 6. But what to do here?

If dockerfiles based on our image are really picky they repin their packages, but if they are depending on what is in the package.... We can update:

pip == 23.2
relstorage == 3.5.0
psycopg2 == 2.9.6
python-ldap == 3.4.3
plone.volto==3.1.0a9

plone-backend/Dockerfile

Lines 6 to 10 in 533916d

	ENV PIP_VERSION=23.2
	ENV PLONE_VERSION=5.2.13
	ENV PLONE_VOLTO="plone.volto==3.1.0a4"
	ENV EXTRA_PACKAGES="relstorage==3.4.5 psycopg2==2.9.3 python-ldap==3.4.0"

The image for 6.0.0 with ADDONS takes forever to boot

In comparison with the previous images:

Running rc1:

docker run -i --rm -e ZSERVER_HOST=0.0.0.0 -e ZSERVER_PORT=55001 -p 55001:55001 -e 'ADDONS= plone.app.robotframework==2.0.0b2 plone.app.testing==7.0.0b2' -e APPLY_PROFILES=plone.app.contenttypes:plone-content,plone.restapi:default,plone.volto:default-homepage -e CONFIGURE_PACKAGES=plone.app.contenttypes,plone.restapi,plone.volto,plone.volto.cors plone/plone-backend:6.0.0rc1 ./bin/robot-server plone.app.robotframework.testing.VOLTO_ROBOT_TESTING

Running 6.0.0 final:

docker run -i --rm -e ZSERVER_HOST=0.0.0.0 -e ZSERVER_PORT=55001 -p 55001:55001 -e 'ADDONS= plone.app.robotframework==2.0.0b2 plone.app.testing==7.0.0b2' -e APPLY_PROFILES=plone.app.contenttypes:plone-content,plone.restapi:default,plone.volto:default-homepage -e CONFIGURE_PACKAGES=plone.app.contenttypes,plone.restapi,plone.volto,plone.volto.cors plone/plone-backend:6.0.0 ./bin/robot-server plone.app.robotframework.testing.VOLTO_ROBOT_TESTING

For me, M1Max first command runs in 8s, the latter in 47s... No clue why, it seems that pip takes much more "Requirement already satisfied:" checks than before.

/cc @jensens @ericof we did update pip to latest and Python 3.11, right? Any idea if that could be the reason?

having css issue after using tls

tested the plone backend without tls, All the css is working fine. but after using the tls on the test domain, i am having css issue. i couldnot figure out the where to make changes. i couldnot find the env relating to this.

used image plone/plone-backend:6.0.0b1

uploaded the screenshot of the issue.

Container Scanning

This should work in GH Actions too. I use it in my projects, so would be good for our images as well.
https://gitlab.com/gitlab-org/gitlab-foss/-/blob/master/lib/gitlab/ci/templates/Security/Container-Scanning.gitlab-ci.yml

Don't use root user when running containers

It's a best practice to not use the root user in containers (see https://pythonspeed.com/articles/root-capabilities-docker-security/)

This image currently adds a plone user but should also set it as the active user with https://docs.docker.com/engine/reference/builder/#user

Move README content into Plone 6 docs, leaving only version information

READMEs should have only minimal information and not duplicate that which is already published in Plone 6 docs. This README currently has some non-duplicated information that should be moved into the docs, and replaced with a link to the docs.

See #67 (comment) for discussion, and suggested changes in https://github.com/plone/plone-backend/pull/67/files.

plone-backend image is not responding correctly to SIGTERM

While testing the cookiecutter-plone-starter docker compose setup today I noticed that when I stop the compose setup, the backend container takes 10-11 seconds to stop. This is the 10 second timeout when the application in side the container doesn't respond correclty to the 'nice' way of asking to stop (I think SIGTERM) and then a SIGKILL follows.

This might have to do something with our entrypoint script that starts the instancne and somewhere along the process chain the signals are not correctly passed along. Any takers? ;-)

precompile all python files to increase startup speed?

actually we have two options:

1. small images size, slower startup (like now)
2. larger image size with pre-compiled python files and faster startup.

I would precompile in first stage already and have all on one layer on copy.

Hint: python -m compileall .

DEVELOP packages should be installed before ADDONS

In docker-entrypoint.sh, 6.x branch

$@ in `start` verb on docker-entrypoint.sh

We'd like to be able to tune Waitress with command line parameters (does not seem like it takes variables), so it would be good if the start verb located near the end of the entrypoint script actually supported "$@" so that we may pass parameters to it.

Create docs on docs.plone.org for this image

The README here lands on Docker Hub, so it should contain basic about the image and Plone.
In depth information has to be moved to docs.plone.org (6-dev branch)
@stevepiercy knows better where to put it there. Lets keep this issue open until this is finished.

Align Dockerfile.nighlty with Dockerfile

Currently they work different and should be aligned again to the more efficient way of current Dockerfile.

why use wheels instead of venv?

followup to #60

after investigating deeper into docker multistage I came across the topic of wheels vs venv.
we use both and I would like to understand why.

according to this article https://pythonspeed.com/articles/multi-stage-docker-python/ a venv is perfectly capable of being passed from one stage to the other - as we do using the wheels directory.

this made me lift my eyebrow:

"Another solution to the problem is to build all the packages as wheels—binary packages with the compiled code—and then copy the wheel files over to the build stage and install them there. This works, but doesn’t really add much."

a second source I found: https://testdriven.io/blog/docker-best-practices/#using-python-virtual-environments indicates even more concrete:

"you may want to use a virtual environment in a multi-stage build rather than building wheel files."

is there any other benefit of using wheels over venv?

Upgrade system before `apt-get install`

We should always deliver most recent software using apt-get upgrade in both stages.

Update base image to Python 3.10

currently we are on 3.9.

Include "make " on apt-get install

Since we build all on make we should install it, so a derived container dont.

plone-backend/6.0/6.0.0a2/Dockerfile.python39

Line 30 in e59d0e9

&& runDeps="git libjpeg62 libopenjp2-7 libpq5 libtiff5 libxml2 libxslt1.1 lynx netcat poppler-utils rsync wv busybox gosu" \

Support Docker Content Trust (DCT)

https://docs.docker.com/engine/security/trust/

Add command for packing the ZODB

Packing is done with different commands depending on whether ZEO or Relstorage is used. For Relstorage it needs a modified conf file that only includes the relstorage section. It would be nice for the image to include the necessary configuration and an entrypoint which runs the right command.

(Or are there good reasons to create a separate image for this?)

Scheduled images building

Using a GH Actions schedule we should build fresh images, i.e. like weekly.

Fix nightly image generation

Since we made the 6.0.x branch default, nightly images are not created anymore.

Healthscheck on HTTP level?

I propose to use our /ok check here:

plone-backend/6.0/6.0.0a2/Dockerfile.python39

Line 55 in eb86fd1

HEALTHCHECK --interval=10s --timeout=5s --start-period=30s CMD nc -z -w5 127.0.0.1 8080 || exit 1

like so using busybox wget:

HEALTHCHECK --interval=10s --timeout=5s --start-period=30s CMD  wget -q -T 5 http://127.0.0.1:8080/ok -O - | grep OK || exit 1

Opinions?

vulnerable module `expat/libexpat` in base image

Dockerfile uses python:${PYTHON_VERSION}-slim-bullseye as base image:

plone-backend/Dockerfile

Line 3 in f490f14

FROM python:${PYTHON_VERSION}-slim-bullseye as base

This image has a vulnerable module: expat/libexpat:

In libexpat through 2.4.9, there is a use-after free caused by overeager destruction of a shared DTD in XML_ExternalEntityParserCreate in out-of-memory situations.

How to fix?
Upgrade Debian:11 expat to version 2.2.10-2+deb11u5 or higher.

This affects all "python:3.x-slim-bullseye" images:

https://snyk.io/test/docker/python%3A3.7-slim-bullseye
https://snyk.io/test/docker/python%3A3.8-slim-bullseye
https://snyk.io/test/docker/python%3A3.9-slim-bullseye
https://snyk.io/test/docker/python%3A3.10-slim-bullseye

setuptools version and installation step during installation of Plone

When creating a venv with python3 -m venv the packages pip and setuptools are installed in specific versions (e.g. pip-20.3.4 and setuptools-44.1.1).

The command /app/bin/pip install -U "pip==${PIP_VERSION}" wheel updates pip to the version specified in ${PIP_VERSION} and installs wheel to the current version (i.e. pip-22.3.1 and wheel-0.38.4) but setuptools remains installed (i.e. in version setuptools-44.1.1)

This version (i.e. setuptools-44.1.1) is used later to build the wheels (e.g. for plone.volto, Chameleon, repoze.xmliter, Zope2, ZODB3, future, sgmllib3k) when installing Plone with /app/bin/pip install Plone ${EXTRA_PACKAGES} -c /app/constraints.txt.

Only after the wheels are build, the installed version setuptools-44.1.1 is uninstalled and setuptools-65.7.0 is installed according to the version specified in /app/constraints.txt

Im not sure if there might be any problems when building the wheels with setuptools in a differnt version as later installed.

Should that be the case I'd suggest to install setuptools in the version specified in /app/constraints.txt before installing Plone.

Furthermore and since /app/constraints.txt already specifies the versions of pip, setuptools, and wheel (i.e. pip-22.3.1, setuptools-65.7.0, wheel-0.38.4) I'd also suggest to update pip and installing wheel and setuptools using -c /app/constraints.txt. This would make the use of the variable ${PIP_VERSION} redundant.

Consider replacing the following line #18 in Dockerfile.builder:

/app/bin/pip install -U "pip==${PIP_VERSION}" wheel

with

/app/bin/pip install -U pip wheel setuptools -c /app/constraints.txt

and removing the variable declaration PIP_VERSION=22.3.1 in line #7 in Dockerfile.builder:

DEVELOP should be properly handled by `pip install -e`

If we assume I have multiple addons specified:

DEVELOP="src/addon1 src/addon2"

then the pip install -e should be called like:

bin/pip install -e src/add1 -e src/addon2

Add python-ldap on build.

Same as with psycopg2 which we already have included python-ldap has no binary wheels and needs compilation.

Plone faster up than postgresql dbinit

If we're in a RelStorage environment Postgresql needs time for it initial dbinit.

Docker unfortunately removed to wait for a healthcheck, it only checks if a container is up, so we can not wait for it.

Now, Plone is starting up, because before postgres states its running before dbinit and while dbinit RelStorage tries to create the tables and stored procedures needed.
Postgres, after finished with dbinit, restarts itself. In Plone/RelStorage now the initialization is eventually only half done. Then Plone tries to connect to a partly created structure and fails.
dbinit is fast as well, mostly under 2-5s.

I propose to delay startup of Plone in case if Relstorage is used based on an environment variable. default 5s.
dinit runs once, so after initialization the variable can be set to 0s for the cluster.

Or are better ideas around?

Invalid URL for README "Extending from this image" in docker entrypoint

When running the backend container an invalid URL is being printed to the console. In 95246aa the whole section about "Extending from this image" was removed.

Maybe, (a) point to the official documentation, or (b) remove the link from the entrypoint script.

plone-backend/skeleton/docker-entrypoint.sh

Line 95 in 6b46109

echo "Read about it: https://github.com/plone/plone-backend/#extending-from-this-image"

Multi-platform builds

We need to support ARM64 build as well, given the awful performance of emulated images in M1 chips.

Would it be hard? Didn't find a good source of knowledge on the accepted way to achieve this. We are building locally, so no auto build in dockerhub, right? Of course, I can build and push them if needed.

I tried to build the images, and I had to do an amendment, so the install of relstorage do not happen in the base image, but on the builder, and stored in the wheelhouse as well, since it needs compilation in M1s. Is there any reason for installing relstorage at a later point?

/cc @jensens @ericof @silviot @avoinea

BUILDOUT_EXTENDS: Option missing

I noticed that it is no longer possible to extend the plone-backend docker images using the BUILDOUT_EXTENDS option.

The official plone image could be extended using the following dockerfile.

FROM plone:5.2

COPY extended_buildout.cfg /plone/instance/

RUN buildout -c extended_buildout.cfg

Here is an example of extended_buildout.cfg

[sources]

politikus.contenttypes = git https://github.com/Sinar/politikus.contenttypes
popolo.contenttypes = git https://github.com/Sinar/popolo.contenttypes
ocds.contenttypes = git https://github.com/Sinar/ocds.contenttypes
politikus.bods = git https://github.com/Sinar/politikus.bods
politikus.theme = git https://github.com/Sinar/politikus.theme
collective.vocabularies.iso = git https://github.com/Sinar/collective.vocabularies.iso
politikus.extractives = git https://github.com/Sinar/politikus.extractives branch=main
politikus.naturalresource = git https://github.com/Sinar/politikus.naturalresource branch=main
plone.app.z3cform  = git https://github.com/enfold/plone.app.z3cform/ branch=3.2.2+enfold1

[buildout]

extends = buildout.cfg 

extensions = mr.developer
always-checkout = force
auto-checkout = *

eggs += 
    eea.facetednavigation
    politikus.contenttypes
    popolo.contenttypes
    politikus.bods
    ocds.contenttypes
    politikus.theme
    plone.app.imagecropping
    politikus.extractives
    politikus.naturalresource
    collective.vocabularies.iso
    Products.PloneHotfix20210518
    collective.relationhelpers
    plone.app.z3cform
    plone.app.locales
    plone.volto
    

languages = en km 

environment-vars =
    zope_i18n_compile_mo_files true

    
[versions]
Products.PloneHotfix20210518 = 1.4
eea.faceted.vocabularies = 7.0
eea.facetednavigation = 14.6
eea.jquery = 11.2
plone.app.imagecropping = 2.2.2
plone.app.jquery = 1.11.2
politikus.contenttypes==1.0a1
popolo.contenttypes==1.0a1
collective.dexteritytextindexer = 2.4.0
politikus.theme==1.0a1
collective.themefragments = 2.12
plone.app.themingplugins = 1.1
collective.relationhelpers = 1.5
plone.app.z3cform = 3.2.2+enfold1

parts +=
    i18ndude
    precompile
   
[i18ndude]
unzip = true
recipe = zc.recipe.egg
eggs = i18ndude  


[precompile]
recipe = plone.recipe.precompiler
eggs =
    ${instance:eggs}
    plone.app.locales
extra-paths = ${buildout:directory}/locales
compile-mo-files = true

Healthcheck: Check if wget from busybox is enough

Busybox wget has limited features and does not support timeouts. Check is we need the real one or better use curl or if w/o timeout works in all situations.

System Upgrade?

I would do an apt-get upgrade -y here. Any reason why we don't?

plone-backend/6.0/6.0.0a2/Dockerfile.python39

Lines 9 to 11 in e59d0e9

	RUN apt-get update \
	&& buildDeps="dpkg-dev gcc libbz2-dev libc6-dev libffi-dev libjpeg62-turbo-dev libldap2-dev libopenjp2-7-dev libpcre3-dev libpq-dev libsasl2-dev libssl-dev libtiff5-dev libxml2-dev libxslt1-dev wget zlib1g-dev python3-dev build-essential" \
	&& apt-get install -y --no-install-recommends $buildDeps\