digitalfabrik / integreat-cms

Simplified content management back end for the Integreat App - a multilingual information platform for newcomers

Integreat Django CMS

Integreat - The mobile guide for newcomers. Multilingual. Offline. Open Source.

This content management system helps local integration experts to provide multilingual information for newcomers.

Development Setup

This section provides a brief overview of setting up the development environment. We support various environments: you can set up everything locally using your preferred package manager, use it as a devcontainer, or utilize the nix-flake. Please clone the repository with the following snippet before starting to setup your development environment.

git clone [email protected]:digitalfabrik/integreat-cms.git
cd integreat-cms

Choosing your setup method

From the three provided development setup options, choose yours based on personal preference and familiarity with the used tools. If you are unsure, the following might help you make a decision. Please note that these are only suggestions.

• using VSCode or PyCharm? → Devcontainer
• on NixOS, already using nix or do not want to install docker? → Nix Flake
• conflicting version of tools required on your system? → Devcontainer or Nix Flake
• do not want to use additional tooling and have no conflicting tooling? → Local Setup

Local Setup

To configure your development environment on your system, please follow these steps carefully.

1. Ensure that the following packages are installed alongside your preferred IDE:
   • npm version 7 or later
   • nodejs version 18 or later
   • python3 version 3.11 or later
   • python3-pip (Debian-based distributions) / python-pip (Arch-based distributions)
   • python3-venv (only on Debian-based distributions)
   • gettext for translation features
   • Either postgresql or docker to run a local database server
2. Execute tools/install.sh to download all dependencies.
3. Execute tools/migrate.sh to apply all database schema migrations.
4. Optionally, run tools/loadtestdata.sh to apply a predefined set of test data.

Devcontainer

To configure your development container, please follow these steps carefully.

1. Make sure you have Docker and VSCode (not VSCodium) installed on your machine.
2. Open the project in VSCode.
3. If you're opening VSCode for the first time, you'll be prompted to install the "Dev Containers" extension. Click "Install" to proceed.
4. Open the command palette (Ctrl + Shift + P or Cmd + Shift + P on macOS) and search for "> Remote-Containers: Open Folder in Container".
5. VSCode will open the project in a new container, install all further required tools and load the testdata.

Known Limitations

The perfect is the enemy of the good; thus, this section illuminates aspects of our evolving development setup.

Sharing git username and email

A known limitation exists where certain versions of Visual Studio Code (VSCode) may not copy the user's .gitconfig file correctly into the Devcontainer environment. In such cases, when you attempt to commit changes within the Devcontainer, you may be prompted to enter your Git username and email every time. This can be inconvenient and disrupt the workflow. However, there is a workaround for this issue. You can resolve it by appending the content of your personal .gitconfig file, located at $HOME/.gitconfig, to the end of the repository-specific .git/config file, which in this case would be integreat-cms/.git/config. By doing so, you ensure that the necessary Git configuration settings are correctly applied within the Devcontainer environment, allowing for a smoother development experience.

For more information, refer to this issue in the VSCode Remote Extension repository.

Docker permissions

The user must be in the docker group on linux, VSCode does not allow to optionally enter sudo password.

Nix Flake

To configure your development environment through the provided nix flake, follow these steps carefully.

1. Install nix. Depending on your distribution or operating system, multiple ways might be available to do this; one recommended way is to use the install script provided by zero-to-nix.com:

   curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install

2. Run nix develop inside your project directory. You should see nix pulling in the required dependencies for running the project.
3. Execute tools/install.sh to download and install all project dependencies.

Please note that closing the shell in which you ran nix develop will destroy your development environment, i.e. any time you want to work on the project again, you will need to execute nix develop beforehand, and then start your code editor (code ., nvim,...) from within that same shell.

Known Limitations

On MacOS, installing libmagic separately through brew install libmagic might be required.

Run development server

Run the development server using /tools/run.sh, then open your browser and go to http://localhost:8000. The default login credentials are username: "root" and password: "root1234".

Documentation

For detailed instructions, tutorials and the source code reference have a look at our great documentation:

📓 https://digitalfabrik.github.io/integreat-cms/

Alternatively, you can generate it yourself using the tools/make_docs.sh script.

Project Architecture / Reference

• Integreat CMS: The main package of the integreat-cms with the following sub-packages:
  • API: This app provides wrapper functions around all API routes and classes mapping the cms models to API JSON responses.
  • CMS: This app contains all database models, views, forms and templates forming the content management system for backend users.
  • Core: This is the project’s main app which contains all configuration files.
  • Firebase API: This app provides wrapper functions around the Firebase API to send push notifications.
  • GVZ API: This app provides wrapper functions around our Gemeindeverzeichnis API to automatically import coordinates and region aliases.
  • Nominatim API: This app provides wrapper functions around our Nominatim API to automatically import region bounding boxes.
  • Sitemap: This app dynamically generates a sitemap.xml for the webapp.
  • SUMM.AI API: This app provides wrapper functions around the SUMM.AI API for automatic translations into Easy German.
  • XLIFF: This app allows (de-)serialization of translations from/to XLIFF (XML Localization Interchange File Format) for standardised exchange with translation agencies.
• Tests: This app contains all tests to verify integreat-cms works as intended

To better understand the overall intention it might also be helpful to look at the wiki for municipalities (GER) that teaches how to use our CMS.

License

Copyright © 2018 Tür an Tür - Digitalfabrik gGmbH and individual contributors. All rights reserved.

This project is licensed under the Apache 2.0 License, see LICENSE and NOTICE.md.

integreat-cms's Issues

API: Region list feedback

Endpoint for posting feedback on the list of available regions.

https://github.com/Integreat/cms/wiki/REST-APIv3-Documentation#cities

PN: view sent notifications

Display sent push notifications in a list per language.

The list should contain the following information: Title, Date+Time, user that sent the message

API: package site/page resources

Endpoint which packages (tar) all resources required for a page or site. This includes linked PDFs and sourced images.

Included should be all resources hosted on the CMS. External resources should not be bundled.

For this, the content of https://github.com/Integreat/cms/wiki/REST-APIv3-Documentation#pageseventslocationsdisclaimer needs to be analyzed and then all resources bundled.

Media management

Cities use/include images and PDFs in pages. Therefore back end users need a convenient way to manage these files. Most commons file types:

• PDF
• PNG
• JPEG

Other common image formats need to be supported as well.

The following additional features should be supported:

• Data should be stored in a directory defined in the settings.py
• enable replacement of files (update files without changing download URL)
• add alt text to enhance accessibility (probably as required field)
• security checks of uploaded files
• Nice to have feature: data deduplication. Users are uploading lots of duplicated content. We could save a lot of disk space with deduplication of stored files.
• Uploaded files should not immediately be publicly available.
• Scaling of images for the apps
• Optimazation, for PNG lossless, for example with optipng, for JPEG with defined compression levels, i.e. 85. See #118
• Please have a look at caching/storing rules of the app. Ask app team about current implementation.

API: FCM endpoint

Endpoint for getting sent Firebase Cloud Messaging push notifications.

https://github.com/Integreat/cms/wiki/REST-APIv3-Documentation#firebase-messages

API: all pages endpoint per language

Endpoint for all pages of one language. This must be Integreat API v3 compatible.

Pages whose parent page is not public should not be delivered either.

https://github.com/Integreat/cms/wiki/REST-APIv3-Documentation#pageseventslocationsdisclaimer

Create data migration tool

Create a tool that can migrate all data from the old CMS (Wordpress) to this CMS.

Required data (this list needs to be checked for completeness):

• Pages with publishing state, history, revision state, linked translations and hierarchy, attached pages
• Users (maybe send only invitation link to new system, w/o migrating passwords and 2FA codes)
• Events
• Locations
• Push notifications

Events: Create, Edit, Delete

The creation and editing of events should need different privileges than editing pages.

We do not want to version events.

An event should contain (completeness needs to be discussed):

• Date
• Time
• Location
• Coordinates
• Description
• Title
• Associated organizations
• Attached photo / thumbnail

Admin: duplicate regions

Duplicate a site with all its content except for users.

API: page feedback

Implement endpoint for posting feedback on a page.

The following definition should be supported: https://github.com/Integreat/cms/wiki/REST-APIv3-Documentation#pageeventdisclaimer

The API should accept an JSON object as defined in the above link, but also HTTP POST variables with the same names.

API: implement HSDS/HSDA

See https://openreferral.readthedocs.io/en/latest/

https://github.com/openreferral/specification

Chat for authors

Create a chat that can be used to communicate between CMS users. It should be possible to create different channels, for example within one site, or a global chat. If messages are new (unread), they should be highlighted. A timestamp and the sender name should be displayed besides the message.

The chat should be integrated in the Dashboard.

Admin: manage users

Create, edit and delete users with e-mail, login, password, site access, role assignments.

Assign users to sites.

See also #45.

Pages: list of archived pages

It is necessary to keep a list of of deleted pages pages. It is sufficient to keep the ID in store, but drop the content.

This feature is required for API v0/v1/v2 compatibility. See #11. It is possible that we do not need APIv0 to v2 compatibility. There are very few users left using the old API. We have to evaluate the situation before taking on this issue.

Pages: Overview as tree

Create an overview of all pages in a tree.

This should include:

• tree view
• Drag & Drop page to other location in tree
• Revision status
• Translation status
• Exclude archived pages
• show only pages of current site

Analytics overview in dashboard

Show Matomo statistics for a region in dashboard for back end users.

The Dashboard should display a chart of API calls per language and day during the last 30 days. The data can be fetched from the Matomo API. Each region has its own access token, that needs to be configured in the settings for the region. The setting is managed by the superadmins.

Admin: region management

• add
• change
• hide
• delete
  regions

Pages: Attach content

Attach other pages from the same or other sites at the beginning or end of a page.

The other, attached pages should be shown while editing but not be editable.

Use case: different sites share common text templates which need to be amended by local addresses.

Translation coverage analytics

Generate report of translation coverage. This must include:

• A list of all outdated translations

Optional:

• Chart of coverage percentage per language and day for the last month

API: offer list feedback

Endpoint for posting feedback on available extras.

https://github.com/Integreat/cms/wiki/REST-APIv3-Documentation#extras-1

API: search result feedback

Endpoint for posting feedback on search results.

https://github.com/Integreat/cms/wiki/REST-APIv3-Documentation#request-11

API: single page endpoint

Endpoint for single pages. This must be Integreat APIv3 compatible. The page is selected by either a permalink or page ID.

https://github.com/Integreat/cms/wiki/REST-APIv3-Documentation#pageeventdisclaimer

API: extras endpoint

Endpoint to get all activated Extras for a site. This must be Integreat API v3 compatible.

https://github.com/Integreat/cms/wiki/REST-APIv3-Documentation#extras

API: offer feedback

Endpoint for posting feedback on an extra.

https://github.com/Integreat/cms/wiki/REST-APIv3-Documentation#extra

The API should accept an JSON object as defined in the above link, but also HTTP POST variables with the same names.

Add Matomo integration

Matomo has to track API requests for pages, including if all or just a single page was sent.

The following data must be tracked:

• Endpoint
• Time of request

Optional data:

• Client used (Web or native app)
• Numbbers of pages delivered

How do wecompare single page vs all page API calls? Each page delivered could be one action, while one API call is a visitor.

The statistics need to be visible for the admin of the instance. The view has to contain:

• Chart of page requests per hour and language for the last 24 hours
• Chart of page requests per day and language for the last 30 days
• Chart of page requests per month and language for the last year

Pages: export PDF

Export selected list of pages into PDF. Option to include table of content. Cache created PDFs.

It is probably a good thing to use the export PDF API for this feature. This ensures, that apps and back end produce the same results.

If possible we can include customization features for regions. For example, the cities could include their logos into the header or footer. This is covered in #213

API: sites endpoint

Endpoint of available sites. This must be compatible to Integreat API v3.

https://github.com/Integreat/cms/wiki/REST-APIv3-Documentation#locationsmultisitesinstances

Create login

Log in with username/e-mail and password.

Superadmins can be redirected to the back end. If no site is selected for users, present them with a site selection. If has only access to one site, redirect to this site immediately.

Pages: save automatically during editing

This is a feature that exists in Wordpress and should be implemented before launch.

Permission Based Access Control

THIS NEEDS TO BE DISCUSSED: Do we want to differentiate roles and groups or just have one of them?

Each part of the CMS requires specific privileges. These should be collected in roles. Roles are then assigned to users.

Roles:

• Superadmin
• Admin (for single site) (Verwalter)
• Trusted (is allowed to publish pages / events)
• Events (is allowed to create/edit events)
• Pages (is allowed to create/edit pages)

Additionally, it must be possible to allow some users to edit pages in parts of the tree without assigning the role "Pages".

PN: sending

Send push notification to target group. Automatically translate to other languages if wanted by user. We want to support globally set topics, for example default "news", "refugees", or "supporters". The old channel structure needs to be supported: /topics/[BLOG_ID]-[LANG_CODE]-[TOPIC]. For details, see https://github.com/Integreat/cms/wiki/Firebase-Notifications

Support saving draft messages.

Pages: Create, Edit, Publish, Archive, Delete

Add functionality to create, edit, publish, archive and delete a page. Changes to pages have to be versioned.

For editing, there should be a WYSIWYG editor as well as a simple markup language.
The editor should support arabic text direction by default.

Enable minor edits which do not change the translation status.

It must be possible to add media to a page and embed images. See #48

For API v0/v1/v2 compatibility, the page ID of deleted pages must be tracked. Alternatively, deletion must be disallowed in favor of archiving. See #12

This page should also list the corresponding feedback.

• Create
• Edit
• Publish
• Archive
• Delete

Activated offers in Dashboard

The manager (Verwalter) of a region should easily see which offers are activated. This should be a very simple list of activated offers, including the offer icons used in the App.

Pages: review process

Implement a process for reviewing pages before publication or publication of changes.

View feedback

Overview page of feedback ordered by submission time.

Distinction between web and native app. It should also be shown how much up and downvotes there are in sum for this page.

It should be possible to search for for feedback about individual pages.

Task Breakdown:

• Create new template for feedback that extends _base.html
• Create a new view for feedback
• List all feedback, mark new feedback
• Add link to object which the feedback refers to

API: languages endpoint

Endpoint to get all languages for a site. This must be Integreat API v3 compatible.

https://github.com/Integreat/cms/wiki/REST-APIv3-Documentation#languages

Add Imprint form

Add a form to edit the imprint.
This involves the form-model, form views and form templates.
This should be similar to the page form, so you can probably simply copy + paste existing code.

Depends on #357 ✔️

Setup tailwindcss

Currently tailwindcss is only loaded statically. However, it should be loaded dynamically as specified at https://tailwindcss.com/docs/installation.

Translation: export XLIFF

Enable exporting pages to XLIFF file.

Take translation state of pages into account. Unchanged pages or minor changes do not require translations.

RSS-Feed on the Region Dashboard

Motivation

As we are supporting a RSS-Feed on the current wordpress, we should deliver the same feature within our release. It should show the last blog entries from our website.
It not necessary to support a "already read" function. Showing the latest 5 news should be sufficient.

Proposed Solution

It should be kind of equal to the current widget:

Add translation coverage to "To-Do" dashboard

The Dashboard should contain some basic overview of the current translation status.

This could be a pie chart that visualizes 3 numbers: pages with up to date translations, pages with outdated translations, not translated pages.

This should be part of the new "To Do" dashboard, see #2107 (comment).

Translation: connect automatic translation API

Enable connecting automatic translation APIs, for example Google, Linguee, or Bing.

API: offer details endpoint

Endpoint to get specifics for one activated offer. This must be Integreat API v3 compatible.

https://github.com/Integreat/cms/wiki/REST-APIv3-Documentation#extra

Translation: import XLIFF

Import translations for pages from XLIFF file. Importing should be handled similarly as manually editing a page. That means a new page version is created and potentially requires a review.

Pages: New approach to control the arrangement of pages

Currently there are two ways to determine the arrangement of the pages:

1. Tree view: Drag & Drop
2. Detailed view: Determining the order and the parent page

In particular, defining the order is not satisfactory, as the numbers are always assigned at different intervals.

My suggestion:

• define the parent page via select field
• then drag & drop between the existing children's pages

Are there any better suggestions?

Events: overview list

List all events for content managers of a region. Only events in the future should be visible by default. It should be possible to also display older events when wanted by the user.

The overview should contain basic information about the event: title, start and end date/time. It should also contain quick links for editing and deletion.

API: events feedback

Endpoint for posting feedback on events.

https://github.com/Integreat/cms/wiki/REST-APIv3-Documentation#pageeventdisclaimer

The API should accept an JSON object as defined in the above link, but also HTTP POST variables with the same names.

Show latest feedback in dashboard

Show admins and content creators the latest feedback from app users in dashboard. This should be a list of the last ~5 messages. In some cases, the feedback only contains a thumbs up or down, in other cases it can contain a message. The title of the page or event concerned should be displayed as well and linked to the editing page.

Additional option: Add link that sends the message string to translate.google.com or deepl.com for translation. This can be useful if the feedback is given in a language the back end user does not understand. It is perfectly fine, if the link opens the translation website in a new tab. No need to fetch a translation via the API.

API: parents endpoint

Endpoint to get parents of a page up to the root. This must be compatible to Integreat API v3.

https://github.com/Integreat/cms/wiki/REST-APIv3-Documentation#parents-of-a-post

@maxammann @Simple-codinger is this used by any app? Otherwise we could move this to another milestone.

API: children of page endpoint

Endpoint to get all children of a page. This must be compatible to Integreat API v3.

https://github.com/Integreat/cms/wiki/REST-APIv3-Documentation#children-of-a-post

@maxammann @Simple-codinger is this used by any current app? Otherwise this can be moved to another milestone.