unicef / iogt

IoGT

Internet of Good Things (IoGT) is developed as a public good under a BSD-2 open-source license.

The development uses the Python programming language, building on the Django server-side web framework combined with the Wagtail content management system.

IoGT 2.x requires:

• Wagtail 3.0.3
• Django 3.2
• Python 3.9+
• PostgreSQL (14+), MySQL and SQLite (3.38+) as database backends

Getting started

The easiest way to get started is to run IoGT using Docker. Make sure you have Docker Desktop installed, or at the very least the Docker Engine and Docker Compose (which are included in Docker Desktop).

Check out the code.

git clone https://github.com/unicef/iogt.git
cd iogt

Create the database. By default, an SQLite database will be created. This will take a few minutes on the first run because the IoGT container image needs to be created from scratch.

docker compose run --rm django python manage.py migrate

Create a superuser account for administration purposes.

docker compose run --rm django python manage.py createsuperuser

Compile .po language files stored in locale/

docker compose run --rm django python manage.py compilemessages

Start the server.

docker compose up -d

Check the logs.

docker compose logs -f

Once running, navigate to http://localhost:8000/ in your browser. Log in to the admin UI at http://localhost:8000/admin/ - use the superuser name and password you created earlier.

Shut everything down.

docker compose down

Optional

Setting up test data

It is possible to automatically populate the database with example data for a basic test site.

docker compose run --rm django python manage.py create_initial_data

Optionally, create the main menu automatically as well.

docker compose run --rm django python manage.py autopopulate_main_menus

Configuring the Chatbot

Follow instructions here

Content transfer between sites

Articles can be transferred from other IoGT sites - details.

Adding new localizable strings to the code base

After adding new strings to the code base that are user-facing (see https://docs.djangoproject.com/en/4.0/topics/i18n/translation/ ), follow the following process:

1. Run ./manage.py translation_tracking
2. Review and update common/translation_utils/translation_status.csv. Strings that only appear in the admin backend don't need to be tagged as translate, only those that face users. See here for details.
3. Rerun ./manage.py translation_tracking

This process updates PO files as necessary and compiles a list of strings that appear in the translation manager in the admin backend.

Remark: The file common/translation_utils/translations.csv is NOT updated in the process. Even though it has a column "is in use", its data is currently not updated automatically, and similarly, new strings are not added automatically.

Apache Superset Setup and Configuration

• See here

Page caching

Page caching is provided by Wagtail Cache and is deactivated by default. To start caching pages, follow the activation intructions.

Elasticsearch

To use Elasticsearch as a search backend instead of the relational database see the setup instructions.

Migrating content from IoGT v1

Follow instructions here

Add/Update Package

Follow instructions here

Running Tests

Run the following commands:

make tests
make selenium-test

In parallel, for example, with 4 processes:

IOGT_TEST_PARALLEL=4 make test
IOGT_TEST_PARALLEL=4 make selenium-test

More details of the Selenium tests can be found in the Selenium test README.

iogt's Issues

Features to Drop in New CMS

Pages > Personae
Settings > Site Permissions

Use initial design from IoGT 3 prototype in version 2 Wagtail site

Using Django/Wagtail templating and template inheritance create a base.html template that includes the header and footer sections that are common to all pages.

The new design is currently online at https://iogt-pwa.web.app/

Corresponding code can be found in the unicef/iogt-pwa version 3 prototype. The article content and navigation in the prototype are currently hardcoded instead of being read from the CMS.

As CMS Admin I can manage site settings

Existing Features to Keep:

• Logo editing
• Checkbox to show only translated pages
• GA and FB Analytics Links
• Content Rotation Settings
• Social Media Footer Page Links
• Social Media Article Sharing Buttons

Unsure about Existing Features / Need to Investigate:

• Article tag settings
• Service directory API settings

Features to Add:

• Color Scheme

As a CMS editor, I can upload multimedia files on the site for users to stream from the front end.

Multi-media files can be uploaded as MP3 & MP4 format. These files can then be accessed when editing articles and loaded as feature content. When file uploads exceed 6MB a warning sign appears that the format may not be accessible on low end devices.

(bonus points: can the MP4 function as a banner on the homepage?)

As a CMS editor, I can add, edit and remove an individual user

All existing features remain.

In v3.0 and beyond it would be great to see information links next to each user group that provides a summary of the user group for the CMS editor to quickly understand user permissions assigned.

Draft Sections and Articles are shown in navigation

Sections and Articles that have not been published are currently being shown on the home page and in the navigation (but a 404 error is given when the article is clicked since the Article has not been published).

Community Moderator user group allows enrolled front-end users to assist with comment moderation.

They will have the ability to delete or respond to users with canned responses (developed by CMS editors on backend). In their profile, they can view the total number of comments moderated (total and in the last week).

Implement language translation and investigate improved tools

Site administrator resposible to translation of overall site.

Country office responsible for local translation of content.

Investigate user interface tools for working with translations, e.g. wagtail-localize.org

As CMS editor, I can add and remove footer pages

Footer pages may include terms & conditions, about IoGT, etc and are pages that appear on the site footer. They should be fully editable pages, as articles.

As a CMS editor, I can modify the site's logo with an image upload from my gallery

Image size specifications should be noted in the CMS as an information box / hover .

Basic search

Users should be able to search for content.

• Users can enter search terms in input box in the site header.
• Users are shown a page of results.
• Search index should use Elasticsearch as a backend.

As a front-end user, I can report comments that violate community policies

Community members can self-moderate by reporting inappropriate comments.
Reported comments appear as an alert in the CMS.

As a CMS Editor I can add a single featured content to a Section

Sections should have a "Featured Content" field which allows for the selection of one page (article, survey, poll, quiz etc.) that is within that section. This is optional, a section doesn't have to have a featured content.

On the site, the featured content would then be displayed at the top of the section page before anything else (subsections, child articles).

As the CMS Editor I can design custom fields in the user profile for front-end users to answer when registering.

When users create accounts, they can be asked to complete a series of custom fields. Only user name and password are mandatory.

As CMS Editor, I can edit features of sections

Sections and subordinate sub-sections should have the same features for customization. These should include:
0. Title
1. Image (which appears at the top of the section's landing page)
This is an image selection from the image gallery and allows for single image upload

2. Commenting state (open / closed)
As is.
3. Slug
This is an editable field that is pre-populated with the title of the section
4. Page title
This one line field determines the title that appears in the tab on your web browser. It is editable but prepopulated with the title of the Section.
5. Search description
This feature does not currently function or aid in improved search results on the site, but it should
6. Tags
This feature does not currently function, but should.
7. Color Setting
This is currently done very poorly. Users should be able to select from a color wheel or insert a hex color

the color sets the style for the whole section

8. Activate Up Next & Recommended Articles

9. Scheduled Publishing

10. Icon This is a new feature. The updated site design requires the use of icons to assist users w/ navigating through sections. Use of icons aims to ease the user experience for low literacy audiences. Icons should be pre-loaded into all sites from the icon library and available for selection. COs should have the toption to upload their own icons, too.

As a front-end user, when I click on a link that takes me off of the IoGT site, a pop-up may appear that alerts me that I'm leaving a free-data environment.

When users accessing IoGT on a whitelisted IP address click on links to leave the site, such as a link to a WHO article, they should receive a pop up alert that they may incur data charges. In the CMS, administrators should be able to toggle this feature off and on. This pop-up language should be included in the multi-language support of the site.

Investigate RapidPro integration

Standard Wagtail CMS Installation

This repository will be used for the development of version 2 of Internet of Good Things (IoGT).

First task - replace the current contents of this old repo with a standard Wagtail 2.12 install and enable Wagtail v2 API.

The intention is to document even these initial steps to allow everyone in the project to follow along with the process. Some other work for the v2 backend has already begun in parallel and will be brought over to this official repository.

As the CMS editor, I can create at least 4 children sections and support orphaned content.

The CMS should allow for Section > Sub-section > Sub-sub section > Sub-sub-sub section > article/poll/survey/quiz. Content can appear within any of the aforementioned parent sub-sections. Content can also be orphaned such that select pages are only accessible through direct link.

Improvements to IoGT design

• Tidy up current CSS for different media types
• Work with client to improve the overall design
• Ensure that navigation works for users who have JavaScript disabled (e.g. users of some feature phones)

As a CMS editor, I can configure robust surveys to allow front-end users to share insights or learnings.

Surveys should allow for a feature image to appear of surveys.
Surveys should allow for editable description to appear at the start of surveys.
Surveys should allow for the following field options:

• Single line text
• Multi line text
• Positive number
• Number
• Date
• Date / time
• Email
• Checkbox (i.e. 1 option, check to consent)
• Checkboxes (i.e. check all that apply)
• Radio buttons (multiple choice with only 1 answer)
• Drop Down (multiple choice with only 1 answer)

Positive number, radio buttons & drop down should provide options for skip logic within the survey configuration.

• All fields should be marked as required or optional (with optional as the default).
• All fields should have an associated "Admin Name" which relates to the CSV file header when exporting survey results.
• All fields should have optional, editable information / help texts (e.g. "check all that apply") which appear on HTML view as italic, light gray text and in javascript enabled views as an "i" with a circle around it. Text will appear on hover.

Polls, surveys and quizzes should have

• All existing validation / error messages when surveys are saved and required fields are not met. Such as adding a field without an admin name or adding a multiple choice field but not assigning any options.
• Editable submit and homepage button text that edits either the button to select when beginning a survey or submitting a survey. By default these read "start" and "submit" - and should be incorporated in the translation solution to ensure that all features of the site are translated appropriately.
• the option of sitting either within a parent section or sub-section or as a child to an article. The option to have a click-button to begin or direct display (either onto the body of the article as an optional content module or in the parent section or sub-section) should be available.
• the option to require login or accept anonymous responses
• the option to allow multiple responses per user or not
• the ability to format the confirmation page to include text ,images, media, and links as an article.
• Scheduled publishing options (post and expiry date)

User account registration / management

Visitors of IoGT sites should be able to sign-up for an account to allow for personalisation and community involvement.

• Visitors should be able to create a user account for themselves.
• As part of the sign-up process visitors may be asked a set of questions, which may be different for each site, and may be optional or mandatory.
• Users should be able to change details of their account (name, username, password)
• Users should be able to initiate a password reset process.

Publish content to multiple sites

• Automated process to make content available in single tentant sites

Requirement: Develop content and ready in multiple languages, then push Spanish version to Spain, Mexico etc. (in draft - in an orphaned section because countries organise content differently). The pushed version is a copy and doesn't make it's way back to the central editor, it can be edited/customised locally.

• Multi tentant site - simple system to author for multiple site simultaneously (causes issues)

Implementation of analytics

Implement Google Analytics and/or investigate a suitable alternative.

Currently access to Google Analytics is not giving updated/full information.

Some results are not currently appearing in Google Analytics and are being manually added to data studio.

• Frequently visited sections
• Types device
• User time on site
• Bounce rate

RapidPro is is using PowerBI.

There is a Google Analytics -> PBI plugin.

Poll and survey data analytics

As CMS Editor I can move content pieces (sub-sections, articles, polls, surveys, quizzes) on the site

The current reorganization feature in the CMS works well. CMS editors can select any content from the site and move it and all subordinate pages:

When "Move" is selected, you're taken to a page where you can navigate to the parent section or sub-section that you want to move content to.

It might be better, however, to have a master site heirarchy that allows editors to drag-and-drop sub-sections around. This would be easier for site managers at country office level to manage independently.

As a front-end user I can post comments when I'm logged in and reply to other users' comments. I can like comments.

This feature already exists, but the UX is not great: comments should appear align left v align right so that users can see what's a comment v what's a reply.

New feature: liking comments. Comments on smart phones and above can be viewed by most likes or most recent.

In v3.0 and beyond, users should be able to upvote / downvote and like comments.

Comments

Registered users should be allowed to comment on content.

• Commenting can be enabled or disabled on any page.
• Users can post comments.
• Users can reply to comments.
• Users can 'like' comments.
• Moderators can review, filter and export comments.
• Moderators can provide custom or canned responses to comments.
• Community moderators can flag inappropriate comments.
• Moderators should be alerted to flagged comments.

As a CMS Editor, I can configure quizzes for front-end users to test their knowledge

Formatted similarly to surveys but with the option to identify correct responses for positive number, number, date, drop-down, and radio button fields. The quiz feature comes with an automated feedback loop for users sharing their score. In the CMS, editors can craft messages for when the user answers incorrectly.

As a site admin, I can set data cleaning / data dump schedules for user profiles & survey data

This issue requires a conversation to hash out more.

Design clean-up

The existing markup and styling was copied over from Preact SPA that was developed by fairly inexperienced developers. Before further development of the front-end, it would be beneficial to review the existing markup and styles. Some deficiencies have been identified already:

• HTML duplicated to cater for different device types
• Inconsistent media queries, targeting slightly different device breakpoints
• CSS is not very well organised (might benefit from BEM methodology)

There may be other issues, and suggestions for improvement would be welcome for our consideration.

The app is designed to work primarily on low-end devices (feature phones and budget smartphones), but should support all devices. There is also a requirement for the app to be fully functional on devices/browsers that do not support JavaScript.

As a CMS editor, I can bulk upload users and assign them to a custom user group

As CMS editor, I can add / edit / save to draft / publish articles with robust features for promotion.

As CMS editor,
I can add articles to any parent section or sub-section. Article features which currently exist and should remain include:

• Title
• Feature Image
• Tags
• Body (with pre-formatted features for lists, numbered lists, text, images, media, and page links)
• Commenting Settings
• Recommended Articles
• [ ] Recommended Section (Edit by @dannyparsons: Not needed, see response from Laurie: Better to have a single recommendation and the editor can pick whether the link takes the user to a section or article)
• [ ] Featuring in Homepage (Edit by Danny: in discussion with Laurie agreed that this moves to a setting on the homepage instead of setting on each article)
• Status (Publish, Draft, Unpublished)
• Privacy
• Edit Lock
• Go Live Date Time / Unpublish Date Time

Features which should be removed include:

• Article > Content > Subtitle
• Article > Promote > Feature in Latest
• Article > Promote > Featuring in Section

TBD Existing Features with Unknown Functionality
-Social Media
-Tags for navigation
-Reaction Questions

• Article > Promote > Topic of the Day
• Article > Promote > Metadata
• Article > Promote > Common Page Configuration > Show in Menus
• Article > Promote > Common Page Configuration > Search Description

Features which should be enhanced include:

• Style code should be able to be overwritten in the body
• Additional formatting options for body text including multiple header formats (right now, there's only H1 which makes text bold)
• Adding a video or audio file should include an upload option, like as in images. Right now, you have to first load the files in the media section and then they can be accessed when editing pages)
• Hyperlinking text to internal or external pages. Right now, inserting a page link results in the page title appearing. This is not editable. The editor should be able to link text such as "Learn more about vaccines." and link to the page "About Vaccines."
• The CMS editor should be able to insert an image and have the image appear as a hyperlink.

New Features Include:

• Add a body feature to embed a survey directly on the page - not necessarily at the bottom.

Nice to have Features (v3.0 beyond) include:

• Explore multi-column formatting for desktop view

As a front end visitor, I can create and edit my account

Features to Keep in New CMS

Definitely keep:

• Settings > Redirect - you can redirect paths to specific sites or pages (e.g. redirect /new to a specific poll) https://docs.wagtail.io/en/stable/editor_manual/managing_redirects.html

• Settings > Site permissions - looks like you can modify access for users to the sites specified in Settings > Sites (mentioned below), looks similar to #16. Feature mentioned in #17

• Settings > Segment user groups - looks like a really bad UI for creating a group of used (which can later be used in segments) - could be very useful (hopefully with a change in UI if possible)

• Settings > Collections - can create collections of images, documents etc. and limit access to users or groups of users – would likely be useful in conjunction with the above
  https://docs.wagtail.io/en/stable/editor_manual/documents_images_snippets/collections.html

• Pages > Security questions - shows up at account registration time, we should keep it

• Pages > Tags - looks like there's a way to use these tags on the backend, but I'm not sure whether it refers to these tag pages or just tags that you type in elsewhere. It seems like it could be very useful if we figure out how to use it
  https://docs.wagtail.io/en/v2.0/reference/pages/model_recipes.html#tagging
  https://learnwagtail.com/tutorials/adding-tags-to-wagtail-cms-pages/

• CMS landing page > Segments - looks like a way to segment users into groups based on criteria like the time or day they visit the website or the number of visits to a page of the website – data from here could be quite useful, so we should keep it
  https://wagtail-personalisation.readthedocs.io/en/latest/editor_guide/segments_dashboard.html

• Article > Promote > Metadata - looks like tags again using which we can do interesting things on the backend, so we should keep this

• Article > Promote > Featuring in section - looks like it shows up along with the main section picture and description. For me the article I was testing on appeared in the main page with its parent page

Maybe keep:

• Settings > Sites - I believe this may allow one to test locally and modify the site that the [LIVE] box goes to – it may be worth keeping if we can figure out how to use it
• Settings > Persona - this may be a way to collect data about what subsets of users are looking at. If that's the case, this would probably be worth keeping
  https://pypi.org/project/molo.usermetadata/

Use Wagtail 2.11.x (Long Term Support release)

Modify the wagtail version in requirements.txt to use the Wagtail Long Term Support (LTS) release.

This is a minor code change, but it requires all developers to delete their local test database db.sqlite3 (so that Wagtail migrations can only be applied up to the LTS release) - and also update or delete their virtual environment.

The new requirement should be:
wagtail==2.11.*

As the CMS editor, I can set pages & sections to public, private to select user groups, or password-protected.

In the current CMS, editors can set any section, sub-section, article, survey other content's view permissions as follows:

This feature is used frequently and should remain. It is an important part of our 'proofing' which allows us to have content on the site under review by the Ministry without public visibility.

As CMS Editor I can reorganize site content

This includes both the reorganization of articles, sub-sections, banners, and footers to appear in a specific order (currently done via drag-and-drop within the parent section) AND also includes moving content from one section or sub-section to another. Content moving is fine as-is, but in v3.0 and beyond it would be good to see a content hierarchy or tree within the CMS that allows for easier drag-and-drop reorganization.

As a CMS editor, I can enable polls to allow for anonymous submissions.

Currently, polls function to allow users to vote and receive immediate feedback: how did their answer stack up against their peers or other community members? However, the current CMS requires users to be logged in before voting in a poll. The current CMS allows for anonymous survey submissions, but not polls. Polls should be configured to allow anonymous submissions but restrict multiple submissions from the same device.

Polls, surveys and quizzes should have

• the option of sitting either within a parent section or sub-section or as a child to an article. The option to have a click-button to begin or direct display (either onto the body of the article as an optional content module or in the parent section or sub-section) should be available.
• the option to require login or accept anonymous responses
• the option to allow multiple responses per user or not
• the ability to format the confirmation page to include text ,images, media, and links as an article.

As a comment moderator, I can review comments, export comments, filter comments and provide custom or canned responses

Offline access for users with intermittent network access

For users with intermittent network access it would be very useful to download content for offline use. Users with devices/browsers which implement features of Progressive Web Applications (PWA) could take advantage of Service Workers to enable this.

We would like to offer users a choice of how much content to cache for offline use:

• bare minimum: core assets + article text (no article images) for previously visited articles
• all content for previously visited articles
• bulk download of all content

As a CMS editor, I can upload files to the IoGT site for front-end users to download to their device.

As with multi-media files, I can upload files to IoGT and host them on the server. These can be inserted into articles as hyperlinks within the body of the article for download. When documents are added as part of the body of a text, they appear as a button, rather than hyperlinked text.

As the CMS editor, I have can create user groups and assign groups access to select sections of the site.

Comments are moderated w/ AI to protect users

As a data analyst, I can export survey, poll and quiz data by form or by user as a CSV file or as an external API to PowerBI

As a data analyst, I can export survey, poll and quiz data by individual survey, poll or quiz. For example, I can export all results from a single survey in a CSV file. I can also export results from all three form types for a single user or user group.

Adding group restriction to Page(s) results in an infinite loop

This issue is the result of an open bug wagtail/wagtail#4297 in the Wagtail project. It's very strange how this hasn't been closed yet.

Thankfully, there is a workaround on the same link for this.

As site administrator I can view the total number of users registered, users by group and export a list of users with key profile information

Note that currently you can export users but it goes to a predetermined email address which is now defunct. Users should be able to download to their device.

Customisable homepage

The homepage should be editable to be able to choose which articles/surveys/polls/quizzes appear on the homepage.

Currently this is done through editing the article by going to Promote > FEATURING IN HOMEPAGE

Instead, the desired functionality is for this to be done by editing the homepage and not by having to go to each individual article. Selecting items for the homepage should be done in a similar way as "Recommended Articles" are chosen where you can add articles/surveys/polls/quizzes through a search and be in a specific order.

As a CMS editor, I can load banner images such that content is promoted / featured on the IoGT homepage.

Examples of current sites relying on banners include:
ec.goodinternet.org
zw.goodinternet.org
sa.goodinternet.org
pacific.goodinternet.org

In the CMS, a banner section allows the editor to add, remove, & reorganize banners. Current banner upload screen is pasted below. All fields remain.

Adding subtitle text results in the appearance of text at the bottom of the rotating banner on java-enabled sites.

Features to drop include:
Settings > Show in Menu
Settings > Search Description
Promote

As a CMS editor, I can bulk upload images and organize them with folders or tags.

As CMS editor, I can visit the "Images" gallery in the CMS and bulk upload images. The image gallery currently only alllows for tags to be added after images are loaded and tags must be added 1 at a time. Tags should be bulk-added as part of the upload process. If possible, it would be ideal to have better organization options for images. The bulk upload feature works well right now, but navigating through images when you want to add them onto an article is more complicated.

As an example, when I edit an article and attempt to add an image to the body or as a feature image, I must manually scroll through the entire library to find the image unless tags have been added to filter results. Improved navigation through this image gallery would be ideal.

As the CMS editor, I can select the color scheme for my site using HEX codes

CMS editors can override the default color scheme for IoGT in the settings tab by selecting the default colors for various features and replacing them with a HEX code.