ietf-tools / www

A customized CMS for the IETF website

A customized CMS for the IETF website

• Changelog
• Contributing
• Install
• Requirements
• Deployment

---

Requirements

• macOS / Windows: Docker for Desktop
• Linux: Docker + Docker-Compose

Install

Quick start

First, clone this repository.

Run in Docker locally

This project uses Docker to build and run the code, both frontend and backend. So the only requirement to run it locally is a recent version of Docker with docker-compose.

How to run (with a database dump)

1. Obtain a recent database dump with name like ietfa.*.gz and place in docker/database/ directory. Otherwise, it will start with a fresh database.

2. Obtain and unarchive media files into media/ folder.

3. Run docker compose up. It will build and start the frontend builder (yarn run start) and the backend (python manage.py runserver analog), along with a Postgresql database. The first run will take a while because the database dump needs to be restored.

4. After the frontend compilation finishes, the website should become available at http://localhost:8001

5. Create a super user on Python application docker instance to access http://localhost:8001/admin

   docker exec -ti www-application-1 python manage.py createsuperuser

6. To destroy everything (i.e. start the database from scratch) run docker compose down.

How to run (without a database dump)

1. Run docker compose up. It will build and start the frontend builder (yarn run start) and the backend (python manage.py runserver analog), along with a Postgresql database. The first run will take a while because the database dump needs to be restored.

2. Create an admin user

   docker exec -ti www-application-1 python manage.py createsuperuser

3. Log into http://localhost:8001/admin

4. Create a new "Home Page" (page type must be Home Page) and publish.

5. Go to http://localhost:8001/admin/sites/ and select localhost.

6. Select the new "Home Page" as the root page and save.

7. The website should become available at http://localhost:8001

8. To destroy everything (i.e. start the database from scratch) run docker compose down.

Backend details

The backend configuration resides in ietf/settings/docker, inheriting some settings from base.py. Configuration is done with environment variables with sane checks, i.e. if a variable is required but not set, the application won't start.

Analytics settings

To send analytics data to Matomo set the following configuration options in the local settings file.

MATOMO_DOMAIN_PATH = "analytics.ietf.org"
MATOMO_SITE_ID = "<site id>"

On the frontend (separate from Docker if needed)

Note: Running these steps before the Docker instructions above may prevent the frontend docker instance to run properly.

Install NVM and Yarn and then run these commands from the repo directory,

First, install dependencies:

# Make sure you use the right node version.
nvm install
nvm use
# Install project dependencies
yarn install

Then, build the frontend static files using one of these commands

# Builds and watches the frontend assets (use this when developing).
yarn run start

# Builds frontend development (non-optimized) assets, without watching
yarn run build

# Builds frontend production assets.
yarn run dist

Other available commands can be viewed with

yarn run

Multi site setup

This repo contains code for both the IETF and IAB websites, which are intended to run as separate sites. If you don't have a DB dump, or your dump does not have the IAB site set up, these are the steps you need to have a paralell site running:

• Set up a page using the IAB homepage template at the root level (/admin/pages)
• Configure a site with that page as its root (http://localhost:8001/admin/sites/)
• In settings -> layout settings (http://localhost:8001/admin/settings/utils/layoutsettings/2/), select your new site and make sure that the base template is set to IAB
• Header and footer links are populated in the same way as the IETF website. The header contains pages that have the 'show in menu' checkbox ticked in the 'promote' tab. Footer links are set in settings -> footer links.

Upgrading dependencies

Dependencies are managed using pip-tools. They are specified in requirements/*.in and version-pinned in requirements/*.txt. To update the pins, run:

docker compose run --rm application requirements/compile -U

Deployment

Production: IETF, IAB

Staging: IETF, IAB

Dev (automatically build from main branch): https://ws-main.dev.ietf.org/

Testing changes on sandbox

• Use Build and release GHA.
• Select the branch that you want to deploy.
• Make sure Create Production Release is not ticked.
• Tick Deploy to Sandbox.
• Click Run Workflow button.

If the main branch is selected, changes will be deployed to https://ws-main.dev.ietf.org/

Changes will be available on a subdomain compiled from the branch name if any other branch is selected. If a / is present in the branch name, the first part will be stripped off and any other / are replaced with -. For example, deployment from the feat/foobar branch will be available on https://ws-foobar.dev.ietf.org, branch fix/foo-bar will be available on https://ws-foo-bar.dev.ietf.org and branch fix/foo/bar will be available on https://ws-foo-bar.dev.ietf.org.

www's Issues

Back out datatracker integration

The attempt to integrate with the datatracker has not been as useful as was envisioned early on and has created operational/maintenance headaches. It can be backed out.

Inconsistent elasticsearch results when searching to insert a link

[reported by Jay Daley] When inserting a page link or document there is often an error message that nothing matching the search text can be found even when a matching page or document exists.

This can be worked around by restarting the search or adding additional characters in the search box.

Remove “gray box” below “Blog” menu item when there are no secondary-menu items

When all submenu items are removed from the News & blog top-level menu item (via the Settings->Secondary menu controls in Wagtail admin), there remains a gray box appearing on roll over. See screen shots.

When there are no secondary menu items, there should be no gray box.

With the new approach to IESG statements, I believe it would be OK to remove the "News and Blog" menu options under the Secondary menu controls.

consider additional security for forms on www.ietf.org

Web forms on www.ietf.org currently do not have any mitigation against automated submissions.

Some amount of "spam form" is currently received.

Current published form pages (and submission emails) are:

https://www.ietf.org/forms/wg-webex-account-request/ ([email protected])

https://www.ietf.org/contact/contact-form/ ([email protected])

https://www.ietf.org/forms/keyword-suggestions/ ([email protected])

Additional information about the volume of spam is needed to understand the urgency of the current situation, but some systemic approach to mitigating form spam should be considered.

Change the blog applications use of querystrings to filter blogs to use segments in the URL path

The current scheme using URLs like /blog/whatever?primary_key=3&secondary_key=27 defeats CDN caching and the ability for editors to use wagtail's tools for linking to internal pages.

Instead, the app's use of the URL space should be modified to something similar to:
/blog/news/ (filtering to news entries)
/blog/<other secondary topic>/ (filtering to the set of secondary topics)
/blog/entry/whatever

The notion of PrimaryTopic can go away (news can be moved to a SecondaryTopic, which can just become Topic). Topics will need slugs.

Add new snippet to include "Note Well"

(EDIT: See #30 (comment) for the requirements)

The snippet viewer for MailingListSignup objects uses a custom view and template to show a page with Note Well text with the link to the signup page (or email address) for the mailing list in question.

At the moment, the template contains an older version of the Note Well text.

We have a Note Well page at /about/note-well. Perhaps this custom view could leverage the bits used to serve that page so there was only one local copy to maintain?

glossary page template update request

The glossary page (https://www7.ietf.org/about/glossary/) currently displays Glossary snippets on pages of 10 terms each (which then need to be clicked through through).

Instead, the glossary page should simply display all terms on a single page to make browsing terms easier.

(the page currently works as expected, so this is an enhancement request)

Blog page shows Roles inconsistently

At the time of report, https://www.ietf.org/blog/mud/?secondary_topic=8& shows "chair" roles for Eliot and Brian in the right side-bar, but not on the blog article itself.
If there is no role, the separator between it and name should be suppressed.

Mailing list signup Snippet does not work as expected.

A bug was reported on www.ietf.org via the RT queue ([email protected]) and I’m not sure how to handle it.

Here’s the pointer to the original ticket (the text of which is below):

https://www.ietf.org/rt/Ticket/Display.html?id=278525

Here are's what I've found after poking at wwwtest.ietf.org. The issue seems to be generalized with the “Mailing list signups” snippets.

Instead of linking to the URL specified in a snippet, the link is to something like:

https://wwwtest.ietf.org/misc/disclaimer/5/

(The last number will change depending on the Snippet.)

On wwwtest.ietf.org, I’ve tried creating a brand new Mailing list signup snippet using both the “pick a mailing list” and and it too has the problem. The example I created is at the bottom (currently) of these pages:

https://wwwtest.ietf.org/how/runningcode/hackathons/107-hackathon/
(Using QUIC in the WG mailing list sign up selection feature)

https://wwwtest.ietf.org/how/runningcode/hackathons/106-hackathon/
(Using the Sign Up field to specify a URL)

Screenshots attached.

previews of blog pages fail

The blog page templates currently use a collated published date for the article being viewed to decide which other blog page summaries to show in the right hand column. For unpublished blog pages, this leads to a crash on preview since coalesced_published_date will be None.

See https://github.com/ietf-tools/wagtail_website/blob/5a5115e583abe4d4dce83514ff1f68bcf5b326a8/ietf/blog/models.py#L253

right side blog index does not show most recent posts

The right side list of blog posts should show the most recent posts to the IETF blog, regardless of when the post being viewed in the main part of the page was published.

Instead, the posts shown on the right side seem to be the ones published just before the one being viewed in the body of the webpage. For example when viewing this post:

https://www.ietf.org/blog/meeting-network-monitoring/?

The more recent "IETF 106 Highlights" post should be at the top of the list on the right.

Looking further back, when viewing this post:

https://www.ietf.org/blog/blind-men-and-elephant/

the posts published just before it are visible in the right side.

Add the ablility to filter the full IESG statement index page by topic

See Subject

Return fewer other-blog pages for each blog page render

Currently each blog view page puts the titles and authors for every other blog entry into both the feed sectionwhat's used to show blog listings on mobile and what's used for the right scrolling sidebar. This should be limited to many fewer. The 3-5 most recent before the entry being looked at for the sidebar, and those in the last 3 months for the feed would be a good initial target.

There should be an index page where viewers can find all the blog entries.

Insufficient vertical space after last list item

The gap after the last item of an ordered list or unordered list is too small and so the following text is too close. Screenshots of editing and display view attached.

Integrate Matomo

Integrate matomo consistent with the Analytics Proposal

There are unused requirements

requirements.txt contains packages that were used in earlier versions of the project, but are now not required at all. At the moment 'celery' is an example. All the requirements need to be checked to see if they are still relevant.

Consider revising Area templates

The original idea behind Area pages was to provide a general overview of IETF areas with links to related items (working groups, recent RFCs, Active drafts) in the Datatracker. This fit with the general goal of connecting the public-facing www.ietf.org with the "work" in the IETF, as represented by objects in Datatracker. However, the implementation details of the page template (e.g. using alphabetical sorting to list RFCs and WGs) meant that the auto-generated content of the pages are not useful.

A mock up, just to provide an idea of what the templated does, is staged here:
https://wwwtest.ietf.org/topics/areas/security/

Ideally, there would be a way of keeping the general idea behind the pages, but reconsidering the implementation details so that the pages would be more useful or representative.

For example, is there a way to list all the WGs in an area? (maybe omit the description, but include the name, abbreviation, and links to the main page and charter page in Datatracker.

Is it possible to leverage the "Newly adopted I-Ds in WGs" information to highlight new work in WGs related to an area?

For RFCs, is there a way to list the most recently published Standards-track RFCs that were the product of an Area WG?

If there isn't a way to adapt the template so it can be useful, it might be worth removing it to avoid possible unintentional use.

"Show all" link on right-side blog index not clickable on short posts

When a blog post body is short and the post index right side of a blog post page must be scrolled to the reach it, the "Show all" link displays but is not clickable. This is not a problem on longer blog post pages when the "Show all: link scrolls farther up the page.

The link should be seen and clickable regardless of the length of the blog post.

An example is here:

https://www.ietf.org/blog/ietf108-online/
Attached is a screenshot.

Update list item styling

Currently, unordered list items appear in slightly smaller font and with insufficient vertical space after the last list item.

List item font should be same as regular paragraph and vertical spacing before the first item of a list and after the last item of a list should be the same as between regular paragraphs.

Remove badly defined mixin from the project sass files

See https://github.com/ietf-tools/wagtail_website/blob/master/ietf/static/css/mixins.scss#L330.
Note that the definition is commented out.

This results in sass compilation failures using any more recent versions of libsass (the error was not being reported by the version in use):

CompileError at /
Error: Mixin "keyframes" does not accept a content block.
on line 1074:14 of ietf/static/css/main.scss

@include keyframes(slide-down) {

-------------^

Can not or there is nothing to edit at https://www.ietf.org/admin/pages/1230/edit/

The page comes up in the wagtail interface and looks like its there (the live page is fine, has content and all) but there is no actual content in the wagtail edit page that can be seen or edited. The "Introduction" section at the top of the editing page is the only section that shows up with anything in it.

Consider revising Topic templates

The Topic templates were intended to work with Area pages to describe and relate general topics being discussed in "the general market" (e.g. IoT) to the work of the IETF. However, the non-use of Area templates has meant that the layout logic of the Topic template no longer applies. For example, the peer-page menu provided on the side is too far down to be useful, and the space in the main part of the page body intended for Area pages, now is blank.

Here is an example of the Topic page template in action
https://www.ietf.org/topics/netmgmt/

Here is an example of the Topic page template how it (roughly) was intended to be used:
https://wwwtest.ietf.org/topics/netmgmt/

There would still be value in connecting general topics of interest in the Internet (IoT, new transport technologies, Privacy & security) with the work of the IETF, but it may be better to consider how that might be done differently. For example, could Blog posts (or other pages) be tagged to relate to one or more Topics of interest with the latest 3-5 (Page title and Search description) being listed in the body?

Styling around the next/previous entry controls on blog pages is broken

The container for the next and previous entry boxes at the bottom of the page extends all the way across the screen and is above the containers for the blog entries in the right scrollbar.

So, if you mouse to the right of the "Next" box, the cursor sees no link (try moving the mouse on a blog entry just above and just below the top of the Next box).

If the blog entry is small enough to fit entirely under that container, you can't click on it. If it happens to be the last entry, there's no way to move it to where you can click on it.

Datatracker search function has a typo in constructed URLs

The rcs parameter should say rfcs

docker-entry.sh has a too-agressive chown -R

The chown should be made much more specific. The mountpoints for volumes should be moved out of /code.

links to RFCs broken

It looks like links on several pages to RFCs that were previously linked no longer work.

For example the link to RFC 2026 at the top of:

https://www.ietf.org/standards/process/informal/

Does not work, though it still appears to be a link. Here's a snapshot of that page where the link works:

https://web.archive.org/web/20190912203705/https://www.ietf.org/standards/process/informal/

It is also the case that in the Wagtail editing interface, the text looks like a link, but it is not possible to edit the link by choosing the "link" icon (see screenshot attached). The link can be undone by clicking the "unlink" icon.

This appears in a variety of links on this page, and also on other pages ( ) but not for every link. For example at the top of https://www.ietf.org/standards/process/ of the four parts of text that look like a link, 2 links work and 2 do not.

Make date display on Event page templates optional

A feature of the current www.ietf.org wagtail site is that it automatically populates the “Upcoming events” section of the front page based on the dates indicated in pages using the Event page template. This removes the need to manage manually that part of the website.

However, when it is desirable to feature event pages other than the next 2 upcoming, the date field in those pages must be manipulated to get them to display on the front page.

This mostly works (see the screen shot of “Interim meetings” now featured). However, a side effect is that the “fake dates” are now also shown on the Event page itself, even though those dates may not be strictly correct. (See the screenshot of the current Interim meetings page.)

One way to address this would be to make the date fields optional to display, with (say) a check box in the editing view to indicate whether or not it should be displayed on the page itself. This need not affect the behavior of other page templates that use the date information. (I don’t think the Event date information is actually displayed on any other page at the moment).

The view for BlogPage is database inefficient

The view needs to make better use of prefetch and similar tools to cut down on the number of queries generated by each page view

Pagination of filtered glossary items which render in one page makes no sense

At https://www.ietf.org/about/glossary/?query=Blue%20sheets, there are stanzas that look like:

`

                            «
                        

                        1

                        
                            »

`
These are artifacts of paginating glossary search results. When there is more than one page of results, these blocks have links to the other pages. When there's only one page, however, the block is meaningless ans should be suppressed.

CSRF cookie error for cached Form pages

On this page:

https://www.ietf.org/topics/keywords/

When a form page (https://www.ietf.org/topics/keywords/keyword-suggestions/) is accessed via the call to action snippet, submitting the form returns a CSRF error. (screenshot attached) for a submission that seems valid.

When the same form page is accessed via a link embedded in the "Suggestions for additional keywords” text on the page, the form operates correctly.

The form page should work the same when accessed from any link on the www.ietf.org site.

Date filtering on iesg statements is not functioning

The new iesg statement index page pays attention to querystrings (related to #10). The queryset is missing an annotation.

On the post pages, the filters accept dates, but don't construct querystrings in the results URL.

The post pages aren't providing a datepicker.

Migrate to Django 2

Recent Wagtail releases require Django 2

suggestion for RFC Snippet adder dialogue

For the "NEW RFC" webpage (https://www.ietf.org/admin/snippets/snippets/rfc/add/) there are currently no indications (AFAICT) about what values ought to added for the NAME and TITLE pages.

It would be helpful to have a tip similar to what appears for the RFC field. Screen shot attached to illustrate.

Add log rotation to the container

Right now, the containers logs are growing monotonically

Print CSS for key pages (I-D checklist, Guidelines, Tao, etc.)

Particularly for long pages such as the Tao and the I-D checklist and guidelines, the website should offer a more streamlined version for printing. Attached are examples of the current printed version and a better approach provided when using "reader view" in Safari.
READER- IETF | The Tao of IETF: A Novice's Guide to the Internet Engineering Task Force.pdf
PRINTED IETF | The Tao of IETF- A Novice's Guide to the Internet Engineering Task Force copy.pdf

404 page served through CDN should not mention triggering URL

CDN caching of www.ietf.org makes the wagtail 404 page incorrect in most cases: It mentions the URL that triggered the 404, but if the page has been cached by the CDN previously, the linked URL shown from cache will most likely be wrong.

I just followed https://ietf.org/how/to/cook/eggs/, and the wagtail page indicated that the URL I'd followed was something else: https://www.ietf.org/rfc/rfc1738.txt (because that's the latest version cached by the CDN). So it might be argued that we should fix wagtail's 404 page to either not mention the URL, or to not be cached by the CDN.

improve form resiliency to spam

There are spam avoidance techniques we are not yet taking advantage of.

Some of the share providers are returning links to www7

Find and eradicate the places www7 is leaking into what's being served.

Several pages are not currently editable

We are running into the drafttail issue discussed in this thread:
wagtail/wagtail#4602

Around 68 StandardPages are affected.

Re-implement custom snippets

See https://docs.wagtail.io/en/v2.8/releases/2.2.html#upgrade-considerations.

Support for the deprecated behavior is removed in wagtail 2.4.

This is currently used by the snippets choosers.

Odd behavior with filter dropdown on blog pages

If you load /blog, then click on either the filter box (labelled 'All'(, then click that box again, the way the list of options is exposed changes (it does not go away), and the behavior changes such that if you move the cursor to the bottom of the list, the main text gets reflowed below the list.

resize of page causes body content to be pushed down

More details here:
https://mailarchive.ietf.org/arch/msg/tao-discuss/TCv732G_QZQl4jEEbz81IDPNJrY

"Appears to be a javascript issue"

Also, with javascript off, the "drop down" section menu feature does not really render very elegantly. Screenshot attached.

Ordering of "next" and "previous" blog pages reversed

At the bottom of a blog page, the "next" button links to the blog page that was published just before the current page. The "previous" button links to the blog page published just after the current page. Wrapping around to the start/end of the list is consistent with this ordering.

This seems backwards to me - I'd expect "previous" to be the page published before the current one.

Apologies if this is intentional and I've misunderstood the intent.

Move closer to the current wagtail release

We need to move closer to the current releases of wagtail. That will require moving to django2 (see #20).

Wagtail was at 2.6.1 on the date this issue was filed.

A new page with any snippet content will fail to save

Creating a new page with content that includes a snippet fails to save with an error that looks like:
save() prohibited to prevent data loss due to unsaved related object 'page'.

This is probably a longstanding but hidden bug exposed by the python 3 upgrade.
It highlights the need for regression tests (of which there are currently none).

The qcluster-based pages to refresh bits of the datatracker mirror do not work

There are admin pages that show counts of the objects in the classes that mirror datatracker classes. They have update buttons on them. The code behind that update button does not work correctly. All of this infrastructure should be removed (the need for qcluster will be removed with it).

At the moment the mirrored objects are maintained by a cron job.

In the long run, the interface with the datatracker should change such that the mirrored classes are not needed (at least in their current form).

Inelegant responsive resize mode

There is a point in resizing the viewport of a page when the header extends full width but the page has no right hand sidebar the page body does not use the full width but leaves a blank sidebar.

Linked text in page bodies does not linewrap

linked text in page bodies does not line wrap, even when there are spaces included.

This causes odd-looking line breaks, which seem to be in error.

Text should wrap as it would if it were not linked, for example at a space between words.

A screenshot shows an example from a recent IETF Blog post:

https://www.ietf.org/blog/ietf108-irtf/

Another screenshot shows an example of the desired behavior from:

https://daringfireball.net/linked/2020/08/26/covid19-aerosols

Search queries with embedded nulls cause 500 error

A null in the querystring of a search will causes a 500 error (due to psycopg/postgres implementation decisions). Such searches should just return the empty search result page.

Wagtail error/exception handling needs improvement

NOTE: This ticket and the text below was originally copied from the Torchbox ticketing system:

https://projects.torchbox.com/projects/ietf/tickets/211

which requires a password and may no longer be active.

-Greg

++

Hi Tom -

Yup. Check the logs. I should have done that first. Sorry. But it turns out I still would have had to contact you. There was indeed a configuration issue, on our side.... but...

The proximate cause was that Wagtail was sending from "[email protected]", but use of that address was restricted to Mailman only. So the SMTP library got a rejection message from Postfix (mail transport.)

The portion of the log that illustrates this is:

File "/usr/lib64/python2.7/smtplib.py", line 747, in sendmail raise SMTPRecipientsRefused(senderrs)
SMTPRecipientsRefused: {u'[email protected]': (553, '5.7.1 [email protected]: Sender address rejected: not logged in')}
I've opened permission for that address, and the process works now.... for now....

But I suggest there still ought to be better exception handling for this than a crash. If there is a problem with sending the email, a more polite failure message, and a referral of the user to https://www.ietf.org/contact/ with a request to contact us, would probably be a more desirable option.

I don't know the current status of anything, but any time there is an interaction with server processes, there ought to be some kind of try/catch handling. I think? I leave it up to you to determine what's best here.

Thanks so much!
Glen

+++

Hi Glen,

We think the best approach is to design and implement an enhanced 500 error page for internal server errors. We've spent around an hour looking into this for you this month.

If you are happy with this approach, we'll spend further time providing a design and development estimate for the work and let you know how long we expect this error message to take.

To speed up the estimate, it would help if you could provide some messaging for the error message - or if you prefer we could recommend some copy for you.

Many thanks,
Beth