mysociety / research-repository

A place for mySociety to keep its research in a sensibly organised and presented manner.

Research Repository

mySociety and its partners do lots of research. This is a place for us to list that research in a coherent manner for easy finding and citation.

Development

Codespaces

The most simple development environment is open up the repo in codespaces.

This runs a prebuilt container with the data populated (the state that 'local development with docker and vscode' would eventually get to).

A Github action will push changes to the GitHub repository back to git.mysociety.org. git.mysociety.org is still the primary, and changes forced there will override the Github repository.

script/server will then start the dev server at http://localhost:8000

Local development with docker and vscode

After cloning the repo, open up the repo and (if the 'Visual Studio Code Remote - Containers' extention is installed), you will be prompted to reopen the repo in the devcontainer. Do this, wait for it to complete. Once the window opens, it will automatically run script/populate (will need to wait a bit longer for this dialogue to finish).

script/server will then start the dev server at http://localhost:8000

Local development with docker

Run docker-compose up. This will build an application container and PostgreSQL containers These will run in the foreground, so you will see console output in the shell from the containers. You can stop the containers by hitting control-C. If you'd rather run in the background, add the -d switch; if you do this you can stop the environment with docker-compose down.

You can then run docker-compose exec app script/populate to load a set of test data.

Accessing admin

The Admin password to the live site is available on mySociety password store - you can then create a user account on the django backend.

https://research.mysociety.org/admin/

To create a token to use the remote zip upload, use script/token <username> in the terminal.

Models and concepts

ResearchItem

The core of the site is the ResearchItem - this holds all the information about an individual piece of research.

It has a ‘published’ boolean that is required to be ticked before appearing on any public listing - but the page itself can be accessed through its slug at any time.

Featured items appear on the homepage and on the research page on the main mysociety site.

New items need an image for the page (hero) and a thumbnail - the thumbnail can be automatically generated from the hero on the admin page.

Outputs from stringprint can be uploaded as a zip into ‘archive zip’ - this will create required research outputs to link and the link to the TOC.

Less self explanatory fields:

• Table of contents url - an external json can be used to create a table of contents (mostly for use in stringprint imports) - this will then be stored in the table of contents cache.
• table of Contents cache - this stores the json from above - but can also be used for arbitrary markdown to create tables of contents for other kinds of documents. TOCs will appear below the abstract, but will be included in the abstract if the code [TOC] is used.

ResearchItems have Outputs, Tags and Authors

Output

Outputs control the links and files associated with a ResearchItem They can either have a link or a file associated with them and count the number of times they have been accessed. The Top_order refers to the order of items under the image at the top (left to right). -1 is do not display. The lowest value item is also the link in the image itself is clicked. Order - controls the outputs on the righthand bar (generally less important).

Tag

Tags are the way ResearchItems and their display are managed on the site. They are many-to-many, but their exact role is modified by their properties. They have a page that shows their associated items at /section/[tagslug]

• Display - unchecking allows hidden categories that are used for internal management but do not appear in sidebar of items (e.g. creating associations for embedding on other pages, or to raise the association of items for the purpose of the related items detector).
• Top order - if the category should appear as one of the options in the top bar of the ‘research’ page.
• Is series - adds a ‘part of [tag]’ near the top of a research item, for more tightly connected groups of ResearchItems.
• Display with years - if unticked, will not separate items by year (looks better if a category only has a few items).
• Featured - tags can also be featured, this displays on the featured page on the home page and embedded pages like a Researchitem (This is the one place the thumbnail or date is used).

The default tag (where the research link on the top bar leads - is set in the one ‘Site’ object).

Authors

Fairly self-explanatory but the admin a lot of different options and allow author names to be transformed into links to their site via a customisable link behaviour.

Pages App

Content on the rest of the site is also customisable via the admin.

• The single ‘site’ object controls the title, twitter handle and site description, as well as the default tag for displaying research.
• Page objects allow for non-research item pages of material. The html is embedded in the admin and has a customisable social image and description. A nav order above 0 will put it in the top bar.
• Links allow links to external sites to be put in the top bar.

Embedding in external sites

As explored on wiki page, items can be embedded in other pages - either as iframes or by reformatting the data.

This involves a very basic api for accessing data. For instance, referencing this url creates a json with the 6 most recent featured items:

/embed/featured/limit:6/format:json

. ‘Featured’ can also be any tag slug. These can also be stacked - for instance /blog-posts/community/ shows blog posts that are also in the community tag.

Excluding ‘format:json’ just displays a webpage, that can be used in an iframe (how the stringprint sites show related items).

research-repository's Issues

Citations should also be provided in BibTeX format

We already load the pages for publications with plenty of metadata, and provide a ready-to-go citation in Chicago style. We should probably include a BibTeX summary for those who know how to use it, eg for Participatory Budgeting:

@misc {rumbul_parsons_bramley_participatory-budgeting_2018,
    author  = "Rumbul, Rebecca and Parsons, Alex and Bramley, Jen",
    title   = "Participatory Budgeting",
    month   = "jan",
    year    = "2018",
    url     = "https://research.mysociety.org/publications/participatory-budgeting",
    journal = "mySociety Research"
}

TICTeC Attendees List

clean up ampersands appearing as & in titles

Looks like maybe titles are being escaped twice?
e.g. title before this section renders/reads as

Affluent Countries: The US & UK

...but note ampersands elsewhere in the running text are appearing OK (yay).

I'm raising this as an issue here just in case it's a consequence of your autoconversion process that I think was used to generate that (wonderful) page; it might just be a thing to fix in that one document, I dunno, in which case sorry for polluting the wrong repo with an issue.

Migrate to HTTPS

Via Google Webmaster Tools:

To owner of http://research.mysociety.org,

Starting October 2017, Chrome (version 62) will show a 'NOT SECURE' warning when users enter text in a form on an HTTP page, and for all HTTP pages in Incognito mode.
The following URLs on your site include text input fields (such as < input type="text" > or < input type="email" >) that will trigger the new Chrome warning. Review these examples to see where these warnings will appear, so that you can take action to help protect users’ data. This list is not exhaustive.

http://research.mysociety.org/

The new warning is part of a long term plan to mark all pages served over HTTP as 'not secure'.

Here’s how to fix this problem:

Migrate to HTTPS

To prevent the “Not Secure” notification from appearing when Chrome users visit your site, only collect user input data on pages served using HTTPS.

I'm not entirely familiar with the necessary process for this, but am happy to do the wiki reading to figure it out (I don't believe it's anything too nasty). @sagepe, can you foresee a problem with this? Does the submodule-pulling for static research projects cause an issue with this?

Item Page Design

On item pages for the research site (E.g. https://research.mysociety.org/publications/participatory-budgeting)

@zarino suggested moving the picture above the title (as it started) - but also changing the aspect ratio to take up less space.

As this site has also gained more content since the first round, it might also benefit from a general Design Think™ about if it's still displaying things nicely.

Upgrade Django

We're using Django 1.11, this went EOL on April 1st, so we need to upgrade.

The current LTS, 2.2, is supported until 2022 and seems like a reasonable target.

Seperating different kinds of content

There's currently a problem where blog posts look bad alongside the thumbnails for reports. While we've been talking about ways of making the blog post thumbnails nicer - thinking about it the deeper problem is that there's enough content we should be displaying these separately (allowing for different levels of authority).

In the multi-layer-tags - I've adjusted the structure of the site to allow tags to be used as filters - so different kinds of content can be presented separately in the navigation.

Currently this looks like this:

Which is functional, but needs nicer formatting of the UL to work as navigation.

@zarino : if you're on board with the basic approach, can making that look like navigation be an internal design task in a future sprint?

Implement script-to-rule-them-all

Bring research repo into convention on startup and population scripts https://github.com/github/scripts-to-rule-them-all

Make It Pop

The new research site (http://repository.dev.mysociety.org/) is currently based entirely off the GitHub pages theme (https://github.com/mysociety/mysociety-docs-theme), and could do with some design love.

Of particular note:

• We need to think of a good way to list research.
• Static pages (eg research strategy) currently just store and output HTML, so the potential is there to do Cool Stuff.
• Changing models to include things like hero images (if you want) is dead easy, let @jacksonj04 know.
• URL structure, navigation etc is completely up in the air. If you think something will work better a different way, we can change it.

Multi Year Display

Multiple years with one item should compress to a single line. e.g.

Bad social preview on pages

E.g. newsletter social card should be a bit nicer:

Cross report search

We now have a lot of reports, with a lot of information, but that's dependent on me remembering what report has what.

Stringprint outputs a json file that it uses itself for search - but this could be pulled together across all reports into a single site wide search. This could expand to cover linked blog posts and build a knowledge base.

Sharing buttons

Would be good to have sharing buttons on the repo pages.

Resolve virtual environment issue

Virtual environment currently sits above the main repository structure - would be nice to be able to move everything up a level.

@jacksonj04 : This was presumably in response to a problem in alternative structure, any more details?

Problems with FixMyBlock discovery report listing

1. The "Contents" links on https://research.mysociety.org/publications/fixmyblock-issue-reporting-tower-blocks-discovery lead to URLs like https://research.mysociety.org/sites/fixmyblock-discovery/#background which are a 404.
2. The pale blue summary box in the "Tower block safety and the law" section of https://research.mysociety.org/html/fixmyblock-discovery/l/5.9256.6ivu3.7jhzg.hsiey.5w_z- is a duplicate of the previous blue summary box about "Data on tower blocks in the UK".

Add Test Data

Add test data for initial population of test environments.

Server Error (500) showing under Related Research heading in Access to Nature Prototyping Report

Under the "Related Research" heading, at the bottom of the page at:

https://research.mysociety.org/html/prototyping-nature/

I currently see:

Server Error (500)

It's trying to embed

https://research.mysociety.org/embed/related:prototyping-nature-test-2

rather than, I assume:

https://research.mysociety.org/embed/related:prototyping-nature

Better handling for adding blogs (internal and external)

Overhead of adding in a single blog post is quite high.

Two components of this ticket:

• Checkbox to make a repository page 'skipable' where it adds no content (toc, multi formats) over just linking straight to the blog post. This will make it redirect to the URL of the only output.
• Similar to stringprint import, an automated flow for loading a mysociety blog post into the database (extracting hero, author).

Better social media pullout

https://research.mysociety.org/

Currently:

Make the output buttons look more like buttons or links

I always struggle to find these. I think it’s because my brain filters them out as "tags", like on a blog page?

Not sure what the ideal improvement is. I think it’s some combination of:

1. Putting them in a more prominent location, eg right below the report title
2. Adding more space between the links
3. Making them look more like buttons – more horizontal padding / less vertical padding, filled-in backgrounds
4. Making them look more like links – blue colour, underlined, with a little "file" icon depicting the format for each one

Social images not being shared over https correctly

Sort out favicon, robots.txt and sitemap.xml

These aren't currently being served correctly, and it looks like sitemap.xml isn't being generated. Should they be served from web/ or from static/, and do we need to review how at least robots.txt and sitemap.xml are being managed?