keithamoss / demsausage

Democracy Sausage

This is the source for the govhack-science website highlighing sciency datasets at govhack-science.github.io.

Content served by Jekyll, with layout and nice bits stolen shamelessly from Rasmus Andersson.

Planning

GovHack Data Model

Documentation

• Jekyll
• Liquid Templating Engine
• Liquid For Designers

Requirements

• Ruby
• RubyGems
• NodeJS
• Python
• Jekyll

Installation

https://jekyllrb.com/docs/installation/ https://jekyllrb.com/docs/github-pages/

Our friends at GitHub have provided the github-pages gem which is used to manage Jekyll and its dependencies on GitHub Pages. Using it in your projects means that when you deploy your site to GitHub Pages, you will not be caught by unexpected differences between various versions of the gems. To use the currently-deployed version of the gem in your project, add the following to your Gemfile

The best way to install Jekyll is via RubyGems. At the terminal prompt, simply run the following command to install Jekyll:

$ gem install github-pages

All of the GitHub pages flavour of Jekyll, and its gem dependencies are automatically installed by the above command, so you won’t have to worry about them at all.

Run

https://jekyllrb.com/docs/usage/

jekyll serve

If you receive a message like:

Could not find gem 'github-pages (= 67)' in any of the gem sources listed in your Gemfile or available on this machine. (Bundler::GemNotFound)

Then it means GitHub has upgraded their github-pages gem version, so just run bundle install from the root directory to upgrade to the latest set of deps GitHub is using.

demsausage's Issues

Address geocoding cost issue

For the Victorian election users used our Google-based geocoder to make:

• 38.2k autocomplete requests ($153)
• 5.7k geocoding requests ($40)

This didn't cost us anything because we get $200/month free from Google, but the total cost was $193 for the month. With NSW having a greater population we can expect this will start to actually cost money for their election.

Let's look at what we can do to address this.

e.g. By reducing how much we use the autocomplete geocoder e.g. Only when the user stops typing for a few seconds OR only when the user hits enter

If we want to look at other geocoders @stevage has a list here: https://getlon.lat

https://addressfinder.com.au came up at (you only get charged for geocoding requests, not autocompletes). Based on current pricing, we'd need the Pro 3 plan and pay $326 for 5.5k geocodes.

AB#116

WIP Todo List

Foo

• Bug Fixes
  • What caused the Failed to fetch error? Maybe make the setTimeout hack more robust?
  • What caused Type error?
  • What caused Network error when fetching?
• Enhancements
  • Cache the MapBox Static Images in the DB to avoid unnecessary use of our API quota OR nix them and design a different chooser UI
  • Cleanup has_other from Canning and earlier
  • Why isn't Mitcham PS turning up in search properly (c.f. Twitter)
  • Better loading screen

After Tasmania

• Load SA polling place data
  • Reply to email from St Gabriel’s
• Questions
  • What's required to make admin work better on mobile?
  • Historically are there (m)any polling places that have foods that aren't sausage/cake, but no sausage and cake?
  • Google logout issue
• Migrate to new infrastructure
  • Can we keep the API on GS if we move the two static sites?
  • How to handle things that require cron job-type functionality (emails for admins, database backups)
  • Move the two static sites to S3 behind CloudFront with automatic deploys from Travis.
    • CloudFront vs CloudFlare. Does CloudFront do the fancy gzipping and content optimisation that CloudFlare does?
  • Add HTTPS versions of the site to Google and Bing's webmaster tools.
• Enhancements: Admin Console
  • A button to manually trigger GeoJSON generation for an election
  • Daily emails for admins
  • Add a link to the main site on the Admin Console's ResponsiveDrawer
• Enhancements: Public Site
  • Add geolocation for finding stalls (+ Google Analytics events)
  • What are the pre-conditions to get fullscreen via manifest.js?
  • Log Google Analytics events for using the ElectionChooser and loading an election (or part of Election Chooser?) and finding a polling place
  • Make the election chooser change the URL to something human readable like /south-australia-2018. This would also make the back button work with tabs.
  • Make stall descriptions a textarea
  • We're not showing the free text field on finder or querying a polling place
  • Clarify the Add Stall stall description field to talk less about food and more about the purpose. Add the free text field as an additional entry option for e.g. cold drinks
  • Add GA events to map querying
  • Think about election switching across Map, Find, and Sausagelytics wrt. active elections and historical
  • Add an "No active elections" state to Add Stall. Maybe telegraph upcoming elections?
  • AddStall as a Stepper?

Sprint to the line

• Make selecting polling places on the map work
• Quick tidy up of the submitted/approved email notices
• Quick SQL injection/unicode test
• Make BottomNav active icons work
• Move first_report/latest_report to the server and capture unofficial polling places
• The "sometimes polling places don't load on the map until you pan/zoom" issue
• Client-side error capture
• Server-side error capture
• Google Analytics
• Does import deal with detecting dupes? Does it fail or warn/check? (It auto merges and flags errors.)
• Add static pages
• MVP of desktop responsive view
• Fix the issue with has_bbq in markPendingStallAsReadAndAddUnofficialPollingPlace()
• Display stall info on the stall cards too
• Deploy and test on more browsers, devices, and an iPad
  • AppBar doesn't fit the logo + title on an iPhone 5S (Chrome)
  • EmptyState on Finder appears whilst there are GoogleAutocomplete results
  • Clicking on items in the ResponsiveDrawer doesn't close the drawer
  • Maybe disable autoFocus on AddStall
  • Clarify that we won't spam you
• Create Tassie election. Test end-to-end process.
• Ship it! Announce!
• Is CloudFlare working on the main domain?
• Update "Add Stall" form to search the actual polling places if polling_places_loaded is true.
• Get and load polling place data for Tassie
  • Website description issue on Google and Bing
  • Problem editing http://admin.democracysausage.org/election/15/polling_places/61/edit
  • Fix setting first_report/last_report for all polling places loaded
  • Fix pendingStallCount notice on ElectionPollingPlaceLoaderContainer
  • Make the unknown icons appear below everything else
  • http://admin.democracysausage.org/stalls/12/ breaks if has_other is null
  • Data load issue: [Divisions](https://www.tec.tas.gov.au/House_of_Assembly_Elections/StateElection2018/PollingPlaceLists.html) didn't come with the KML file
• Smarter pre-fill of info ("Merge"?) to make the approval process faster for high-volume elections. The goal being that users can approve stalls on their phone in < 1 minute.
• Support multiple active elections at once
  • Support automatically turning off elections. Is the whole concept of active even valid then? c.f. Election opening dates
  • Add an active election chooser to AddStall. Update all uses of currentElection to use stalls.elections_id. Likewise, map querying. GeoJSON loading how? Display an election card even if there's only one election? How does the public map allow choosing between elections?
  • Admin UI
    • Refactor elections.elections
    • Update reference to elections.current_election_id
    • Add short_name as an editable field
    • Add react-responsive for better Drawer and BottomNav (Election, Pending Stalls, Polling Places)
    • Review responsiveness on mobile
    • Update Pending Stalls to group by election
    • Update Stall Approver to clearly show the election being edited
    • Can we refactor to make the "two elections on one day" workflow nicer for admins?
  • How to support having a "primary active election"?
  • Add an 'active election' indicator to the public site?
  • Setup Batman By-election, Reply to email, and Tweet promoting SA and Batman being live
• Make BottomSheet sticky
• Support 'Has Run Out' icons again

Adding stalls before polling places are loaded

• Add polling_places_loaded INT DEFAULT 0. Update interface and SQLite CREATE TABLE definitions.
• Add a control to switch polling_places_loaded on the elections manager
• When an election is created, create an empty table for it. Ensure an empty GeoJSON file is also created.
• Check code that relies on db_table_name being empty to work - it will always be set now
• Create "Add Stall" form with an address lookup if polling_places_loaded is false.
• Briefly tells users why address lookup
• Try to pull the place name for polling_place_name from address lookup. Also state.
• Update the pending stall approver to check for duplicates (by location) and prompt the user to edit the relevant polling place. Block traditional 'approving'. Allow to 'delete'.
• Deal with two dupes in the queue and approving one?
• If polling_places_loaded is false, APPROVING a stall will create a new 'fake' polling place in its table. Update GeoJSON. Also update its polling_place_id so it'll in turn be updated by the import process. And null out stall_location_info since it's now tied to a polling place that will be migrated by the import process.
• Can we safely remove division from the required fields for a polling place? Ditto polling_place_premises from stalls.
• What about running the first import while there are pending stalls that are unofficial
• During batch loading of new polling places: Add a 'CHECK' status that logs the old (user-entered) name and address and the new official name and address.

Admin Console

Manage Elections

• Create Election, including:
  • Election opening date (the date from which we accept polling places)
• Edit Election
• Set Active Election
• Add/replace polling booths for election: GUI or script -> both to automatically update/replace in Carto. Complexity re updating booths closer to release date when we've mapped stuff already?

Polling Booths

• Better "Approve and Save" workflow for stalls
• Search for polling places (autocomplete)
• Edit Booth
• Edit Polling Place Type en masse
• On approval, the user that submitted the stall should be emailed a confirmation
• Polling booth info card
• Stall info card
• Record user name, email, booth location (at least LGA - for targeted comms) once booth is approved
• Automatically derive polling place type from stored data (e.g. list of public schools) and historical sausage data
• Predictive chance of sausage based on historical data, maybe on type of polling place?
• ESS / Snag Voting integration for Grant's fundraising efforts. Ads?
• Sticky polling place search box for faster turnover
• Focusing the polling place searcher should highlight everything in there (efficiency)
• Loader: Handle duplicate polling places (Usually caused by the next task...)
• Loader: What to do about polling places spanning multiple or all divisions?
• Write better copy for the "Stall Submitted" and "Stall Approved" emails.
• Download as CSV/XLS

Public Site

• Make it faster and more lightweight!
• Close accepting booths once election day is over
• Election opening date (the date from which we accept polling places)
• What can we do to make dupey polling places (e.g. Uniting Church) more obvious as to which one the user has selected?

Fix the Safari iOS bug on the Add Stall screen

Check commit history. Did we already quickly patch this?

And I think the email field requirement is tripping folks up. Handle required fields failing validation in the browser better if they're at the bottom of the screen.

Make the frontend work (mostly) without a backend

Future future goal.

When we do work on updating the DemSausage frontends the goal should be to make it functional without any backend server (for key functionality like search and querying). Secondary functionality can just fail gracefully (like adding a stall == an email link).

AB#125

Enhancements to polling place loading

• Add a diff check before loading that the user has to confirm. Useful for getting an idea of how many polling places are changing for the nth load.
• Add a report of which unknown fields were dropped.
• Support loading JSON?
  • Think about the workflow here a bit more. Historically, we've often had to munge addresses a bit in Excel first. So a JSON ingest doesn't really help much here. Automatic address munging would, though - can that be automated?
• Add a geocoder that tries to (1) geocode based on data from past elections, and failing that, (2) based on the Mapbox geocoder. If the geocoder returns two reasonably confident results then display a GUI for the user to define which is the right location.
  • Think about the UX of a geocoder.
  • Should it be part of the loading process or a distinct first step?
  • Test based on Qld 2017 data (in which we had issues a few issues using a batch geocoder that put a few polling places in the wrong location).

AB#130

Cron timings

• Reminders at 8AM and 8PM
• The 8AM reminder only 2 weeks out from an election

Think about how we're using the information we collect about the purpose of stalls

We collect a lot of info about what the stalls are raising funds for, but then we don't do anything with it except for showing it on the site.

Have a think as a team about how we can better highlight some of these through social channels, on the website, et cetera.

AB#118

Supporting API integration by third parties

• Advertise the API in a high-level sort of way
• Disclaimer for API users about what we expect (see WhatsApp for examples)
• Disclaimer for folks submitting stalls warning of fact and maybe pointing to other disclaimer
• Decide data licensing position
• Looking into DRF's support for API keys and throttling
• Consider CORS changes that would be required for third party sites to integrate
• Add drf-gis fields to the API doco (now possible tfranzel/drf-spectacular#38 (comment))

Tech Options

• Roll our own vs use something like RapidAPI

Documentation

• How permanent polling place and stall URLs work and caveats therein (#18)
• How polling place loading works and caveats therein (per G)

AB#127

Line breaks in stall descriptions aren't carried across to the stall info cards

Allow people to embed the Democracy Sausage map on their site

Media sites like to embed us via screenshots - but they don't always credit us or take a good screenshot.

If we can make their jobs easier and offer an easy embed functionality we might be able to prevent some of this.

Considerations:

• Do we build a little static maps API that generates a point-in-time snapshot they can link to as an image? (With our logo et cetera)
• Do we build a traditional <iframe> embed view of the site that they can drop into their CMS? (This means we'll need to deal with the additional load.)

AB#115

Give stalls and polling places on the public site permanent URLs

Right now ourselves and users can't link to stalls or polling places directly, only to elections. We should be able to link to the polling place details card through a permanent URL like:

https://democracysausage.org/victorian_election_2018/polling_places/Name/Premises/State
https://democracysausage.org/victorian_election_2018/polling_places/ec_id (where available - depends on the electoral commissions)

And to stalls through a permanent URL like:

https://democracysausage.org/victorian_election_2018/stalls/stall_id

Using the polling place name as the unique key will avoid issues with the polling place ids changing and make integration by third parties easier (no ids to update, and they'll have this info anyway).

Issues

Polling places: Upon investigation, combining name, premises, and state is the only way to get a globally unique polling place identifier. Proposed URL format revised.

Considerations

• Poling places: Do we only support polling place name or premises as well? We could support both under the same URL and just try name first?
• Polling places: Polling place name is unlikely to change after the initial load, but premises name may. This makes it not really a permanent URL, but close enough for our needs.

Design

• Both URLs will end up at the same UI screen, but will fetch their polling place info differently.
• The UI should display a "View on map" button.
• If a stall is is still pending its link will work, but we'll just display the standard "We don't know anything about this polling place yet" message.
• We'll need a nice error state for "Couldn't find polling place" to catch edge cases where the polling place name / premises changes.
• People submitting stalls should receive a link to their stall in the "stall approved" email
• Users should be able to get a link to a polling place when they query it
• Admins should be able to get a link to a stall when they're approving it
• Admins should should be able to get a link to a polling place when they're editing it or viewing the favourites list

Test URLs

Web UI

https://test.democracysausage.org/federal_election_2019/polling_places/Bayswater%20North/Hillcrest%20Primary%20School/WA/

https://test.democracysausage.org/federal_election_2019/polling_places/8143

https://test.democracysausage.org/federal_election_2019/stalls/1000

API

https://test.democracysausage.org/api/0.1/polling_places/lookup/?election_id=27&format=json&name=Bayswater%20North&premises=Hillcrest%20Primary%20School&state=WA

https://test.democracysausage.org/api/0.1/polling_places/lookup/?election_id=27&format=json&ec_id=8143

https://test.democracysausage.org/api/0.1/polling_places/stall_lookup/?format=json&stall_id=1040

Fix the iPhone bug on the Add Stall form with the submit button

Users on certain models of iPhones still can scroll down to the bottom of the page to add a stall.

Verify in Safari, Chrome, and on H's phone.

Make a new COAS logo sans the reference to ESS

AB#157

Notify the administrators when a stall is submitted on election day

At the last few elections we noticed folks would use the Add Stall form to send in information about stalls on election day itself (and election-day-eve too).

Alas, we don't monitor that part of the admin site a lot on election day - so it's easy for us to miss them.

Let's think about how we can notify administrators about pending stalls that need approving election day. A few ideas:

• Add logic that emails us each time a stall is added if current_day = election_day. This could be a whole new email or BCCing us on the "You stall has been submitted" email that users get.
• Fancy push notifications to browsers (once Scremsong is up and running and we have the infrastructure for that).
• Updating the favicon for the admin site with a number indicator badge so they can see it if the tab is open.

AB#109

Add a simple counter for # of stalls / # of polling places

Somewhere on the Admin Site.

Dashboard (root) or Election Manager.

After South Australia

• Did Add Stall close down automatically at 8pm on election day?
• Check the raw access logs for the API
• Write requirements for the Twitter GUI with H
• Handle nuking all of those spammy links so they actually 404/301 never was here
• What to do about basemaps for the next Federal? Finding a balance between basemap and geocoding services could be tricky. (How does Carto measure page views in their 75k limit?)
• How can we get to event value in GA?

General Bug Fixes

• Find a better fix for the GeoJSON not always loading issue

UI Fixes

• Fix the main UI containers so they only fill 100% and take up the proper amount of space
• Find a better fix for the Fruit phone BottomNav gap issues (Safari Responsive Design Mode ftw)
• page-content on iPhone 5S (Chrome) doesn't include the bottom padding to allow scrolling down. (Hackily fixed for now by adding 100px of padding to the bottom of all pages.)
• Better flag errors on AddStall (e.g. Failing to provide Contact mail)
• LGG3 doesn't appear to load Roboto

SEO

• What can we do with server-side rendering or similar to make it more Crawler friendly? Is there a simpler approach? Is it even required these days?
• Submit a sitemap to Google et al.
• Investigate SEO tweaks - we're not even on the first page of Bing

For Consideration

• What could go on the Admin Console homepage that would be useful to see? (Alternatively, default to an existing page)
• Did we ever load that public school dataset?
• During polling place loading, what happens if there's a difference in two historical data points?
• Would a "View on map" option be useful on Finder and PollingPlaceQuerying?
• New feature idea: Let me know when there are stalls available near me. For people in the lead-up - either tell them election eve or election morning. Email or notification. Look at traffic patterns and events prior to validate.

New Features

• Filter to find vegan et cetera
• A place to show images et cetera that we want to highlight (per election? per polling place?)

UI Enhancements

• Include more prominent calls to action to:
  • Submit booths in the critical pre-election period (form-based)
  • Report booths on election day
  • Look at our Redbubble!
• Think about how we can add clearer call to action when users discover a stall. The message is passive at the moment.
  • Think about how we can use the high/med/low chance of sausage as a prompt for submissions. Think about how we can better engage and prompt users like "Do you know about this stall?" "Can you help with this stall?" Maybe link to a "How to help us collect data" guide (Talk to P&Cs, Google)
• Revamp the design of PollingPlaceCardMini. Stalls without info overshadow those with. (c.f. Google's Election Site) and the expander icon takes a lot of space on those without stuff to expand (ideally, let's have nothing to show that needs an expander). Also, show some historical information about the booth would be nice. Also, we may have photo(s) to show from this election (or even past?). Also maybe color-coding by has food/has nothing/is unknown. Make the Red Cross of Shame clearer (it's just an icon at the moment).
• Think about adding a nudge banner (see SausageMap.tsx) to the Map and/or Finder to prompt users to use their GPS. (Depending on results of interactions with the current GPS mechanism from the last elections)
  • c.f. https://developers.google.com/web/fundamentals/native-hardware/user-location/#gently_nudge_users_to_grant_permission_to_their_location, https://uxplanet.org/mobile-ux-design-the-right-ways-to-ask-users-for-permissions-6cdd9ab25c27, https://material.io/guidelines/patterns/permissions.html#permissions-request-patterns
• Should stall have unique URLs on public and a Share button with that fancy thing like The Sizzle has?

Refactoring

• Run Lighthouse - take any quick wins and log the rest as issues if they warrant attention (check GA page timings to corroborate)
• Integrate LogRocket to get screenshots when errors occur (ref)
• Can we reduce the JS size without going full code splitting? (OpenLayers - are we including it all or is tree shaking working? Would vendor/app bundles with async loading help wrt. caching?)
• Investigate switching geocoding to Mapbox and whether we can use their directions APIs for what we wanted to do for finding polling places.
  • Do we have to use their basemap if we use their APIs?
  • Did enough people seem to use Finder to warrant it?
• Better share common code between admin and public
• Think about how we can host the API for free/<= $5/month in a better place (c.f. security, moving to a better language, automatic db backups). But probably don't rewrite it or re-engineer with PostGIS. The time to do so and re-test everything (or write tests) isn't worth it at this point.

AB#101

Add endpoint for all polling places for election (alternatively, include stall name in map API endpoint)

I was able to link our dataset to Democracy Sausage's quite easily, without spamming the Democracy Sausage API, by pulling the /map endpoint and matching on coordinates.

However, this hasn't been perfect - there's a couple of hundred polling places that weren't linked, and upon inspection have differing coordinates between our systems.

If there was a /polling_places/all endpoint, this would work nicely and allow the most thorough matching.

Alternatively, if the /map endpoint included polling place name, this could be used as another way to match polling places.

Polling place loading - address handling

Make polling place loading handle merging separate address component fields together into a single field. It's now the most time consuming part of the loading process.

Get increased rate limits from Twitter

And a blue tick?

Rewrite the API in Django

@todo

• Implement public API endpoints
• Implement admin API endpoints
  • Download polling places as a CSV file (here, here, and here)
  • Upload and ingest a polling places CSV file
  • Edit a polling place type
  • Edit a polling place
  • Approve a stall (+ send email)
  • Decline a stall
  • Email opt in/out
  • Trigger Election GeoJSON regeneration
  • Optimise GeoJSON generation (too many db queries per noms?)
  • Mailgun webhooks
  • Cron job to email us nightly
  • Better error handling from DRF to React
• Check for remaining legacy API calls. Remove hard-coded paths.
• Go over PHP code function-by-function to check for any special logic (like is_active had). Remove legacy API code.
• Test Admin and Public sites
• Update and re-run db migration script again
• Squash and nuke all migrations and re-run fresh db setup
• DevOps #48
• Setup local prod
• Bump Supervisord, UWSGI, Django, and Nginx timeout limits up to account for loading polling places
• Make changes to digitalocean-stack
• Deploy as api2.democracysausage.org
• Tell the API user about forthcoming changes
• Apply and verify Mailgun web hook changes; Verify stall approval emails
• Deploy to PROD
• Deploy as api.democracysausage.org alongside new admin and public sites
• Test Admin and Public sites (check static assets are available)
• Load a test election
• Finish fixing the chance_of_sausage migration
• Get folks to login again. Set is_staff=True and is_superuser=True.
• Check db backups are working again
• Docker hub and private things in images
• Decommission old PHP api site
• Verify cron is working
• CORB
• Move API to /api
  • Move Sausage API
  • Update cookies for Sausage
• Fix election dates
• Location Enter key bug on Add Stall form
• Use a VERSION file (and put app version for backend and frontend in the Redux store so we see it in Sentry.io)
• CF cache purging on redeploy
• Look over the CloudFlare rules and think about how caching is working through the levels.
• Verify how CloudFlare treats static resources + icons now that we've removed those special page rules
• Did CF cache polling places for use before or did we bypass? What would CF cost to resolve the cookies and page rules issues (if they are still both issues?)
• CF doesn't like long-running requests (The backend process still completes, but we don't see the logs in response.)
  • Log results to a table using a "job id"
  • Use Celery (Move polling place loading; Use for scheduled jobs instead of cron)
• Tweak cron timing. Arrives at 14:00 - should be 20:00.
• Check login behaviour on mobile
• Include polling place name / premises / stall name in emails to better differentiate from our side

Enhancements

• Explore django-rest-swagger
• Validate which API endpoints are publicly exposed (e.g. DELETE to purge cache)
• Use Nginx Memcached for GeoJSON to avoid hitting Django at all

Setup media@, name@ emails from DemSausage

Can we do this all in Mailgun? Can folks do a "Send from" via Mailgun?

Retire the GMail and have it forward to a new info@ or hi@?

Add fuzzy text searching to the admin polling place editor

AB#117

Make pretty a Word document template for media releases

Can we sustainably support by-elections?

• Do / Not Do / Do Differently?
• I'm happy to source data and load for the handful of polling places they have (max 60 minutes, could automate more if we keep doing this)
• Happy to promote and approve stalls before-hand (low effort, can be delegated to interested locals where available)
• What about on-the-day though? Remote session work, but I'm wary of making Democracy Sausage a chore vs a thing we look forward to getting together and doing a few times year.
• There may be options for changing the stall submission and collection model to make it less work on us - but can we really get away from spending time on the day collecting and responding to tweets.
• Think outside the box.

AB#60

Web share API for perma polling place links

What's involved in getting Twitter and Facebook social cards working so that when we and others link to us users get a nice preview of the election map that was linked to?

This could also involve stall preview cards (per #18), not just the election map.

Could also use the Web Share API alongside this work for folks to share the map natively on mobile.
https://webkit.org/blog/8718/new-webkit-features-in-safari-12-1/

AB#120

Upgrade deps and switch to yarn

Our JavaScript deps are really out of date and we're using an old version of TypeScript (2.5.3). We should also switch to yarn from npm.

Had an initial go at upgrading deps and and switching to yarn at FOSS4G SoTM Oceania 2018, but ran into too many conflicts and errors and bailed out.

Lets take the current working install and upgrade TypeScript and then incrementally upgrade our deps. This would be a good time to switch to the latest Material UI too (if our material-ui-*) deps have been updated.

Edit: And let's upgrade node at the same time.

Switch to Travis CI build stages

We have two deploys in .travis.yml that trigger the after_deploy command twice - this leads to two simultaneous rebuilds of our Digital Ocean stack.

We've worked around this for now by using after_success, but this isn't ideal because it runs after script, not after deploy. This works for now because the S3 deploys are quicker than the Digital Ocean droplet startup sequence, so the deploys have finished pushing the new versions to S3 by the time the new droplet pulls them.

Build Stages are the Travis CI solution to running one task after all preceding tasks have finished.

AB#110

Link search results into the map

Users should be able to get to the map from a set of search results.

The map should at least zoom to the bounding box of the search results, if not actually highlight them, and show an icon at the lat,lon the user searched from.

Make it easier to specific the geo bounding box when creating/editing elections

A way of automatically/manually setting the default election lat/lon based on the loaded data.

At the moment we're manually setting the lat,lon centre of the election that the user is zoomed to.

More volunteers that aren't me could create elections themselves if we could make this more user-friendly.

Ideas:

• When loading data - Set it based on the centroid of the loaded data (if it's not already set)
• When creating an election - Allow the user to specify a point on the map through an interactive

AB#108

Add a Get data button

So folks can download it themselves.

Talk it over with the group.

What license? CC-BY?

What about the polling place data - that has varying (or no) license depending on the electoral commission involved.

AB#121

Write a script to backup and blow away the dev database and replace it with the latest PROD snapshot

For bonus points: Add an option to replace it with a live snapshot of PROD

Use a VERSION file/flag

• Use a VERSION file/flag and
• put app version for backend and frontend in the Redux store so we see it in Sentry.io

AB#129

UX and functionality ideas from ESS

How stall were displayed, what information was collected, et cetera.

Functionality

SnagVotes

Folks could vote on a stall.

Stalls and polling places as lists

Prompt users to add stalls when viewing a booth

AB#119

Redesign polling place cards on the public site to better utilise space and better show the most important details about the polling place and stall

Like, descriptions aren't even visible!

This could involve removing the stall name field - what role does it really play?

Discuss with the team.

AB#111

Handle dupe detection in the data loader

The data loader should hard fail if there are duplicate polling places.

Electoral Commissions commonly duplicate polling place records when a single polling place serves multiple districts/divisions.

At the moment we're just throwing warnings to users when this happens - but we never want duplicate polling places, so we should hard fail the import instead.

Think about SSR

https://news.ycombinator.com/item?id=18924724

AB#124

Looking into DRF's support for API keys and throttling

Also consider CORS changes that would be required for third party sites to integrate.

AB#132

Look into replacing Mailgun with a service like Mailchimp

So we don't have to manage mailing lists and can send pretty emails

AB#126

During South Australia

• Seaview High Polling Booth Chance of Sausage bug
• Update the language used in the emails to make it clearer how to make edits to stalls
• Update the language used around Chance of Sausage with a "let us know if we're wrong " CTA.
• Confusion re the link in the submitted/approved email "Yes, that is correct. The link in the email went somewhere else though and i was worried it would be wrong in the map. Thanks for checking."
• Reimplement a more streamlined Sausagelytics
  • ElectionChooser stats?
  • A list of polling places per <something>

    "Do you have a list of sites that are dedicated to serving snags? It saves me having to pull the information from the map."

• Update the Media and About pages
• Add a 'How can I help' section that talks about what skills we need (design et al.) and how to help crowdsource info in the lead-up
• Discuss our approach to by-elections
• Discuss integrating photos for elections and polling places
• Discuss how we could let people edit their own stalls once submitted
• Discuss what the actual action is for users who know a booth has noms "because they've been voting there for 20 years", but don't know enough to report a stall in full detail
• Implement new Chance of Sausage metric per WhatsApp discussion and new language (with a focus on CTAs for "tell us about the stall!")
  • Do we need a different "Add Stall" form for the day for folks who don't own stalls to submit crowdsourced info?
• Discuss 'blog' section idea per WhatsApp discussion
• What do about Instagram

AB#107

Post-Victorian election link the # of votes cast at each booth to compute % of population covered

AB#123

Allow users to filter stalls by the types of food on offer

All of the optionals: Vego, et cetera.

Changes

• Make the filter an AND, not an OR
• Can we do x2 res versions of the icons?
• Vanilla versions of sausage, cake, sausage + cake icons
• "Green plus" versions of sausage, cake, sausage + cake icons for "has other options cases"
• "Run out" versions of sausage and cake icons
• Sidebar Map Find Add Stall bug on https://test.democracysausage.org/federal_election_2019
• Search, zoom, switch elections

Notes

• filter ui mockup
• https://dribbble.com/search?q=filter
• material-ui filter ui

Make the Add Stall button more prominent

I think the original thought behind this was a small number of folks finding it hard to work out how to add a stall.

Move the polling place loader to use Celery

CF doesn't like long-running requests (The backend process still completes, but we don't see the logs in response.)

Use Celery (Move polling place loading; Use for scheduled jobs instead of cron)

AB#128

Add a GitHub link

AB#114

Rework the UX of Step 2 on the Add Stall form

It seems to be confusing a few users and stopping them from proceeding past that step to submit a stall.

AB#113

API tests

• Write tests for all API endpoints and end-to-end tests around creating elections, approving/denying stalls, editing polling places, and loading polling places

AB#36

Change the UI around "Chance of sausage"

Currently the "Chance of Sausage" indicator on stall cards only says LOW, MEDIUM, or HIGH.

Suggestion from H:

• Say how many historical data points it's based on so folks can judge the accuracy of the prediction
• Add wording around "no data available" if the chance is NULL

AB#133

Users who submit stalls should be able to edit them again

As a stall submitter, I want to be able to edit my stall once I've submitted it, so that I can update the stall details myself if something changes.

As a stall approver, I want to be able to see which pieces of stall information have changed since we approved it, so that I can more easily decide which changes to apply to the polling place.

Problem

Currently there's no process for stall editing via the website. Users have to email us directly and we have to manually apply edits via the Polling Place Editor.

For the 2018 Victorian election we had 40-odd requests for updates to stalls that we had to manually process and respond to ourselves. That's no good - let's optimise that!

Proposal

Stall Approval

• Add an approved_on datetimestamp to stalls that is updated when a stall is approved (views.py or signals.py?)
• Check that IP-based history is actually enabled and working (a quick glance suggests not)

Stall Submitted/Approval Email

• Re-purpose the mail_confirm_key code for the confirm_key for the stall approval URL
• Do HMAC properly using the stall_id and election_id as the message
• Create a URL on the public site like https://democracysausage.org/edit-stall/?confirm_key=foobar
• Update the stall submitted and stall approval email templates to include the new URL and a blurb about stall updates

Stalls API

• Add an endpoint that checks that a given confirm_key is valid and isn't super old (?) when the stall page is loaded. Also that the election is still live.

Stall Edit Form

• Check the confirm_key validation endpoint on componentDidMount
• If invalid, display a message
• If valid:
  • Display a summary of the polling place it is attached to as an information card
  • If it's an unofficial stall that hasn't yet been approved, display the stall_location_info
  • If not yet approved, display a banner noting this
  • Display a stall editing form with the information currently in stalls (be it approved or otherwise - so they can edit yet-to-be-approved stalls). Consider re-using the same component as Add Stall, just with a different Container. The user won't be able to edit the stall location or their email.
  • Apply client-side form validation logic per the Add Stall form
  • Apply backend form validation logic per the Add Stall form
    If the edits are valid:
  • Set the stall's status to Pending (regardless of what it was before)
    If the edits are saved successfully:
  • Display a message similar to the Add Stall message so the user knows it won't appear on the map straightaway

QAing an Edited Stall

• Display a banner indicating this stall was edited since it was approved
• Mark or highlight any stall fields that changed since it was last approved (using django-simple-history) and the approved_on datetimestamp. This should allow us to capture multiple edits to a stall between approvals.
• If the stall has never been approved, don't show the banner and don't show the diff. (Treat it like a regular stall.)
• If the stall is approved, apply the usual logic (from part 1 above) except for the email template. Use a new "your stall has been updated" template that says something like "If this wasn't you, please email [email protected] and let them know!". Include all of the information that was approved (current state - no diffing).
• If the stall is declined, still apply the usual workflow?

Other

• Add a way for us to get the edit stall URL for a stall so we can easily send it to people to self-serve who don't realise they can edit their own stalls

Deploying

• Update env vars on PROD
• Update Google Cloud Console
• Update and test Mailgun webhooks
• Run UPDATE demsausage.app_elections SET election_day = election_day + interval '-16 hour';
• Run UPDATE demsausage.app_stalls SET approved_on = reported_timestamp WHERE status = 'Approved';
• Check cookies
• Check POSTs and PATCHes on both sites
• Check mail opt out (junk my historical stalls first - if any)

Fix the shade of purple used on the loading image

The loading page's shade of purple is different to that of the DemSausage logo.

Let's fix the colour or change the logo to a transparent PNG.

Allow users to be given access to manage just a specific election

To enable people to remotely manage specific elections - e.g. by-elections.

Election Managers

• Can be appointed to manage individual elections (e.g. local government elections, by-elections)
• For elections they have access to they will have rights to:
  • Add/upload polling booths
  • Approve pending booths
  • Edit polling booths
• They will receive emails notices that there are pending stalls
• Investigate whether electoral commissions release polling booth data for these smaller elections.
  • It may be necessary to develop a simple GUI to let them map polling places by-hand where there no authoritative list of polling booths.
• Individuals to be managers can be recruited from social media
• Consider what tools/materials they would need to perform this role and promote DS in their local communities.

Manage Users

• Create User
• Edit User
• Delete User
• Set user access level: Super User, Election Manager

AB#131