Coder Social home page Coder Social logo

cds-snc / notification-api Goto Github PK

View Code? Open in Web Editor NEW
48.0 22.0 20.0 27.44 MB

GC Notify API | GC Notification API

Home Page: https://notification.canada.ca

License: MIT License

Makefile 0.05% Python 98.11% Dockerfile 0.12% Shell 0.47% Mako 0.01% PowerShell 0.02% Procfile 0.01% JavaScript 1.21%
notification-canada-ca notify-core beta

notification-api's Introduction

Notification

Contains:

  • the public-facing REST API for Notification built on the GOV.UK Notify platform, which teams can integrate with using their clients
  • an internal-only REST API built using Flask to manage services, users, templates, etc (this is what the admin app talks to)
  • asynchronous workers built using Celery to put things on queues and read them off to be processed, sent to providers, updated, etc

Functional constraints

  • We currently do not support sending of letters
  • We currently do not receive a response if text messages were delivered or not

Setting Up

For any issues during the following instructions, make sure to review the Frequent problems section toward the end of the document.

Local installation instruction (Use Dev Containers)

  1. Install VS Code

brew install --cask visual-studio-code

  1. Install Docker

brew install --cask docker

  1. Install the Remote-Containers extension

  2. In VS Code run "Remote-Containers: Open Folder in Container..." and select this repository folder

  3. Run the service

make run

To run the queues

Run make run-celery-local or make run-celery-local-filtered. Note that the "filtered" option does not show the beat worker logs nor most scheduled tasks (this makes it easier to trace notification sending).

Python version

This codebase is Python 3 only. At the moment we run 3.10.8 in production. You will run into problems if you try to use Python 3.4 or older.

To run Performance tests

Ask your teamate for the following keys and add to .env

PERF_TEST_AUTH_HEADER =
PERF_TEST_BULK_EMAIL_TEMPLATE_ID=
PERF_TEST_EMAIL_WITH_LINK_TEMPLATE_ID=
PERF_TEST_EMAIL_TEMPLATE_ID=
PERF_TEST_EMAIL_WITH_ATTACHMENT_TEMPLATE_ID=
PERF_TEST_SMS_TEMPLATE_ID =

And run the performance tests using. We generally test with 3000 users every 20 seconds (but use your best judgement).

locust -f tests-perf/locust/locust-notifications.py

To update application dependencies

poetry.lock file is generated from the pyproject.toml in order to pin versions of all nested dependencies. If pyproject.toml has been changed (or we want to update the unpinned nested dependencies) poetry.lock should be regenerated with

poetry lock --no-update

poetry.lock should be committed alongside pyproject.toml changes.

Using Local Jinja for testing template changes

Jinja templates used in this repo: email_template.jinja2

Jinja templates are pulled in from the notification-utils repo. To test jinja changes locally (without needing to update the upstream), follow this procedure:

  1. Create a jinja_templates folder in the project root directory. This folder name is already gitignored and won't be tracked.

  2. Copy the jinja template files from notification-utils into the jinja_templates folder created in step 1

  3. Set a new .ENV variable: USE_LOCAL_JINJA_TEMPLATES=True

  4. Make markup changes, and see them locally!

  5. When finished, copy any changed jinja files back to notification-utils, and push up the PR for your changes in that repo.

  6. Remove USE_LOCAL_JINJA_TEMPLATES=True from your .env file, and delete any jinja in jinja_templates. Deleting the folder and jinja files is not required, but recommended. Make sure you're pulling up-to-date jinja from notification-utils the next time you need to make changes.

Testing

To help debug full code paths of emails and SMS, we have a special email and phone number set in the application's configuration. As it stands at the moment these are the following:

Notification Type Test destination
Email [email protected]
SMS +16135550123

Whereas the smoke test emails and long codes might not get through the whole GCNotify data treatment, these will and have proper database fields populated. This is useful for proper stress tests where the notifications shouldn't merely touch the API front-door but also get through the Celery workers processing.

Frequent problems

Problem: No postgres role exists.

Solution: If the command complains you don't have a postgres role existing, execute the following command and retry the above afterward:

createuser -l -s postgres

Problem : E999 SyntaxError: invalid syntax when running flake8

Solution : Check that you are in your correct virtualenv, with python 3.10


Problem: ImportError: failed to find libmagic. Check your installation

Solution:Install libmagic, ex: brew install libmagic


Problem: assert 'test_notification_api' in db.engine.url.database, 'dont run tests against main db

Solution: Do not specify a database in your .env


Problem: sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) fe_sendauth: no password supplied

Solution: Ensure SQLALCHEMY_DATABASE_URI supplied in pytest.ini or your .env file is valid to your local database with user access, (pytest.ini takes precedence)


Problem: Messages are in the queue but not sending

Solution: Check that celery is running.

notification-api's People

Contributors

allait avatar andrewleith avatar antoineaugusti avatar ashimali avatar bandesz avatar crystalpea avatar gov-cjwaszczuk avatar imdadahad avatar jimleroyer avatar jzbahrai avatar kentsanggds avatar klssmith avatar leohemsted avatar mannickutd avatar maxneuvians avatar minglis avatar mohdnr avatar patheard avatar pcraig3 avatar pyup-bot avatar quis avatar renovate[bot] avatar richardc0 avatar sakisv avatar sastels avatar servingupaces avatar smcmurtry avatar timarney avatar venusbb avatar whabanks avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

notification-api's Issues

How does TBS in Brief happen?

What?

  • Book an interview with the TBS in Brief team to find out how they do it
  • Bryan has the information to get in touch with the TBS Internal Comms group which is under SCMA (Strategic Communication and Ministerial Affairs)
  • Find out their process, how it is sent, what problems they have
  • Log their use case in Drive

Why?

  • They are close to us and their use case is a valid one to explore

Content Audit of Notification

What?

  • Find UI things we need to change
  • Find UI things we need to translate
  • Find code of UI things to Translate
  • Find all the things we need to re-write such as content in footer or kill
  • Find where it says letters so we can get rid of them
  • What pages should we kill?
  • Should we merge some pages?
  • Put this all in a Google doc or a github wiki for the team to reference

Why?

  • We don't want to confuse people

How much do other currently implemented solutions cost?

What?

  • How much does it cost the government to send email notifications to people now?
  • MailChimp costs for news letters?
  • What other services are used there and what are there costs? Such as outlook mailing lists? How much does that actually cost SSC? is it trivial?
  • Is there a labour cost we are replacing by allowing them to send from software directly with our API? What does that equate to?
  • If anyone is sending SMS messages, how much does it cost them?

Why?

  • We should know how much people currently are paying out of their budgets for when they need to send people email notifications so we can use this as a performance indicator of money saved.

Document what we had to change to implement our fork of the Notify application and get to work on our infrastructure.

What?

  • Create a wiki article about what changes we had to make to get it up and running on K8S
  • Wiki about What did we need to change with Email and SMS to get them in Canada? our providers?
  • What do we need to change to make french work for the UI and templates

Why?

  • Documentation will make it easier to recreate what we have done to modify the product for our purpose. There is a lot of existing documentation on Gov.uk we can borrow when we need documentation, but we need to stay on top of new stuff we do so we can merge documentation back later.

Find out who sends IT bulletin

What?

  • Find out who runs Services - IT Service Desk_Bureau de service de la TI
  • Interview them about their Use case
  • Log their use case in Drive

Why?

  • TBS is easy for us to reach out to and collect information from. They may want to use our solution and so not only does this research but it can help us find new clients.

Does our Copy of Notify come with an unsubscribe mechanism?

Dependancy: https://github.com/cds-snc/CANotifier/issues/28

What?

  • If we send emails to people they need to be able to unsubscribe easily, Does the Notify fork have that?
  • SMS needs to be able to unsubscribe too, is there a mechanism for that? Can they text back something to unsub them? do we need to send a message explaining that when we SMS some one for the first time?
    If no:
  • Create a simple way for people to unsub from emails, put a herf in emails that takes them out of the list and store them in a db as to block them from getting future emails from the CANotifier System (Just to be safe, we could isolate it to that user or department in the future but lets keep it simple for now.)
  • SMS needs a way too, like a user replies with No or Non to remove them. We could ask them yes or no
  • Needs to be in both official languages

Why?

  • We aren't sure if CASL applies to government communications but we don't want to be accused of sending unsolicited communications people can't get out of and have it be true.

How does the Government current send Email and SMS notifications?

What?

  • We need to know how the government sends stuff out currently (I believe the current most popular solution is outlook mailing lists)
  • a Sample size of different groups and how they send notifications

Why?

  • We need to know what we are trying to displace and be better than

Update the UI to support switching to French

What?

  • Put in toggle to switch the UI to French that sets a session variable.
  • Detects browser language

Why?

  • Official Language support
  • Usability testing
  • I imagine we will have French users

We need to validate that the UI is accessible before usability testing

Dependant #43

What?

  • Once we have it up and running we need to validate that the Notify system is WCAG 2.1 Level AA compliant
  • Run it through our Accessibility testing software if possible if not, let me know. I will Talk to Julianna about a sweep so we can integrate that feedback into design.

Why?

  • If we add stuff to it that isn't we will just end up making more work for ourselves. It should be so let's validate it.

When uploading a CSV of recipients I need to be able to have a variable that reflects the Users Preferred language so that I know which Template to send them

What?

  • Column in a CSV upload French or English needs to be supported, could have values like Eng, Fr, English, French, Francais, E, F, 1,0,2 and we need to be able to have users match that up to a template if they are sending a communication in both official languages
  • Needs to be able to be utilized by the API so that a connected service using us for notifications can send it in a language
  • Upload interface needs to reflect that this is possible as does the ability to create two templates so we need a design that can be implemented for this.
  • We need to update the existing Notify design to support these additions

Why?

  • All the currently used solutions and private sector solutions thing of bilingualism as an after thought, we need to make it a core feature to boost our value prop of the software.
  • If users opt into a news letter they often select the language they which to receive communications in.

Notes

  • One email with both languages is a good first step, this can happen later, but is a valid GoC federal usecase

When sending messages I want to send them in both official languages when I know a users language prefs so that I can be official Language compliant

Dependancy: https://github.com/cds-snc/CANotifier/issues/20

What?

  • Implement our designs for the two use cases of french messages
  • CSV uploads or API calls can have a users language. Based on this is sends either the English or French template message, unsub messages, et al.
  • Template creation, means you need to create two now

Why?

  • French support is a key differentiator and Value prop for this tool as the private solutions consider this an after thought.

Are we allowed to Upload and store information about recipients uploaded by Senders?

What?

  • Are we in the clear if we let people upload and send information to recipients if they have already collected the info and are cleared to use it for that purpose?
  • How secure does it need to be?
  • Is it ok to be stored in logs or should it be scrubbed?
  • If our infrastructure (AWS?) cleared for protected B are we ok?
  • The software shows messages that didn't send. Do they matter since they didn't make it? Can we assume the info is wrong?

Why?
Here is what came out of consultation with Legal Services.
It boils down to “how many pieces of information we can collect about an individual” before we can cross reference data and infer identity. The answer is two. As long as we are not collecting more than two “identifiers”, we are compliant.

An Example would be
Identifier #1: Language of user
Identifier #2: Email address

How can we actually send Email reliably to a mass of people?

What?

  • Can we adapt CANotify to use AWS SES https://aws.amazon.com/ses/ for sending emails? instead of what it currently uses? Does it use that? if so Great!
  • In the UI Is there anyway for the Sender to configure the reply address? or the API?
  • Anyway to configure SPF records?

Why?

  • We need emails to go out and not trigger spam blacklisting by email services so that government agencies can be confident the emails will get through.
  • We may one day need to support replies to the emails sent so if it doesn't have that it may be a future feature.

Reducing a Spam score of a message

  • Less emails coming from one place to an email service (E.G. Sending 1000 emails to Gmail in 5 seconds is going to get us blocked)
  • Allowing a user to set a reply too address so that it isn't No Reply can also reduce spam scoring
  • Fill a proper subject line
  • Avoid HTML errors in the messages
  • Use normal text in a URL instead of the URL

Password security, Banlist Upgrade

What?

  • Says 8 characters, no other rules
  • There is a black list of common passwords such as '1111111'
  • We need to enhance this for GoC spec to remove Canada works and departments

Why?

  • The system will have protected info in it and we need to ensure its security.

How does the Privacy act impact the service

What?

  • How does the Privacy act impact what we can and can't allow with the service?
  • Are we liable for people miss-using the service?
  • Legal opinion of how this impacts our work
  • Ontario Notify team said that they can't put names in the emails since they are not encrypted, is that true for us?
  • Should we hide any personal information in the logs?

Why?

  • We wish to be compliant with all laws to reduce risk and meet ethical requirements

When trying to get Notification used by Government departments a memorable brand could improve peoples opinions

What?

  • We need styling for UI elements
  • A colour scheme for link colours(Current #005ea5), link hover, dashboard boxes(#005ea5), Success states, failure states(#b10d1e), no failures(#01823b), search buttons (#01823b), Link&page highlights(#ffbf47), and box/button highlights(#dee0e2)
  • Font styles (Currently Arial/Helvetica has been used to replace GDS proprietary font)
  • Drop shadows like they have on buttons seem to be used to denote a button or a highlight box.

Why?

  • We need something that reflects the nature and ownership of the service
  • We want to give it a Government of Canada or CDS feel rather than a GDS Feel

Notes

image

image

When evaluating Privacy we need to know if Emails are stored in Logs

What?

  • GDS says they have an option to scrub the logs. It looks like info is only kept for 7 days anyway.
  • Do we have that option?
  • Can we turn that on or is it on by default?
  • It stores the emails to show which ones bounces, can we still save bounces since they didn't work the emails don't reveal PII?

Why?

  • If we aren't storing emails in Logs it reduces privacy complexity

When working on Notify we need to have it on K8S to ensure service reliability so we don't have to worry about it as much

What?

  • Standup/Migrate CANotify on K8S
  • Using whatever infrastructure provider we need to best support protected A/B, is that AWS?
  • .yaml file to manage docker containers? (I hope this still uses docker like Gov.uk notify)

Why?

  • If we set it up on K8S proper it will improve the service accessibility and reliability.
  • We can have new containers appear when the service goes does and also stand up security goals, branch reviews, et al.

What sort of content does the template maker support?

What?

  • Dependant on standing up CANotify and the web interface or at least looking at the code
  • Does it support Markdown? HTML? Plain Text? French Characters?
  • How can we send Images?
  • Gov.uk Says attachments are an optional feature, how do they work in our implementation?
  • How do the Variables work? Tags from the CSV?

Why?

  • The senders will care about what they can send and it directly impacts the use cases we can support

When users upload a CSV of recipients we need to warn them that they must be allowed to use the email for this express purpose in order to be compliant with the Privacy Act

What?

  • https://www.priv.gc.ca/en/privacy-topics/privacy-laws-in-canada/the-privacy-act/
  • We need a warning message on the page that states users must be compliant with the privacy acts rules around Fair use of the emails of numbers they are uploading
  • We need to figure out how the privacy act applies to SMS and Email notifications and write elements to put in the UI
  • Figure out where in the UI we need them

Why?

  • Legal Requirement

Update Favicon From Crown to Mapleleaf favicon

What?

  • Crown favicon should be canada.ca maple leaf fav icon
  • Can we change the Page Meta Title too so it doesn't say Gov.uk Notify?

Why?

  • We don't want people to think we are monarchists

When sending messages to recipients are there limits to the information that can be in the messages?

What?

  • Example, I don't think we can upload and send peoples SIN numbers to them, but maybe we can?
  • ODS said that they can't include peoples names, do we have that limitation?
  • What is taboo or off limits from an info classification level? does it vary from Unclassified, Prot A, Prot B?
  • Are there other things we can't send because of Privacy or consent notifications?
  • We need a message in the UI telling them this, warning the senders

Why?

  • We are assuming some liability with this tool and to avoid that we need to advise them what they can't use the system for to cover ourselves.

When testing and using Notify we need a public facing URL to drive traffic to.

What?

  • put it up on canotify.cds-snc.ca
  • Should we just have it as a dev review URL for now?
  • Validate that we have domain verification for @cds-snc.ca on AWS for sending emails

Why?

  • Min req is We need people to be able to access it so they can usability test and review so if we have a URL to send to people I don't mind what it is.
  • Emails need to work as well so we need a default domain for them to come from

Get rid of Gov.uk Branding in UI

What?

  • Colours are pulled in via an NPM from a deprecated UK repo
  • We need to update the colours to CDS agreeable ones (Use the colour pallette from digital.canada.ca, Black, white, Dark grey, yellow highlights) As a placer holder until we get design help
  • Replace Gov.uk UI logos with GoC logo

Why?

Remove hardcoded Google Analytics

The templates have hardcoded Google Analytics values. These need to point to our Google Analytics account + be pulled in via .env vars.

Screen Shot 2019-07-04 at 8 36 39 AM

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.