alphagov / notifications-api

GOV.UK Notify API

GOV.UK Notify API

Contains:

• the public-facing REST API for GOV.UK Notify, which teams can integrate with using our clients
• an internal-only REST API built using Flask to manage services, users, templates, etc (this is what the admin app talks to)
• asynchronous workers built using Celery to put things on queues and read them off to be processed, sent to providers, updated, etc

Setting Up

Python version

We run python 3.11 both locally and in production.

psycopg2

Follow these instructions on Mac M1 machines.

AWS credentials

To run the API you will need appropriate AWS credentials. See the Wiki for more details.

environment.sh

Creating and edit an environment.sh file.

echo "
export NOTIFY_ENVIRONMENT='development'

export MMG_API_KEY='MMG_API_KEY'
export FIRETEXT_API_KEY='FIRETEXT_ACTUAL_KEY'
export NOTIFICATION_QUEUE_PREFIX='YOUR_OWN_PREFIX'

export FLASK_APP=application.py
export FLASK_DEBUG=1
export WERKZEUG_DEBUG_PIN=off
"> environment.sh

Things to change:

• Replace YOUR_OWN_PREFIX with local_dev_<first name>.
• Run the following in the credentials repo to get the API keys.

notify-pass credentials/firetext
notify-pass credentials/mmg

Postgres

This app requires Postgres to run.

If you are using notifications-local, the correct Postgres version will be provided automatically by the docker-compose file.

If you are running this app manually, you will need to manage Postgres yourself. Install Postgres.app. Check the docker-compose file above to find the correct Postgres version to use.

When our unit tests are run in Concourse, Postgres is based into the container via the concourse_tests step of docker/Dockerfile.

Redis

To switch redis on you'll need to install it locally. On a Mac you can do:

# assuming you use Homebrew
brew install redis
brew services start redis

To use redis caching you need to switch it on with an environment variable:

export REDIS_ENABLED=1

Pre-commit

We use pre-commit to ensure that committed code meets basic standards for formatting, and will make basic fixes for you to save time and aggravation.

Install pre-commit system-wide with, eg brew install pre-commit. Then, install the hooks in this repository with pre-commit install --install-hooks.

To run the application

# install dependencies, etc.
make bootstrap

# run the web app
make run-flask

# run the background tasks
make run-celery

# run scheduled tasks (optional)
make run-celery-beat

We've had problems running Celery locally due to one of its dependencies: pycurl. Due to the complexity of the issue, we also support running Celery via Docker:

# install dependencies, etc.
make bootstrap-with-docker

# run the background tasks
make run-celery-with-docker

# run scheduled tasks
make run-celery-beat-with-docker

To test the application

# install dependencies, etc.
make bootstrap

make test

To run one off tasks

Tasks are run through the flask command - run flask --help for more information. There are two sections we need to care about: flask db contains alembic migration commands, and flask command contains all of our custom commands. For example, to purge all dynamically generated functional test data, do the following:

Locally

flask command purge_functional_test_data -u <functional tests user name prefix>

To execute in ecs, you can use the ecs-exec.sh script

./scripts/ecs-exec/ecs-exec.sh
<select notify-api>
flask command purge_functional_test_data -u <functional tests user name prefix>

All flask commands and command options have a --help command if you need more information.

Further documentation

• Writing public APIs
• Updating dependencies

notifications-api's Issues

Bound time check doesn't take into consideration time loading private key

I have noticed a sporadic issue when we get back a clock skew error that does not appear to be related to our clock time.

I believe the issue is that the time taken from receiving the request to decoding and validating the jwt (using the __bound__ value) isn't taken into consideration - and there is a database operation in between these things.

I believe the fix is to take a note of the time at the start of requires_auth which can then be passed into _decode_jwt_token before then being passed into decode_jwt_token so it can then be taken into account when calculating the leeway on date/time validation.

log exceptions from celery

To do so, we need to extend the Task object to make it handle failures, since by default it doesn't.

Using this snippet we can get exceptions logged.

from celery import Task

class LogExceptionTask(Task):
    def on_failure(self, exc, task_id, args, kwargs, einfo):
        current_app.logger.exception('Celery task failed')
        super(LogErrorsTask).on_failure(exc, task_id, args, kwargs, einfo)

@notify_celery.task(base=LogExceptionTask, ...)

Poppulate ft-notification-status

I'd like know how I can call the "migrate-data-to-ft-notification-status", I can't look data in table ft_notification_status.

• DB Select
  notification_api=# select * from ft_notification_status;
  bst_date | template_id | service_id | job_id | notification_type | key_type | notification_status | notification_count | created_at | updated
  _at
  ----------+-------------+------------+--------+-------------------+----------+---------------------+--------------------+------------+--------

---

(0 rows)

notification_api=# select count(*) from notifications;
-[ RECORD 1 ]
count | 105

• CELERY CALLS:
  [tasks]
  . celery.backend_cleanup
  . celery.chain
  . celery.chord
  . celery.chord_unlock
  . celery.chunks
  . celery.group
  . celery.map
  . celery.starmap
  . check-for-missing-rows-in-completed-jobs
  . check-for-services-with-high-failure-rates-or-sending-to-tv-numbers
  . check-job-status
  . check-precompiled-letter-state
  . check-templated-letter-state
  . collate-letter-pdfs-to-be-sent
  . create-fake-letter-response-file
  . create-letters-pdf
  . create-nightly-billing
  . create-nightly-billing-for-day
  . create-nightly-notification-status
  . create-nightly-notification-status-for-day
  . delete-email-notifications
  . delete-inbound-sms
  . delete-invitations
  . delete-letter-notifications
  . delete-notifications-older-than-retention
  . delete-sms-notifications
  . delete-verify-codes
  . deliver_email
  . deliver_govbrcpf
  . deliver_sms
  . get-pdf-for-templated-letter
  . process-incomplete-jobs
  . process-job
  . process-returned-letters-list
  . process-sanitised-letter
  . process-ses-result
  . process-sms-client-response
  . process-virus-scan-error
  . process-virus-scan-failed
  . raise-alert-if-letter-notifications-still-sending
  . raise-alert-if-no-letter-ack-file
  . record-daily-sorted-counts
  . remove_letter_jobs
  . remove_sms_email_jobs
  . replay-created-notifications
  . run-scheduled-jobs
  . sanitise-letter
  . save-api-email
  . save-email
  . save-letter
  . save-sms
  . send-complaint
  . send-daily-performance-platform-stats
  . send-delivery-status
  . send-inbound-sms
  . switch-current-sms-provider-on-slow-delivery
  . tend-providers-back-to-middle
  . timeout-sending-notifications
  . update-billable-units-for-letter
  . update-letter-notifications-statuses
  . update-letter-notifications-to-error
  . update-letter-notifications-to-sent

What I need do for work good this report?

consider push notifications

hey!

i just came across this service, and wondered if you have already considered push notifications? they have the realtime-ness of SMS, but within the rate-limits, they're usually free.

Non-ASCII characters in email addresses

We use notify with our Register a Food Business service at the FSA.

We recently had an issue where sending an email using notify failed because the email address for the business contain a vowel with an umlaut.

Could someone advise if this can be fixed or is a problem with our implementation please?

Allow sending to specific template version

In addition to specifying the template id, it would also be useful if the send an email / send an SMS endpoints and clients supported an optional template version number so that new versions of templates could be authored and used alongside older versions, especially when parameters change.

This would help with supporting templates across environments.

e.g.
POST /v2/notifications/email { "email_address": "[email protected]", "template_id": "f33517ff-2a88-4f6e-b855-c550268ce08a", "template_version": 2, }

Broken link in README

The example link in item 2 to "Update the jenkins deployment job in the notifications-aws repo (example)" is giving an error 404. I think the repo is private or moved.

Using a list token in an email template

I'm currently doing work on behalf of CCS and are looking for some feedback personalising an email using a list token.

I have a token in my email template ((errors)) for example. Using the Java client I am supplying an empty list like so: personalisation.put("errors", Collections.<String> emptyList()). There are occasions where I don't have content for this token and other information in the email will suffice.

This token is not being replaced and ((errors)) remains in the email. I don't want to send a list of 1 with an empty string as a bullet point will be rendered. Can you advise if the API can handle an empty list?

Not all notifications showing up on the API

We are using the GC Notify service which is the public-facing REST API for Notification built on the GOV.UK Notify platform.

We use the service to send email notifications using various templates. In addition, we retrieve notification details from the API endpoint /v2/notifications for persistent delivery status logging. The API endpoint returns some notifications but not all which were generated in the last 7 days. The notifications missing from the API call belong to a specific template.

Is there a criteria for notifications to be returned from /v2/notifications endpoint?

Security Feature Request: placeholders which explicitly prevent markdown injection

Hi there,
At current, placeholders are able to be transformed into markdown if they are populated as such.

e.g. if you send [link](dodgy-link here) into ((your-placeholder-here)) it will render in your email.

This presents a compelling opportunity for an adversarial attacker to utilise gov notify's relative respectability as a phishing vector which both appears legitimate and gets through spam filters.

A potential suggestion would be to have specific fields that are populated in user input denoted within the template using some type of new syntax (e.g. ({safe-placeholder})) which gives protection at the template level.

Gov notify could then choose to not send that email, throw an error or even alert the responsible template holder.

It would also allow your team to spot adversarial behaviour patterns via logging.

Hope this is helpful.

Allow templates to be added and edited via the API

For deploying and moving between environments, it would be useful if templates could be edited or created via the REST API (or client) rather than a manual job.

log task id in celery tasks

by cleaning up the logs we've lost the context of what specific task run a log line was from -

inside a task, you can use self.request.task.id to get a unique task ID, but i think we need to extend the logging stuff from utils if we want to use this - probably inside the app/celery/celery.py::NotifyCelery class.