hammady / wwpray

A website that shows the prayer times for a preconfigured list of masjids in a tabular format. Visitors can subscribe for email notifications.

Where When Pray: A serverless application for prayer time scraping and notification

A website that shows the prayer times for a preconfigured list of masjids in a tabular format. The data is scraped from the masjid websites periodically. Visitors can subscribe to one or more masjids and receive email notifications when the prayer times change.

Motivation

With each masjid having their own schedule for prayers that also change sometimes every few days, the only way to get accurate information is to visit each masjid's website. This application aims to solve this problem by scraping the prayer times from the masjid websites and notifying the subscribers when the prayer times change. Some solutions, like the awesome TMA or The Masjid App, already exist but they require the masjid management to update the prayer times on their website, which is not always the case. I decided to create this application to solve this problem for myself and others by offering a simple website that displays all the prayer times in one page.

Architecture

The application is built using serverless components on AWS:

1. Scraper: A Lambda function triggered once daily and scrapes the prayer times from the masjid websites. The data is then stored in a JSON file in S3.
2. Notifier: A Lambda function triggered as soon as a new JSON file is created (typically from the Scraper) and compares it to the previous JSON file. If there are any changes, it sends an email notification to the subscribers. Finally, it replaces the previous JSON file with the new one. Each masjid has its own list of subscribers.
3. Generator: A Lambda function triggered as soon as a new JSON file is created (typically from the Notifier) and generates a static HTML site from it. The HTML site is then uploaded to S3 and served via CloudFront. The website lists all the masjids and their prayer times in a tabular format. It has a form to subscribe to notifications where it submits to the Subscriber Lambda function (see below).
4. Subscriber: A Lambda function triggered when a user submits the subscription form which adds the user to the masjid's list of subscribers.
5. Forwarder: A Lambda function triggered when a user sends an email to the feedback email address. It forwards the feedback email to the admin.

Email notifications

Subscribers are stored in Amazon SES as contacts with a separate topic for each masjid. The email notification contains a link in the footer to unsubscribe from the notifications. The link target is handled by SES and allows the user to opt in/out from any topic. The subscription form target adds the user as a new/updated contact to the corresponding topic in Amazon SES. Amazon SES only supports 1 contact list per AWS account and maximum of 20 topics which is good enough at a small scale. For a wider support of masjids, different deployment using separate AWS accounts will be required, unless subscriptions are handled differently.

Masjids database

Each masjid has its own Python class which contains some metadata about the masjid and the code that scrapes its website. This is a simple approach that may be replaced in the future with a more sophisticated solution. To support a wider range of masjids, the code can be modified later to only operate on a subset of the masjids and deployed under different subdomains, where visitors will be typically interested in the masjids in their area.

wwpray's Issues

Detect changes in Jumuaas

Make a logo for wwpray with favicon

Currently, the app name is only displayed as text, and the website favicon is Svelte's favicon. We can make it shine by designing our own.

Once the logo is designed, we can use Real favicon generator to have a cross-browser & cross-platform favicon for wwpray.

Add more masjids

Support for Zero-JS Form Submissions with Query Params for Success and Error Messages

We are willing not to require JS for users to use the website. We only use JS to enhance the experience, but the basic functionality isn't JS dependent. So, we are not using AJAX requests. We rely on HTML form elements.

The subscription endpoint is a POST method that takes email and topics as query parameters and returns a FormData payload with "message" and "topics" properties.

We use a form element to handle subscriptions. Here is a minimized version of the code:

<form
	method="POST"
	action={SUBSCRIPTIONS_BASE_URL}
>
	<input type="email" name="email" />
	<input type="checkbox" name="topics" value="MNNexus" />
	<input type="checkbox" name="topics" value="Shalimar" />
	<button type="submit"> Subscribe </button>
</form>

Upon submission, this will be done:

• Because the method is set as POST, the form element will send a request SUBSCRIPTIONS_BASE_URL without any query params but send a FormData payload like the following:

  email: [email protected]
  topics: MNNexus
  topics: Shalimar

• The method redirects the user to SUBSCRIPTIONS_BASE_URL, and this is the end. The user left wwpray page.

To enhance compatibility, the following changes could be made to SUBSCRIPTIONS_BASE_URL:

• Change the endpoint method to GET, or read data as FormData instead of query params.
• If subscription is successful, redirect to back to the original page with message= and topics= query params
• If subscription failed, redirect with error_message= query param.

Skip notifying subscribers on Maghrib changes

Because it happens almost daily for all masjids.

Move SES templates to the CloudFormation definitions

Currently, I am using a third-party to manage the templates.
With CloudFormation, I can do the same if I just include the template definitions in serverless.yaml along with other CF definitions. This also add traceability for all template changes as they will be checked in source control and go through the future CI/CD process to get deployed.

Investigate issues in notifier preventing email sending

Traceback (most recent call last):
  File "/opt/python/serverless_aws_lambda_sdk/instrument/__init__.py", line 580, in _handler
    result = user_handler(event, context)
  File "/var/task/serverless_sdk/__init__.py", line 144, in wrapped_handler
    return user_handler(event, context)
  File "/var/task/handler.py", line 107, in run
    if validate_topics(new_topics) is False:
  File "/var/task/handler.py", line 90, in validate_topics
    valid_topics = get_valid_topics()
  File "/var/task/handler.py", line 86, in get_valid_topics
    for topic in ses_client.get_contact_list(ContactListName=contact_list_name)["Topics"]
KeyError: 'Topics'

Probably related to the deletion of all topics.

Highlight changes

Show (Changed) or color in a different color for the changed entries in iqamas/Jumaas.

Follow-up on moving out of the SES sandbox to production

Case pending with Amazon support.

Side-bar remembers previous selections

When I click on subscribe it auto-selects the current masjid which is ok, but when I close the sidebar without clicking on subscribe, it will remember that masjid and it will be auto-checked when I select another masjid. I expect the side-bar to be reset the checkboxes when I close it.

Centralize common configuration in serverless.yaml files

Many configs are common between services like "email_from", "feedback_url", ...
Instead of copying them in related services, it is better to centralize their configuration somewhere.

Create an alternative compact layout

This layout will show all the data in a single table.

Optimize notifier to send emails in parallel

Currently, the notifier sends emails sequentially. There is a lot of wasted time in the function waiting for network (probably 99%!). This increases the cost of the function and won't scale at all if we have thousands of subscribers as it will be very expensive and may go beyond the max allowed timeout (if any).
We should send the emails asynchronously. The caveat here is that we are still bound by the maximum sending rate from SES (currently 14 requests per second), so we can only optimize the speed up to a certain limit.
It would be nice if the function would read the sending rate using SES API in the beginning then control the number of threads/coroutines making sure it is not sending too fast.

Replace dummy implementation for generator with actual generation

Currently the generator service does the following:

1. Downloads the input files
2. Generates some files (dummy generation)
3. Uploads the output directory

It is required to replace number 2 with an actual implementation for the static website generation.

Investigate retry policy for all lambdas

Enable subscriptions (email notifications)

The user can subscribe to email notifications for any masjid by submitting their email and list of topics. The base URL should be read from the environment.

Deploy the prod stage using CI/CD

Add more metadata to masjids

Currently the only metadata is the masjid name, and because it is used as the topic name, it has some restrictions (cannot have spaces). We can add more metadata like display name, URL, and address.

Change scraping path in TMA scraper

TMA does not update the displayed table on midnight, but rather on 9am EST.
This table gets its data from the JSON path: .masjid[prayer]. The better way is to get the data from the full year schedule which is available in the path: .masjid.iqamas[nearest_day].

Many of the texts are Tile Cased

Like Pm Am, ... even jumuas. I think this is unnecessary. If we want to title case the prayer names, that would strictly apply to this column only in all tables, or have a lookup.

Optimize generation of static site

The generator takes 30+ seconds, most of them spent in generating the static site.
See below, the gap is where it is busy running the build. Can we make it faster?

Error building svelte

When invoking the function locally, I get this:

> [email protected] build
> vite build


vite v4.4.11 building SSR bundle for production...
✓ 73 modules transformed.
✓ built in 540ms
Could not resolve "./notified.json" from "src/routes/+page.server.ts"
file: /Users/hossam/Workspace/wwpray/generator/svelte/src/routes/+page.server.ts
error during build:
RollupError: Could not resolve "./notified.json" from "src/routes/+page.server.ts"
    at error (file:///Users/hossam/Workspace/wwpray/generator/svelte/node_modules/rollup/dist/es/shared/node-entry.js:2287:30)
    at ModuleLoader.handleInvalidResolvedId (file:///Users/hossam/Workspace/wwpray/generator/svelte/node_modules/rollup/dist/es/shared/node-entry.js:24860:24)
    at ModuleLoader.resolveDynamicImport (file:///Users/hossam/Workspace/wwpray/generator/svelte/node_modules/rollup/dist/es/shared/node-entry.js:24920:58)
    at async file:///Users/hossam/Workspace/wwpray/generator/svelte/node_modules/rollup/dist/es/shared/node-entry.js:24807:32
{
    "errorMessage": "Command failed: npm run build",
    "errorType": "Error",
    "stackTrace": [
        "Error: Command failed: npm run build",
        "    at checkExecSyncError (node:child_process:887:11)",
        "    at execSync (node:child_process:959:15)",
        "    at /opt/homebrew/lib/node_modules/serverless/node_modules/cli-progress-footer/lib/private/cli-progress-footer/disable-props.js:87:41",
        "    at handleSubProcessSync (/opt/homebrew/lib/node_modules/serverless/node_modules/cli-progress-footer/lib/private/cli-progress-footer/disable-props.js:33:15)",
        "    at childProcess.execSync (/opt/homebrew/lib/node_modules/serverless/node_modules/cli-progress-footer/lib/private/cli-progress-footer/disable-props.js:86:11)",
        "    at module.exports (/Users/hossam/Workspace/wwpray/generator/generate.js:15:5)",
        "    at module.exports.handler (/Users/hossam/Workspace/wwpray/generator/index.js:65:9)"
    ]
}

I run this command to invoke in the generator directory:
serverless invoke local --function generator

Add about us, fork on Github and send us feedback links

Use masjid display name in email notifications and SES topic descriptions

Explain what wwpray does

• Expand the name to Where & When to Pray.
• Explain how it works in an intro paragraph
• Explain what subscribe does with no spam policy and unsubscribe immediately anytime

Add Arehman-ICNA masjid to scrape sources

https://aric-icna.ca/

Parse iqama times to be properly sortable

With the introduction of #9, we need to ensure prayer times are sorted correctly as timestamps rather than strings. This avoids cases where 08 coming before 7 and 1 pm coming before 12 pm.
Times should be parsed as time components for today and in the correct timezone for each masjid. The output should be the number of seconds since midnight in UTC.

Highlight the next prayer in all masjids

This basically answers the main question: Where And When to pray! We can even put this info above all tables.

Weird toast behaviour

When I try to register an email that is not verified as a sending identity on SES, I get Internal server error appearing in 2 toasts, one green and one red. See below:

When I quickly subscribe again, it returns 200 and shows yet another green toast along with the older two which is ok, but the older green one never disappears.

Setup a toast notification system

We need a friendly way to show notifications to the user. Showing toasts in the top center of the screen could be sufficient.

Toasts require JS to be shown. In case JS is disabled, we show an Alert/Flash message.

Changed on subtracts one extra day

Take a look at the current values for DarFoundation:

                "maghrib": {
                    "time": "6:22 PM",
                    "changed": true,
                    "changed_on": "2023-10-28"
                },
                "isha": {
                    "time": "7:45 PM",
                    "changed": true,
                    "changed_on": "2023-10-28"
                }

Take a look now on wwpray.net:

These should show today in the interface.

Now is 28-10 10:40am EST.

Create new lambda to receive feedback

Either receive feedback via form and the lambda will forward it as an email to the admins, or configure SES to receive emails and forward them.

Optimize scraper to run scraping in parallel

Currently, the scraper scrapes websites sequentially. There is a lot of wasted time in the function waiting for network (probably 99%!). This increases the cost of the function even though it is limited to 20 websites only.
It would be better to do async requests and write the final file once all scrapers finish.

Reserve a new domain

Show last updated timestamp in relative time format

Disable actual email sending for dev stage

Assign domain aliases to the CloudFront distros

Search immediately on type

With all the data loaded locally, it should be lightning fast to search immediately on every change in the search box. Currently the user has to press Enter.

Use masjid id in TMA sources instead of slug

Slugs may change. Ids are immutable.

Handle last_updated timestamp for each masjid separatley

After implementing #46, some masjids will fail to update, and the global last_updated flag will be misleading. We will still keep it as it denotes the last update for all masjids, and it is used in triggering the generator function. However, we should add to the json schema last_updated for each masjid. If a masjid fails to update, the generated data file should copy values from the previous scraping trigger. If it succeeds, new values are generated with a fresh last_updated timestamp.

Show subscribed topics in toast

Subscribed topics do not show in the green toast, I think we should show them too. They were separated in a different response item so that clients show them however they like.

Show/hide Jumuaas automatically

Show/hide Jumuaas automatically when it is Thursday evening to Friday evening and hide it otherwise. Users can always toggle.

Change frequency of scraping to once a day

This makes the "Changed" flag more meaningful as it will mean "changed since yesterday".

Use JS API to generate static site

AWS Lambda has limitations running npm commands, or probably commands in general (see below for the error when invoked on aws, it works perfectly fine locally). The original plan was to build the static site using a node function (JS API), not a shell command. The current implementation is a hack where it invokes a shell command from a node function.
There are 2 pieces in this puzzle:

1. Installing the dev dependencies automatically. This can be done by merging the 2 package.json files, the root one in the generator service and the one in the svelte directory. I hope there was a cleaner method in node, but we may need to get around by doing this.
2. Invoking the build using JS API. I found this: https://v2.vitejs.dev/guide/api-javascript.html#build. It seems possible with some configuration I guess to accommodate for svelte static generation.

---

Error logs:

10:26:44:353 PM - Oct 12 2023
AWS LAMBDA
dev
wwpray
generator
generator-dev-generator
npm ERR! code ENOENT
10:26:44:353 PM - Oct 12 2023
AWS LAMBDA
dev
wwpray
generator
generator-dev-generator
npm ERR! syscall mkdir
10:26:44:354 PM - Oct 12 2023
AWS LAMBDA
dev
wwpray
generator
generator-dev-generator
npm ERR! path /home/sbx_user1051
10:26:44:354 PM - Oct 12 2023
AWS LAMBDA
dev
wwpray
generator
generator-dev-generator
npm ERR! errno -2
10:26:44:370 PM - Oct 12 2023
AWS LAMBDA
dev
wwpray
generator
generator-dev-generator
npm ERR! enoent ENOENT: no such file or directory, mkdir '/home/sbx_user1051'
10:26:44:370 PM - Oct 12 2023
AWS LAMBDA
dev
wwpray
generator
generator-dev-generator
npm ERR! enoent This is related to npm not being able to find a file.
10:26:44:370 PM - Oct 12 2023
AWS LAMBDA
dev
wwpray
generator
generator-dev-generator
npm ERR! enoent 
10:26:44:372 PM - Oct 12 2023
AWS LAMBDA
dev
wwpray
generator
generator-dev-generator
npm ERR! Log files were not written due to an error writing to the directory: /home/sbx_user1051/.npm/_logs

FYI, the file-system for lambda is read-only. The only writable location is /tmp. I am not sure what led to the creation of the above folder. It could be easily fixable from the npm command.

Host static site using CloudFront

Determine paths to cache.

Website looks messy on real mobile devices

Looks messy on mobile, there is extra padding, and the bell icon has no reason to be there. Also, when side-over is open, the scrolling functionality behaves in a weird way.

Add Jumua layout

The idea is that Jumas are excluded from the prayer layout. We discussed putting them in their own layout.
When we implement #9, we can default to this layout on Fridays (starting after latest Isha on Thursday and ending after latest Juma on Friday).

Style the email notifications

Set timeout for each masjid scraping

Some masjid websites will not respond in a timely manner during the http request which breaks the whole function due to timeout. Allow a limited timeout for each request and compute the required function timeout accordingly. If a masjid request fails, move on to the next one without update.

Email notification sent even with no changes