ada-anu / metabase-to-wordpress-updates

METABASE-to-WORDPRESS-updates

Code to update the ADA wordpress website (https://ada.edu.au) with newly published, and recently updated, datasets.

The metabase API is used to query results from the Dataverse postgres database for new and updated datasets.

The wordpress API is then used to write new posts to the ada website. The categories of these new posts are set such that the new posts are displayed in the correct section of the wordpress site.

The code also updates the ADA twitter feed to let twitter users know when there are new and updated ADA datasets.

metabase-to-wordpress-updates's Issues

As an ADA website owner, if a dataset is updated to a new major version and the ‘recently updated dataset’ post has not expired, I want to leave it in the ‘recently updated dataset’ section but highlight its new version so that it is obvious to external users that the dataset had been updated

Question: can visual tags of ‘updated’ ‘version: x.0’ be added to the card displaying the post? see #9

Question: should posts be sorted by publish date so that the most recent update is the first from the left (top left corner if multiple rows)? Perhaps that is the default.
Or should posts be left in their original order?

As an ADA twitter account owner, if there are multiple new/updated datasets on Dataverse, I want them all to appear in one tweet so that the twitter feed is not overwhelmed with individual tweets if there are many new/updated datasets.

If there are more than y (3?) new datasets, list the most recent y, and add a link to dataverse with text ‘Check Dataverse for recently published/updated datasets’ to the tweet.

Question: If there is at least one tweet on a specific day (ie. At least one new or one major version update has occurred on a specific day), should there be a single tweet that is continuously destroyed/created with new info if there are subsequent new/updated datasets published?
i. Tweets have ids but is it possible to determine which tweet on a specific day is for announcing new/updated datasets so that this tweet can be destroyed/recreated with each new dataset?

1. Can create tweet and somehow get the id of the tweet just created and store that somewhere for that particular day and then delete/recreate as necessary, each time tracking the new id.

=> Mingjing’s proposed solution is to always create new tweets and not delete old ones.

As an ADA website owner, I want the ADA twitter account to be updated with newly published datasets and new versions so that twitter followers are made aware of what’s new on Dataverse and to keep the twitter feed on ADA website fresh.

As an ADA website owner, I want any new and/or updated datasets to display on the ADA website within x hours or x minutes so that visitors to the site are aware of new and updated datasets in a reasonably timely manner.

Ideally, this refresh of the ADA website would occur when a dataset is published ie. with a push notification from Dataverse.

Question: Is there a way to implement this functionality as a push rather than polling?
Question: What is considered timely? 15 minutes? An hour? Once a day? see issue #3

As an ADA website owner, I want new/updated dataset statements to be created as posts in wordpress so the post can be managed, modified, archived, deleted, etc.

Example: precondition: script polls every hour
i. dataset x is updated twice in the same hour from v1 -> v2, v2->v3
ii. result: want ADA post to reflect v3 only

As an ADA website owner, I want a ‘default’ ‘check back soon’ post in each of the NEW and UPDATED sections that will be visible if there are less than x datasets in the corresponding category so there is always something visible in the corresponding section.

See #1

As an ADA website owner, if a dataset that is ‘newly published’ is updated to a new major version and the ‘newly published dataset’ post has not expired, I want to leave it in the ‘newly published dataset’ section but highlight its new version so that it is obvious to external users that the dataset is ‘new’

Q: can visual tags of ‘new’ ‘updated’ ‘version: x.0’ be added to the card displaying the post? see #9

Not sure if visual tags add anything visually. There is already a lot of text and the area where the card lies indicates if it is new or updated. Tags may just be visual clutter. Closing.

As an ADA website owner, I want an external user to see basic information about a newly published dataset

The post should include the following:
a. Title
b. Description (full description shows in actual post; excerpt shows in Blog module ‘card’)
c. Version (major and minor)
i. This is in the sql details
d. DOI
e. Link to dataverse dataset that will open dataset in new tab/window
f. Date published/updated?

Q: can visual tags of ‘new’ (or ‘updated’) be added to the card displaying the post? see #9

Q: do we sort the posts by release date so that the most recent update is the first from the left (top left corner if multiple rows)?
Or do we leave the cards in their original order?
Answer: Sort posts by release date so most recent is the first from the left

Can visual tags of ‘new’ ‘updated’ ‘version: x.0’ be added to the card displaying the post?

As an ADA website owner, I want the most recent x newly-published datasets to appear in the ‘Newly Published Datasets’ section of the website.

Similar to issue #5

Question: Should finite number (x) be shown or all?

Answer: The most recent 20 are shown for both new and updated datasets.

Question:
Should these posts expire after a certain amount of time?
If they are to expire, how long after publishing post?

Answer: They don't expire but just aren't shown once they are no longer in the 20 most recent.

As a developer, implement a script scheduled to run every <chosen time interval> to poll the metabase query and to write/update posts with the worpdress API.

There doesn't seem to be a push notification to initiate the update to ada.edu.au and twitter.

So write and deploy a (python?) script that runs every x [interval of time] by virtue of scheduling. see #3

Schedule this script as a cronjob on an ADA VM (dataverse-dev.ada.edu.au?)

deploy application to docker container

Create a docker image to allow application to be deployed anywhere

As an ADA website owner, I want there to be 2 sections on the ADA website so that site visitors can identify NEW datasets and UPDATED datasets on dataverse PROD

The ada website needs to be updated with datasets from dataverse that are:

1. NEW
2. UPDATED - (major versions only)

To implement this there will be 2 sections on the front page in the ada website:

NEW Datasets
UPDATED Datasets

As an ADA website owner, I want the polling script to update the ADA website in a timely manner so website visitors are made aware of recently new and updated datasets. of

Question: What is considered timely? 15 minutes? An hour? Once a day?

Answer: For testing, will run it every 15 minutes. Time period will have to be decided for production mode.

As an ADA website owner, I want the most recent y updated datasets (major version only) to appear in the ‘Recently Updated Datasets’ section of the website.

Only new major versions (v2 +) should be displayed.

Similar to issue #4
Question: Should finite number (x) be shown or all?

Answer: The 20 most recent major version updates are shown.

Question:
Should these posts expire after a certain amount of time?
If they are to expire, how long after publishing post?
Answer: They don't expire but just don't show if they are not in the 20 most recent major version updates.