kartoza / sawps

SANBI WPS

🦏 SANBI WPS

The SANBI Wildlife Protection System is a platform to track the population levels of endangered wildlife.

The work is funded by the South African National Biodiversity Institute

This project is implemented by Kartoza under contract to IDS.

📒 License

This program is free software; you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version.

Read the full license.

🧑🏽‍💻 Development

Please follow the Kartoza coding standards.

🏃‍♂️ How To Run Project with Visual Studio Code

Copy and customize environnment file: Create a copy of the deployment/.template.env file and name it deployment/.env. Update the deployment/.env file with your project-specific settings.

Ensure Dev Containers extension is installed

Build and open project in devcontainer: press Cmd+Shift+P (macOS) or Ctrl+Shift+P (Windows/Linux) to open the Command Palette. Type Dev Containers: Rebuild and Reopen in Container and select it to reopen the folder inside the devcontainer.

Change the permissions of the django_project folder:

chmod -R a+rw django_project

Install dependencies inside the container: Press Cmd+Shift+P (macOS) or Ctrl+Shift+P (Windows/Linux), type Tasks: Run Tasks and select it. Choose React: Install dependencies to install the necessary dependencies.

Run the migrations inside the container: Press Cmd+Shift+P (macOS) or Ctrl+Shift+P (Windows/Linux), type Tasks: Run Tasks and select it. Choose Django: Migrate.

👩‍💻 Open the developer site

Run the project inside the container: Press Cmd+Shift+D (macOS) or Ctrl+Shift+D (Windows/Linux), choose Django+React in the RUN AND DEBUG dropdown.

Review the list of forwarded ports from the container to your local host:

Open your web browser and go to localhost:8000 to view the running application.

👩‍🏭 Create a super user

💽 Restoring layer schema for map

The full dump file for layer schema is ~5.3GB, meanwhile compact dump file is only 24MB.

Download Full Dump File for Layer.

Download Compact Dump File for Layer.

Preview for compact dump file:

Copy the dump file to db container. Then run pg_restore from inside db container to restore the dump file.

docker cp sanbi_layer_db_compact.dump deployment-db-1:/home/sanbi_layer_db_compact.dump
docker exec -it deployment-db-1 /bin/bash
cd /home
pg_restore -h 127.0.0.1 -U docker -d django -n layer sanbi_layer_db_compact.dump

💻 Resources

Database ERD

Figma Board

sawps's Issues

Wireframe: Admin Environment (Admin)

We need a administration environment page to be able to link to:

• User Administration
• Roles and Permissions
• Add Spatial Layers
• Add / Configure Species
• Data Security
• Specific species Events Content

Should link to the following pages:

• Users
• Roles and permissions
• Spatial layers
• Species
• Data Security

Review Darwin Core Spec

We need to get a copy of the Darwin Core specification to and inspect the document to ensure the platform is Darwin core compliant.

Link sent by Matthew in the sprint: https://dwc.tdwg.org/terms/

User Requirements 5: Add and configure system species (Admin form)

We need to create an add species form:

The end user will need to specify species kept on their property and will likely be entered using this form. When entering species the information should link to a property selected from a spatial database which automatically collects information (like street, city, area). The form should be editable / changeable to be able to meet future requirements.

User Requirement 9: add, or update species population and/or activity data

The user should be able to select species information from an online form linked to the property.
The user should be able to download a form which can be edited and uploaded to capture species information manually.

test ticket

The user should be able to select species information to be captured form an online form.
The user should be able to download a form which can be edited and uploaded to capture species information manually.

Basemap: filtering data

Going through all acquired potential data for basemap and filtering out what are usable and required layers.

User Requirements 14: Configure reminder notifications (Admin)

System administrators must be able to configure triggered emails to remind system users to provide updated data at specified times.

Notifications should be sent to the data providers' email addresses stored as part of their system profile

User requrments 36/37/38: Extract data as CSV/Extract data as report/ Save extracted data

A side panel on the interactive map that allows for the visualization of the data and for the user to download, export and view reports as follows:

• Extract data as CSV
• Extract data as report
• Save extracted data

Pillar 1: Data Access and Control (wireframeing)

This overarching milestone ticket that will then be broken down into sub-tickets to create a mockup required for each feature in the overall platform wireframe
the following functionality is required in Pillar 1:

• Add a new user account
• Configure roles and permissions
• Assign role to account
• Add spatial layers
• Add and configure system species
• Data security

User Requirements 8: Add profile information (species and property)

In the user profile, they will have a link to a properties manager where the user can access a list of their properties. They can create, edit and add information to each property, sutch as species, population, and incidents.

It will be essential to have a map associated with this functionality that the user can use the select and or digitize their property and assign data to it through various forms and spreadsheet uploads.

User Requirement 8: Add profile information (species and property)

We need a page for users to be able to add and edit profile information

User Requirement 15: Log in (also UR6 : data security)

Users shoul be able to login to their user account

Use Case 2: Configure roles and permissions

We need a roles and permissions page to be able to:

• List roles and permissions information
• add roles and permissions

This should link to a add roles and permissions form where administrator can edit/add user access to information.

Use Case 1/3: Add new user account/Assign roles & permissions to user accounts

We need a users page to be able to:

• View and configure user permissions
• Admin add a user
• Link data provider properties

Should link to:

• Add user form

The administrator should be able to add users and information. The system should also collect data related to users and be displayed in a logical manner. The complete list of information collected from the users are still TBC but include validated personal information (email and cellphone) spatial information (property location and boundaries and species information (species of animals kept). Before a user can access the system the user MUST also agree that information can be collected with in their limited right and stored in a POPIA compliant manner.

User Requirement 11: View own species data (map, dashboard, table)

User should be able to view / filter and edit their own data

A user should be able to define their property by means of uploading a shapefile

Some of the users might need to do a bulk upload of data and would need to be able to upload data for multiple properties. The functionality to define multiple properties in by a means other than selecting cadastral bounds or by manually digitizing a property is also required.

User requirement 14: System communications option for Admins

A system by which to send communications to the platform users as an admin

Set-up Deployment environment

We need a working deployment environment setup by 1 April for developers to be able to start.

Wireframe: interactive map viewer and tool

We need a environment where spatial information can be viewed. The full functionality of this "map viewer" is TBC but includes:

• Selecting properties form a map
• Defining property borders
• View and filter species information

User Requirement 27: View species population data, as trends

System requirement.

User Requirements 14: Configure reminder notifications (user)

The user will receive automatic notifications from the system but will also be able the create reminders for them selves to update and upload data periodically.

Pillar 2: Data Collection, onboarding & transformation

This is an overarching milestone ticket for Pillar two that will be broken down into sub-tickets to create the wireframes mockup for the following required functionality:

• Create new account
• Add profile information (species and property)
• Add, or update species population and/or activity data
• Upload species population and/or activity data
• View own species data
• Edit account information
• Edit profile information
• Configure reminder notifications
• Log in
• Pull data from other systems or online databases
• Species habitat and biology information
• Provincial habitat information view

Basemap: Acquiring data for basemap

Need to get vector layers:

• Roads
• Rivers
• Dams
• Protected Areas
• Place Names
• Cadastral Boundaries
• Ecosystem Type
• Critical Biodiversity Layers
• Overlay for larger scales instead of the aerial imagery - went with a simple polygon of South Africa Admin boundary 0

Need to get raster:

• Satellite Image tiles (it was suggested to use a pre-existing service like Google satellite tiles or something similar)

User Requirements 5: Add and configure system species (Admin)

We need to create a species list page to be able to:

• List all species and information
• Add Species

Will link to an add Species form

Basemap: Styling Layers

Style the following layers:

• Roads
• Rivers
• Dams
• Protected Areas
• Place Names
• Cadastral Boundaries
• Ecosystem Type
• Critical Biodiversity Layers
• Overlay for larger scales instead of the aerial imagery - went with a simple polygon of South Africa Admin boundary 0

We need to create a training document

We need to create documentation for users to be trained on how to use the platform

Add spatial layers and filters (admin)

In the administration environment, the user needs to be able to add spatial layers that can be used as filters for the map interface/ tool

We need a manual testing an validation process

We need a manual testing an validation process. We can copy the manual testing and validation process from planet

User Requirement 14: Make system mailing lists (admin)

A way to make mailing lists for system communications by administrators.

Use Case 1: Add new User Account

Test if a new user has been added. Tests if an Administrator can add, view, update and assign permissions to all users on the system.

User Requirement 2: Configure roles and permissions (form)

A form for capturing new roles and permissions information to assign to users.

Graphic Design and Site Component Creation

Creating the theme and style aspects of the wireframe for the platform and the components to create the wireframe within the theme.

User Case 4: Add spatial layers (Admin)

We need a spatial layers page to be able to:

• List all spatial layers and information
• Add layers

Discuss different login method for users with certain roles / security clearance

The login for data upload users should be simple but yet secure in order to encourage users from private backgrounds to upload data to the platform. Using high security login methods like a yubikey might be to complicated for average or inexperienced users and discourage users from using the platform.

Users with access to sensitive information (e.g. like SANBI admin team) should however need to follow the most secure login process and authentication methods in order to secure the platform.

We will need to have different login and authenticating methods to satisfy different levels of security for the platform.

Use Case 1: Add new User Account

Adding a user account needs to implement. A general user needs to be able to register and the system automatically creates them a user account with base privileges. Administrators needs to be able to add, view, update and assign permissions to all users on the system.

Admin Add User:


User Registration Flow:

Basemp: Getting local instance of tegola/postgis/maputnik up and running to style basemap using maputnik

Setting up a local instance of tegola, maputnik and a postgis database to style the acquired layers in maputnik. This is done in order to create a json style file to be uploaded on the final server.

User case 4: Add spatial layers forms (admin)

A form that points the platform to the spatial information required for the system is likely stored and managed in Geoserver.

We need to think here about context layers vs, filtering layers

Use Case 1: Add new User Account

Test automation:
Users should be able to register on the platform and Administrators should be able to create User Accounts. Test if a new user can be added. Tests if an Administrator can add, view, update and assign permissions to all users on the system.

Wiring up the mock-up to make it interactive.

Adding the prototype links, animations, and interactions to the mockup.

Visualization Metrics for Dashboard

We need to know what metrics need to be used to visualize data on the dashboard.

Clarification of terms from the user document

SANBI Functionality Questions:

1. What databases should the system pull data from?

2. CITES

3. ?

4. ?

5. What high-
   level summary

statistics and
data
visualisations
would be
useful on the
first page?

3. What
   information
   should be in
   the summary
   reports the
   users and

decision-
makers can

download?
4. Can you provide us with a sample data collection/ upload template? What data beside population
is captured etc.
5. Can you provide some sample or example data visualisations and explain some of the metrics so
that we can understand better and so that we can incorporate them into the wireframe?
• View the number of properties per population category (What is meant by population
category? Does it mean species?)
• View the number of populations per population category (What is meant by population
category? Does it mean species?)
• View the number of populations per population category, per area (What is meant by
population category? Does it mean species?)
• View species population data as Trends (e.g. population growth/loss over time? What other
population metrics apply/are needed and can be visualised as trends? Do you want these to
be automatically calculated by the system and shown on a dashboard that the user can view
and download? The requirements mention BI and the user creating their own visualisations
is this Power BI?)
• View species activity data, as totals, by method (What is a species activity, just a definition
so we know we are on the right track and what does method mean in this instance. )
• View activity data, as % of total population (What is an activity defined as?)

Basemap: Upload data to dedicated PostGIS server

• Generate XYZ tiles
• Find out to which server/service the upload must occur
• Upload XYZ tiles

Create a Data Model of the platform

We need a data model of the platform

Add a new user account (admin)

Adding a new user by the admin and adding and updating user information in the admin environment.

User Requirement 6/7: Data security/Create new account (Wireframe: Registration from)

We need a registration page for users to be able to register a new user account. User email and cellphone number should be validated

User Requirement 16: Pull data from other systems or online databases (admin)

Several DB will be used by the system to periodically pull data from the sources referenced in section 6.8.

A list of these default harvested DB's will be available in the administration environment, however, there will be the functionality to add and remove harvesters as needed by an admin.

Pillar 3: Data filtering, visualisation & export (wireframe)

This is an overarching milestone ticket for Pillar 3 Data filtering, visualization & export it will be broken down into sub tickets that cover the functionality required in the mockup as follows:

• Filter by fauna species name
• Filter by time period (annual)
• View species population data, as totals, per area
• View species population data, asdensity, per area (leopard)
• View number of properties per population category
• View number of populations per population category
• View number of populations per population category, per area
• View species population density per property
• View species population data, as trends
• View species population data, as a map
• View species population data, as a map
• View species activity data, as totals, by method
• View species activity data, as totals, by method, by area
• View activity data, as % of total population
• View species activity data, as trends
• View all species activity data, as a table
• View species population data, as a map
• Extract data as CSV
• Extract data as report
• Save extracted data

We need two factor authentication to secure the platform

We need to secure the user authentication so their account cannot be hacked.
Review the link below:
https://pypi.org/project/django-mfa2/

Use Case 1: Add new User Account

In the data model a table for user information including
Name
Title
Surname
Date of registration
Organization
Role
Email address
Cell phone number
Data use permissions (this may need to be added)
Mailing lists
User ID
Event Type (log in, log out)
Timestamp
IP address

User Requirement 28: View species population data, as a map

A page that allows the platform user to:

• Filter by fauna species name
• Filter by time period (annual)
• View species population data, as totals, per area
• View species population data, asdensity, per area (leopard)
• View number of properties per population category
• View number of populations per population category
• View number of populations per population category, per area
• View species population density per property
• View species population data, as trends
• View species population data, as a map
• View species population data, as a map
• View species activity data, as totals, by method
• View species activity data, as totals, by method, by area
• View activity data, as % of total population
• View species activity data, as trends
• View all species activity data, as a table
• View species population data, as a map

Through an interactive map with layers that can toggle on and off and different layers to use as filters.