issa16 / cogs3

Project Management System

cogs3

Project Management System

• Development
• Sequence Diagrams

Development

1. Checkout the source code.

   mkdir -p ~/code && cd $_
   git clone [email protected]:issa16/cogs3.git

2. Create an env file.

   cd cogs3
   mv cogs3/.template_env cogs3/.env

3. Build and run.

   docker-compose up -d

   Note: If docker-compose hangs when creating the cogs3_web_1 container, you may need to add the cogs3 directory to the list of file sharing resources.

4. Obtain an SSH connection to the cogs3_web_1 container.

   docker ps
   docker exec -it <id_of_cogs3_web_1_container> bash

5. Run the database migrations.

   python3 manage.py migrate

6. Load data fixtures into the database.

   ./load_fixtures

7. Create an admin user.

   python3 manage.py createsuperuser

8. Run the unit tests.

   python3 manage.py test -v 3

9. Generate coverage report.

   coverage run manage.py test
   coverage html

10. Enter http://localhost:5000/ in a browser to see the application running.

Sequence Diagrams

• User Role Sequences
• Tech Lead Role Sequences
• RSE Role Sequences
• Sysadmin Role Sequences

cogs3's Issues

Refactor unit tests to use test data fixtures.

Description
Refactor unit tests to use test data fixtures instead of creating model instances inside setUp() functions.
Test data fixtures should reside in app/fixtures/tests/fixture_name.json

Apps

• dashboard
• institution
• system
• users
• project
• openldap
• security

Phase 2 release to production

TODO

Before release to prod

• Run database migrations locally against a psuedo copy of prod
  Issues

1. OperationalError "Incorrect string value Fixed database character set (utf8).

2. django.db.utils.IntegrityError: (1452, 'Cannot add or update a child row: a foreign key constraint fails (cogs3.#sql-3da_b, CONSTRAINT project_project_attributions_id_0460870a_fk_funding_a FOREIGN KEY (attributions_id) REFERENCES funding_attribution (id))')

---

• Test LDAP actions

After release to prod

• Ensure database migrations have run correctly
• Test LDAP actions
• Create a test project and user at each institution

Add RAM requirements to project application form.

Description:
Can RAM requirements also be added to the list, we have some users who are likely to request low numbers of cores but a lot of RAM

Enable users to update personal information

Description

Users should be able to access a view for editing their profile and personal information.

Tests

Unit tests for the interaction between the form and the model
Integration test for accessing and submitting the form

email_user function duplicated code

The version in common/util.py should be able to replace the one in users/notifications.py completely (check?). (phase-2)

Restrict project applications to institutional users only.

Description

Restrict project applications to institutional users only.

Tests

• Integration tests to ensure the 'Create Project Application' button and form is only visible to institutional users.

Create a Django management command to import existing users.

Create a Django management command to import:

• Shibboleth users
• External users

Work branch: https://github.com/tystakartografen/cogs3/tree/user-migration

Replace signals by overriding the model's save and delete methods.

Description
For maintainability, replace all signals by overriding the model's save and delete methods.

Add the ability to submit project membership requests using SCW, HPCW or ARCCA project codes.

The user should have the ability to submit a project membership request using a SCW, HPCW or ARCCA project code.

Support external collaborator login.

Description
Allow pre-approved external collaborators to login to the Django application directly and not via a shibboleth identity provider.

Define Project Funding Sources

Description
Project funding sources should be represented as a separate objects, so each project has multiple funding sources, and each funding source has a PI.

@M4rkD @rantahar

Rename the PI in a project context to "Project Supervisor" or similar

Given that issue #54 would allow multiple funding sources for a given project, which is indeed the case in existing projects, the term 'PI' in the project context may lose some of it's original meaning.

Does it make sense to retain this information (i.e. who is "in charge" of this project), but give it a friendlier and less confusing name, to reflect it's purpose? This is purely a change of nomenclature, keeping existing functionality.

One suggestion could be "Project Supervisor"? With a tick box which allows the tech lead to specify is they themselves are the project supervisor.

The terminology "Principle Investigator" would then be restricted to the funding requests, to avoid confusion of multiple definitions of the same term.

Institution could be derived from technical lead email

Would it make sense to derive institution from the technical lead email address? Or is this a feature to allow a users to create a project belonging to a different institution?

I would suggest that this is a limited use case, which could create a lot of edge cases which we would have to test. Might make our lives easier to derive it from the technical lead email address and remove it from the form.

Send email notification when project status set to rejected, suspended or closed.

Description
When a project is rejected, suspended or closed the project owner should be sent an email notification with a short explanation as to the decision.

• Add a 'comment' field to the project model.

Send email notification on project / project membership status update.

Description
The relevant users should be sent an email notification upon a project or project membership status change.

Signal: Project status.
Notify: Project owner.

Signal: Project membership.
Notify: User requesting membership.

Not compatible with python 3.7

'async' is a reserved keyword in python3.7, but it is used as a function argument in some of the dependencies.

Either set 3.6.x as the only working python version (in the readme), or find which package is causing the problem exactly in requirements.txt and upgrade it?

Create a development docker image.

Description
Create a development docker image to assist potential contributors.

Redirect after login not working

Reproduction:

1. Logout
2. Visit a link to a page (e.g. https://my.supercomputing.wales/projects/create/)
3. Login page shows expected URL (e.g. ?next=/en/projects/create/)
4. After login, user is redirected to dashboard page (i.e. the initial link is ignored)

No root cause investigation done yet.

Unhandled error 500 when creating projects with duplicate titles

Creating a project with a title which exists in the database violates the unique key constraint. This is unhandled and returns with code 500.

Date picker uses different date format to date field

In the project request form, the Start date and End date fields use the jQuery date picker, which outputs a date in the MM/DD/YYYY format. However, the model expects a date in the format DD/MM/YYYY, so valid dates are stored incorrectly 40% of the time and are rejected 60% of the time. Either the jQuery widget should be told to use the DD/MM/YYYY format, or both the widget and the model should standardise on ISO 8601 (YYYY-MM-DD).

Define user terms and conditions.

Description
Define user terms and conditions.

Read email address from shib session and not registration form.

Description
Either read the email address from the shib session or ensure the email address submitted matches that of the email address stored in the shib session when registering a user.

permission_required pages redirect issue

Generic views with permission_required do not redirect correctly when permission_required condition is not met and a user is logged in. Too many redirects error message thrown.

@rantahar

Project owner permissions not revoked

Project owner group to the tech_lead of a project in signals/project. However is not removed from "previous" tech_lead.

Any attempt to change tech_lead could result in previous tech lead still having project_owner permissions

PI identified by email in project creation form

In the Project object and the project creation form, identifying the PI by email be more unique than a free form name field. PI name should be replaced by a "PI email" field, which should be validated as an email address. Only email addresses from an institution should be allowed.

Only one project can be in an awaiting approval state at a time

Attempting to create a new projects while there is another project in an awaiting approval state, violates the unique key constraint on project code (since both are null).

Project owners can invite users to join an approved project.

Description
Project owners can invite users to join an approved project.

Create a 'project approval' group.

Description
Create a 'project_approval' group and assign the required permissions in order for the group to have the ability to approve projects applications.

A member of the 'project_approval' group should also be able to see a listing of pending/unapproved projects per institution.

Removing a user from a project doesn't work

When a user is deleted from a project via Cogs they still show up as a member of the project from the groups command. I tried removing a user (a.abm7) from project scw1122 a few days ago but groups still shows membership of that group/project and presumably they can still run jobs against it. I'm not sure if this is a bug with Cogs itself or some of the supporting LDAP scripts.

Allocation storage detail not shown in project view

Allocation storage is a field on the database which is not populated by the project form, but displayed in a single project view. The form instead requests "Home storage in GB" and "Scratch storage in GB", which I believe should be displayed in the project view instead.

Null project code throws unique constraint error.

Description

• Submit a project application, the project code will be set to an empty string.
• Submit another project application, the project code will be an empty string. This will result in a unique constraint error.

The project codes must be unique but not when the project code is an empty string.

ARCCA displayed for Swansea users

The ARCCA textbox is displayed for all users. I suspect that Swansea users would not know what this is.

Since the system goes live for Cardiff only first, then this is not an immediate concern, but I believe this should be addressed before Swansea goes live.

PI email stored as a user

This has been discussed, adding here for visibility.

There are some possible issue, that has been flagged up here. The key one is that we need to validate the PI of any funding attributed is aware of the attribution and agrees to it in writing, in a way that can be audited. The other is more of a "nice-to-have" than an issue, wanting to allow the PIs to log in and engage with the system should they desire, but making this optional.

There needs to be some discussion around the mechanisms that could be put in place to support this, and at what point these mechanisms would make sense to implement.

Here is a possible approach that would make sense to me, but would need some discussion:

Each project has one or more "funding sources", funding sources have a PI, a total amount, an attributed amount, a status (application/granted/rejected) and a start date. Funding sources can be added at any point, as the project grows or changes over time, and can periodically be reviewed by (for example) by an attribution board. Addition of each funding source would (eventually) require an "OK" from the PI of that funding source. PIs get emails with a mailto link when they're declared as a PI on funding, which they can click to get a pre-populated email which they can email to us to give their consent.

Multiple funding sources, with potentially different PIs, can then co-exist nicely within a single "project", which is a situation that we do see in some bigger, longer-running groups here. In the current approach, in this situation, if a group started generating multiple funding streams over several years, and wanted to attribute them, they'd have to have a project per funding stream projects - which could very quickly become a headache for sysadmins.

Another thought, who is the "Project PI" is, in a project has multiple funding streams? Is this then the "PI" in the sense of the person who is submitting the application for compute-time resources? Maybe this makes sense? Or maybe the concept of per-project PI needs to be revisited and firmed up a bit?

Add home and scratch disk space requirements to project application form.

Description
Can the project application form differentiate home and scratch in disk space requirements, we have some users who store vast amounts of data on scratch but very little on home/groups.

LDAP system allocation integration.

Description

Integrate the LDAP system allocation API to support Hawk and Sunbird.

Extend shibboleth user profile.

Description
Add the following fields to the shibboleth user profile.

• College/Department.
• ORCID
• Scopus URL
• Homepage
• Cronfa URL

Add the ability to attach documents to project applications.

The user should have the option to attach any supporting documents to a project application form.