fronteerio / grasshopper Goto Github PK

Please add me as contributor. Made some changes to the source and would like to commit here.

Potential spots

• When outputting events via iCal they should be formatted in UTC (and the server should run as UTC)

I have 5 organisational units (module type), each with the same parent. 2 of those organisational units have event series associated with them.

When I request /api/orgunit?app=1&includeSeries=false&parent=1112&type=module I get the correct results: 5 organisational units without series included.

When I request /api/orgunit?app=1&includeSeries=true&parent=1112&type=module I get incorrect results (note the only difference being includeSeries). I'm expecting 5 organisational units to return, 2 of them having an Array of event series, 3 of them no event series. However, what I get is only 2 organisational units, the 2 that have event series associated to them. I would expect the 3 without event series to show up as well.

A set of REST APIs will be made available through which the application administrator can configure a T&C agreement for the application.

When a user signs into Grasshopper for the first time, they will need to accept the T&C before they are able to use the application. Every time the T&C are updated, users will be required to re-accept the T&C.

A dedicated REST API endpoint will be developed to assist in the yearly rollover of Timetable data to the new academic year. This endpoint will be responsible for calculating the offset of each event and converting the data of the event to the new academic year. The endpoint will also be responsible for creating the user accounts for all organisers associated to the rolled over events.

The rollover REST API should be available to application administrators only.

Endpoints:

• GET /api/orgunit/:id/export?type=<csv|json>
• POST /api/orgunit/:id/import

Export endpoint

This endpoint will dump all the direct organisational units and their series/events under a specified organisational unit.

Within the TimeTable context this can be used to get the data for a specific part:

• All the modules under that part
• All the series in the modules of that part
• All the events in the series of the modules of that part

URL:

GET /api/orgunit/:id/export?type=json
TODO: Define CSV format

Data:

[
  {
    "id": 10                                                      // The module id
    "externalId": "eng-first-elec",                    // The module's external id as given to us by the engineering department
    "displayName": "Electricity",                     // The module's displayName
    "published": true,                                       // Whether the module is published to students or not
    "series": [
      {
        "id": 42,                                                // The series id
        "externalId": "eng-first-elec-01",          // The series external id as given to us by the engineering department
        "displayName": "Practicals",               // The series displayName
        "description": null,                               // The series description
        "events": [
          {
             "id":   1123,                                   // The event id in Timetable
             "externalId": "eng-first-elec-01-20150314",          //  The event id as in engineering their system
             "displayName":  "Practical",
             "description": null,
             "notes": null,
             "type": "lab",
             "start": "2014-03-14T14:00:00.000Z",
             "end": "2014-03-14T15:00:00.000Z",
             "organisers": [
               // A person who doesn't have a Cambridge Raven account but helps in running this lab / lecture / ..
               "Jack Jones",
               //  The following is a person who does have a Raven account
               {
                 "id":   523,
                 "shibbolethId":  "sg555",       // The Raven ID / Shibboleth identifier
                 "displayName": "Simon Gaeremynck",
               }
             ]
           },
           {
             "id":   1124,                                                           // The event id in Timetable
             "externalId": "eng-first-elec-01-20150321",          //  The event id as in engineering their system
             "displayName":  "Practical",
             "description": null,
             "notes": null,
             "type": "lab",
             "start": "2014-03-21T14:00:00.000Z",
             "end": "2014-03-21T15:00:00.000Z",
             "organisers": [
               "Jack Jones",            // A person who doesn't have a Cambridge Raven account but helps in running this lab / lecture / ..
               //  The following is a person who does have a Raven account
               {
                 "id":   523,
                 "shibbolethId":  "sg555",       // The Raven ID / Shibboleth identifier
                 "displayName": "Simon Gaeremynck",
               }
             ]
           },
         ]
       }
     ]
  },
  // A second module..
]

Import endpoint

This endpoint will take a set of data and ingest it into the system. It allows an application administrator to import data to a specific organisational unit. The idea is that the import API is able to take in the same data as generated by the export API. This means that users could take a dump through the export
API, manipulate it (but retain the formatting, ids, ..) and update an entire batch of modules, series and events in one go

The following would be supported:

• Creating new modules, series and/or events. If a module, serie or event has an externalId that we can't resolve to something we know about, it will be auto created
• Updating existing modules, series and/or events their metadata. Datapoints such as display names, descriptions, start/end dates, etc.. can all be updated by changing the value.

The externalId is the primary identifier that will be used to resolve organisational units, series and events. The id field will only be used when externalId is omitted.

Both CSV and JSON are supported.

URL:

POST /api/orgunit/:id/import

Data:

See the export endpoint

Provides a ton of (configurable) linting rules: https://www.npmjs.com/package/jscs

This enables the ability to archive all the data in an entire application. It will output the following data hierarchy:

• All organisational units
• All series
• All events

Both CSV and JSON are supported

Endpoint:

GET /api/apps/:id/export

Perform a complete review of the UI to help us assess if we can take on responsibility for bug fixes to the UI.

With node 6:

$ npm i
[snip]
npm ERR! 404 Registry returned 404 for GET on https://registry.npmjs.org/gh-context
npm ERR! 404 
npm ERR! 404  'gh-context' is not in the npm registry.
npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
npm ERR! 404 It was specified as a dependency of 'grasshopper'
npm ERR! 404 
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.

With 7:

$ npm i
[snip]
npm ERR! 404 Registry returned 404 for GET on https://registry.npmjs.org/gh-admins
npm ERR! 404 
npm ERR! 404  'gh-admins' is not in the npm registry.
npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
npm ERR! 404 It was specified as a dependency of 'grasshopper'
npm ERR! 404 
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.

The Sequelize package has been updated to 3.0.0 and breaks the current master branch. https://www.npmjs.com/package/sequelize

A set of REST APIs will be made available through which the administrators for an application can be managed. These REST aPIs can be used to make regular users application administrators or revoke the administrator status of an existing application administrator. The manage application administrator REST APIs will only be available to application and global administrators.

Note that there should be NO logic whatsoever attached to this in the backend. This is purely a flag that will be used by the UI.

In order to make it easier for external applications and developers to interact with the Timetable REST APIs, the following types of OAuth support should be added:

• 2-legged OAuth, allowing for server-to-server communication
• 3-legged OAuth, allowing for third-party services to interact with the system on behalf of a user

2-legged OAuth can be used for integration between Timetable and a different trusted system. An example of such integration would be an integration of the Timetable data into Moodle courses.
3-legged OAuth will provided external application developers a way to ask permission to interact with the user's calendar on their behalf.

The implementation of OAuth scopes is excluded from this package of work. Therefore, a user will be able to use the full REST API once an OAuth access token has been obtained. Users will be able to revoke a third-party integration at any time.

The UI wants to add a property that uniquely identifies each user in mixpanel across Cambridge applications. The simplest thing we can do is to salt the crsid with a known key and md5 hash it.

This can be an extra field on the User model

I believe we can use Sequelize's new remove[AS][Plural] method once it's been released.
See sequelize/sequelize#2338 (comment)

Right now we're doing:

• get all organisers
• add in new organisers
• remove existing organisers
• persist entire set of organisers

That new method might allow us to cut down the amount of data that goes back and forth. There might be similar areas where we can do this (user subscriptions maybe?)

Global/Application administrators should be able to identify which groups a user is in.

The user can determine this himself already through:
/api/orgunit?includePermission=true

Maybe that parameter could become includePermission=<true | false | userId>.

or we could have a new endpoint ..

Once #60 (statsd impl) gets merged, a little bit of time should be spent to set up a default dashboard. It should contain the following information:

Would it be possible to allow the back-end to use a custom config.js; using a flag when the application is being started? Something in the lines of node app.js -c /path/to/config.js.

Sequelize 2.1.0 has completely removed their callback-based result passing and moved to using promises completely.

According to the maintainer we can either use promises in our DAOs or hack something on top of Sequelize.Promise.

A REST API will be made available through which application administrators can provision user accounts using a CSV file import. This user import API can be used as a one-off user import at the beginning of the academic year, be can also be used programatically to synchronise the user accounts throughout the year.

The CSC file should contain:

• The CRSID of the user (=shibboleth identifier)
• The full name of the user
  • The email address of the user

User accounts will be created for those users that do not have an account yet. Users that already have an existing user account will not be updated. The user import REST API will only be available to application administrators.

When displaying a user's calendar, the UI wants to show Belongs to course - subject; part. In order to do that, the UI needs the parent id of the module that the serie in a calendar belongs to. Using that ID they can construct the path in the orgunits tree they fetched in an other request.

For example, assume the following scenario:

There is an organisational / serie tree like this:

Engineering (course) [1]
   IA (part) [2]
      Statistics 101 (module) [3]
         Functions (serie) [1]
NST (course) [4]
   Biology (subject) [5]
      IIA (part) [6]
         Statistics 101 (module) [7]
            Functions (serie - borrowed from engineering) [1]
   Ecology (subject) [8]
      IIA (part) [9]
         Statistics 101 (module) [10]
            Functions (serie - borrowed from engineering) [1]

When students retrieve their calendars and click on the event a popover is shown. The following cases should be supported:

• An engineering student should see "Belongs to Engineering; part IA"
• A biology student should see "Belongs to NST - Biology; part IIA"
• An ecology student should see "Belongs to NST - Ecology; part IIA"

I haven't thought this entirely through yet, but I imagine that when a user subscribes to a serie, we'll need to record the context of that subscription (the module, the serie belongs to).

• The engineering student would do: POST /api/series/1/subscribe?contextId=3
• The biology student would do: POST /api/series/1/subscribe?contextId=7
• The ecology student would do: POST /api/series/1/subscribe?contextId=10

When we retrieve the calendar, we can then include the module the serie belonged to. The ParentId of the module will point to the part, this can then be used by the UI to construct the "belongs to ..." string.

Take the following example:

var shibExternalIdAttributes = 'eppn persistent-id targeted-id';
var headers = {
      "host": "shib-sp.grasshopper.fronteer.io",
      "pragma": "no-cache",
      "cache-control": "no-cache",
      "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
      "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36",
      "dnt": "1",
      "referer": "https://idp.testshib.org/idp/profile/SAML2/Redirect/SSO",
      "accept-encoding": "gzip, deflate, sdch",
      "accept-language": "en-US,en;q=0.8,nl;q=0.6",
      "cookie": "express:sess=eyJwYXNzcG9ydCI6e319; express:sess.sig=lD72sfPoT4t1NXc9TLwDEW3VsXg; shibboleth=s%3A1.g%2FaFChXR33E3zo6bTAlUmXTfRnukEcoLJB0xZs%2BVauU; _shibsession_64656661756c7468747470733a2f2f736869622d73702e6772617373686f707065722e66726f6e746565722e696f2f73686962626f6c657468=_40bfd1309004250cb84d7a1402eab08b",
      "shib-cookie-name": "",
      "shib-session-id": "_40bfd1309004250cb84d7a1402eab08b",
      "shib-session-index": "_c59be7dd840da62d819f2f35b989d779",
      "shib-identity-provider": "https://idp.testshib.org/idp/shibboleth",
      "shib-authentication-method": "urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport",
      "shib-authentication-instant": "2015-04-07T19:12:14.588Z",
      "shib-authncontext-class": "urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport",
      "shib-authncontext-decl": "",
      "shib-assertion-count": "",
      "eppn": "[email protected]",
      "affiliation": "[email protected];[email protected]",
      "unscoped-affiliation": "Member;Staff",
      "entitlement": "urn:mace:dir:entitlement:common-lib-terms",
      "targeted-id": "",
      "persistent-id": "https://idp.testshib.org/idp/shibboleth!https://shib-sp.grasshopper.fronteer.io/shibboleth!0zmBYdEakGvgdMh7P6+cusXCUm8=",
      "shib-application-id": "default",
      "remote_user": "[email protected]",
      "via": "1.1 shib-sp.grasshopper.fronteer.io",
      "x-forwarded-for": "131.111.184.27",
      "x-forwarded-host": "shib-sp.grasshopper.fronteer.io",
      "x-forwarded-server": "shib-sp.grasshopper.fronteer.io",
      "connection": "Keep-Alive"
    }

_getBestAttributeValue will always return the persistent-id value and NOT the eppn value as it equals the remote_user attibute. This is probably not what we want.

Don't forget to add functionality/tests for the patterns

In my branch, I've had to switch back to cookie-session 1.0.2 as the latest version is causing sessions to no longer work. I spent a bunch of time trying to figure out why it breaks, but wasn't able to get to the bottom of it in a reasonable amount of time.

We should investigate how we can move back to latest

This ticket's main purpose is to figure out whether we really need a config API.

The main benefits of not having a config would be:

• not having to write a config api troll face
• not having to worry about cache invalidation. I imaging that once we do introduce a config, we might want to cache it on the app servers. We then need a mechanisme to invalidate these caches, for which we'll probably need some kind of pub/sub mechanisme. This would introduce an extra component just for the pub/sub purpose. I'd rather avoid that complexity if necessary.

So far, we need some kind of "configuration" for the following things:

• Authentication
  • Local authentication enabled
  • Local registration enabled
  • Shibboleth authentication enabled
  • Shibboleth IdP
  • Shibboleth name attribute resolution
  • Shibboleth email attribute resolution
  • Other auth strategies...
• Events
  • Whether regular users can create events
• Series
  • Whether regular users can create series
• Skinning
  • Store/get skin values
• Terms and conditions
  • Store multiple versions of a T&C page

This should simply subscribe to all the series (directly) under the organisational unit

A bit of an obvious oversight, but we should probably add the event description in the ical dump.
The resulting iCal description for an event should look like:

{{type}} with {{organiser1}}, {{organiser2}}
{{description}}

{{notes}}

In a sufficiently secure environment, this isn't really a problem but we should just listen on 127.0.0.1 and let apache deal with exposing the necessary bits of logic to the outside world.