illumidesk / illumidesk

IllumiDesk enhances your LMS with Jupyter Notebooks + Auto-Grading.

IllumiDesk

This monorepo is used to maintain IllumiDesk's authenticators, spawners, and microservices. This setup assumes that all services are running with Kubernetes. Please refer to our help guides for more information.

Overview

Jupyter Notebooks are a great education tool for a variety of subjects since it offers instructors and learners a unified document standard to combine markdown, code, and rich visualizations. With the proper setup, Jupyter Notebooks allow organizations to enhance their learning experiences.

When combined with the nbgrader package instructors are able to automate much of tasks associated with grading and providing feedback for their users.

Why?

Running a multi-user setup using JupyterHub and nbgrader with containers requires some additional setup. Some of the questions this distribution attempts to answer are:

• How do we manage authentication when the user isn't a system user within the JupyterHub or Jupyter Notebook container?
• How do we manage permissions for student and instructor folders?
• How do we securely syncronize information with the Learning Management System (LMS) using the LTI 1.1 and LTI 1.3 standards?
• How do we improve the developer experience to provide more consistency with versions used in production, such as with Kubernetes?
• How should deployment tools reflect these container-based requirements and also (to the extent possible) offer users an option that is cloud-vendor agnostic?

Our goal is to remove these obstacles so that you can get on with the teaching!

Prerequisites

Kubernetes v1.17+.

Quick Start

This setup only supports Kubernetes-based installations at this time. Refer to the helm-chart repo for installation instructions.

Development Installation

Refer to the contributing guide located in the root of this repo.

Building the JupyterHubs

1. Build the JupyterHub for local testing with docker-compose or docker:

make build-hubs

1. Build the JupyterHub for local testing with Kubernetes:

make build-hubs-k8

General Guidelines

This project enforces the Contributor Covenant. Be kind and build a nice open source community with us. ++

License

Please refer to the included license in this repository's root directory.

illumidesk's Issues

Port factories used for testing to pytest fixtures

Instead of using factory methods, define pytest fixtures. Having factory methods as fixtures help with discoverability, particularly when using the pytest --fixtures command.

Example factory:

def factory_auth_state_dict(
    username: str = 'foo', course_id: str = 'intro101', lms_user_id: str = 'abc123', user_role: str = 'Learner'
) -> Dict[str, str]:
    return {
        'name': username,
        'auth_state': {'course_id': course_id, 'lms_user_id': lms_user_id, 'user_role': user_role,},
    }

With a pytest fixture:

@pytest.fixture
def factory_auth_state_dict(
    username: str = 'foo', course_id: str = 'intro101', lms_user_id: str = 'abc123', user_role: str = 'Learner'
) -> Dict[str, str]:
    return {
        'name': username,
        'auth_state': {'course_id': course_id, 'lms_user_id': lms_user_id, 'user_role': user_role,},
    }

Catch value error when launching application with non-unicode characters

Overview

Launching the application with non-unicode characters is returns an internal server error. For example, converting the / character to %2F when adding url parameters to the launch request will return a 500 error.

This error is expected, however, the cause should be displayed as a 400 (bad request) error with the description to allow the user to take corrective actions.

Solution

Display a bad request message and guide the user on how they could reformat the request with proper encoding.

Create assignments with deep linking

Allow users to create assignments within their LTI 1.3 compatible LMS from the shared grader notebook.

Move replay attack validation to the LTIUtils class

The LTI13LoginHandler should also check for replay attacks by validating nonce/timestamp combinations. Move the OrderedDict() that persists nonces to the LTIUtils class. Refactor LTI11LaunchValidator and implement with the LTI13LaunchValidator.

Provide support for the Open RStudio workspace type

Description

Sometimes content creators and instructors would like the flexibility to provide additional workspace types to their end-users.

This feature request's scope is focused on providing the RStudio as a workspace type option.

Solution

Offer the RStudio as an option with a workspace type selector when using the profile spawner.

Scope and Tasks

• This initial version does not provide support for Auto Grading features from within the RStudio. Auto-grading support for R-based labs/assignments is provided by the default Jupyter classic and Jupyter Lab images.
• Document how to override the profile spawner's selector to spawn/access the workspace automatically with query parameters and/or LTI custom fields.

Add option to provide users with additional workspace types

Overview

In addition to allowing users to select their desired workspace type, allow users to specify the resources to run the container-based environments, such as CPU and memory options.

Solution

• Allow administrators to select the workspace's spawner. For example, Jupyter Notebook with GPU using Kubernetes Spawner or THEIA IDE with Docker Swarm spawner.
• Allow users to specify multiple named workspaces, each with its own settings
• Allow users to launch workspaces using a standard form or pass query parameters to spawn container with the desired options

Add option to deploy stack with AWS EFS

Overview

By default, the ansible-playbook does not concern itself with wither or not the local file system is mounted with a shared file system such as NFS. In some cases (particularly for IllumiDesk staff) the option to deploy the stack to a remote host that uses AWS EFS mounts provides a more stable system.

Scope

Update the ansible-playbook to allow the user to specify whether or not they would like to mount stack's host folders located on the EC2 instance to the AWS EFS service. This option requires an active AWS account and may only be used with regions that support EFS services (update docs accordingly).

Segregate JWKS (public key) endpoint from LTI 1.3 configuration JSON URL endpoing

• Publish JSON Web Keys at: https://my.illumidesk.com/hub/jwks
• Publish Configuration URL at: https://my.illumidesk.com/hub/lti13/config

Error sending grades with LTI 1.3

Describe the bug
Unable to send grades using the developer key (LTI 1.3) with the Canvas LMS.

To Reproduce

1. Add a developer key with the LTI option
2. Add https://.illumidesk.com/hub/lti13/config as the configuration URL
3. Confirm JWKS endpoint has the ../hub/lti13/jwks path

Expected behavior
Grades post to Canvas LMS for a given assignment.

Stack Trace

DEBUG:illumidesk.lti13.auth:Getting lms access token with parameters {'iss': '161500000000000001', 'sub': '161500000000000001', 'aud': 'https://app.thisismetis.com/login/oauth2/token', 'iat': 1594664984, 'exp': 1594665049, 'jti': 'e4c98abe-a6cc-4f44-ab27-69136ba6b793'}
DEBUG:illumidesk.lti13.auth:Obtaining token b'eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6InZIeExybnlMNGVHOGhIdnoxekQ3SHVsQkVvTV9iejhuYnNuMGZXRlFTU3MifQ.eyJpc3MiOiIxNjE1MDAwMDAwMDAwMDAwMDEiLCJzdWIiOiIxNjE1MDAwMDAwMDAwMDAwMDEiLCJhdWQiOiJodHRwczovL2FwcC50aGlzaXNtZXRpcy5jb20vbG9naW4vb2F1dGgyL3Rva2VuIiwiaWF0IjoxNTk0NjY0OTg0LCJleHAiOjE1OTQ2NjUwNDksImp0aSI6ImU0Yzk4YWJlLWE2Y2MtNGY0NC1hYjI3LTY5MTM2YmE2Yjc5MyJ9.WlZytMiy_AcQTwcRgW6X3HXTK--bWd28R1iUhAGd6p8J7iIVz821w6W_T3VINh34Q1U0AZSZAF2fat7SgnhfXya7RLlprzZlrDsqiIuGdRZDkDjm8X6LceYdrJsCp4IDge-LbbQEmq5oKHZOSM4eZBDa43iAyx39Xnt-lxk5DEKAnvDhX0WMioUbXKjAx1-goTxsbBBPI6uy_JftCK65ohOXWAmUvuUoDrz9Ph6T3Dety5ucCbqYBtvVLtcd3-ult0DPssvi8gp11SFl_GgNRBkPAAN4_aEkp6npoilguPk_KlKJ-DFya-NVbZFFCzRHAKer_yE0TWWx3S6EutGdRhAaCE7hZJ5ae7iiiIGQ0xcdRZZu7g_QTGgaKjj6cm_V9QSqv5ib8wIQyRmXHFGmZ221PIPdL-2NBfXBJVLIg5mw684EG7xHGd0zO5kkgjunmxRu8p2nFC2j445NGcoTlTxA4sUF017JwCKTyjbhPXmENy2CIvGtdFJ6vHX37zKX9Sc0saxQlT3aV61_-WxsD-6AJ8A0g4QXWUNcx3UwgsseaqM__Z3py6TD4Epc_2cgF9E_pj3VGprdSfgWj2fcCLpBVblyiPdHFvBU8tLpfaTWh6up0G5R-sK9FaZiL6SVFe_Lg82pNVoy0pmFcp8akFfyK8LO6oETHGtDTUiL_r8'
DEBUG:illumidesk.lti13.auth:Scope is https://purl.imsglobal.org/spec/lti-ags/scope/score https://purl.imsglobal.org/spec/lti-ags/scope/lineitem https://purl.imsglobal.org/spec/lti-ags/scope/result.readonly https://purl.imsglobal.org/spec/lti-ags/scope/lineitem.readonly
DEBUG:illumidesk.lti13.auth:OAuth parameters are {'grant_type': 'client_credentials', 'client_assertion_type': 'urn:ietf:params:oauth:client-assertion-type:jwt-bearer', 'client_assertion': 'eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6InZIeExybnlMNGVHOGhIdnoxekQ3SHVsQkVvTV9iejhuYnNuMGZXRlFTU3MifQ.eyJpc3MiOiIxNjE1MDAwMDAwMDAwMDAwMDEiLCJzdWIiOiIxNjE1MDAwMDAwMDAwMDAwMDEiLCJhdWQiOiJodHRwczovL2FwcC50aGlzaXNtZXRpcy5jb20vbG9naW4vb2F1dGgyL3Rva2VuIiwiaWF0IjoxNTk0NjY0OTg0LCJleHAiOjE1OTQ2NjUwNDksImp0aSI6ImU0Yzk4YWJlLWE2Y2MtNGY0NC1hYjI3LTY5MTM2YmE2Yjc5MyJ9.WlZytMiy_AcQTwcRgW6X3HXTK--bWd28R1iUhAGd6p8J7iIVz821w6W_T3VINh34Q1U0AZSZAF2fat7SgnhfXya7RLlprzZlrDsqiIuGdRZDkDjm8X6LceYdrJsCp4IDge-LbbQEmq5oKHZOSM4eZBDa43iAyx39Xnt-lxk5DEKAnvDhX0WMioUbXKjAx1-goTxsbBBPI6uy_JftCK65ohOXWAmUvuUoDrz9Ph6T3Dety5ucCbqYBtvVLtcd3-ult0DPssvi8gp11SFl_GgNRBkPAAN4_aEkp6npoilguPk_KlKJ-DFya-NVbZFFCzRHAKer_yE0TWWx3S6EutGdRhAaCE7hZJ5ae7iiiIGQ0xcdRZZu7g_QTGgaKjj6cm_V9QSqv5ib8wIQyRmXHFGmZ221PIPdL-2NBfXBJVLIg5mw684EG7xHGd0zO5kkgjunmxRu8p2nFC2j445NGcoTlTxA4sUF017JwCKTyjbhPXmENy2CIvGtdFJ6vHX37zKX9Sc0saxQlT3aV61_-WxsD-6AJ8A0g4QXWUNcx3UwgsseaqM__Z3py6TD4Epc_2cgF9E_pj3VGprdSfgWj2fcCLpBVblyiPdHFvBU8tLpfaTWh6up0G5R-sK9FaZiL6SVFe_Lg82pNVoy0pmFcp8akFfyK8LO6oETHGtDTUiL_r8', 'scope': 'https://purl.imsglobal.org/spec/lti-ags/scope/score https://purl.imsglobal.org/spec/lti-ags/scope/lineitem https://purl.imsglobal.org/spec/lti-ags/scope/result.readonly https://purl.imsglobal.org/spec/lti-ags/scope/lineitem.readonly'}
[D 2020-07-13 18:29:50.257 JupyterHub handlers:114] public_key is b'-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAxJcOmxiYxjMG8TzMYNVi\nT/p2zzm5myZ22gSHeixPZHcDixl3Ly7QG4Pl8/mMZyR5gAvGEWTv2oaMW032Beox\nm4EPONwbJ1xh7UjVCgEtzxD2LGkS7JPO0sd3jfcEvLt8/Dw/JukNms5fnDWIAAAi\nCMHsfjazoJNvvdwCsMo33JTfBailSRxdOMFrix9ufzbz+sYVtO2WmwWKyst6ApOC\nS/SMxNbZzNX1qGltl0SR/wvtYWMGW90io6j7fq4LmFjEerkQnKWBOv0bur8zaSPa\ndUBfzDHinRZClhZcilPoIfTSfdzVPEZSaUjYELPnRO63Xlc51CASvpEzWR8kYxig\n3FBrX5ZxEnpY7i50X0nk2RsrlkQZojqLBcm7EkXFbybXMnK5HlW25EQIbIcBTccD\nqfYGXW6XMeEES+/vQj2N6NWO4x5GLCNaeYhtCq7c36hRqcODyS35onwUbDe4EEjn\nO3Fx/dYA6smVdQ5TGgMwJlRgAQsbt7EKjytj+dKmMtl1DtxNAsiZsPNkYG4jWMu4\n7r8P/VcbkU7LicPCSjKXlVrr91IskRo20WV/e3qsxJOYVKBl+1+krJq/dFt+DNsn\n/AVO0KaiXJurxb3QGK4bOA0+rmb8aYgIuXshcodQqBg+o6AsXt6jIcSEcx321BK2\nF0ZO9e4XQ3y2t7V4j8wglyECAwEAAQ==\n-----END PUBLIC KEY-----'
[D 2020-07-13 18:29:50.257 JupyterHub handlers:117] the jwks is {'e': 'AQAB', 'kid': 'vHxLrnyL4eG8hHvz1zD7HulBEoM_bz8nbsn0fWFQSSs', 'kty': 'RSA', 'n': 'xJcOmxiYxjMG8TzMYNViT_p2zzm5myZ22gSHeixPZHcDixl3Ly7QG4Pl8_mMZyR5gAvGEWTv2oaMW032Beoxm4EPONwbJ1xh7UjVCgEtzxD2LGkS7JPO0sd3jfcEvLt8_Dw_JukNms5fnDWIAAAiCMHsfjazoJNvvdwCsMo33JTfBailSRxdOMFrix9ufzbz-sYVtO2WmwWKyst6ApOCS_SMxNbZzNX1qGltl0SR_wvtYWMGW90io6j7fq4LmFjEerkQnKWBOv0bur8zaSPadUBfzDHinRZClhZcilPoIfTSfdzVPEZSaUjYELPnRO63Xlc51CASvpEzWR8kYxig3FBrX5ZxEnpY7i50X0nk2RsrlkQZojqLBcm7EkXFbybXMnK5HlW25EQIbIcBTccDqfYGXW6XMeEES-_vQj2N6NWO4x5GLCNaeYhtCq7c36hRqcODyS35onwUbDe4EEjnO3Fx_dYA6smVdQ5TGgMwJlRgAQsbt7EKjytj-dKmMtl1DtxNAsiZsPNkYG4jWMu47r8P_VcbkU7LicPCSjKXlVrr91IskRo20WV_e3qsxJOYVKBl-1-krJq_dFt-DNsn_AVO0KaiXJurxb3QGK4bOA0-rmb8aYgIuXshcodQqBg-o6AsXt6jIcSEcx321BK2F0ZO9e4XQ3y2t7V4j8wglyE', 'alg': 'RS256', 'use': 'sig'}
[I 2020-07-13 18:29:50.258 JupyterHub log:181] 200 GET /hub/lti13/jwks (@172.31.9.201) 180.37ms
INFO:illumidesk.lti13.auth:Error by obtaining a token with lms. Detail: b'{"error":"invalid_request","error_description":"JWS signature invalid."}'
[E 2020-07-13 18:29:50.343 JupyterHub web:1792] Uncaught exception POST /hub/submit-grades/ds-2020-06/quiz1 (172.31.35.96)
    HTTPServerRequest(protocol='https', host='metis.illumidesk.com', method='POST', uri='/hub/submit-grades/ds-2020-06/quiz1', version='HTTP/1.1', remote_ip='172.31.35.96')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1703, in _execute
        result = await result
      File "/usr/local/lib/python3.6/dist-packages/illumidesk/grades/handlers.py", line 51, in post
        await lti_grade_sender.send_grades()
      File "/usr/local/lib/python3.6/dist-packages/illumidesk/grades/senders.py", line 208, in send_grades
        await self._set_access_token_header()
      File "/usr/local/lib/python3.6/dist-packages/illumidesk/grades/senders.py", line 191, in _set_access_token_header
        token = await get_lms_access_token(self.lms_token_url, self.private_key_path, self.lms_client_id)
      File "/usr/local/lib/python3.6/dist-packages/illumidesk/lti13/auth.py", line 66, in get_lms_access_token
        resp = await client.fetch(token_endpoint, method='POST', body=body, headers=None)
    tornado.httpclient.HTTPClientError: HTTP 400: Bad Request

Update LTI 1.3 JSON configuration with User.id variable substitution

Update the LTI 1.3 standard configuration to add a new custom claim key, named lms_user_id which uses the value provided by the standard $User.id variable defined in LTI's 1.3 variable substitution message name/values.

Update Travis-CI after success script to build and push docker images

Update after success script used with Travis-CI:

• When merging to master build notebook and hub images
• Tag images with current master hash commit
• Tag images with latest tag
• Push notebook and hub images to DockerHub

Add automated load testing

Trigger automated load testing (most likely with a dedicated load testing service).

Verify and add documentation on how to send grades to Open edX

Verify sending grades to Open edX with LTI 1.1, add documentation.

Ansible error when pulling base notebook image from DockerHub

Overview

Running the ansible playbook with the make deploy command returns an error when running the notebook task.

To reproduce

Run make deploy from the latest master release.

Expected behavior

An ansible-playbook run without errors.

Actual behavior

An error that states that the dockerhub jupyter/docker-stacks tag 8e8cbd0a0af7 is not recognized.

Add default ENTRYPOINT and CMD to setup-course image

Add the command which is currently used by docker-compose to launch the stack to the setup-course Dockerfile itself, which will allow us to run this service stand alone, i.e. docker run ... which is useful for testing and integration tests.

Document setup instructions and end-user guides for dashboard servers

Update documentation to describe how end-users can create visualizations and share them within their class settings

Python interpreter not correctly set on remote host

Describe the bug

On some occasions, the ansible-playbook ... command returns a deprecation warning stating that the python interpreter is set to Python2 instead of Python3 even though the python_interpreter key in ansible.cfg is set to /usr/local/bin/python3.

To Reproduce

Run make deploy or the standard ansible-playbook -i hosts ansible/provisioning.yml -e 'ansible_python_interpreter=/usr/bin/python3' command from the repo's root. The following message should appear in the gathering facts task:

TASK [Gathering Facts] *********************************************************************
[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host illumidesk should use 
/usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior 
Ansible releases. A future Ansible release will default to using the discovered platform 
python for this host. See 
https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for 
more information. This feature will be removed in version 2.12. Deprecation warnings can be
 disabled by setting deprecation_warnings=False in ansible.cfg.

Expected behavior

Run the make deploy or the ansible-playbook command as instructed above and not see the deprecation notice.

Alternative Solutions

Various alternative solutions were tested based on official documentation such as discovering the python interpreter version automatically. Running the command with the -e ... flag is also an option but adds more room for configuration mistakes.

Migrate all UI elements to JupyterLab

Provide consistency with Jupyter-based workspaces by migrating from the classic notebook interface to the lab interface.

Send grades to LMS using LTI 1.3

Overview

Send grades to the LMS by posting a score.

Scope

Maintain the same feature set as sending grades with LTI 1.1:

• Same UI (in classic notebook - grader console / form grader) to send grades column with button(s) to send grades by assignment
• Any additional UI elements to verify sending grades will be dealt with separately

Add support for Kubernetes

Overview

Add Kubernetes support for development environments as well as staging/production environments.

Requirements (in no particular order)

• Provide the necessary documentation and setup options to set up development environments that are compatible with Kubernetes
  • Dev setup with K3's and update contributor guide
• Use the JupyterHub helm-chart as a sub chart zero-to-k8s
• Customize Helm chart based on IllumiDesk requirements
  • Workspace base image references
  • Custom authenticator(s)
  • Custom spawner(s)
  • PV/PVC for storage
  • Environment variable references
  • AWS specific manifests, such as aws-iam-authenticator and alb-ingress
  • Add option for internal NFS service for customers that don't want or have AWS EFS
  • Add option for internal Postgres service for customers that don't want or have AWS RDS. The JupyterHub DB can continue to support SQLite as an option but the grader must use Postgres due to thread-safety.
• Update setup-course service to create new grader services and folder structure when receiving requests for new courses
• Add EFS/NFS PV/PVC and attach end-user workspaces to this storage option
• Update IllumiDesk JupyterHub image to use the illumidesk/jhub-k8:latest image.
• Refactor existing IllumiDesk base spawner class to inherit from the KubeSpawner
• Refactor jupyterhub base image to install IllumiDesk package with pip, either with a GitHub release or with pypi vs local copy/install using zip
• Providing conditions for create custom configuration YMLs used by Helm to deploy and update stacks (epic, requires POCs to determine a) interface contracts between admin-dashboard and backend and b) implementation of how to create custom config so that Helm can use it.

Refactor IllumiDesk custom spawners

Refactor IllumiDesk custom spawners to simplify classes.

• IllumiDeskBaseDockerSpawner(DockerSpawner)
• IllumiDeskRoleDockerSpawner(DockerSpawner)
• IllumiDeskWorkSpaceDockerSpawner(DockerSpawner)

Set admin_users configuration option with ansible hosts configuration file

Some users need access to the Admin menu to manage end-user servers.

Add instructions and configuration options to allow users to add users to the standard JupyterHub.admin_users group.

Allow users to launch other workspace types

In addition to Jupyter Notebooks, allow users to launch other workspace types such as VS Code and THEIA IDEs.

Migrate repo documentation to readthedocs

• Migrate documentation within Readme to the docs folder
• Publish to readthedocs.org
• Add docs tests

End-user guides are not managed within this monorepo. These documents are managed in the docs repo.

Course names with hyphens lead to 400 errors when sending grades to the LMS

Describe the bug
Course names which contain hyphen or underscore characters lead to 400 errors

To Reproduce
Create a course which includes a hyphen or underscore character in the string

Expected behavior
Successfully send grades to the LMS (assuming all other requirements are met)

Additional context
The SendGradesHandler regular expressions to accept/reject post that contains the course name and the assignment name using the <course_id>/<assignment_name> path.

Update JupyterHub base image

Update JupyterHub base image to version 1.2 now that the issue with the page assets has been resolved with the docker image.

Add UI tests

Ensure UI elements are behaving as expected with webdriver-based tests.

Document setup instructions and end-user guides for Postgres labs option

Update documentation so course administrators and/or instructors are able to spawn and manage container-based Postgres containers for their lab environments.

Remove superfluous grader image check with role spawner

We no longer need to check if the user_role key from the authentication dict has the Grader role, so we can remove this code from the auth_state_hook in the IllumiDeskRoleDockerSpawner class.

course_id is not optional with lti11_enabled = false and lit13_enabled=false

Describe the bug
The hosts.example file says that the course_id variable is optional. If you don't fill it out and keep lti11_enabled and lti13_enabled to false, then the docker-compose.yml template creates an invalid YAML file.

Either, course_id needs to be in the required section of the hosts.example or there needs to be an additional check for an empty course_id on this line: https://github.com/IllumiDesk/illumidesk/blob/master/ansible/roles/jupyterhub/templates/docker-compose.yml.j2#L43 . I'm not sure which is the correct solution given all the other downstream effects of those changes.

To Reproduce
Steps to reproduce the behavior:

1. Clone master.
2. copy hosts.example to hosts.
3. change just the ansible_host, ansible_port, ansible_user, ansible_ssh_private_key_file, and ansible_password to whatever it should be
4. make deploy

Expected behavior
Successful install and launch of all the docker containers

Actual Behavior

TASK [launch : launch with docker-compose] ***************************************************************************
fatal: [illumidesk]: FAILED! => {"changed": true, "cmd": ["docker-compose", "up", "-d"], "delta": "0:00:00.562821", "end": "2020-05-25 08:28:47.208785", "msg": "non-zero return code", "rc": 1, "start": "2020-05-25 08:28:46.645964", "stderr": "yaml.parser.ParserError: while parsing a block mapping\n  in \"./docker-compose.yml\", line 4, column 3\nexpected <block end>, but found ':'\n  in \"./docker-compose.yml\", line 43, column 3", "stderr_lines": ["yaml.parser.ParserError: while parsing a block mapping", "  in \"./docker-compose.yml\", line 4, column 3", "expected <block end>, but found ':'", "  in \"./docker-compose.yml\", line 43, column 3"], "stdout": "", "stdout_lines": []}

Additional context

Update nbgrader configuration to support the Julia language

Update the nbgrader_config.py that is copied to the shared grader notebook directory. This update needs to take place in both the ansible-playbook task and with the setup-course service.

Current configuration:

c = get_config()

c.CourseDirectory.root = '...'
c.CourseDirectory.course_id = '...'
c.ClearSolutions.code_stub = {
    "python": "# your code here\nraise NotImplementedError",
    "javascript": "// your code here\nthrow new Error();"
}

New Configuration:

c = get_config()

c.CourseDirectory.root = '...'
c.CourseDirectory.course_id = '...'
c.ClearSolutions.code_stub = {
    "python": "# your code here\nraise NotImplementedError",
    "javascript": "// your code here\nthrow new Error();",
    "julia": "# your code here\nthrow(ErrorException())"
}

Update jupyterhub_api methods with types

Add types for parameters and returns where applicable using Python's native typing.

Provide support for the VS Code workspace type

Description

Sometimes content creators and instructors would like the flexibility to provide additional workspace types to their end-users.

This feature request's scope is focused on providing the VS Code as a workspace type option.

Solution

Offer the VS Code as an option with a workspace type selector when using the profile spawner.

Scope and Tasks

• This initial version does not provide support for Auto Grading features from within the VS Code. This may be dealt with at a future date by providing a VS Code compatible extension to hook into the existing grading artifacts.
• Document how to override the profile spawner's selector to spawn/access the workspace automatically with query parameters and/or LTI custom fields.

Provide support for the THEIA IDE workspace type

Description

Sometimes content creators and instructors would like the flexibility to provide additional workspace types to their end-users.

This feature request's scope is focused on providing the THEIA IDE as a workspace type option.

Solution

Offer the THEIA IDE as an option with a workspace type selector when using the profile spawner. The alternative option is to provide additional workspace types with the jupyter-server-proxy from within the Jupyter Notebook, however, most users prefer direct access to streamline the user experience.

Scope and Tasks

• This initial version does not provide support for Auto Grading features from within the THEIA IDE. This may be dealt with at a future date by providing a VS Code compatible extension to hook into the existing grading artifacts.
• Document how to override the profile spawner's selector to spawn/access the workspace automatically with query parameters and/or LTI custom fields.

Add Java to workspace base image

Add Java and Jupyter Kernel to base image.

Update LTI13LoginHandler

• Remove set_state_cookie method
• Get nonce value from request argument
• Get redirect_uri from request argument instead of determining it dynamically
• Remove the get_state() method since the target_link_uri from the JWT is the one that counts
• Update/add tests

Increase LTI 1.3 handler test coverage

Improve test coverage for LTI 1.3 login, callback, and JSON configuration handlers.

Update LTI 1.1 authenticator to consider Student string

The Open edX LMS does not send user roles with the "Learner" string. Update the authenticator to also consider the "Student" string.

Java kernel installation error

The base dockerfile errors out during the build due to an error with the cleanup step. Change into the /tmp working directory and manage the installation within that directory. Then, remove the archive with a relative path.

Add docstrings and types to lms_grades.py

Add docstrings and types to all classes and functions defined in src/illumidesk/handlers/lms_grades.py.

Build setup-course image using base ubuntu image

The setup-course image takes a long time to build. Build the image with the layers built by the JupyterHub image to save time and disk space.

Refactor LTI 1.3 handlers

Refactor LTI 1.3 login handler and callback handlers:

• Update authorize_redirect so that the implementation is consistent with oauthenticator's handlers
• Add keys available in the extra_params as configuration options as properties to the handler class
• Ensure all required fields to initiate the LTI 1.3 authentication flow are included in the initial login request

Create new repo to maintain workspace images

Migrate docker notebooks and workspaces maintenance to its own repo.

Include option to clone repositories when launching new workspaces with LTI launch requests

Overview

Users in many cases have their content stored within git-based source control solutions, such as GitHub, GitLab, and BitBucket. This feature allows users to provide query parameters to clone and merge code located within remote git repositories when launching workspaces.

Solution

Integrate the standard nbgitpuller package with the orchestrator and the end-user workspaces to allow instructors/content creators to assign specific information to clone and start workspaces using external content:

• Git repository URL
• Branch
• File path
• Workspace type (one of classic notebook, jupyterlab, rstudio, theiaide)

Real time chat support

integrate dependencies required to allow users to chat with each other in real-time from within their workspace environment(s).

Deploy stack with built images

For dev environments, pull docker images instead of building them from scratch. This will speed up the process for deployments and provide more consistency.

• Adjust CI/CD to build/tag/push images to DockerHub with merges to master branch
• Update ansible-playbook to pull images from DockerHub
• Update documentation

Invalid outcome service url when launching tool from course menu

Describe the bug

Under certain use-cases, LMS administrators and instructors may place the IllumiDesk launch menu item in the main course navigation menu. Students that launch the IllumiDesk app from the main course menu will receive a 500 error due to the fact that the lis_outcome_service_url argument is missing from the launch request. In Canvas, the lis_outcome_service_url is only included when launching the application from the Assignment section.

To Reproduce

Steps to reproduce the behavior:

1. Login in as a user with the Learner role
2. Launch IllumiDesk application by clicking on the Course menu

Expected behavior

Launch the application and redirect the user to their initial redirect page (set by custom_next_url, next_url, or by updating application settings to the default launch page).

Actual behavior

• User interface: 500 error
• Logs:

[D 2020-06-02 14:12:28.524 JupyterHub authenticator:219] Assigned username is: student1
[E 2020-06-02 14:12:28.524 JupyterHub web:1788] Uncaught exception POST /hub/lti/launch (172.31.9.201)
    HTTPServerRequest(protocol='https', host='ernesto.illumidesk.com', method='POST', uri='/hub/lti/launch', version='HTTP/1.1', remote_ip='172.31.9.201')
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/tornado/web.py", line 1703, in _execute
        result = await result
      File "/usr/local/lib/python3.8/dist-packages/illumidesk/authenticators/handlers.py", line 35, in post
        user = await self.login_user()  # noqa: F841
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 699, in login_user
        authenticated = await self.authenticate(data)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/auth.py", line 383, in get_authenticated_user
        authenticated = await maybe_future(self.authenticate(handler, data))
      File "/usr/local/lib/python3.8/dist-packages/illumidesk/authenticators/authenticator.py", line 233, in authenticate
        control_file.register_data(assignment_name, lis_outcome_service_url, lms_user_id, lis_result_sourcedid)
    UnboundLocalError: local variable 'lis_outcome_service_url' referenced before assignment

Screenshots

Additional context

Tested with Canvas LMS and using LTI 1.1.

Add configuration option to enable or disable fetch feedback from student's assignment list tab

Overview

Some organizations don't use the manual feedback option provided by the grader. By default, the fetch button is enabled for all students, regardless of whether or not feedback has been submitted (aka generated and released) to the student.

Solution

Offer an option to disable the Fetch feedback button from the Assignment List extension. This should also disable the generate/release manual feedback from the Instructor's perspective since students will not be able to fetch generated and released feedback.

Update configuration settings to remove repetitive configuration options

Overview

Update ansible-playbook to remove repetitive settings options.

Additional context

With the current setup anytime we add a new spawner, authenticator, etc we are forced to add the same settings to the different configuration files. As the configuration options get more complex it makes more sense to leverage template options to define settings independently and then have ansible take care of creating the required configuration files.

Refactor ansible-based yaml configuration file to organize options by roles

Overview

As the number of configuration options gets larger keeping the configuration options organized by role will help the administrator's user experience.

Configuration Definition

Current Structure

all:
  hosts:
    illumidesk:
      ansible_host: 34.221.91.112
      ansible_port: 22
      ansible_user: ubuntu
      ...

Proposed Structure

all:
  hosts:
    illumidesk:
        jupyterhub:
            ...
        reverse-proxy:
            ...
        workspaces:
            ...
        spawners:
            ...
        authenticators:
            ...
        labs:
            ...

Notes

• Provide the option to update Python3 packages included with the requirements.txt file
• The labs section enables/disables the environment for the user(s), such as the Postgres DB, used for lab environments
• The Spawners section should only include one spawner type, unless otherwise noted (mutually exclusive). For example, the IllumiDeskDockerSpawner and the IllumiDeskProfileSpawner cannot both be enabled (set to true). If more than one is enabled, then we can either use the first one enabled in the configuration file (alerting the user as such) or error out of the ansible-playbook with the reason.
• Same comment as above for authenticators