illumidesk / async-nbgrader

nbgrader extension to manage assignment auto-grading asynchronously with a light queue based system.

Async nbgrader

A Jupyter Notebook server extension which adds:

• Async capabilities to nbgrader's auto-grading service.
• Export grades as a CSV file for the Canvas LMS

Installation

1. Clone this repo and install the pacakage with pip:

git clone https://github.com/illumidesk/async-nbgrader
cd async-nbgrader
pip install -e .

1. Create and activate your virtual environment:

virtualenv -p python3 venv
source venv/bin/activate

1. Install and Activate Extensions

Install and activate client and server extensions:

jupyter nbextension install --sys-prefix --py async_nbgrader --overwrite
jupyter nbextension enable --sys-prefix --py async_nbgrader
jupyter serverextension enable --sys-prefix --py async_nbgrader

Run the Auto-Grader

This package leverages the same Python API available with nbgrader. Therefore no additional changes are required to run the auto-grader. For example, once you have collected assignments, run the following command to auto-grade assignments with the async-nbgrader server extension:

nbgrader autograde "<assignment-name>"

Toggle to use the standard syncronous autograder

This package includes the option to set an environment variable to toggle whether to use the asyncronous version of the autograder or the syncrononous version (default is async):

Environment Variable	Description	Default
NBGRADER_ASYNC_MODE	Used to set whether the autograder runs syncronous or asyncronous mode	"true"

Export Grades as a CSV

Follow the steps below to export grades from the nbgrader database to a *.csv (default is canvas_grades.csv) file:

1. Export the Canvas LMS grades as a CSV file from your Course's gradebook by following these instructions.

2. Copy the CSV file to a location where the ild command has access to the exported CSV. (The ild command is used to export grades from the nbgrader database).

3. Run the ild export command to export grades from the nbgrader database:

ild export --canvas_import=/path/to/my/grades-from-canvas.csv --canvas_export=/path/to/my/grades-for-canvas.csv

By default, the ild export command uses the canvas.csv as the file to import grades from and the canvas_grades.csv to export grades to. You can override these values with the --canvas_import and the --canvas_export flags to designate the path and file name for the CSV file to import and export, respectively.

The ild command is a wrapper for the nbgrader command. Therefore all other flags included with the nbgrader CLI are available with the ild command, such as the --debug flag.

If successful, the output in the terminal should look similar to:

[ExportApp | WARNING] No nbgrader_config.py file found (rerun with --debug to see where nbgrader is looking)
[ExportApp | INFO] Using exporter: CanvasCsvExportPlugin
[ExportApp | INFO] Exporting grades to canvas_grades.csv
[ExportApp | INFO] Skipping second row
[ExportApp | INFO] Finding student with ID 358
[ExportApp | INFO] Finding submission of Student '358' for Assignment 'postgres-migration-test'

Contributing

For general contribution guidelines, please refer to IllumiDesk's contributing guidelines.

The async_nbgrader package installs the nbgrader package as a required dependency, therefore you should not have to install it explicitly.

The async_nbgrader package overrides nbgrader's default auto-grading service (included with the Formgrader extension) by converting the grading service from a syncronous service to an asyncronous service. Thefore it's a good idea to get familiar with the nbgrader documentation (although not a must) to setup your local environment by following these instructions.

Use pytest to run tests:

pytest -v

License

Apache 2.0

async-nbgrader's Issues

Automate linting, code style, coverage reports, and run unit tests

• Add flake8
• Add black
• Add isort
• Integrate codecov
• Add pre-commit hook
• Run unit tests with GitHub Actions

Update package to add configurable to use legacy ansync setup with SQLAlchemy

Add a configurable to toggle between AMQP based queues and SQLAlchemy jobs options.

Update README.md development setup instructions

Update the development setup instructions to clarify the steps to install the packages required for the development environment and how to install the package in editable mode.

Update unit test to remove tests based on Selenium

Replace tests based on Selenium with tests that use nbgrader's RESTful API endpoints

New Architecture for Auto Grading Job services

Overview

IllumiDesk needs to re-design the auto-grading services to improve scalability and decouple the service from the rest of the LMS system.

Desires and Goals

• The auto-grading service can be called from any client using standard APIs (RESTful, etc)
• Like the rest of the system, requests should be authenticated
• Leverage native Kubernetes services to the extent possible to manage jobs
• Maintain compatibility (at least for the first iteration) with the nbgrader database schema and nbgrader ipynb schema
• Allow system administrators to adjust CPU / Mem settings for auto-grading containers
• Enable notifications based on job status (completed, failed, in-progress, etc)
• Log auto-grading jobs to a central location via stdout/stderr (thus allowing administrators to integrate the logging tool(s) of their choice.

Notes

• The nbgrader auto-grading feature is tightly coupled with the Jupyter Notebook. The async-nbgrader project helps but need to take this a step further and siphon off the auto-grading service into a more traditional queue/task-based job architecture.
• The queue/job architecture setup naturally has many options. We are more inclined to stick with a Kubernetes-native approach. This of course has pros/cons. If we use something with native Kubernetes (Argo, K8s jobs, etc) then it's more complex to setup and maintain, however it would allow us to provision this with any cloud vendor if we chose to do so and/or offer this with the IllumiDesk Enterprise version. Perhaps when managing this with our SaaS solution we could combine K8s with a managed service, such as Fargate via plugins/configs/middleware but that is wait-and-see.
• The queue/job architecture should be generic enough so that we can receive notifications based on the status of the desired state and alert the end-user and/or internal IllumiDesk collaborators based on certain rules.
• We could use the gofer_service and gofer_submit projects as sources of inspiration.

Nice to haves

• Once the auto-grading has been completed, allow users to configure whether or not they need to auto-submit grades to their LMS either with LTI 1.3 or third-party integration service (Zapier, generic Webhooks, etc).

Diagrams and Architecture

This diagram has a basic first draft of how the re-architecture could look.

Resources

• For the background workers which can be configured by the end-user, we can use Horizontal Pod Autoscaler dependent on external metrics: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-metrics-not-related-to-kubernetes-objects (this will need change in our helm-chart)
• Background worker with kubernetes - https://kubernetes.io/docs/tasks/job/coarse-parallel-processing-work-queue/

Complications identified

• Worker containers would need shared volume access of the jupyter, in order to read the notebook. This is on basis of the current solution where we are using file storage.
• Acquiring locks on basis of Notebook so that multiple workers don't write to the same notebook

Automate testing with GitHub Actions

• Add automation for GitHub Actions