openedx / credentials Goto Github PK
View Code? Open in Web Editor NEWService hosting course and program certificates
License: GNU Affero General Public License v3.0
Service hosting course and program certificates
License: GNU Affero General Public License v3.0
The current jenkins jobs for copying course certificate data to Credentials are unreliable and result in data drift.
Work to deprecate the Old Mongo Modulestore is currently blocked from completion because signature image artifacts for course certificates are stored in Mongo's GridFS.
There is a vision that web certificates for courses will migrate to the Credentials IDA. This work is a small piece of that, specified to unblock the DEPR work while remaining aligned with the overall vision.
This work should consider:
(needs correction) AFAIU, currently the course certifications forEach learner are stored in a (mutable) SQLdb in a server farm (somewhere amazon-knows-where). How can this be made more accountable?
Given such case, Would it be possible to issue a key-pair, where either 1. the public key forEach learner is augmented for each additional course certification, or 2. use a more ZKP scheme to prove just the required course cert to the other party.
Would be even better if the proof can be done offline anywhere, so it could replace a stamped/hand-signed piece of paper once for all.
The django.core.cache.backends.memcached.MemcachedCache backend is removed so we now are replacing this with PymemcacheCache
in all of our IDAs, so we need to replace this for credentials also.
https://docs.djangoproject.com/en/4.2/releases/4.1/#features-removed-in-4-1
This work represents a review of the existing Credentials service and the creation of design documents and other artifacts for generalized mechanism for enabling Credentials sharing from the Open edX platform to external networks.
The design requirements should be agreed to during the design review, however, here are some initial thoughts:
AC:
Learner ownership of digitally verifiable and shareable assertions of attainment -- credentials -- is aligned with goals of learner-centricity and open platforms. The Open edX platform has existing mechanisms for sharing Open edX credentials, however, they do not support recent innovations around distributed ledgers. Both the EU and the W3C have been focused on development standards for shareable, verifiable credentials based on distributed ledgers. Credentials are connected to a self-sovereign digital identity and can be shared and independently varied to support processes like applying to degree programs, applying for employment, etc.
The Initiative proposes to
Project-level metrics:
Impact: Increase the value of attainments achieved on the Open edX platform by allowing learners to present them in order to prove their attainment.
Measure: Deployment of reference implementation to Open edX instances, enablement of the feature, use by learners
TBW
TBW
TBW
unknown
tCRIL
No response
Issue for the Overall initiative
Starting to gather information for supporting integration between the Open edX platform and the EU verifiable digital credentials standard
EBSI
https://www.youtube.com/watch?v=m2uj7fgb2JI
https://www.youtube.com/watch?v=ATXCzY-GM_U
https://ec.europa.eu/digital-building-blocks/wikis/display/EBSI/
https://ec.europa.eu/digital-building-blocks/wikis/display/EBSIDOC/EBSI+Documentation+Home
https://ec.europa.eu/digital-building-blocks/wikis/display/EBSIDOC/Data+Models+and+Schemas
https://ec.europa.eu/digital-building-blocks/wikis/display/EBSIDOC/Demonstrator
https://ec.europa.eu/digital-building-blocks/wikis/display/EBSI/Conformant+wallets
https://api.preprod.ebsi.eu/docs/
EDC
https://europa.eu/europass/en
https://europa.eu/europass/digital-credentials/issuer/#/home
https://europa.eu/europass/digital-credentials/viewer/#/home
https://github.com/european-commission-empl/European-Learning-Model
https://europa.eu/europass/system/files/2020-11/EDCI-Diplomawildcards-171120-1131-2172.pdf
This repository is using Ubuntu 20.04 for testing. That version of Ubuntu will be out of support before Teak. Therefore this repo needs to be updated to testing with Ubuntu 24.04 before Sumac is cut to allow everyone sufficient time to switch to the new version.
Update this repository to test with Ubuntu 24.04 so that we can make the switch.
Note: In some cases, it may not make sense to test with both the old and the new version. For example, if the workflow is running linting or publishing to a package manager. In these cases, simply update the workflow to run on the newer version or opt to set it to ubuntu-latest
instead if it doesn't matter what version it's running on. If you're unsure, reach out to the maintenance working group in #wg-maintenance in slack for guidance.
Known affected workflow files:
CHAPI is a technology that aims to standardize the protocol for adding verified credentials to digital wallets. The objective of this issue is to develop a point of view on whether supporting CHAPI would allow the platform to access more wallets.
Some resources:
https://chapi.io/
https://w3c-ccg.github.io/credential-handler-api/
https://playground.chapi.io/
This issue a placeholder for the scope that will be defined in the Discovery work, #1735
In order to have support for Django 4.2, we first need to run tests on Django 4.2 in the credentials service. For updating CI, we should run tox and GitHub actions modernizers.
Tox Modernizer: tox_moderniser_django42.py
Github Action Modernizer: github_actions_modernizer_django42.py
Before starting an implementation for Verifiable Credentials sharing based on the design created in #1728, It is necessary to validate the required fields that should be included into VC/OBv3 claim.
Draft mapping of the Program and Course certificates data stored in the Credentials IDA to VC fields, that are used by MITxPRO and DCC: https://openedx.atlassian.net/wiki/spaces/OEPM/pages/3490840577/Verifiable+Credentials+design#Forming-verifiable-credentials-or-digital-credentials-using-the-learners'-data
Abstract
Node 16's support will end on September 11, 2023. So we have to upgrade node JS to v18 before that.
Extras
Project should use .nvmrc
to have a consistent node version
We have to update Readme accordingly.
While installing the docker image using tutor-credentials
nightly, the image build fails during the requirements installation step. The error is:
WARNING: lxml 5.1.1 does not provide the extra 'html-clean'
WARNING: lxml 5.1.1 does not provide the extra 'html_clean'
ERROR: Exception:
Traceback (most recent call last):
File "/openedx/venv/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
status = run_func(*args)
File "/openedx/venv/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 248, in wrapper
return func(self, options, args)
File "/openedx/venv/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 377, in run
requirement_set = resolver.resolve(
File "/openedx/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
result = self._result = resolver.resolve(
File "/openedx/venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "/openedx/venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 427, in resolve
failure_causes = self._attempt_to_pin_criterion(name)
File "/openedx/venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 249, in _attempt_to_pin_criterion
satisfied = all(
File "/openedx/venv/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 250, in <genexpr>
self._p.is_satisfied_by(requirement=r, candidate=candidate)
File "/openedx/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/provider.py", line 240, in is_satisfied_by
return requirement.is_satisfied_by(candidate)
File "/openedx/venv/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/requirements.py", line 83, in is_satisfied_by
assert candidate.name == self.name, (
AssertionError: Internal issue: Candidate is not for this requirement lxml[html-clean,html-clean] vs lxml[html-clean]
This is likely because Lxml 5.1.1 is being installed which does not contain the html-clean extra.
Lxml 5.2.1 should be the package being installed here instead but the requirements do not list it as such because of the constraint
credentials/requirements/constraints.txt
Lines 22 to 25 in 5000131
that was pinned in #2430. edx-i18n-tools
has been updated as per this PR and it is safe to remove the constraint now.
Currently, there is no official tutor-credentials plugin. There is a community one, but it is out of date and the docker image seems to be missing: https://github.com/pcliupc/tutor-credentials
There is an issue that was closed accepting this plugin as the official community plugin for credentials.
It looks at though the plugin has not been touched since it was created.
After the Node 16 upgrade our non-React JS tests stopped working during CI. After some research, we were made aware that this may be an issue with Firefox not being captured correctly running in a GitHub/GitHub Actions + Karma + Node 16 environment (see karma-runner/karma-firefox-launcher#245).
When we discovered the issue we attempted to move away from Firefox and use a headless version of Chromium instead. We were able to get this working in CI, but ran into issues locally when installing a new dependency (puppeteer) used to facilitate the use of Chromium with Karma.
Currently, we have a method of running the tests locally that we do manually during development. Ideally, we'd like to get these tests working in CI again.
Potential Options:
This will be tracked internally at 2U by https://2u-internal.atlassian.net/browse/APER-1976.
The jobs started failing a few weeks ago. AFAIK, we have not triaged the error to know why they are failing. The original error alert triggered on August 14th. This job has continued to fail each Sunday since the original failure (so a re-run doesn’t look like it will fix the issue).
The logs has an error that looks like:
20:18:22 2022-08-14 20:18:22,667 INFO 43769 [utils.py:290] - Opened PR: https://github.com/openedx/credentials/pull/1719
20:18:23 2022-08-14 20:18:23,214 INFO 43769 [utils.py:424] - Deleting branch credentials:transifex-bot-update-translations2022-08-14.
20:18:23 Traceback (most recent call last):
20:18:23 File "transifex/pull.py", line 120, in <module>
20:18:23 pull(
20:18:23 File "transifex/pull.py", line 84, in pull
20:18:23 raise RuntimeError('Failed to compile messages.')
20:18:23 RuntimeError: Failed to compile messages.
A link to a failing PR from this job: #1740 .
Looking at our CI runs I also see an error that looks like:
msgfmt: found 1 fatal error
INFO:i18n.validate:Different tags in source and translation
msgid: Issued {month} {year}
-----> Emitido {mes}{año}
-----> "{month}", "{year}" vs "{año}", "{mes}"
This looks like we may have had a templated string in Transifex that shouldn’t be translated.
We think this may need to be fixed inside of Transifex.
The Credentials service was recently updated to support publishing program certificate events to the Event Bus (EB).
At the time, publishing events to the EB required code. There is work being done to enable publishing of events to the EB via configuration.
Work is planned to convert Credentials to a publish-by-config approach. This means we will be deprecating and removing the following settings:
PROGRAM_CERTIFICATE_EVENTS_KAFKA_TOPIC_NAME
: topic names are part of the EVENT_BUS_PRODUCER_CONFIG
and a separate setting will no longer be neededSEND_PROGRAM_CERTIFICATE_AWARDED_SIGNAL
: removed in favor of a single setting (SEND_LEARNING_PROGRAM_CERTIFICATE_LIFECYCLE_EVENTS_TO_BUS
) to enable publishing program certificate events to the EBSEND_PROGRAM_CERTIFICATE_REVOKED_SIGNAL
: (see above)Work will be tracked as part of an internal 2U JIRA ticket: https://2u-internal.atlassian.net/browse/APER-2966
This ticket may cover multiple independent bugs and need to be broken up. The "Issue date" is the only translated string. We're not certain how best to handle this.
This was logged for course certificates, and would be fixed in edx-platform: https://github.com/edx/edx-platform/blob/40964045e4cc284a8761419d2dc3cdd8e5cd1c6e/lms/djangoapps/certificates/views/webview.py#L123
Learners with browsers using a right to left language as their default will see the certificate jumbled:
Steps to reproduce
Also to note, usually the Issue date is in the browser default language instead of the certificate language (this is the only bit actually translated)
Version 3.0 of the Open Badges specification is expected to be approved in the June of 2022. This specification will allow learner to share their accomplishments across numerous networks, supporting many learner-centric use cases like:
Additionally, support for Open Badges 3.0 will allow the Open edX platform to share credentials with the DCC/W3C network described in #1731 . Thus this issue is a prerequisite for that item and will represent most or all of the scope of that issue.
The specification for Open Badges 3.0 is available here: https://github.com/IMSGlobal/openbadges-specification/tree/develop/ob_v3p0
This repo's acceptance tests still use the bok-choy framework, which was deprecated in Feb 2022. The bok-choy repository is no longer being maintained, and it will soon be archived.
Once the bok-choy repo is archived, the acceptance tests should continue working, as we will not be removing the package's releases from PyPI. For long-term maintainability of this repo, though, the tests should eventually be upgraded to a non-bok-choy testing framework or removed.
This issue a placeholder for the scope that will be defined in the Discovery work, #1735
The Digital Credentials Consortium is a network of leading international universities designing an open infrastructure for academic credentials.
Further details are available here.
This issue proposes designing and implementing an integration between the Open edX platform and the DCC sign-and-verify service and the mobile-based wallet application.
The DCC credential specification was created by a consortium that includes a number of edX partners and Open edX users. The specification is aligned with W3C verifiable credentials initiatives. There is a clear connection between open learning and open credentials standards that emphasize learner ownership and control of attainments.
There is an existing, adjacent, implementation of integration with the DCC sign-and-verify service and mobile wallet. However, there is no direct integration with the Open edX platform. The existing integration is written in python and works with the Django framework. Both technologies are core to the Open edX platform architecture.
Prior art:
The design should consider integrating with Open edX existing credential store in a loosely coupled way. DCC integration should be optional. Enabling it and configuring it should use project standards like feature flags.
Ideally, the integration can leverage the existing library. Ideally the Open edX project developers and ODL can collaborate on enhancing and maintaining the library.
To be written
In order to transfer ownership of CourseCertificate signature assets from the LMS to the Credentials IDA:
There are several libraries that provide critical, low-level functionality that will be necessary for implementing VC Sign and Verify in python. We will need to understand which libraries exist and, ideally, choose ones that are full-featured and well-support.
If we have to implement any of these libraries ourselves, we should not that and consider the ROI for that choice.
The list as we currently understand it is:
On 2024-05-08 an update was made to the public Dockerfile to support Python 3.11 (#2474).
After this merged to the master
branch, I noticed that the Build and Push Docker Images
workflow has started to fail.
Before merging my changes, I had verified a successful build of a docker image from the repo's base Dockerfile. Upon further inspection, I realized that we are building multiple versions of the image for multiple platforms (linux/amd64
& linux/arm64
, configured here -- https://github.com/openedx/credentials/actions/runs/9014850121/workflow#L53).
This issue seems specifically related to the image build for linux/arm64
. It is having an issue finding a compatible version of the didkit
dependency for the arm64
image.
Logs from a failed workflow: https://github.com/openedx/credentials/actions/runs/9014850121/job/24768334656.
We first build the linux/amd64
version, which is able to locate and install didkit
v. 0.3.2:
#26 4.081 Collecting didkit==0.3.2 (from -r requirements/production.txt (line 77))
#26 4.103 Downloading didkit-0.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.8 kB)
...
#26 15.47 Downloading didkit-0.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (10.6 MB)
#26 15.58 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.6/10.6 MB 95.0 MB/s eta 0:00:00
...
Installing collected packages: webencodings, text-unidecode, pytz, python-memcached, pypng, polib, openedx-atlas, itypes, fontawesomefree, django-webpack-loader, django-sortedm2m, zope-interface, zope-event, urllib3, uritemplate, typing-extensions, sqlparse, six, simplejson, semantic-version, pyyaml, python-slugify, pymemcache, pyjwt, pygments, pycparser, psutil, pillow, pbr, path, oauthlib, newrelic, mysqlclient, markupsafe, markdown, lxml, jmespath, inflection, idna, gunicorn, greenlet, fastavro, dnspython, django-ratelimit, didkit, ...
...
Successfully installed asgiref-3.8.1 attrs-23.2.0 backoff-2.2.1 bleach-6.1.0 boto3-1.34.99 botocore-1.34.99 certifi-2024.2.2 cffi-1.16.0 charset-normalizer-3.3.2 code-annotations-1.8.0 coreapi-2.3.3 coreschema-0.0.4 cryptography-42.0.7 defusedxml-0.8.0rc2 didkit-0.3.2...
However, during the image build for linux/arm64
, we run into issues:
#50 21.22 ERROR: Could not find a version that satisfies the requirement didkit==0.3.2 (from versions: 0.0.1, 0.2.1)
#50 21.22 ERROR: No matching distribution found for didkit==0.3.2
...
21.22 ERROR: Could not find a version that satisfies the requirement didkit==0.3.2 (from versions: 0.0.1, 0.2.1)
21.22 ERROR: No matching distribution found for didkit==0.3.2
------
Dockerfile:67
--------------------
65 | # Dependencies are installed as root so they cannot be modified by the application user.
66 | RUN pip install -r requirements/pip_tools.txt
67 | >>> RUN pip install -r requirements/production.txt
68 |
69 | RUN mkdir -p /edx/var/log
--------------------
ERROR: failed to solve: process "/bin/sh -c pip install -r requirements/production.txt" did not complete successfully: exit code: 1
I'm not sure if this is caused by an incompatibility between Python 3.11 and didkit
. I have not rolled back the changes to the public Dockerfile as Python 3.11 was a requirement for the Redwood release of Open edX.
As part of the process of changing ownership of CourseCertificate signature assets, existing certificates and the code that creates certificates in the LMS will potentially need to be changes to support the new canonical asset source.
This issue current represent both the required discovery and the execution of making certificates function as they do today with a new asset source.
fix SASS related to print views for program certs which is breaking.
During discussion about the future of Credentials IDA and it's functionality, it came up to discussion that dependency on Discovery brings a lot of complexity for operating Discovery. Discovery is used to provide Programs and Course Runs data to form a Learner Record for a user. Furthermore, programs can be configured only in Discovery IDA, therefore the dependency between Credentials and Discovery can't be removed without changes to programs implementation in Open edX.
It was decided to evaluate the possibility of reimagining the Program's implementation, so this functionality can be moved out of Discovery IDA and integrated directly into LMS, for example via a separate plugin. This would reduce operational complexity, but also make it possible to evolve and provide more value for community installations. Additionally,
This ticket can be used for tracking purposes for this initiative.
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
Warning
These dependencies are deprecated:
Datasource | Name | Replacement PR? |
---|---|---|
npm | @babel/plugin-proposal-object-rest-spread |
These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.
Dockerfile
ubuntu focal
.github/workflows/add-depr-ticket-to-depr-board.yml
.github/workflows/add-remove-label-on-comment.yml
.github/workflows/add-to-project.yml
.github/workflows/ci.yml
actions/checkout v4
actions/setup-python v5
actions/checkout v4
actions/setup-python v5
actions/setup-node v4
actions/checkout v4
actions/setup-python v5
actions/setup-node v4
codecov/codecov-action v4
actions/checkout v4
actions/setup-python v5
.github/workflows/commitlint.yml
.github/workflows/migrations-mysql8-check.yml
actions/checkout v4
actions/setup-python v5
actions/cache v4
.github/workflows/push-docker-image.yml
actions/checkout v4
actions/github-script v7
docker/setup-buildx-action v3
docker/setup-qemu-action v3
docker/login-action v3
docker/build-push-action v5
docker/build-push-action v5
.github/workflows/self-assign-issue.yml
.github/workflows/upgrade-python-requirements.yml
package.json
@openedx/paragon ^22.0.0
bi-app-sass 1.1.0
css-loader 7.1.2
css-minimizer-webpack-plugin 7.0.0
file-loader 6.2.0
mini-css-extract-plugin 2.9.1
sass 1.78.0
sass-loader 16.0.1
url-loader 4.1.1
webpack 5.94.0
webpack-bundle-tracker 3.1.1
@babel/core 7.25.2
@babel/eslint-parser 7.25.1
@babel/plugin-proposal-object-rest-spread 7.20.7
@babel/plugin-transform-modules-commonjs 7.24.8
@babel/plugin-transform-object-assign 7.24.7
@babel/preset-env 7.25.4
@edx/eslint-config 4.2.0
babel-loader 9.1.3
eslint 8.57.0
eslint-plugin-import 2.30.0
eslint-plugin-jsx-a11y 6.10.0
jasmine-core 5.3.0
jasmine-jquery 2.1.1
karma 6.4.4
karma-coverage 2.2.1
karma-firefox-launcher 2.1.3
karma-jasmine 5.1.0
karma-jasmine-jquery-2 0.1.1
karma-spec-reporter 0.0.36
karma-webpack 5.0.1
webpack-cli 5.1.4
.nvmrc
node 20
requirements.txt
requirements/monitoring/requirements.txt
The Credentials IDA is targeting support for Python 3.12 for the Open edX Sumac release. The Sumac release is targeting a 2024-10-09
date.
The Open edX credentials service is currently used only for program credentials.
In 2020, OpenCraft conducted a technical discovery that provides high-level proposals for extending the credentials application. This proposal contains the results of the discovery.
Marco Morales:
This has been stuck behind me writing detailed product requirements for a long time, though the project is based on work done before. Also, our addition of program records 2 years ago (edx/credentials) can be seen here. A preview of a learner "My Achievements" view is shown in this Invisionapp as well, but that was never built.
Marco's requirements:
Building on the ideas in the "work done before" document before here are a few milestones to consider / size:
TBD
Deliverables/milestones:
Nutmeg or Olive
TBD - Q3/Q4 2022
OpenCraft
The initial discovery was prepared by Jill Vogel ([email protected]) -- feel free to reach out to her with any questions.
Under the effort of Django 4.2 Upgrade, complete all of the following steps to complete the upgrade.
Django 4.2
tox.ini
, and GitHub Actions
workflows.Django
version in the requirements to Django==4.2
make upgrade
to update all dependencies for Django 4.2
.Django 4.2
and Django 3.2
to contain backward compatibility.Django 3.2
.19 aug 2022
25 sept 2022
30 Jan 2023 (date subject to change but will be after Olive cutoff)
August 2023
Quince
The Credentials front end uses an outdated approach to React JS usage, and still uses django templating.
Entire front end: https://github.com/openedx/credentials/tree/master/credentials/templates
We are replacing the UI with an MFE: https://github.com/edx/frontend-app-learner-record/
Core functionality will be 1:1. Exceptions include:
No response
No response
No response
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.