zenodo / zenodo-rdm

Zenodo, powered by InvenioRDM

Zenodo - Research. Shared.

Zenodo, a CERN service, is an open dependable home for the long-tail of science, enabling researchers to share and preserve any research outputs in any size, any format and from any science.

Powered by Invenio

Zenodo is a small layer on top of Invenio, a ​free software suite enabling you to run your own ​digital library or document repository on the web.

Installation

See INSTALL.rst

Developer documentation

See https://zenodo.readthedocs.io/

License

Copyright (C) 2009-2017 CERN.

Zenodo is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

Zenodo is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with Zenodo; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.

In applying this license, CERN does not waive the privileges and immunities granted to it by virtue of its status as an Intergovernmental Organization or submit itself to any jurisdiction.

Zenodo (R)

Zenodo (R) and the Zenodo logo are trademarked by CERN and are not covered by the GPLv2 license. Thus, if you instantiate Zenodo on your own servers you must change the branding to your own.

Please consider creating your own overlay to Invenio and look at Zenodo for inspiration.

zenodo-rdm's Issues

support form: add error and success notifications for frontend

The new support form does not display any success / error messages to the user. The only error messages that are displayed come from Formik form validation (using Yup)

The new support form is currently implemented in #61

In this issue:

• Evaluate how error messages (not validation) are displayed in the support form
• Display a notification when an error occurs in the backend (e.g. e-mail not sent) or when the form was sent (in case of success).

See the current form

migration: record access links in `_parent` in transform

See how we migrate restricted access links. This is different in legacy, where we track links per record version and not on the parent-level.

Remove excess search button CSS

When inveniosoftware/invenio-app-rdm#1933 is released, remove the excess search-button CSS from site.variables and site.overrides. See TODO comment in the mentioned files.

theme: header menu styling

Since "Communities" is added via Flask-Menu, we have an entry displayed in the top bar, but the styling is off:

Two options to deal with this:

• Drop Flask-Menu usage completely (i.e. don't render a menubar)
• Fix styling and adjust header to properly use Flask-Menu to be consistent in the search page as well

theming: profile pages

Preparation

• make sure the layout is defined - do we keep Zenodo or RDM styling/layout?
• list all the pages that need customisation

Pages

TBD

Decisions

TBD

deposit form: evaluate resource types

are the resource types available in invenio-rdm suitable for the demo site ?

add license

theming and content: search page

Preparation

• make sure we keep the layout from RDM - A: yes
• gather all the additional information about changes/decisions regarding the elements of the search (fill the #decisions section below)

Elements

• facets
  • access rights - do we keep them, are they curated? - A: we keep the facet
  • vocabularies - do we need to migrate the vocabs ? A: only the task with resource types (separate tasks)
  • keywords - are they curated, should be moved to vocabularies? A: should be added as facet - they are a subset of subjects (custom subject = keyword) - check if the mapping has correct type (keyword) for the subjects.
• styling/theming/layout - do we keep the RDM styling or we use Zenodo's? A: RDM
• search item - do we keep RDM styling or we use Zenodo's? A: RDM, add many versions information on the search items (priority: now)

Decisions

• We keep the access rights facet
• Migration of vocabs: only the task with resource types (separate tasks)
• Keywords should be added as facet - they are a subset of subjects (custom subject = keyword) - check if the mapping has correct type (keyword) for the subjects.
• RDM styling/layout
• Search item keep RDM style, add many versions information on the search items (priority: now)

search-ui: remove padding on header

migration: record communities in `_parent` in transform

    # TODO: Enable communities after handling "communities" stream, because we will
    #       need their PK for inserting entries in `rdm_`
    # comms = data.get("communities")
    # if comms:
    #     r["communities"] = {
    #         "ids": [c for c in comms if c != "zenodo"],
    #         "default": comms[0],
    #     }

Create a Zenodo profile for describo

The goal of this issue is to create a describo profile to describe a deposit in InvenioRDM.

The agreed fields for this iteration are the following:

• title
• author
• license
• keywords (subjects)
• datePublished

---

Tasks:

• create a profile compliant with Describo rules.

• serve the file as a static file in zenodo-rdm.

---

Useful links:

• https://github.com/Arkisto-Platform/describo-online/wiki/Writing-a-domain-specific-profile
• https://github.com/UTS-eResearch/describo/blob/master/src/components/profiles/paradisec.describo.profile.json
• https://schema.org/ (property definitions)

frontpage: fix white color in footer sections

vs zenodo sandbox

create a cron task to wipe showcased data

set a cron task to wipe and recreate the showcase data

migration: search redirections

If we compare the Zenodo search guide and the one that comes with InvenioRDM, there are quite some changes in terms of nested fields (title vs metadata.title) and semantically equivalent fields (keyword vs subjects).

Many of these changes are related to the different search index mappings, but also migrated/changed fields. We have to:

• Identify the differences between mappings
• Make a decision to either...
  • Use the query parser to translate the fields (see users example)
  • Use copy_to in the mappings to create fields "aliases"; not ideal since it means overriding mappings (which I'm not sure if it's possible atm)

Manually create fixtures for ZenodoRDM

• Add a fixture mechanism to load communities from a json. See for example the users in invenio-rdm-records. Also see how do we add new users.
• The json format can be inspired from https://github.com/zenodo/zenodo/blob/master/zenodo/modules/fixtures/data/communities.json
• Add in zenodo-rdm the communities fixture to load

theming: community details

Preparation

• make sure the layout is defined
• gather all the additional information about changes/decisions regarding the elements of the frontpage (fill the #decisions section below)
• define elements which need changes

Elements to customise

TBD

Decisions

TBD

Postponed as the layout needs a bigger discussion.

theming: deposit form

Preparation

• make sure the layout is defined - do we keep the RDM layout and styling?
• gather all the additional information about changes/decisions regarding the elements of the frontpage (fill the #decisions section below)

Elements

• right side panel - access rights
• fields (define order)

Decisions

1. Hide access buttons to metadata (side panel)
2. hide metadata only button

For this task: Add config variables. Fix the backend part in a different task.

migration: add rights/licenses in `_metadata` in transform

        def _rights(self, data):
            license_id = data["$ref"].rsplit("/", 1)[-1].lower()
            # TODO: Map legacy license IDs (or create aliases in RDM)
            return [{"id": license_id}]

        # TODO: check if license exists or create as custom
        # if data.get("license"):
        #     r["rights"] = self._rights(data["license"])

upgrade base image and config

• Move readiness/liveness probe to /api/ping.
  • Done in inveniordm, inveniordm-qa, zenodo-rdm and zenodo-rdm-qa
• Moved services to pull from CERN's registry instead of dockerhub
• Base image is upgraded to almalinux.

theming: communities home page

Preparation

• make sure the layout is defined, do we keep the Zenodo layout, styling ?
• gather all the additional information about changes/decisions regarding the elements of the communities (fill the #decisions section below)

Elements

• community search result item Part of #48
• top result item
• side panels (information boxes)

Decisions

• Keep RDM layout, i.e. don't keep the side-panel
• Restyle community-cards
• Add "Featured communities"-slideshow below header (To be added when we have the component!) inveniosoftware/invenio-communities#819

ui: missing GitHub/ORCiD option in sign-up page

Legacy:

RDM:

theming: static pages

Preparation

• make sure the layout is defined
• gather all the additional information about changes/decisions regarding the elements of the frontpage (fill the #decisions section below)
• list all the static pages we need to add

Elements

• error pages
• static page

Decisions

TBD

Static pages to be migrated

Decisions: Static pages in zenodo are external entities, they are linked from the instance and will be kept ?

migrate legacy API to new instance

more details: https://github.com/zenodo/rdm-project/issues/14

Depends on inveniosoftware/invenio-rdm-records#1139.

Endpoints

• Metadata
  • GET /api/deposit/depositions
  • POST /api/deposit/depositions
  • GET /api/deposit/depositions/<id>
  • PUT /api/deposit/depositions/<id>
• Actions
  • POST /api/deposit/depositions/<id>/actions/publish
  • POST /api/deposit/depositions/<id>/actions/edit
    • Need to check if works in all cases.
  • #181
    • We need to check the difference between this, and DELETE draft.
• #182
  • POST /api/deposit/depositions/<id>/files
  • GET /api/deposit/depositions/<id>/files
  • PUT /api/deposit/depositions/<id>/files
    • Used to sort files order. Not sure if still relevant/possible or used.
  • GET /api/deposit/depositions/<id>/files/<key>
  • PUT /api/deposit/depositions/<id>/files/<key>
    • Used rename a file. Not sure if still relevant/possible or used.
  • DELETE /api/deposit/depositions/<id>/files/<key>
• New files API
  • Blocker: we should be able to disable the legacy /api/files/... endpoints, so that we can override them in ZenodoRDM for draft/record bucket access and ops.
  • inveniosoftware/invenio-files-rest#286
  • GET /api/files/<bucket_id>/<key>
  • PUT /api/files/<bucket_id>/<key>
  • DELETE /api/files/<bucket_id>/<key>
  • #183

Metadata and REST API responses

• created
• modified
• title
• id
• record_id
• #185
• submitted
• owner
• files
  • created
  • updated
  • key
  • checksum
  • size
  • mimetype
  • links
• doi
• doi_url
• metadata
  • upload_type
  • publication_type
  • image_type
  • publication_date
  • title
  • description
  • creators
  • #186
    • contributors
    • #180
    • license
    • embargo_date
    • doi
    • prereserve_doi
    • keywords
    • related_identifiers
    • grants
    • subjects
    • version
    • method
  • #187
    • access_conditions
    • notes
    • references
    • communities
    • journal_title
    • journal_volume
    • journal_issue
    • journal_pages
    • conference_title
    • conference_acronym
    • conference_dates
    • conference_place
    • conference_url
    • conference_session
    • conference_session_part
    • imprint_publisher
    • imprint_isbn
    • imprint_place
    • partof_title
    • partof_pages
    • thesis_supervisors
    • thesis_university
    • locations
    • dates
• links
  • self
  • html
  • files
  • publish
  • edit
  • discard
  • #189
    • latest_draft
      • This is used when there's a newer version being edited
    • latest_draft_html
      • This is used when there's a newer version being edited

Other

• #184
• #188

frontpage: choose better icons

The icons are coming from the "stock" InvenioRDM frontpage, they should be changed to something more fitting to their text

js: add eslint-config-invenio

support form: add missing tests

Currently the support form is missing tests. E.g. test SupportForm.form_handler and test the endpoint directly (not the view functions).

For this issue:

• Add a test for SupportForm.form_handler
• Add a test for testing the support form endpoint
• (Optional) Add other tests that might be relevant for the module

support form: add re-captcha

The new support form does not have a re-captcha yet implemented.

The new support form is currently implemented in #61

In this issue:

• Evaluate how re-captcha is currently implemented
• Migrate the re-captcha to the new form

See the current re-captcha in action: https://sandbox.zenodo.org/support
See the current implementation in:

• https://github.com/zenodo/zenodo/blob/master/zenodo/modules/support/templates/zenodo_support/contact_form.html

global: replace missing features with "/coming-soon" links

As done in https://inveniordm.web.cern.ch

support form: fix styling

The new support form is not properly styled in comparison with the form found in production .

The new support form is currently implemented in #61

E.g. Send Request button is missplaced, the grid layout is too big, etc.

theming: front page elements

Preparation

• make sure the layout is defined
• gather all the additional information about changes/decisions regarding the elements of the frontpage (fill the #decisions section below)

Elements

• footer
• header
• side panels (information boxes)
• main panel
  • recent uploads - frontend
  • featured communities - frontend

Decisions

• Keep color of information boxes in the side panel (creates a nice and clear differentiation between the main content and the side panel)
• Main panel: inveniosoftware/invenio-app-rdm#1932

devops: when deploying recreate demo data

Every time https://zenodo-rdm.web.cern.ch is deployed, the database should be wiped and new demo data should be re-created:

• Record
• Communities + members + records (pending, accepted, rejected, etc.)
• Clean Redis sessions

migration: URL redirections

Footer in small viewport

The footer takes up almost the entire screen when viewed on smaller viewports.

Must be rectified to a compact view

configure github and orcid login

theming: community search page

Preparation

• make sure the layout is defined - do we keep the RDM styling or Zenodo's? Zenodo does not have the dedicated search page - the search is on the communities home #47
• gather all the additional information about changes/decisions regarding the elements of the frontpage (fill the #decisions section below)

Elements

• facets
  • styling
  • which facets do we need
• search result item - styling

Decisions

We leave it as it is, just make sure we display ALL the available information from the community metadata

frontpage: align with new rdm-template

A new layout structure for the frontpage is added in inveniosoftware/invenio-app-rdm#1946, which will affect the current Zenodo-frontpage.

Make sure layout blocks are aligned.

zenodo: add custom metadata

https://github.com/zenodo/rdm-project/issues/1

theme: footer should be fixed to bottom

support form: address support form comments

Address comments in #61

We are creating a separate issue so we can unblock other developments that depend on the support form

create a zenodo-rdm-qa paas instance

• spawn QA instance, to have separate testing environment (the sandbox link will become known to the public and we need a stable environment - sandbox - to showcase Zenodo-RDM
• Create a gitlab repo to store the values. See for reference https://gitlab.cern.ch/cds-team/cds-rdm-openshift.
  • Done: https://gitlab.cern.ch/zenodo/zenodo-rdm-openshift
• Investigate how to properly tag the image to differentiate sandbox from prod deployment.

support form: add files error message / highlighting

Support form does not show any error message when a file is rejected (e.g. file size is too big). The component from react-dropzonerejects the file and an error message is printed in the console.

The support form is currently implemented in #61

In this task:

• Add error highlighting to the files field when a file is rejected.

See #61 for more details
See current support form (https://www.sandbox.zenodo.org/support):

community details: make sure all demo data fields look pretty on the page

Make sure the description, curation_policy and page fields on community details page are visible.
Currently, the communities.yaml file consists of plain text for these fields (see the comment)

Steps to be done:

• show all the fields on the page (currently only the description is visible)
• fix/create(?) the html parsing for yaml file at the community details page
• replace the plain text in description, curation_policy and page fields in community.yaml file with the original html script.

See links to copy-paste the original htmls from (in the same order as in the file):

• https://zenodo.org/api/communities/biosyslit
• https://zenodo.org/api/communities/cernopenlab
• https://zenodo.org/api/communities/linguistics
• https://zenodo.org/api/communities/aas
• https://zenodo.org/api/communities/astrothesis
• https://zenodo.org/api/communities/briefideas
• https://zenodo.org/api/communities/ecfunded
• https://zenodo.org/api/communities/lara
• https://zenodo.org/api/communities/tops
• https://zenodo.org/api/communities/cs19
• https://zenodo.org/api/communities/covid-19
• https://zenodo.org/api/communities/wind_energy

prepare sandbox for zenodo presentation

• add a banner indicating that it is a demo website
• dump 0.5M - 1M records belonging to Biodiversity and NASA communities
• load records to Zenodo RDM sandbox

zenodo-rdm: align sign/sign up pages

• Add Privacy notice in sign in form
• Change sign up template to match https://sandbox.zenodo.org/signup/

support form: add translations

Currently the support form does not have any translation.

The support form is currently implemented in #61

In this task:

• Make sure that i18next is properly installed and bundled (webpack.py).
• Mark strings for translation (i18next.t('something') in the frontend.)
• Mark backend strings fo translation (e.g. error messages)
• Add translations

theming: record details page

Preparation

• make sure the layout is defined: do we keep zenodo's layout?
• gather all the additional information about changes/decisions regarding the elements of the frontpage (fill the #decisions section below)
• list all the details page elements, define which ones we need to migrate from zenodo
• do we keep the right side panels ?
• what needs to be taken into account in the main panel (content area?) what sections do we need? in which order?
• files section: do we need it for the demo site? do we keep RDM style?
• citations: do we place it in the main content (as in RDM?) or we place it on the right side panel (currently in Zenodo)

Elements

• side panels
• metadata
• files
• citations
• ?

Decisions

• We leave the metrics and "indexed in" sections out for now
• The citations stay in the main section
• Citations search box will be added in https://github.com/zenodo/rdm-project/issues/26

docker: build also branch images

Currently, only the master branch is built and tagged as the zenodo-rdm:latest image. It should be possible to also build automatically e.g. a qa or prod branches into zenodo-rdm:qa and zenodo-rdm:prod images.

Optionally, we can also introduce triggering manual builds via a repository dispatch request and the client_payload parameter.

support form: integrate zenodo implementation

Information

See https://github.com/zenodo/rdm-project/issues/7

Actions

• FIX draft PR: #61
• Evaluate what needs to be done to move the support form to InvenioRDM e.g migrate bootstrap to SUI, angular to react etc.
• Evaluate if it makes sense to move the support template to InvenioRDM and have each instance provide its own template or only parts of it needs to be configurable

The HTML template of the form is defined in: zenodo/modules/support/templates/zenodo_support/contact_form.html

1. The sanitization of the form data is carried on in a script defined in the same file.

The function dedicated to handling the submit and rendering the template is on: zenodo/modules/support/views.py

1. The categories of the support form are defined in the configuration of the application.
2. The handling of the template consists of performing a last check of the data, then wrap it and send it in two emails, one confirmation email for the user and another support ticket for Zenodo.

Accounts: Style email templates + other configuration

deposit form: creatibutors orcid dropdown link should open in new tab