eudat-b2share / b2share

B2SHARE software for the EUDAT CDI services.

B2SHARE

B2SHARE is an user-friendly, secure, robust, reliable, and trusted service to share and publish your research data. B2SHARE is able to add value to your research data via (domain tailored) metadata, and assigning Persistent Identifiers PIDs to ensure long-lasting access and references. B2SHARE is one of the B2 services developed via EUDAT.

Deposit and release your data via the generic interface, or select a community extension including domain-specific metadata fields. Share your data with others in a safe and trusted environment. Scientific communities are able to brand templates, and use their own collections with specific metadata.

B2SHARE is based on Invenio. Invenio enables you to run your own electronic preprint server, your own online library catalogue or a digital document system on the web. It complies with the Open Archive Initiative metadata harvesting protocol.

Main service is running at: https://b2share.eudat.eu/

Developer documentation

A developer documentation is available. It can be built like this:

$ pip install -r requirements.txt # install dependencies
$ pip install -e .[all]
$ cd docs
$ make html
$ # open in your browser docs/_build/html/index.html

Acknowledgement

This work is co-funded by the EOSC-hub project (Horizon 2020) under Grant number 777536.

b2share's Issues

Stopping / Restarting upload of Big Data

(EUDAT-364c on CSC JIRA)

When uploading a file, the file is split into chunks, and then recombined when the entire file is uploaded. When stopping the upload of a large file, the progress bar goes back down to zero. If you resume the upload, the upload starts again from scatch, instead of resuming from where it left off, (or starting again with the first uncompleted chunk.)

If a file upload is permanently halted, the chunks remain in the upload directory.

Running on a virtual machine with limited space and memory, I was unable to test how the upload performs with really large data, but I verified that it works on a file size of 354MB. The upload for that file took about 4 minutes, even though it was on a local machine.

Some links on the home screen are empty

Because we only copy templates/*.html files to the invenio directory we don't have our markdown files there. These markdown files provide the content for about/faq/tou and therefore we need to copy them. This is solved with pull request #48

Signing in with a non-existing account doesn't give clear error

Sign in to https://b2share.eudat.eu/youraccount/login/ with nonsense information. The page just silently reloads giving no feedback on what happened.

• Clear error information needed.
• First check if it is Invenio error or introduced by us (B2SHARE overlay)

Inconsistent placement of search bar in the blue bar (new design)

GUI for DRIHM

• Use feedback from #19.
• Check the elements and values for controlled vocabularies with the community

Improve OAI_DC formats (add more fields)

See file etc/bibformat/format_templates/OAI_DC.xsl, check whether this is the one used for OAI-PMH, add more informations to it.

(Don't) Remove language field from the common set of metadata

Leave it for the subdomains if required.

provide virtual machine for B2Share-development from CSC network where developers can get access

Adequate (end) user documentation

Carl Johan

• As a wiki page here at Github
  • Always up-to-date. Changes from here deployed to the application.
• Integrated as online help/how to in the B2SHARE application

EUDAT-443: Align license label on the right

(Copied from CSC JIRA)

The formatting of the metadata fields is poor: "Description" is aligned to the top of the text entry, and the "Open Access" switch button is the wrong size or should be aligned right. (It's not totally clear which is better.)

EUDAT-411: Add a cron script for cleaning up simplestore upload folder

(Issue copied from CSC JIRA)

Once bibupload is done, the temporary uploaded file can be safely deleted.

• cron job, select e.g. 3 week old files (tunable by a script parameter)
• cleanup-b2share-upload-files.sh
• make use of the Invenio configuration setting for the upload location.

Metadata from B2SHARE is harvested and provided by B2FIND

• As long as we have no records, use the KTH instance for development
• Our OAI-PMH interface looks OK. So what is missing? Is B2FIND woring on it?

When going to youraccount (click your name) an error is generated

GUI for CLARIN

Configure it as suggested in #12 by Ilja. Look into the relevant parts of user feedback in https://jira.csc.fi/browse/EUDAT-364.

Create abuse form - create basic form

The B2SHARE UI must have an section with a CAPTCHA protected form for "Report Abuse or Inappropriate Content" which creates a B2SHARE-Abuse RT Ticket
The Form must contain following fields:
Report Subject: Abuse or Inappropriate content | Copyrighted material | No research data | Illegal content (choose one)
URL of Inappropriate content within B2SHARE
Why is the content Inappropriate? (500 char)
Your claim is based on: The content violates my rights | I am annoyed by the content | I am the legal owner of the content
Your name:
Your affiliation
Your email address
Your address
Your phone number
B2SHARE UI must also contain a valid link to B2SHARE ToU (Terms of Use)

Add user documentation as "User Guide" at the list of links at the bottom

GUI for EUON

Just like #20

Test deployment on stable Invenio

Use the latest stable release of Invenio to test B2SHARE deployment.

If this works, we move to this stable release as our base.

Deploy KTH version

https://test-eudatis.csc.fi/

is now running the same version of Invenio + Simplestore as KTH.
There was some config issues, though. Because the older version of
Invenio/Simplestore uses Jinja 2.6 and the new one Jinja 2.7

From /opt/invenio/lib/python/invenio/jinja2utils.py

had to be commented off:

jinja2._markupsafe.escape = utf8escape

jinja2._markupsafe.Markup = Markup

I guess it's because jinja2 2.6 had _markupsafe-subdir under the jinja2 dir
and jinja2 2.7 uses the MarkupSafe-module. And the requirements for the jinja2 was
installed orginally from the requirements which we got from:

https://raw.github.com/SimpleStore/invenio-scripts/master/install/install-invenio-deps-rh.sh

https://b2share.eudat.eu is still running the version, which is installed from the git.

Check the upload speed issues with Invenio/B2Share

Uploading to B2Share seems to be quite slow. Check what is causing this (http-server config? invenio? hw?

"Lost Password" page leads to a 404 error

(EUDAT-364d on CSC JIRA)

On the login page the is a link marked "Lost your password?" Following that link leads to a 404 error.

Allow user to logout in the new UI

Development virtual machines as clones of the CSC deployment

We need a way for developers to work in the same environment, especially configured in the same way, as the deployment machine uses.

CSC guys already have a common development VM for us, which has the same environment as the public deployment at b2share.eudat.eu.

So we need an efficient way for each developer to request (or make himself) a clone of this common dev virtual machine.

This is a blocker issue. Until it is resolved we can't solve quite a few other things.

accounts seem to be usable even with account policy requiring account activation

accounts seems to be usable even without account activation and account policy is requiring account activation

Use global handle system for the PID link of the record page

If one goes to the record page describing a specific resource (e.g. https://b2share.eudat.eu/record/24) on the right a Metadata area shows up. The PID link inside is a local handle system prefix (currently epic.sara.nl/v2/handles) instead of the global hdl.handle.net/ prefix. Change this to point to the global handle system URL.

Pause while depositing an item(s)

After the metadata fields are filled in and the "Deposit" button is clicked, there is a pause while a PID is retrieved from the EPIC server. There should be some feedback to the user while this is happening, similar to the "Loading Form" message after you select a Domain.

Create abuse form - intergrate with RT and mail

See related ticket: Create abuse form - basic

A confirmation email should be sent to the requestor address above, after that a RT ticket can be created

After that, a confirmation email for created RT ticket is to be sent to the submitter:
" Thank you for submitting an abuse resolve request to B2SHARE Abuse. Your request will be handled by our administrators. You will receive an reply to your request.

**

Note. Also consider how to implement the mail intergation so that email cannot be used by spammers to relay spam,,...

Metadata tooltips do not use the correct attributes

In simplestore-addmeta-table.html, there are options at the bottom of the file for how the tooltip should appear. These are not currently working - instead default values are being used. There is no delay, and the trigger is set to both hover & focus.

Features from KTH instance into the CSC instance

The CSC guys deployed the HEAD revision from our repository, but the instance at http://b2share.eudat.eu has a few diferences compared to Ilja's KTH instance:

• No "Linguistics" metadata option during Submission
• No Search parameters and Browse mode, see screenshot

Find out where the problem is and explain, so that CSC team can fix it.

Z-indexes of search parameter + user link + bee are conflicting

The correct order should be:

• bee: lowest
• username link: middle
• search parameters: highest

persistent XSS-issue

I've now played around a little bit more now with this persistent XSS-issue and it seems that the description/tag-fields are atleast vulnerable, possible others also. How it works is that the attacker can add some malicious code in <script></script>-tags to description/tag field (possible to others also) and after that it can be executed for example when some other user finds the malicious record from the search-records.
Solution is that the input from the fields needs to be also escaped. There are some good ways explained in http://en.wikipedia.org/wiki/Cross-site_scripting
This is more Simplestore-related issue, and not in relation to the reflected cross-size scripting issue which was found in Invenio

HowTo for new users

Add intros to b2share.

http://usablica.github.io/intro.js/

On the search result page, the blue background of the search bar doesn't extend to the horizontal limits

change the contact link on the pages to point support request form

change the contact link on the pages to point support request form @ http://www.eudat.eu/support-request

websearch results are not showing up correctly when something is searched

Browse mode not working

In the search box open the options-ticker. There is "Browse" button down that should show all records. It shows none.

Stored PIDs should use handle prefix

Use CFG_HANDLE_SYSTEM_BASEURL instead of the 'location'.

Add user documentation link

Search result output is throwing a 500

When searching with an empty string. Logs below.

2014-01-14 19:00:50,181 ERROR: Exception on /search [GET] [in /usr/lib/python2.6/site-packages/flask/app.py:1423]
Traceback (most recent call last):
  File "/usr/lib64/python2.6/site-packages/invenio/webinterface_handler_flask.py", line 244, in __call__
    response = self.app.full_dispatch_request()
  File "/usr/lib/python2.6/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib/python2.6/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python2.6/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/python2.6/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/lib64/python2.6/site-packages/invenio/webinterface_handler_flask_utils.py", line 199, in decorated_function
    return render(template_name, **ctx)
  File "/usr/lib/python2.6/site-packages/flask/templating.py", line 128, in render_template
    context, ctx.app)
  File "/usr/lib/python2.6/site-packages/flask/templating.py", line 110, in _render
    rv = template.render(context)
  File "/usr/lib/python2.6/site-packages/jinja2/environment.py", line 969, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.6/site-packages/jinja2/environment.py", line 742, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/invenio/etc/templates/websearch_search.html", line 21, in top-level template code
    {% from "websearch_results.html" import render_search_results with context %}
  File "/usr/lib/python2.6/site-packages/jinja2/environment.py", line 1013, in make_module
    return TemplateModule(self, self.new_context(vars, shared, locals))
  File "/usr/lib/python2.6/site-packages/jinja2/environment.py", line 1070, in __init__
    self._body_stream = list(template.root_render_func(context))
  File "/opt/invenio/etc/templates/websearch_results.html", line 225, in top-level template code
    {{ render_search_results(recids, collection, pagination, format_record) }}
  File "/opt/invenio/etc/templates/page.html", line 114, in top-level template code
    {% block page_body -%}
  File "/opt/invenio/etc/templates/page.html", line 146, in block "page_body"
    {% block body %}
  File "/opt/invenio/etc/templates/websearch_results.html", line 222, in block "body"
    {{ render_search_results(recids, collection, pagination, format_record) }}
  File "/opt/invenio/etc/templates/websearch_results.html", line 159, in template
    {%- if of[0] == 'h' -%}
  File "/usr/lib/python2.6/site-packages/jinja2/environment.py", line 378, in getitem
    return obj[argument]
UndefinedError: 'of' is undefined

Start/Stop Upload

(EUDAT-364b on the CSC JIRA)

When uploading several files, if you stop and start the upload quickly, occasionally, the red "Stop Upload" button is placed to the right of the green "Start Upload" button, and both are briefly visible. I was unable to consistently generate this error, and it does not happen every time.

This is a minor error, as the problem fixes itself, and the files themselves appear to be uploaded correctly.

Collections and other facets empty

There are facets for faceted browsing in the search results. Currently they are empty, even though they should be populated by attributes from the records.

Possibly related to #35 and #36.

Provide (eventually update) the Landing Page, FAQ and the ToU

Some texts are translated to other languages

E.g. some of the buttons, but also other texts. It produces an ugly mixture of english and czech (in my case).
See the following example:

Latest deposits: don't do any html escaping, authors should be links

Registry and Monitoring

• Include the B2SHARE services into the central registry and
• monitoring.
• Clarify the operational policy (specify OLA)

(split the issue as needed, guys)

The ToU, FAQ, About markup documents should be copyed from the wiki during build.

Design update

The new design implemented by Emanuel.

Change /oai2d to answer through normal http, not https

https is not needed for oai2d-interface (as it is public data) and OAI-PMH standard requires http so that path needs to be changed to answer through http.

OAI-PMH interface showing no records

We now have several testing records (search for empty string to list):
https://b2share.eudat.eu/search?p=&action_search=z

OAI-PMH server displays no records:
https://b2share.eudat.eu/oai2d?verb=ListRecords&metadataPrefix=oai_dc

bibupload won't work in CSC instance

On the logs it will generate error:

2013-12-18 08:15:06 --> Input file '/invenioupload/var/tmp-shared/webdeposit_2013-12-18_08:13:57NTUywM.marcxml', input mode 'replace'.
2013-12-18 08:15:06 --> Stage 2 failed: Error while elaborating FFT tags: Incorrect padding
2013-12-18 08:15:06 -->