eudat-b2note / b2note

B2NOTE is a EUDAT pilot service allowing users to "annotate" semantically a document.

B2NOTE

A EUDAT service for creating, storing and managing annotations about online resources.

Annotations are comments and/or indications providing extra information about a resource such as a data-file.

The B2Note prototype service allows terms defined in controlled vocabularies, ontologies and thesauri to be used in annotations.

Click here for B2Note demo.

The data model used in B2Note is based on the W3C web annotation data model.

Branch	Status
Master
Development

Contributors

Name	Affiliation	Contact
Dr. Yann Le Franc	e-Science Data Factory	ylefranc at esciencefactory.com
Dr. Antoine Brémaud	e-Science Data Factory	abremaud at esciencefactory.com
Mr. Pablo Ródenas Barquero	Barcelona Supercomputing Center	pablo.rodenas at bsc.es
Dr. Tomas Kulhanek	e-Science Data Factory	tkulhanek at esciencefactory.com

Instructions for contributing

Please follow along guidelines outlined at:

HOW TO GITHUB [...] by Rich Jones.

With the addition that the name of the worked-on fork-branch eventually submitted for a pull request should be formatted as follows:

contributor_short_id (optional) _ #issue_reference_number _ Code_change_description (omitting spaces)

In short:

1. Log an issue on source repository: EUDAT-B2NOTE/b2note.
   1. Describe contribution objectives.
2. Fork branch of interest (or update fork if previously forked).
   1. Branch out from fork base using naming convention detailed above.
3. Commit changes to branched out code.
4. Submit reasonably documented pull request to branch of interest.
   1. May require preemptive alignment to target branch latest code.

Copyright

Copyright © e-Science Data Factory, 2015-2016

Copyright © Barcelona Supercomputing Center, 2015-2016

This software may not be used, sold, licensed, transferred, copied or reproduced in whole or in part in any manner or form or in or on any media by any person other than in accordance with the terms of the Licence Agreement supplied with the software, or otherwise without the prior written consent of the copyright owners.

This software is distributed WITHOUT ANY WARRANTY, express or implied relating to MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE, except where stated in the Licence Agreement supplied with the software.

Created By:    Yann Le Franc (PhD),          e-Science Data Factory;
               Antoine Brémaud (PhD),        e-Science Data Factory;
               Pablo Ródenas Barquero (BSc), Barcelona Supercomputing Center.

Created for Project:    EUDAT2020

EUDAT receives funding from the European Union's Horizon 2020 programme - DG CONNECT e-Infrastructures. Contract No. 654065

b2note's Issues

Indent exported JSON-LD

Improve user experience by displaying pretty printed JSON of annotation documents JSON-LD.
Either when displayed in the interface or included in the text file made available for download from the "export" menu.

API, return full JSON-LD response

Using event hooks before returning items.

Update data model to W3C web annotation

http://www.w3.org/TR/annotation-model/

API: send list of annotations based on provided target_id

Handle through calls to Eve API containing file_id as parameter.
API calls Django-MongoDBEngine.
DjangoMongoDBEngine retrieves corresponding documents.
DjangoMongoDBEngine formats documents as validated JSON-LD.
DjangoMongoDBEngine sends JSO-LD to API.
API forwards JSON-LD as response.

Design homepage, login, sign-in, profile pages

Set Travis to use MongoDB 3.2 Enterprise

Currently uses MongoDB 3.2 Community version whereas version on the server is 3.2 Enterprise.

Ideally version used in tests should match that used in production instead.

String inputs check as: isinstance( s, (str, unicode) )

In b2note_app > mongo_support_functions

In cases where an object of type string can be accepted as well as one of type unicode, conditions on the input object should become that stated in the above title, replacing:

if type( s ) is unicode ...

with:

if isinstance( s, (str, unicode) ) ...

Review styling of "Create annotation" section

Following implementation of tab layout for new create free-text annotation functionality.

Checking for existing Solr indexed class with "labels" matching string

Will be used to send feedback message to user in case they attempt to create a free-text keyword annotation using an input string (body.value) for which exact match can be found as "labels" of Solr indexed classes.

Output: json document of Solr matches/False.

Inputs: free-text tag body.value (string).

Process:

1. Check input string is valid.
2. Send request (python requests module) to Solr, for documents with labels field matching the input string.
   e.g.

import requests
input_string = "Refrigeration"
r = requests.get('https://b2note.bsc.es/solr/b2note_index/select?q='+  input_string + '&wt=json&indent=true&rows=1000')

3. Return json result for processing (i.e. will display results as suggestions to choose from, with an option to proceed with making a free-text rather than semantic tag).

Copy annotation document

User shall be able to copy any annotation documents from interface, so as to create an new additional identical annotation document except for its provenance info with current user as creator and date created updated.

Define test DB credentials as environment variables

Currently tests are run using hardcoded dummy username and password at script time for getting Django with mongoDB engine to connect to the test-db, otherwise Django related tests fail indicating that authentication fails.

Ideally these credentials should be, in the same way as they are on production side, defined as environment variables to be retrieved by MongoDB to set up the db and user credentials and by Django for setting up the connection to the DB.

Edit user profile information

User shall be able to edit some or all of their profile information.

Function checking duplicate annotation

Will be used to send feedback message to user in case they attempt to create an annotation with a body that is a duplicate of a previously existing annotation for the same target file.

Output: boolean duplicate True/False

Inputs: target_url (str), intended (new) annotation body (dict).

Process:

1. Retrieve all annotations with target jsonld_id that of the target file.
   2a) New annotation to be semantic tag: check whether any of the retrieved annotations uses the same jsonld_id as the intended new one (exclude checking free-text).
   2b) New annotation to be free-text: check whether any of the retrieved annotations uses the same body.value as the intended new one (includes either semantic-tag or free-text).
2. Return boolean result of check.

Typeahead scrollable drop-down menu for result entries

After fixing issue #5 larger (max 10000) set of results can be retrieved from Solr and as many entries displayed in Typeahead.

This implies that we may want the typeahead dropdown menu to become scrollable for the user to be able to conveniently visualise all entries/results/matches to their text input.

Split-up create annotation function

As one core function creating the document with minimum information then returns either handle (e.g. mongodb _id).

Then separate functions calling the core function then adding extra fields corresponding to the cases of "semantic tag" or "free text" annotation.

U.I. free text

Improve the interface_main for inserting free text annotations.

API, monitor service availability

Check API responsiveness similarly to checking Django page responsiveness.

Travis CI fails at Pull request

As all pull requests are done from external repositories, the Travis CI cannot read the environment variables, so the tests fails.
By defining those variables with dummy values in the travis configuration file, the tests will pass.

Serve Sparql endpoint on root-url/sparql

Make Virtuoso SPARQL Query Editor available from web browsers calling: https://b2note.bsc.es/sparql

2 options:

• Include in corresponding Django project view.
• Apache configuration redirect to Open Virtuoso page.

Configure login

1. Install a sqlite database for storing the logins.
2. Integrate login in the site.

Re-establish interface_main.html form action urls

templates > b2note_app > interface_main.html

URLs were modified in attempt to fix html namespace issues when trying to return console interface through API calls which was not a sensible way to proceed.
These need reverting to shortest working form (i.e. can we remove the '..' or '../' without breaking the routing?

and other occurrences...

Function checking length of free-text body value

Will be used to send feedback message to the user in case they attempt to create a free-text tag annotation with a long body value, to check whether their intent is "tagging" or "commenting" (resulting in 2 different types of annotations: "free-text tag" or "(free-text) comment").

Output: boolean True/False.

Inputs: intended (new) annotation body value (string), tag string length check limit (integer, default 60).

Process:

1. Check input is str or unicode.
2. Check whether input string is longer or shorter than tag string length check limit.
3. Return boolean result of check.

Typeahead displays fixed maximum number of results

E.g. typing 'ana' in textarea currently displays only 5 entries to choose from as semantic tags for creating an annotation whereas the outcome of the corresponding query via browser address bar indicates 231matches: https://b2note.bsc.es/solr/b2note_index/select?q=ana&wt=json&indent=true .

{
"responseHeader":{
"status":0,
"QTime":0,
"params":{
"q":"ana",
"indent":"true",
"wt":"json"}},
"response":{"numFound":231,"start":0,"docs":[
{
"indexed_date":"2016-04-15T00:00:00Z",
"labels":"Anatidae",
"synonyms":["waterfowl"],
"text_auto":"Anatidae",
"uris":"http://purl.obolibrary.org/obo/NCBITaxon_8830",
"id":"19174791-7fc0-43b0-a895-2cc72e2b130b",
"version":1531665477077565441},
{
"indexed_date":"2016-04-15T00:00:00Z",
"labels":"analog-to-digital converter",
[...]
}}

Search annotation on keyword

Add readme file

README.md displayed on homepage for providing visitor with essential information about project.
Specifying service objectives, contributors, affiliations, contact details, contributing instructions, license info.

CreateAnnotation function to return False rather than None

b2note_app > mongo_support_functions.py > CreateAnnotation()
Currently returns "None" object in case of malfunctions.
Rather return "False", expected more appropriate to the purpose.
Adapt related tests accordingly.

Console links to individual annotation page

Hyperlinks in the console should direct to particular annotations display page internal to the site rather than external pages of the original semantic terms.

test_create_annotation, also retrieve document

Currently test asserts that an object is returned, its type and content are as expected.

Should also check that the document is created on the DB.
Also compare document contents to returned objects' contents, and each and both to the initial input(s).
That is, using: Annotation.objects.get( id==a.id )

Apply same reasoning to:

• test_create_semantic_tag
• test_create_free_text

Indexing of large ontologies

Report: Current efforts in indexing ontologies from Bioportal API resulted in script interruption on ontologies containing > 600,000 classes.
In these cases, harvesting steps appear to be processed successfully whereas indexing of the class container object results in script execution interrupt.

Fix: split up indexing in smaller chunks by indexing successive portions of the class container object with size limit fixed based on that of largest ontologies successfully indexed i.e. about 250,000 classes.

SQLite DB in the server for user account information

The DB and all the settings should be defined as a environment variables for omitting the sensible data in the code.

API, Travis CI tests

Every custom API function should have a set of two tests adjoined:

1. Testing expected completion upon providing rightful parameter in call.
2. Testing expected error raising upon calling erroneously.

Change Annotations' jsonld_id base url

For consistency with API design: "/b2note.bsc.es/annotation/_id" would rather be "/b2note.bsc.es/annotations/_id".

"Forgot password" link and process

On login form page.

Travis for continuous integration

Put in place basis for Continuous Integration using Travis service.

API, return annotations based on creator nickname

Call provides string object as parameter "creator_nickname" or equivalent.
Fetch from mongoDB annotation documents including as creator the agent with nickname matching parameter string.
Return JSON of fetched annotation list.

Integrate captcha to registration form

Edit annotations

Logged-in owner can: edit (change body, motivation), delete, change type (semantic, free-text).
Logged-in non-owner can: copy then edit copy.

Change jsonld_id attribution method in createFromPOSTinfo support function

Report: Avoid creating document with temporary jsonld_id = "https://b2note.bsc.es/annotation/temporary_id" used for specific retrieval in second steps where final value based on DB _id is attributed to field jsonld_id of Annotation document.

Fix philosophy: Attribute jsonld_id based on MongoDB _id at time of calling save() may or not be possible, otherwise get MongoDB _id or document copy or some unique identifier of the document returned at time of calling save() then retrieve or use document, set jsonld_id field value according to design and call save() again or update() function of some sort...

Include calls to CreateAnnotation in try-except blocks

b2note_app > mongo_support_functions

Calls to CreateAnnotation function from CreateSemanticTag and CreateFreeText functions should happen within try-except blocks.

That is, include statements within the main try-except block and mitigate potential failure of commands with "if" statements.

Reasoning: prevent application crash events while allowing monitoring of malfunctions.

Email verification in registration process

Email should be sent to provided email address containing a link to click that contacts service back and grants full login access to the website.

Add captcha for preventing automated access

Rename "searchapp" folder and app

Initial naming needs updating with more relevant and readily identifiable label.
Mind consequences in url redirects and others, since name may be used internally by Django and possibly for in and out API calls to Django.

API, swagger documentation

Add API documentation support using Swagger framework.

https://github.com/nicolaiarocci/eve-swagger

Edit annotation document information

User shall be able to edit some of the information contained in those annotation documents that they have created.

Clicks on "pen" icon in display table, access form, submit.

Validation script for web annotation format of json documents

JSON-LD and RDF.

annotator_id, sqlite column name

Is name of column correct/automatically set in any way?

Server side:

sqlite> .schema accounts_usercred
CREATE TABLE "accounts_usercred" (
    [...],
    "annotator_id_id" integer NOT NULL
);

Display ontology acronym and term short-form

Aim: The ontology acronym and term short-form should appear after the term label for each entry displayed in the selection list attached to the textarea of the "create annotation" section of the B2Note prototype service interface. Similar to that of Ontobee website, see: http://www.ontobee.org (menu appearing upon typing more than 3 characters in the "Keywords" text-area).

Prerequisite: New Solr core fields dedicated to metadata about ontologies of origin for every term are made part of the schema, and corresponding information is harvested from a repository online.

Plan:

1. The information corresponding to ontology acronym and term short-form should be defined for retrieving alongside that of the term label from solr core in static > js > ontology_typeahead.js > Bloodhound > remote > map > suggestion_set.
2. Content of corresponding variables should be included in the format definition of the typeahead menu-entry Typeahead > Display > Templates > Suggestion.

#This issue report is also intended for use in the process of designing external contributions workflow.

Re-establish motivations

Semantic tag: "tagging".
Free-text comment: "commenting".

CreateSemanticTag function, safeguard from object_json input object

b2note_app > mongo_support_function > CreateSemanticTag

Check that object_json exists and is a string or unicode of length longer than zero before proceeding with loading as python object.

Handle via API: external calls for creating annotation

Current: B2Share sends file information (and potentially e.g. user-related info) as an HTTP POST directed to the "main-interface" Django view.

Aim: External calls get addressed to the Python Eve API which may then relay necessary information to the Django view for display in the interface and the user to be able to trigger the creating of an annotation based on this provided information via the web interface.

Considerations:

• Custom endpoint/function in Eve, add a decorator specifying the url tail: http://webcache.googleusercontent.com/search?q=cache:ub4N5Ta0T7cJ:nicolaiarocci.com/building-custom-endpoint-handlers-with-eve/+&cd=4&hl=es&ct=clnk&gl=es&client=safari
• POST from Eve, http://stackoverflow.com/questions/22191816/submit-request-post-internally-in-python-eve or use Requests?
• Ensure status of request is provided in response to the HTTP call.
• Post-requests event hooks: http://python-eve.org/features.html#post-request-event-hooks