kbss-cvut / 19msmt-distribution Goto Github PK

Use-case same as in 19msmt-distribution:

• we want to share in git some data (e.g. brat-data)
• docker should be able to use and modify the data
• the issue is related to permission settings (e.g. https://stackoverflow.com/questions/23544282/what-is-the-best-way-to-manage-permissions-for-docker-shared-volumes)

Goal:

• analyse possibilities how to achieve use-case and select the best option

When we underspecify the user in keycloak administration ui, i.e. the user has not specified first name, last name and email address, the user is successfully saved in internal keycloak database. However, it is not saved in graphdb server.

Thus we are not able to login in record manager and we have no way to find out why.
There is no warning message in keycloak administration ui and no warning log in keycloak server.
And since the user is not saved in graphdb there is also not any unusual error in record manager server.

Note that within keycloak administration ui the first name, last name and email address are optional fields, see screenshot below.

The best solution for this issue would be to show error message in keycloak administration ui when saving user without those fields.

Input

• gate document containing the original document and its annotations.
• location of brat annotations
• location of brat document segments (the brat tool cannot process big files so it must be split into segments)

Description of the procedure

• transforms annotations found in the GATE input file as annotations in brat annotations location
• translates annotations start and end indexes correctly from a single document to correct segments
• failed transformation of annotations are reported, e.g. the annotation was not found

Create an example fault tree in where some of its events are associated with model event types.
There must be two Root Events associated with an FHA Event type and 5 Component Failure Events.

Root event associated with an FHA event type with properties:

• required failure type
• predicted failure rate
• estimated failure rate
• aircraft type
• lost function - function type/system type
• fault event label
• criticality

Component Based Failure with a SNS component failure

• aircraft type
• predicted failure rate
• estimated failure rate
• lost function - function type/component type
• fault event label - label is constructed according to the pattern "<SNS code> - <component label> Failure"

A/C

• check and report if single select works
• check and report if multiselect select works

kbss-cvut/fta-fmea-ui#72

blocked by #18.

BRAT service:

• [M] parametrized by the configuration of users
• [C] it should allow importing new corpus (is it possible ?)
• [M] trigger to commit annotations (manual / automatic by cron) to commit to GIT repository (possibly parametrized by commit message)

TODO sharing users across services ?

Revise the current state of the integration between GATE and BRAT.

Overview of usage of BRAT and GATE

• BRAT is used for manual annotation of text document.
• BRAT Annotations should be transformed into GATE compatible format to enable automatic discovery of mentions.
• GATE is used to automatically find mentions of terms from input vocabularies in input text.

TODOs

Implement transformation of GATE to Brat annotations

• import annotations from one document into a new system or a system selected by the user.

• FTA-FMEA-UI should support the user to navigate to the annotation of a given element

• tasks:

  • create API to extract annotations from BRAT into the git repository (extract annotated files + commit into git)
  • create SPipes API to return "list of documents" from git annotated corpus repository
  • create SPipes API to return the content of the annotated document in a format compatible with FTA/FMEA tool (with references to annotation)
  • create a button to import from the selected document (fta-fmea tool vs. separated application)

Blocked by #24.

• pipeline for ER that is written in python

kbss-cvut/fta-fmea-ui#85

Implemented by #41.

It is implemented and working at KBSS (i.e. we navigate to the correct component in the annotator)

Possible solution:

• Add number to the uri

process-annotations-service.sh fails due to insufficient memory for JRE. This happens quite often so multiple execution of the script should simulate it.

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 528482304 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:3018), pid=547, tid=548
#
# JRE version:  (11.0.15+10) (build )
# Java VM: OpenJDK 64-Bit Server VM (11.0.15+10-post-Debian-1deb11u1, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
# Core dump will be written. Default location: /brat-data/core
#

---------------  S U M M A R Y ------------

Command Line: /opt/java-libs/gate-tools.jar brat2gate --input-text-file BlueSky/DA42-POH--section-7_9-7_14/DA42-POH-44.txt --brat-server-url https://kbss.felk.cvut.cz/19msmt-demo/annotator --brat-data-home-directory /brat-data

Host: Intel Xeon E3-12xx v2 (Ivy Bridge), 16 cores, 31G, Debian GNU/Linux 11 (bullseye)
Time: Tue Jun 28 13:33:07 2022 UTC elapsed time: 0.009551 seconds (0d 0h 0m 0s)

---------------  T H R E A D  ---------------

Current thread (0x00007fdc4c016000):  JavaThread "Unknown thread" [_thread_in_vm, id=548, stack(0x00007fdc53672000,0x00007fdc53773000)]

Stack: [0x00007fdc53672000,0x00007fdc53773000],  sp=0x00007fdc53771210,  free space=1020k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)

How to investigate:

• try to run the script multiple times while being logged in docker container, to find out if it happens there as well

When importing components the FTA-FMEA the backend should:

• before calling the import document call https://kbss.felk.cvut.cz/19msmt-demo/annotator/process-annotation-service.sh and wait for the service to finish
  Requirements
• the service uri should be configurable trough Docker

(BK)

• where will be RDF data stored? (folder or GraphDB) ?
• if GraphDB, is it enough to reference the repository?

Scenario:

GET /documents   // list all documents

POST /systems/{systemFragment}/documents?documentId=${url}    //imports document into the system
GET  /systems/{systemFragment}

@Matthew-Kulich will this API work for you?

@lalisand

M1) scenario from #9 is implemented, but we cannot trigger annotation processing and we don't compute probabilities (8.3). We have an example to demonstrate it on (@LaChope)

M2) we are able to trigger annotation processing

M3) we are able to trigger automatic annotation (using GATE) of documents

M4) we are able to compute probabilities

UI should look like:
#9

• maybe it is enough to just merge all DA42-POH-38 into one annotated file
• we want to create new folder (e.g. msmt-demo)
• add DA42-POH-38.txt
• add merged DA42-POH-38.ann (from --has-component, has-failure, --has-failure-mode)

Test it in the annotator if it works. Contact @blcham so we script it to generate .ttl.

A/C:

• at KBSS there is running example in dedicated folder

Review existing annotations e.g. https://kbss.felk.cvut.cz/brat/#/19msmt/BlueSky/DA42-POH--has-function/DA42-POH-38.

What needs to be in the example:

• text that can fit on one screen
• use everything from the ontology "FMEA" (i.e. components, failure modes, cause, functions...)
• at least 3 different connected segments of has-component relation (ideally there should at least 2 components that needs to be merged)

Related ontology to annotate:

How we will proceed annotations:

1. annotate only concepts
2. annotate separately on [1] one specific relationship

We will reference [1] in FTA-FMEA tool.

IRIs of created entities, e.g. components, functions and failure modes should be prefixed by the IRI of type of the entity. For example:

• fta:Component instance - fta:Component/instance-1111

Create video based on #9

• put outline here
• create one or multiple videos so we can discuss

parent issue #17

CLI input parameters:

• input ontology - containing terms to be discovered
• input text - text to be annotated
• output - path where to store resulting annotations

• copy folder
• check/change ontology IRIs so it is meaningful
• at the end, we want to have 2 examples
• add to list of documents (modify SPipes scripts)