The localega from nbisweden

To know more about the encryption/decryption scheme used in CRG (Barcelona)

EGA progress presentation at Elixir AHM in Rome

This is the PDF version (so...without animations)

AAI for the ingestion service

Register a service (maybe a test service at first)
implement the Elixir-AAI requirements
define some permissions per user (admin, ...)
Test the installation

Docker image for the User for submission/decryption

Could be used in the submission process.

It would contain the public GPG ring for the LocalEGA node, (alternatively, just download the key from a GPG server), and offer a nice web interface (including help/explanation text) to the user.

The encryption process is hidden to the user.

The web interface could also offer a "decrypt" page, where the user is asked to input their chosen passphrase, or their RSA key in order to decrypt the files that local EGA sent.

NOTE: only interesting if the files are on the user's machine. Not suited if the files are remote.

Check with ES team what is trigger message from central EGA for file ingestion

Vault skel ingestion

Create a service that moves the re-encrypted files from the staging area into the vault

Contact Oscar about the MQ settings

Just checking how they set up their message broker and are aware of some connection settings we should handle.

US16 - Admin access control (?)

As an EGAl admin,
I want to control users accessing the system,
so that I can provide the expected level of access security

Local EGA administration access

Decide on tech for inbox and set it up

Meeting to discuss tech.

Store ingested files in archive and notify Central EGA

Consolidate the Logs

... including errors.

Pass the debug level in the config file

Getting more done in GitHub with ZenHub

Hola! @jhagberg has created a ZenHub account for the NBISweden organization. ZenHub is the only project management tool integrated natively in GitHub – created specifically for fast-moving, software-driven teams.

How do I use ZenHub?

To get set up with ZenHub, all you have to do is download the browser extension and log in with your GitHub account. Once you do, you’ll get access to ZenHub’s complete feature-set immediately.

What can ZenHub do?

ZenHub adds a series of enhancements directly inside the GitHub UI:

Real-time, customizable task boards for GitHub issues;
Multi-Repository burndown charts, estimates, and velocity tracking based on GitHub Milestones;
Personal to-do lists and task prioritization;
Time-saving shortcuts – like a quick repo switcher, a “Move issue” button, and much more.

Add ZenHub to GitHub

Still curious? See more ZenHub features or read user reviews. This issue was written by your friendly ZenHub bot, posted by request from @jhagberg.

API Endpoint for inbox creation

Guarded by AAI

US07 - Store file submission in vault

As an EGAl admin,
I want to store files in a secure way,
so that I can guarantee privacy

A Vault-like area must exist at each Local EGA

Keep track of CSC server hardware and software installation process

Test ingestion on big DemoData

400 Gb

Figure out if the EGA in Spain is run on several machines

Goal: get a feeling whether the microservice architecture is necessary.

If EGA in Spain does not run on several machines and all of it in on one only, then the microservice architecture is not fully leveraged and maybe even an overkill.

It might be useful to structure the code in replaceable or duplicable components, but if services are not scheduled on several machines, we then might implement it all with a simple thread pool.

If the argument is "We might do it in the future", then it might be the wrong argument for the current design, because we do not know the workload for the services yet. Instead, we could implement it in whatever we think will work, and adjust accordingly (or even re-implement, but then we know why) when the time comes.

Demo the ingestion progress so far

I've shown the code running to Anders already.

I could create a small video (of my screen) showing the different parts and how they are connected.

Not sure it is super interesting, cuz it'll quickly become obsolete 😝

Figure out the API endpoints

Facing out or internal. Both.

Create list of things needed from Spain

Draw a list of things we would like from Spain to move further

Setup host machines to install and test the inbox and the worker components

Investigate what is "Vault"

US13 - QC uploaded files

As an EGAl admin,
I want to QC submitted files,
so that I can be sure that files are correct in the Archive

At iteration 1, just file md5 checking would be provided

US06 - Submission AAI

As an EGAl admin,
I want to allow particular users to submit data,
so that I can control who is submitting

In the first iteration, control of user accounts accessing the upload boxes (i.e. FTP server) must be shared with the EGAl domain, being Local, EGAc or both.

Create one unique config file for the ingestion microservice

Instead of one per Spring-component...

E1 Testsuite

Create a testsuite of some sort for the different parts?

We could have one for ingestion, one for re-encryption, etc...

US02 - Metadata submission

As a data owner,
I want to submit the metadata describing files,
so that they are correctly annotated

Initially, the metadata will follow the current EGA standard (SRA, currently). This metadata is public. Metadata would be submitted to EGAc. Further improvements would be inherited from other tasks in WP9 and follow WP5 recommendations.

US01 - Upload files

As data owner,
I want to upload raw data files,
so that they are kept safe

Local EGA Node would provide a file transfer service (FTP, Aspera, Globus…) that will be managed outside the domain boundaries of the Local EGA solution, although linked to it.

Run QA on ingested files (decrypt-checksum-encrypt)

Create demo microservice encrypting file and storing on local disk

Storage + DB

Submission ID + the rest.

Should have been an Epic ---- US06 - Submission AAI

----- Should have been an Epic -----
Gör om, gör rätt

As an EGAl admin,
I want to allow particular users to submit data,
so that I can control who is submitting

In the first iteration, control of user accounts accessing the upload boxes (i.e. FTP server) must be shared with the EGAl domain, being Local, EGAc or both.