sciebo-rds / connexion-plus

Connexion with benefits for microservices

Sciebo RDS

Sciebo RDS (Sciebo Research Data Services) allows researchers to enrich their research data with metadata and to export it directly from Enterprise Sync and Share (EFSS) Systems like Owncloud to Data Repositories like Zenodo and OSF. It acts as a interoperability layer, allowing them to assign a directory on a Sync and Share System to a research project, annotating the contained data to match the format required by the desired Data Repository and publishing the Research Data directly from the Cloud.

No need to download all the data to a desktop computer to cumbersomely upload it to the repository interface again.

Find more information on the official website at www.research-data-services.org.

Screenshots

Configuring a project	Adding metadata
Adding metadata	Adding metadata
Publishing	Connecting repositories

Deployment

Requirements

To deploy Sciebo RDS you will need:

1. A Kubernetes cluster
2. An OwnCloud instance
3. Two different domains

Getting Started

Please refer to our Getting Started Guide to learn how to deploy Sciebo RDS.

Development environment

To deploy a local environment on top of Minikube, including RDS and NextCloud and / or OwnCloud, please refer to this guide.

Monorepo Structure

We use a monorepo to make it easier to track related changes. This means that you will find all relevant files in this Github repository.

The following table will give an impression of the individual parts.

Folder	Description
/charts	Helm charts
/docs	Website with documentation
/getting-started	Files for easy deployment
/RDS	The RDS code
/RDS/layer0_ingress/web/server	The Python Backend
/RDS/layer0_ingress/web/client/packages/codebase	The Vue.js Frontend
/RDS/layer3_central_services	Microservices ResearchManager, TokenStorage
/	Metafiles and configuration

The monorepo only includes those parts of Sciebo RDS that are officially maintained by the University of Münster.

Additional Repositories

There is also an additional repository that functions as a community hub for development of 3rd party connectors, such as Dataverse. These connectors are community managed – if you are planning on developing a connector and sharing it as Open Source, feel free to use this repository.
A third repository is used for the ongoing development of the Nextcloud Plugin.

Contributing and Documentation

Contributions are always welcome! You can find technical documentation and guides on how to extend Sciebo RDS to your needs and connect additional repositories on our website.

We follow the Gitlab-flow. Our default branch is called develop and can be changed exclusively through pull requests via Github.
When a certain amount of features are done, they are merged into release. From there we will set a tag and publish the changes in helm and docker repositories.

Please adhere to our code of conduct.

Feedback and Support

Please use the Github Issues and Github Discussions for feedback and technical questions. We also have a Gitter chat. :)

If you are using Sciebo RDS as a researcher and need user level support, please refer to the administrators of your Sciebo RDS instance. You can find their email adresse in the Help tab of your Sciebo RDS instance.

Acknowledgements

Sciebo RDS relies on describo-online for metadata management.

Authors

Sciebo RDS is a project developed by the University of Münster and funded by the DFG.

You can reach the development team at sciebo.rds <at> uni-muenster.de or on Gitter.

connexion-plus's Issues

Library dependencies not working with python > 3.10

The issue

I am working on developing ports. I am running python 3.10. I come across the following issue when starting a port server:

Traceback (most recent call last):
File "/home/dave/Projects/local-rds/RDS/layer1_adapters_and_ports/port_figshare/src/server.py", line 4, in
from init import app
File "/home/dave/Projects/local-rds/RDS/layer1_adapters_and_ports/port_figshare/src/init.py", line 38, in
app = bootstrap("PortFigshare", all=True)
File "/home/dave/Projects/local-rds/RDS/layer1_adapters_and_ports/port_figshare/src/init.py", line 24, in bootstrap
app = App(name, *args, **kwargs)
File "/home/dave/.local/share/virtualenvs/port_figshare-n1Gvbe7Y/lib/python3.10/site-packages/connexion_plus/Application.py", line 182, in init
from jaeger_client import Config as jConfig
File "/home/dave/.local/share/virtualenvs/port_figshare-n1Gvbe7Y/lib/python3.10/site-packages/jaeger_client/init.py", line 19, in
from .config import Config # noqa
File "/home/dave/.local/share/virtualenvs/port_figshare-n1Gvbe7Y/lib/python3.10/site-packages/jaeger_client/config.py", line 25, in
from .local_agent_net import LocalAgentSender
File "/home/dave/.local/share/virtualenvs/port_figshare-n1Gvbe7Y/lib/python3.10/site-packages/jaeger_client/local_agent_net.py", line 18, in
import tornado.httpclient
File "/home/dave/.local/share/virtualenvs/port_figshare-n1Gvbe7Y/lib/python3.10/site-packages/tornado/httpclient.py", line 50, in
from tornado import gen, httputil, stack_context
File "/home/dave/.local/share/virtualenvs/port_figshare-n1Gvbe7Y/lib/python3.10/site-packages/tornado/httputil.py", line 107, in
class HTTPHeaders(collections.MutableMapping):
AttributeError: module 'collections' has no attribute 'MutableMapping'

My analysis

Apparently MutableMapping has been depricated in the collections lib in python 3.10 or higher:
https://stackoverflow.com/questions/70943244/attributeerror-module-collections-has-no-attribute-mutablemapping

The tornado lib is using this. I am running version 5.1.1.

I tried upgrading it to the latest version 6.2, but this is not possible as another package is depending on tornado version <6.
This is the opentracing-instrumentation 3.3.1 lib.

And this package is a dependency for connexion-plus.

Workaround

A workaround I will be downgrading to python version < 3.10.

Hint for fix

The opentracing-instrumentation lib is deprecated. See:
uber-common/opentracing-python-instrumentation#118
The suggestion is to switch to OpenTelemetry?!

Add opentracing to all functions and classes used within service

Currently, opentracing will only be used for requests between services, but it would be very useful to know, what happens inside of a services while fulfilling the request. So it would super cool, if there would be an option to achieve this in connexion-plus.

Currently, we use python-flask for tracing. If we implement a decorator to get something equal to this:

https://gist.github.com/itsderek23/68d4336f582e3a6c7d58798a6462f191/21a888fee2958ed10ed198c6f360e8c4e91501ee#file-opentracing_flask-py-L17-L26

Then, we can use something like that decorate-all-methods to provide a method, which decorates all classes and functions in a specific file.

Something like this could be done, too

connexion-plus/connexion_plus/Application.py

Line 215 in ce3be19

install_all_patches()

Usage in classes:

from connexion_plus import add_tracing_class

@add_tracing_class
class myCustomClass():
  ...

Usage in functions:

from connexion_plus import add_tracing_fn

@add_tracing_fn
def myFunction():
  pass

from connexion_plus import add_tracing_patches

add_tracing_patches("lib") # patches all classes and functions like before automatically within the given folder