Coder Social home page Coder Social logo

chemotion-converter-app's Introduction

chemotion-converter-app

Development setup

First, ensure that basic Python development prerequisites are installed on the system. For Debian or Ubuntu this can be done using:

apt-get install build-essential python3-dev python3-pip python3-venv

After cloning the repository, we recommend to create a virtual environment for the application:

python3 -m venv env
source env/bin/activate  # this needs to be done in each new terminal session

Then, the application and its dependencies can be installed:

pip install -e .                     # installs the package in editable mode
pip install -r requirements/dev.txt  # only needed for the development setup

The application is configured using environment variables, which can be read from a .env file. The file .env.dev can be used as template. At least FLASK_APP=converter_app.app needs to be set.

The Flask development server can be started now:

flask run

Production setup

For the production deployment, we recommend to create a dedicated user:

groupadd chemotion -g 2002
useradd chemotion -u 2002 -g 2002 -c Chemotion -m -d /srv/chemotion -s /bin/bash

Again we create a virtual environment, but now we install the application directly from GitHub:

python3 -m venv env
source env/bin/activate
pip install git+https://github.com/ComPlat/converter-app

In production, HTTP basic auth is used to seperate clients. First, create a htpasswd file using the sha1 hashing algorithm:

htpasswd -s -c htpasswd client  # create a new file with the client_id "client"
htpasswd -s htpasswd client     # update an existing file for the client_id "client"

The path to this file needs to be provided in .env as HTPASSWD_PATH.

The development server is not suited for a production deployment. Instead we recomend to use a reverse-proxy setup using gunicorn and NGINX.

Create the .env file in /srv/chemotion/.env, but use .env.prod as template, since it contains the variables we need to set for gunicorn.

In order to create the needed log and run directories create a tmpfiles.d config in /etc/tmpfiles.d/chemotion-converter.conf. A sample file can be found in etc/tmpfiles.d/chemotion-converter.conf. Create the directories using:

systemd-tmpfiles --create

Next, create a systemd service file in /etc/systemd/system/chemotion-converter.service. Again, a sample file can be found in etc/systemd/system/chemotion-converter.service. Reload systemd and start (and enable) the service:

systemctl daemon-reload
systemctl start chemotion-converter
systemctl enable chemotion-converter

If the service won't start, journalctl -xf might indicate what is wrong.

The guincorn server listens on the port given in the env file (default: 9000) on localhost. Using the following code in your NGINX configuration, it can be reversed proxied to the /api/v1 route:

    location /api/v1 {
        proxy_pass         http://127.0.0.1:9000/;
        proxy_redirect     off;

        proxy_set_header   Host                 $host;
        proxy_set_header   X-Real-IP            $remote_addr;
        proxy_set_header   X-Forwarded-For      $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto    $scheme;
    }

    client_max_body_size 64M;  # set this to the maximum file size allowed for upload

Acknowledgments

This project has been funded by the DFG.

DFG Logo

Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under the National Research Data Infrastructure – NFDI4Chem – Projektnummer 441958208 since 2020.

chemotion-converter-app's People

Contributors

jochenklar avatar magdan avatar phuang26 avatar pitrem avatar ptrxyz avatar starmanmartin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

chemotion-converter-app's Issues

Read Cfx files

Stakeholder can request a slimmed down version of the CIF files. These files are called .cfx_LANA

Extension: The GCD reader must be extended by further features

  • In addition to the Compound Results the Peak Tables have to be read.
  • The data tables have to be extended:
    • Status Trace (Column Oven Temperature)
    • Status Trace (Injection Unit Temperature)
    • Status Trace (Carrier Gas Pressure)
    • Status Trace (Carrier Gas Flow)
    • Status Trace (Column Flow)
    • Status Trace (Linear Velocity)

pssession reader upgrade

Allow the PSSession reader to accept both versions of PSTrace, specifically versions 5.3 and 5.9.

PD Reader

  • checking .dat files not readable
  • creating uxd Read
  • thinking about Bruker .raw dll Reader

PD.zip

add tensiometry

new method for "tensiometry"
a new type for Y-Axis labeling = Newton

Fix

Unknown error:

  • Excel file will not be converted:
    AttributeError: 'tuple' object has no attribute 'rstrip'
  • .cif should be higher priority than .cfx_lana

Json Reader

    Parse a json File. For all metadata information it uses the whole json path key of the source path as one key with "." separation.
    All arrays of number will be columns of tables. The system will check for all columns of same length
    and combine them to table.

Example:

From :

            {
                "a": "VAL A",
                "b": {
                    "a": "VAL B.A"
                },
                "data": {
                 "X": [1,2,3,4],
                 "Y": [0.1,0.2,0.3,0.4],
                 "X1": [1,2,3],
                 "Y2": [0.1,0.2,0.3]
                }
            }

Expected result:

metadata:

              #.a = Val A
              #.b.a = VAL B.A

data:

              Table 1 => 
                  #.data.X - [1,2,3,4],
                  #.data.Y - [0.1,0.2,0.3,0.4]
              
              Table 2 => 
                  #.data.X1 - [1,2,3],
                  #.data.Y1 - [0.1,0.2,0.3]

Differentation for Error Messages

We need some better kinds of replays from converter to distinguish between ERRORS ->
cannot convert because of

  1. no Reader
  2. no profile
  3. multiple profiles
  4. file not readable (due to currently binary, some pics or encodings)
  5. others

Pylint, Unit Tests and requirement updates

In order to ensure further development of the tool in the future, it is urgently necessary to (I) implement a meaningful maintenance environment (testing and linting) and (II) regularly update the requirements

new data types

new data type:

  • DLS intensity
  • DLS ACF
  • Emissions

new X-Unit:

  • Lag time (microseconds)

new Y-Unit:

  • relative intensity (%)
  • ACF (a.u.)

Improvement: reload auth file upon change

Would it be possible to reload the auth file as soon as it changes // regularly? It looks to me that as of now the service needs to be reloaded when a user is added/changed.

This would help a lot when it comes to running the service interruption free when dynamically spawning ELN instances that share the same converter app service.

Wrong ##DATA TYPE for some of the new converter profiles

Hi Team,
Hi @phuang26

after checking the conversion of the new data formats on https://complat-eln.ioc.kit.edu/ , I found some problems / bugs / issues.

  • SEC and AIF is working but both are using the wrong ##DATA TYPE=CYCLIC VOLTAMMETRY. Maybe they are using and old profile version or the right entry is missing.
  • CDS (some kind of csv file) won't be converted at all although I could see the profile in the list.

Thanks for the help and best wishes.

Read Zipped CIF

In the productive process Cif files are often zipped and sent to stakeholders. It was requested that these mails are also sent directly to the ELN.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.