openenergyplatform / oem2orm Goto Github PK

When cloning the repo and creating and environment for development purposes from requirements.txt, I stumbled upon requirement problems for pandas and sqlalchemy while testing code.

@jh-RLI is it fair to fix both package versions to pandas==1.2.4, sqlalchemy==1.3.16 as noted here or does anything speaks against it? If only the latter is possible then a dev note would be nice.

Adds new bug and feature templates.

When using the package, a parsing error occurred when opening the metadata file in oep_oedialect_oem2orm.py

Changing the argument in line 169 from "r" to "rb" fixed the issue.

with open(metadata_file, "rb") as metadata_json:
        metadata = json.loads(metadata_json.read())
    tables_raw = jmespath.search("resources", metadata)

Requested by @jnnr:

Can we use the same function, but for single tables? This would allow us to not leave the loop here.

What I mean: Do not do this:

When updating metadata on the OEP via oep_oedialect_oem2orm.api_updateMdOnTable status_code of response is checked whether it is 200. If an error occurs (aka status_code != "200") the error gets logged, but no error is thrown.
Thus, a user cannot know whether updated worked or not.
I would suggest throwing a `MetadataError``with information from response if upload fails.

It would be nice if metadata is uploaded next to table in case of uploading a single table.
In case of multiple table upload this is not useful as each table must have its own metadata.

@henhuy, as you noticed, there is always (I assume since you merged master back to develop) a diff between master and develop after I do a release - I follow the Vincent Diressen workflow, but I think I missed something as sometimes the branches are in sync, but most of the time they differ by 2-3 commits, but actually contain the same content. Maybe this is because github creates additional merge commits? Or do i have to use another merge strategy.

Maybe you have an idea how to make it cleaner :)

The documentation is still not complete.

• Include requirements on OEM fields
• Rework Usage

Right now, OEM data to create tables is read from file.
Should be refactored to allow creation from OEM dict as well.

Uploading tables with oem2orm, e.g. using the script from the tutorial section, is not user-friendly due a lack of helpful error messages.

• Inform the user which table is processed. Currently, when the table creation fails and throws a bad/unspecific error message, it is not clear which table is the reason (in case of uploading a bulk of tables)
• Improve the error messages when violating the postgresql naming conventions for column headers and resources-name. Relates to #16 (comment)

the syntax : pip install -i https://test.pypi.org/simple/ oem2orm
doesn't work properly
error message "No module named 'jmespath'"

Some error messages are not shown to the user

1. ce in

   oem2orm/oem2orm/oep_oedialect_oem2orm.py

   Lines 121 to 124 in bbb6511

   	except oedialect.engine.ConnectionException as ce:
   	error_msg = f'Error when uploading table "{table.name}".'
   	logging.error(error_msg)
   	raise DatabaseError(error_msg) from ce

2. null in type field results in internal server error,

Reproduce with:
Metadata meta_dat.json
Error msg:

    if item.split(" ")[-1] == "array":
       ^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'split'

I requested a feature to convert the oemetadata JSON to ORM.
This makes the publication of data easy.

My collegue @henhuy developed this package md2orm and @jh-RLI pushed it to this repo.

Planned toolchain for Data Upload:

1. Create metadata from data
2. Create ORM from metadata using MD2ORM
3. Create ERM usind ERAlchemy
4. Upload data to OEP using oedialect
5. Upload metadata and add tags

When uploading tables to the OEP using the script the column headers are not checked for capital letters and the table is nonetheless added to the DB.

I'm not sure whether this is a bug or problematic.

Example:

https://openenergy-platform.org/dataedit/view/model_draft/wind_turbine_domestic_lod_geoss_tp_oeo

Using this tutorial and this metadata yields the table below on the OEP

Upload with example_df.to_sql(table_name, connection, schema=schema, if_exists='append', index=False) yields the error:

Out of range float values are not JSON compliant
Writing to umas_potentials failed!

and no data will be uploaded.

However, when I use https://modex.rl-institut.de/create_table/ to create and upload the table, this yields

and upload works fine.

The columns id and id_scal both contain integer values.

Final table: https://openenergy-platform.org/dataedit/view/model_draft/umas_potentials

Two errors in setup.py:

1. OEDialect is set to v1.1 but must be 0.1.1 instead
2. SQLAlchemy v1.3.14 is not working with latest OEDialect (>=1.3.16 there), thus this must be upgraded to v1.3.16 (latest version is not working - don't know which one latest working version)

• Fix the old readme displayed on PyPi

• done