engie-group / shaystack

Simplifying Implementation of Haystack

Shift-4-Haystack

The full documentation is here

About Haystack, and who is it for

Haystack project is an open source initiative to standardize semantic data models for Internet Of Things. It enables interoperability between any IoT data producer and consumer, mainly in the Smart Building area.

Haystack core data model is the Grid, it can be serialized in many formats, mainly Zinc, Trio Json and Csv

About this project

This project implements client side haystack code. Useful to parse or dump Haystack files (Zinc, Json, Trio and Csv).

On the server side, it also implements Haystack HTTP API, useful to serve Haystack data you host.

• We implemented different serving options See API Server
  • Each offering two API endpoints:
    • Classical Haystack HTTP API
    • GraphQL API
• We introduced and implemented the Provider concept, which handles various options in terms on haystack data location:
  • Local or remote file system (including AWS S3)
  • Local or remote relational database
  • Local or remote Mongo database
  • Can be extends with optional AWS Time Stream use for Time Series
  • Other custom data location can be handled by extending shaystack.providers.HaystackInterface

Table of contents

• Quick link
• About Haystack, and who is it for
• About this project
• History
• Client side: Haystack client python module
  • Inspect the data with code
  • Python API
  • Data science
• Features
• Server Side: Haystack API Server
  • API Server deployment
    • Installing
    • Choosing and configuring your provider
    • Starting the server
• Using Haystack API with UI
• Using GraphQL API
• Specification extension
• Haystack filter
• Add Haystack API to an existing project
• Using AWS
• Docker
• Using with Excel or PowerBI
• Optional part
• Data types
  • Null, Boolean, Date, Time, Date/Time and strings.
  • Numbers
  • Marker and Remove
  • Bin and XBin
  • Uri
  • Ref
  • Coordinate
  • Collection List, Dict or Grid
• Contributing
• Resources
• License
• TODO

Try it with colab?

Try it with AWS Lambda?

and the documentation of API is here

shaystack's Issues

make typing: ninja: build stopped: subcommand failed.

Describe the bug
Python typing cheking failed when running the command make typing the following error was generated:

/home/ubuntu/shaystack/.pytype/pyi/shaystack/pintutil.pyi 
[6/80] check shaystack.sortabledict
ninja: build stopped: subcommand failed.

To Reproduce
Steps to reproduce the behavior:

1. git clone -b develop https://github.com/engie-group/shaystack.git
2. cd shaystack/
3. make configure
4. conda activate shaystack
5. make typing

Used environment

• conda 4.13.0
• ubuntu 22.04 and same error on 18.04
• GNU Make 4.3

Error traceback

Check typing...
Computing dependencies
Analyzing 60 sources with 0 local dependencies
ninja: Entering directory `.pytype'
[3/80] check shaystack.pintutil
FAILED: /home/ubuntu/shaystack/.pytype/pyi/shaystack/pintutil.pyi 
/home/ubuntu/anaconda3/envs/shaystack/bin/python3.8 -m pytype.single --disable pyi-error --imports_info /home/ubuntu/shaystack/.pytype/imports/shaystack.pintutil.imports --module-name shaystack.pintutil -V 3.8 -o /home/ubuntu/shaystack/.pytype/pyi/shaystack/pintutil.pyi --analyze-annotated --nofail --quick /home/ubuntu/shaystack/shaystack/pintutil.py
File "/home/ubuntu/shaystack/shaystack/pintutil.py", line 24, in _load_haystack_alias: Function importlib.resources.open_text was called with the wrong arguments [wrong-arg-types]
         Expected: (package: Union[module, str], ...)
  Actually passed: (package: None, ...)

For more details, see https://google.github.io/pytype/errors.html#wrong-arg-types
[6/80] check shaystack.sortabledict
ninja: build stopped: subcommand failed.
Leaving directory '.pytype'
make: *** [Makefile:767: .make-typing] Error 1

shaystack is not compatible with Python 3.10

After trying to start shystack under Python 3.10, it reports only a message that
"To start shift-4-haystack, use 'pip install "shaystack[flask]"' or 'pip install "shaystack[flask,graphql]"' and set 'HAYSTACK_PROVIDER' variable"
despite the variable is defined (and HAYSTACK_DB also is defined).
After modification of site-packages/app/main.py to include text of the exception (print(str(ex)), I have encountered several errors from flask and other modules and finally from Python base module:
"cannot import MutableSequence from collections"
After modification of shystack/grid.py to import MutableSequence from collections.abc instead of just collections, I got uncaught exception (python simply crashes after few seconds).

Steps to reproduce the behavior:

1. Install Python 3.10.4 (x64), shaystack, flask==2.0.3 (current one has issues with shaystack), graphql
2. Define HAYSTACK_PROVIDER and HAYSTACK_DB
3. start shystack
4. See error in system log (event viewer in Windows)

Expected behavior
Be able to start shaystack and use its web interface.

Desktop :

• OS: Windows 10 x64

Please help how to launch shaystack.

Be compatible with hayson

Is your feature request related to a problem? Please describe.
The new hayson format is better like Haystack v3 JSon and Yaml

Describe the solution you'd like
Rewrite all the providers with hayson

make configure is not working

Describe the bug
When trying to setup the environment using make configure the following error was generated:

Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): / - failed

Reason behind the error:
Makefile line 280: installing compilers from conda-forge channel
conda create --name "shaystacks" conda-forge compilers python=3.8 -y
removing the compilers allows the make configure command to pass and create the shaystack environment

To Reproduce
Steps to reproduce the behavior:

1. git clone -b develop https://github.com/engie-group/shaystack.git
2. cd shaystack/
3. make configure

Used environment

• conda 4.13.0
• ubuntu 22.04 and same error on 18.04
• GNU Make 4.3

make validate error: ModuleNotFoundError

Describe the bug
When running make validate wich will automativcally run make-test wich will raise the follonwing error when testing the provider db : ModuleNotFoundError: No module named 'supersqlite.third_party.internal.apsw'

To Reproduce
Steps to reproduce the behavior:

1. git clone -b develop https://github.com/engie-group/shaystack.git
2. cd shaystack/
3. make configure
4. conda activate shaystack
5. make validate

Used environment

• conda 4.13.0
• ubuntu 22.04 and same error on 18.04
• GNU Make 4.3

Error traceback

ERROR: tests.test_provider_db.test_update_grid_in_db(<function _test_update_grid_in_db at 0x7f7cd1018820>, 'shaystack.providers.sql', 'sqlite3:///test.db#haystack', 0)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/shaystack/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/ubuntu/shaystack/tests/test_provider_db.py", line 43, in _wrapper
    function(provider, db)
  File "/home/ubuntu/shaystack/tests/test_provider_db.py", line 104, in _test_update_grid_in_db
    with cast(DBHaystackInterface, get_provider(provider_name,
  File "/home/ubuntu/shaystack/shaystack/providers/haystack_interface.py", line 542, in get_provider
    _providers[class_str] = FullInterface(envs)
  File "/home/ubuntu/shaystack/shaystack/providers/haystack_interface.py", line 459, in __init__
    provider_class.__init__(self, envs)
  File "/home/ubuntu/shaystack/shaystack/providers/sql.py", line 133, in __init__
    _import_db_driver(self._parsed_db,
  File "/home/ubuntu/shaystack/shaystack/providers/sql.py", line 84, in _import_db_driver
    mod = importlib.import_module(splitted[0])
  File "/home/ubuntu/anaconda3/envs/shaystack/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/ubuntu/anaconda3/envs/shaystack/lib/python3.8/site-packages/supersqlite/__init__.py", line 47, in <module>
    import supersqlite.third_party.internal.apsw as apsw
ModuleNotFoundError: No module named 'supersqlite.third_party.internal.apsw'

Filter don't work with ==0

Describe the bug
when i use g.filter('floorNum==0') I expect some result but i get 0 row

To Reproduce
Use the code below to generate the grids
`g = shaystack.parse(
"""id:@muz
dis:Musée de la Nouvelle Calédonie
area: 4439m²
site
geoCity: Nouméa
geoPostalCode: 98800
geoCountry: NC
tz: Noumea
primaryFunction: Museum
yearBuilt: 2022

id: @muz.A.rdc
dis: Rez de chaussée bâtiment A
space
floor
floorNum: 0
siteRef: @muz

id: @muz.B.rdc
dis: Rez de chaussée bâtiment B
space
floor
floorNum: 1
siteRef: @muz"""
, mode=shaystack.MODE_TRIO)`
g.filter('floorNum==0')

Expected behavior
When i use g.filter('floorNum==1') I got one Row as i expect
but when i use g.filter('floorNum==0' I got 0 row or i expect 1 row

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: windosw
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

pointWrite and hisWrite implementation is missing in db providers (only dumb ping)

Hello,

This project allows to load static ontology together with startup data (curVal and history) from files (e.g. .zinc format) into SQL database (I started my journey with sqlite3). So I can start shaystack (REST) API server to query curVal's and hisRead that data.

But now I would also like to use p-h pointWrite and hisWrite methods which are defined in providers/haystack_interface.py only as abstract methods, but are not implemented anywhere else (and NotImplemented exceptions are thrown).

I'd like to start shaystack REST API server to hold ontology and data, then load it with some startup ontology hierarchy and then using shaystack clients (http from external code and from my python code) be able to query the ontology, update + insert data and eventually query the current data + history. That's why I need pointWrite and hisWrite. But also I don't see how shaystack' client API can connect to shaystack server (there's no connect method, just read from static files).

Regards!

Can't parse a trio file when a string is begining with a number

Describe the bug
in trio mode when a dis is begining with a number the parsing fail

To Reproduce
Steps to reproduce the behavior:
trio = """---
id: @muz.A.r1
dis: 1er etage bâtiment A
space: M
floor: M
floorNum: 2.0
siteRef: @muz"""
g=shaystack.parse(trio, mode= shaystack.MODE_TRIO)

Expected behavior
I expect a string parsing

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.