astarte-platform / astarte Goto Github PK

View Code? Open in Web Editor NEW

229.0 229.0 40.0 8.88 MB

Core Astarte Repository

Home Page: https://docs.astarte-platform.org/

License: Apache License 2.0

Shell 0.02% Dockerfile 0.64% Elixir 99.29% Euphoria 0.01% Nix 0.05%

hacktoberfest iot iot-framework iot-middleware iot-platform

astarte's People

Contributors

Stargazers

Watchers

Forkers

bettio drf rbino ebostijancic pavinati spencerx ubi-mirrors hazer joustava malli1983 alessandrodon spidey20202022 matt-mazzucato shubham11107 zizango123 franksongfeng davidebriani annopaolo colman-hartinger suryatmodulus szakhlypa harlem88 nanjarapalli ajunlonglive arahmarchak noaccos lucaato samubd agirolami87 rifasofic hibe7 sorru94 osmanhadzic bryanchance eddbbt jessequinn shinnokdisengir dyslabpro rgallor atsetilam

astarte's Issues

Downsample not working with format=disjoint_tables

example.com/appengine/v1/test/devices/test_device_id/interfaces/com.example.ObjectAggregated/param/0/value?format=disjoint_tables&keep_milliseconds=true&since=2018-09-12T18:53:50.302Z&to=2018-10-14T20:32:51.490Z&downsample_to=769

Returns an array with 10000 elements instead of 769.

Add /metrics to Astarte Pairing API

Add prometheus /metrics endpoint.

Provide an Open Source IoT Platform unironically

As of the day this issue was opened, there's plenty of choices when it comes to developing IoT projects ironically, but there's a lack of concrete resources when one has to break through the layers of irony and face production.

We have to make sure developers can rely on a non-ironic solution.

[meta] Report useful information in metadata

Following issues need to be closed:

Add /metrics to Astarte Realm Management

Add prometheus /metrics endpoint.

Add support to groups to Astarte Trigger Engine triggers.

Data Updater Plant crashes with `value_change` triggers on datastream interfaces

Stacktrace (generated with local Astarte deployment with 0.10-snapshot):

astarte-data-updater-plant_1    | 11:48:08.044 [error] GenServer {Registry.DataUpdater, {"test", <<201, 218, 129, 148, 91, 6, 67, 239, 177, 103, 239, 222, 125,
 12, 110, 47>>}} terminating
astarte-data-updater-plant_1    | ** (FunctionClauseError) no function clause matching in Astarte.DataAccess.Data.fetch_property/5
astarte-data-updater-plant_1    |     (astarte_data_access) lib/astarte_data_access/data.ex:37: Astarte.DataAccess.Data.fetch_property({#PID<0.1721.0>, #Refere
nce<0.3067377248.121896968.132174>}, <<201, 218, 129, 148, 91, 6, 67, 239, 177, 103, 239, 222, 125, 12, 110, 47>>, %Astarte.Core.InterfaceDescriptor{aggregatio
n: :individual, automaton: {%{{0, "realValue"} => 1}, %{1 => <<74, 19, 138, 223, 180, 26, 70, 196, 99, 119, 165, 221, 187, 187, 72, 163>>}}, interface_id: <<2,
 224, 206, 164, 78, 230, 132, 21, 68, 58, 174, 138, 98, 46, 5, 104>>, major_version: 0, minor_version: 1, name: "org.astarteplatform.Values", ownership: :devic
e, storage: "individual_datastreams", storage_type: :multi_interface_individual_datastream_dbtable, type: :datastream}, %Astarte.Core.Mapping{allow_unset: fals
e, description: nil, doc: nil, endpoint: "/realValue", endpoint_id: <<74, 19, 138, 223, 180, 26, 70, 196, 99, 119, 165, 221, 187, 187, 72, 163>>, expiry: 0, ex
plicit_timestamp: true, interface_id: <<2, 224, 206, 164, 78, 230, 132, 21, 68, 58, 174, 138, 98, 46, 5, 104>>, path: nil, reliability: :unreliable, retention:
 :discard, type: nil, value_type: :double}, "/realValue")
astarte-data-updater-plant_1    |     (astarte_data_updater_plant) lib/astarte_data_updater_plant/data_updater/impl.ex:346: Astarte.DataUpdaterPlant.DataUpdate
r.Impl.handle_data/6
astarte-data-updater-plant_1    |     (astarte_data_updater_plant) lib/astarte_data_updater_plant/data_updater/server.ex:53: Astarte.DataUpdaterPlant.DataUpdat
er.Server.handle_cast/2
astarte-data-updater-plant_1    |     (stdlib) gen_server.erl:616: :gen_server.try_dispatch/4
astarte-data-updater-plant_1    |     (stdlib) gen_server.erl:686: :gen_server.handle_msg/6
astarte-data-updater-plant_1    |     (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
astarte-data-updater-plant_1    | Last message: {:"$gen_cast", {:handle_data, "org.astarteplatform.Values", "/realValue", <<12, 0, 0, 0, 16, 118, 0, 8, 0, 0, 0
, 0>>, "test-ydqBlFsGQ--xZ-_e-37E77026732BBA-724F1B33", 15735592586521530}}
astarte-data-updater-plant_1    | State: %Astarte.DataUpdaterPlant.DataUpdater.State{connected: true, data_triggers: %{}, datastream_maximum_storage_retention:
 nil, device_id: <<201, 218, 129, 148, 91, 6, 67, 239, 177, 103, 239, 222, 125, 12, 110, 47>>, device_triggers: %{}, interface_ids_to_name: %{}, interfaces: %{
}, interfaces_by_expiry: [], introspection: %{"org.astarteplatform.Values" => 0}, introspection_triggers: %{}, last_device_triggers_refresh: 0, last_seen_messa
ge: 0, mappings: %{}, message_tracker: #PID<0.1707.0>, paths_cache: {32, %{}}, realm: "test", total_received_bytes: 9089, total_received_msgs: 93, volatile_tri
ggers: []}
astarte-data-updater-plant_1    |
astarte-data-updater-plant_1    | =CRASH REPORT==== 12-Nov-2019::11:48:08 ===
astarte-data-updater-plant_1    |   crasher:
astarte-data-updater-plant_1    |     initial call: Elixir.Astarte.DataUpdaterPlant.DataUpdater.Server:init/1
astarte-data-updater-plant_1    |     pid: <0.1858.0>
astarte-data-updater-plant_1    |     registered_name: []
astarte-data-updater-plant_1    |     exception error: no function clause matching
astarte-data-updater-plant_1    |                      'Elixir.Astarte.DataAccess.Data':fetch_property({<0.1721.0>,
astarte-data-updater-plant_1    |                                                                       #Ref<0.3067377248.121896968.132174>},
astarte-data-updater-plant_1    |                                                                      <<201,218,
astarte-data-updater-plant_1    |                                                                        129,148,
astarte-data-updater-plant_1    |                                                                        91,6,67,
astarte-data-updater-plant_1    |                                                                        239,177,
astarte-data-updater-plant_1    |                                                                        103,239,
astarte-data-updater-plant_1    |                                                                        222,125,
astarte-data-updater-plant_1    |                                                                        12,110,
astarte-data-updater-plant_1    |                                                                        47>>,
astarte-data-updater-plant_1    |                                                                      #{'__struct__' =>
astarte-data-updater-plant_1    |                                                                            'Elixir.Astarte.Core.InterfaceDescriptor',
astarte-data-updater-plant_1    |                                                                        aggregation =>
astarte-data-updater-plant_1    |                                                                            individual,
astarte-data-updater-plant_1    |                                                                        automaton =>
astarte-data-updater-plant_1    |                                                                            {#{{0,
astarte-data-updater-plant_1    |                                                                                <<"realValue">>} =>
astarte-data-updater-plant_1    |                                                                                   1},
astarte-data-updater-plant_1    |                                                                             #{1 =>
astarte-data-updater-plant_1    |                                                                                   <<74,19,
astarte-data-updater-plant_1    |                                                                                     138,223,
astarte-data-updater-plant_1    |                                                                                     180,26,
astarte-data-updater-plant_1    |                                                                                     70,196,
astarte-data-updater-plant_1    |                                                                                     99,119,
astarte-data-updater-plant_1    |                                                                                     165,221,
astarte-data-updater-plant_1    |                                                                                     187,187,
astarte-data-updater-plant_1    |                                                                                     72,163>>}},
astarte-data-updater-plant_1    |                                                                        interface_id =>
astarte-data-updater-plant_1    |                                                                            <<2,224,
astarte-data-updater-plant_1    |                                                                              206,164,
astarte-data-updater-plant_1    |                                                                              78,230,
astarte-data-updater-plant_1    |                                                                              132,21,
astarte-data-updater-plant_1    |                                                                              68,58,
astarte-data-updater-plant_1    |                                                                              174,138,
astarte-data-updater-plant_1    |                                                                              98,46,5,
astarte-data-updater-plant_1    |                                                                              104>>,
astarte-data-updater-plant_1    |                                                                        major_version =>
astarte-data-updater-plant_1    |                                                                            0,
astarte-data-updater-plant_1    |                                                                        minor_version =>
astarte-data-updater-plant_1    |                                                                            1,
astarte-data-updater-plant_1    |                                                                        name =>
astarte-data-updater-plant_1    |                                                                            <<"org.astarteplatform.Values">>,
astarte-data-updater-plant_1    |                                                                        ownership =>
astarte-data-updater-plant_1    |                                                                            device,
astarte-data-updater-plant_1    |                                                                        storage =>
astarte-data-updater-plant_1    |                                                                            <<"individual_datastreams">>,
astarte-data-updater-plant_1    |                                                                        storage_type =>
astarte-data-updater-plant_1    |                                                                            multi_interface_individual_datastream_dbtable,
astarte-data-updater-plant_1    |                                                                        type =>
astarte-data-updater-plant_1    |                                                                            datastream},
astarte-data-updater-plant_1    |                                                                      #{'__struct__' =>
astarte-data-updater-plant_1    |                                                                            'Elixir.Astarte.Core.Mapping',
astarte-data-updater-plant_1    |                                                                        allow_unset =>
astarte-data-updater-plant_1    |                                                                            false,
astarte-data-updater-plant_1    |                                                                        description =>
astarte-data-updater-plant_1    |                                                                            nil,
astarte-data-updater-plant_1    |                                                                        doc =>
astarte-data-updater-plant_1    |                                                                            nil,
astarte-data-updater-plant_1    |                                                                        endpoint =>
astarte-data-updater-plant_1    |                                                                            <<"/realValue">>,
astarte-data-updater-plant_1    |                                                                        endpoint_id =>
astarte-data-updater-plant_1    |                                                                            <<74,19,
astarte-data-updater-plant_1    |                                                                              138,223,
astarte-data-updater-plant_1    |                                                                              180,26,
astarte-data-updater-plant_1    |                                                                              70,196,
astarte-data-updater-plant_1    |                                                                              99,119,
astarte-data-updater-plant_1    |                                                                              165,221,
astarte-data-updater-plant_1    |                                                                              187,187,
astarte-data-updater-plant_1    |                                                                              72,163>>,
astarte-data-updater-plant_1    |                                                                        expiry =>
astarte-data-updater-plant_1    |                                                                            0,
astarte-data-updater-plant_1    |                                                                        explicit_timestamp =>
astarte-data-updater-plant_1    |                                                                            true,
astarte-data-updater-plant_1    |                                                                        interface_id =>
astarte-data-updater-plant_1    |                                                                            <<2,224,
astarte-data-updater-plant_1    |                                                                              206,164,
astarte-data-updater-plant_1    |                                                                              78,230,
astarte-data-updater-plant_1    |                                                                              132,21,
astarte-data-updater-plant_1    |                                                                              68,58,
astarte-data-updater-plant_1    |                                                                              174,138,
astarte-data-updater-plant_1    |                                                                              98,46,5,
astarte-data-updater-plant_1    |                                                                              104>>,
astarte-data-updater-plant_1    |                                                                        path =>
astarte-data-updater-plant_1    |                                                                            nil,
astarte-data-updater-plant_1    |                                                                        reliability =>
astarte-data-updater-plant_1    |                                                                            unreliable,
astarte-data-updater-plant_1    |                                                                        retention =>
astarte-data-updater-plant_1    |                                                                            discard,
astarte-data-updater-plant_1    |                                                                        type =>
astarte-data-updater-plant_1    |                                                                            nil,
astarte-data-updater-plant_1    |                                                                        value_type =>
astarte-data-updater-plant_1    |                                                                            double},
astarte-data-updater-plant_1    |                                                                      <<"/realValue">>) (lib/astarte_data_access/data.ex, line
 37)
astarte-data-updater-plant_1    |       in function  'Elixir.Astarte.DataUpdaterPlant.DataUpdater.Impl':handle_data/6 (lib/astarte_data_updater_plant/data_upda
ter/impl.ex, line 346)
astarte-data-updater-plant_1    |       in call from 'Elixir.Astarte.DataUpdaterPlant.DataUpdater.Server':handle_cast/2 (lib/astarte_data_updater_plant/data_up
dater/server.ex, line 53)
astarte-data-updater-plant_1    |       in call from gen_server:try_dispatch/4 (gen_server.erl, line 616)
astarte-data-updater-plant_1    |       in call from gen_server:handle_msg/6 (gen_server.erl, line 686)
astarte-data-updater-plant_1    |     ancestors: ['Elixir.Astarte.DataUpdaterPlant.AMQPDataConsumer',
astarte-data-updater-plant_1    |                   'Elixir.Astarte.DataUpdaterPlant.Supervisor',<0.1527.0>]
astarte-data-updater-plant_1    |     message_queue_len: 0
astarte-data-updater-plant_1    |     messages: []
astarte-data-updater-plant_1    |     links: [<0.1532.0>]
astarte-data-updater-plant_1    |     dictionary: []
astarte-data-updater-plant_1    |     trap_exit: false
astarte-data-updater-plant_1    |     status: running
astarte-data-updater-plant_1    |     heap_size: 2586
astarte-data-updater-plant_1    |     stack_size: 27
astarte-data-updater-plant_1    |     reductions: 6758
astarte-data-updater-plant_1    |   neighbours:

[meta] Fix and improve interfaces validation

Following issues need to be closed:

astarte-platform/astarte_core#2

Add /metrics to Astarte Pairing

Add prometheus /metrics endpoint.

Provide a set of ready to use interfaces

Astarte should provide a set of ready to install and user interfaces that can be used as an example and can be without further effort:

{
    "interface_name": "org.astarte-platform.standard-interfaces.GenericSensor",
    "version_major": 1,
    "version_minor": 0,
    "type": "datastream",
    "ownership": "device",
    "mappings": [
        {
            "endpoint": "/%{sensor_id}/value",
            "type": "double"
        }
    ]
}

    "interface_name": "org.astarte-platform.standard-interfaces.AvailableSensors",
    "version_major": 1,
    "version_minor": 0,
    "type": "property",
    "ownership": "device",
    "mappings": [
        {
            "endpoint": "/%{sensor_id}/name",
            "type": "string"
        },
        {
            "endpoint": "/%{sensor_id}/unit",
            "type": "string"
        }
    ]
}

Add /metrics to Astarte Data Updater Plant

Add prometheus /metrics endpoint.

Add metadata to device

Add a metadata collection to device.
Metadata will allow to store arbitrary string value associated to arbitrary keys.

Example:

{
    "id": "7RrWVRVAQcmChzSokqYwmQ"
    "connected": true,
    [...]
    "metadata": {
        "customer_id": "123456789",
        "hide": "false"
    }
}

Use individual containers rather than "self-contained astarte" in docker-compose

Deprecate the huge docker container.

Implement sensors plot JS example

Implement a JavaScript example which allows to choose a sensor and to plot a chart for the given interval. The example should also display min, max and average values.
All data should be displayed along with units and sensor names.

The example should use cloud.astarte.genericsensors.Values properties interface and cloud.astarte.genericsensors.AvailableSensors (when available). It would be useful to implement it using React as well.

See also #86

Describe some real world use cases

Documentation should contain some how-tos for following scenarios:

A device with some sensors.
A device which collects data from some wireless sensors and delivers them to Astarte (with trade-off discussion: 1 device_id or multiple device_ids).
A remote controllable device (e.g. on/off relay).

Use Scylla in docker-compose based Astarte

Use the docker-compose deployment to validate the use of Scylla

docker-compose doesn't persist CA configuration

If you execute docker-compose down and docker-compose up on an Astarte instance, all certificates emitted before the restart are seen as invalid (with an Unknown CA error).
The devices eventually reconnect since they verify the certificate, but this has to be investigated.

It should be allowed to set custom http headers

Astarte Trigger Engine should allow users to override default headers, such as "Content-Type", also it should allow to add some custom headers like tokens, or etc.

Adding empty endpoint causes internal server error

Steps to reproduce:
Install an interface with an empty string as endpoint.

{
    "interface_name": "myInterface",
    "version_major": 0,
    "version_minor": 1,
    "type": "properties",
    "ownership": "device",
    "mappings": [
        {
            "endpoint": "",
            "type": "longinteger"
        }
    ]
}

AppEngine: return an empty list for valid queries (without any data)

When making a valid query on an time interval (with no data) AppEngine should return an empty list ([]), a 404 is returned instead.

How to reproduce:

Query a device using AppEngine on an interval with no data

Enhance Channels authorization mechanism

Currently, the JOIN and WATCH claim in a token are intertwined. This is bad design from an auth POV, where it is not acceptable that one credential implies another.

A proposed solution could be something like WATCH:<regex room>:<regex path>, which would keep JOIN standalone and WATCH self-fulfilling. As such, a claim like ["JOIN:.*", "WATCH:myroom:mydevice.*"] would be safe and meaningful.

Allow to configure database TTL on each mapping

[meta] Improve failures handling in data updater plant

Following issues need to be closed:

astarte-platform/astarte_data_updater_plant#2

Astarte Channels should also support messages going to Devices

It should be allowed to watch for events on server owned interfaces.
E.g. it should be possible to watch for data going from the server to the device.

Missing /v1 in socket route

Right now to connect to an Astarte channel the url used is
%{appengine_url}/socket

We should uniform it to other Astarte routes like
%{appengine_url}/v1/socket

[meta] Replace Conform and Distillery with Elixir native releases

Conform and Distillery are both deprecated, start using Elixir releases.

Allow access to previous interfaces data

When an interface major on a certain device is replaced by a newer one, older data is not accessible anymore, add an API which allows to retrieve older data too.

Make ^ and $ implicit in auth regexp

^ should be prepended, and $ should be appended server side to auth regexps, so the claims can be shorter and simpler.

Error running Astarte in 5 minutes

Hey! I am trying to go through with 5 minutes tutorial, but I have a problem with test realm creation.

After generating a key, I run this command and get:

openssl rsa -in test_realm.key -pubout -outform PEM -out test_realm.key.pub
unable to load Private Key
4682208876:error:09FFF06C:PEM routines:CRYPTO_internal:no start line:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-22.240.1/libressl-2.6/crypto/pem/pem_lib.c:683:Expecting: ANY PRIVATE KEY

I am running latest mac os x with openssl installed with brew:

openssl version
LibreSSL 2.6.5

How to properly deal with this problem?

Load :any_interface data triggers

RIght now, triggers having as target the special * interface can be installed, but DataUpdater never loads them in its state to use them.

DataUpdater should do something like

any_interface_id = SimpleTriggersProtobufUtils.any_interface_object_id()
populate_triggers_for_object!(new_state, db_client, any_interface_id, :any_interface)

This should probably be done in reload_device_triggers_on_expiry since these triggers don't depend on a specific interface and/or introspection.

Implement sensors viewer JS example

Implement a JavaScript/HTML5 example which allows to display latest received values for each sensor.
It should also display sensor name and units when available.

It is required to implement it in the most simple and clear way, so it can be used as a starting point for third party users.

See also #86

Implement SamplingRate JS example

Implement a JavaScript example which allows to enable and disable sensors and to configure sampling period.

The example should use cloud.astarte.genericsensors.SamplingRate properties interface. It would be useful to implement it using React as well.

See also #86

Write a tutorial based on generic sensors interfaces

Write a tutorial based on generic sensors interfaces with JavaScript snippets from existing examples.

The tutorial should show:

Explain employed interfaces and their mappings
How to install required interfaces using astartectl or the dashboard
How to simulate a device which published data to those interfaces
How to display device status using the dashboard
How to download data using astartectl
API requests examples
JavaScript snippets based on existing examples
How to make a physical device using ESP32 SDK

Add stats to Groups

The same stats to be found for /stats/devices, should also be available for /stats/groups/

Support initial values in Pairing

Pairing already supports delivering an initial introspection. When this is done, it should also be possible to set initial values on all interfaces.

As an example, suppose we're registering a device with an interface com.example.Info in its initial introspection. A pairing API call should support something like:

[...]
"initial_data": {
    "com.example.Info": {
         "serial": "1491414"
    }
}
[...]

After registering the device, calling AppEngine API on the /com.example.Info/serial endpoint should return the value set, even if the device never connected.

Return a meaningful response on intermediate paths when using a datastream on AppEngine

E.g.:
Interface com.something.Etc with /a/path endpoint:

GET /com.something.Etc: OK
GET /com.something.Etc/a: 500
GET /com.something.Etc/a/path: OK

Unreliable mappings can be discarded when database fails

Reliable datastream mappings can be safely discarded when the database fails.
This change avoids to store hundred of messages while the database is not up.

This change will call discard when insert fails.

Stats should also have a "Never Connected" field

Stats should show these counts:

total devices
connected devices
disconnected devices
never connected devices

Implement sensors viewer WebSocket + JS example

Implement a JavaScript example which allows to choose a sensor and to display real time data using a WebSocket.

See also #86

Add support to prev, next, self to metadata

When fetching a limited set of datapoints metadata should report prev, next, self in a similar fashion to devices list, so iterating on the dataset would be easier.

Validate triggers during install

It shouldn't be allowed to install a trigger having a string known_value on an integer mapping

Allow custom http verbs in astarte_trigger_engine

Allow to configure other http methods such as PUT, GET, etc...

Implement server to device object aggregated interfaces

Implement combined simple triggers using operators in astarte_trigger_engine

Original samples count should be reported when using downsampling

Original samples count should be reported when using downsampling, this information is needed to know if additional data points can be retrieved on a smaller time window without performing any additional request.
metadata field should be used for this purpose.

Add /metrics to Astarte Realm Management API

Add prometheus /metrics endpoint.

Add support for the device_error device event

There's already a partial support for an additional device event (other than device_connected and device_disconnected) that is device_error.

Now that support for device logs is being added to the dashboard, it would be nice to use it to be able to expose errors happening inside Data Updater Plant (e.g. invalid topics, invalid data type etc) happening for a specific device, without having to look at the logs.

[meta] org.astarte-platform.InterfaceName is not a valid interface name

- are not allowed in interface names right now, they should be allowed if we want to allow org.astarte-platform.InterfaceName.

I suggest following change:

- is allowed in any domain components (e.g. org.astarte-platform).
- is disallowed in the last interface name token (e.g. InterfaceName).

This will keep the interface name programmer friendly (e.g. interface name can be always mapped to a class or module name), while this limitation might be removed in future releases.

Related issues:

Revamp AppEngine API

AppEngine's API is currently too cumbersome and prevents some use cases from being fulfilled. Come up with a new design with is backwards compatible with 0.10.

Remove Poison dependency from astarte_appengine_api

Remove Poison dependency as soon as phoenix_swagger 0.8.2 is released.
See xerions/phoenix_swagger#238

astarte-platform / astarte Goto Github PK

astarte's People

Contributors

Stargazers

Watchers

Forkers

astarte's Issues

Recommend Projects

Recommend Topics

Recommend Org