microsoft / azure-digital-twins-java

Azure Digital Twins - Java Support Collection

Azure Digital Twins (2018) - Java support collection

This project has been developed for the Azure Digital Twins (2018) version and is discontinued.

Azure Digital Twins (2018) is retiring on 31 December 2020. A new version of Azure Digital Twins is available in General Availability and provides new updates that weren't available in the previous version such as:

• Flexible modelling and general graph topology
• Richer Compute
• Bring your own IoT Hub for easier integration
• Improved scalability

We recommend removing any Azure Digital Twins (2018) resources that are no longer being used and transitioning to the new service as soon as possible, as Azure Digital Twins (2018) resources will no longer be available for use by end of year 2020.

The new Azure Digital Twins provides the capabilities that this project covered originally for Azure Digital Twins (2018).

Please reference the Azure Digital Twins documentation for transitioning into new services.

azure-digital-twins-java's Issues

Twin Reflector Proxy - Create caching concept including cache invalidation

Use-cases:

• Topology
• Sensor data

Strategy:

• Should be based on Spring cache.
• Cache provider to be decided, check if ADT events can be leveraged for cache invalidation.

Twin Reflector Proxy - Ingest telemetry with twin context

More details and context see issue: #16

User Story User Story Ingest telemetry with context`

As Cloud Connector,
I want ingest sensor data (points) with context as events asynchronously into ADT,
so that the device and sensor is created or updated in the ADT topology
and the actual telemetry value is reflected in ADT.
and I don't have to worry about the REST management API limitations (e.g. throttling) and the REST API details (e.g. Create or Update).

User Story Details

Ingest sensor data (points) with twin context (including properties, attributes and relationships)

Protocol

Pre-conditions

• The ontology must be predefined (in ADT version 1 the space types and property-keys are defined)
• ADT IoT Hub needs to be provision

Event Hub Headers

Please see user story #16

Send telemetry data as event (sensor)

Event Hub Message

Headers:

• TwinReflectorProxy-MessageType: ingest
• Source: cbs

Forward all non Twin Reflector Proxy event hub message headers to IoT Hub and prefix them (e.g. TwinReflectorAgent-Source).

Message Payload (Example):

{
  "id": "S1",                               // external Id of the sensor (point)
  "entity-type": "Sensor",
  "attributes": {
     "dataType": "Temperature"
  },
  "properties": [
      {
          "entity-type": "propertykeys",
          "name": "displayName",
          "value": "Temperature XDK"
      }
  ],
  "relationships": [
      {
          "entity-type":     "device",
          "name":            "parent",
          "target-id":       "D1"           // external Id of the device (equipment)
      }
  ],
  "telemetry": "100"                        // any kind of telemetry data
}

Twin Reflector Message Processing

• Query for the sensor by the external Id
• Create or update the sensor via the management API
• Send the telemetry data to the IoT Hub

Write unit tests for Management Client

Twin Reflector Proxy - Provide Docker file/build

Twin Reflector Proxy - Ingest topology updates

User Story Ingest twin elements

As Cloud Connector,
I want ingest topology (e.g. spaces) events asynchronously into ADT,
so that the updates are reflected in the ADT topology
and I don't have to worry about the REST management API limitations (e.g. throttling) and the REST API details (e.g. Create or Update).

User Story Details

Ingest twin instances (Create or update) as event asynchronously into ADT including properties, attributes and relationships. The events are processed from Azure Event Hub and mapped into several ADT REST API calls.

The ADT REST API is defined here.

Context Building Block View

Protocol

Pre-conditions

• The ontology must be predefined (in ADT version 1 the space types and property-keys are defined)
• ADT IoT Hub needs to be provision

Event Hub Headers

Header	Values	Required	Description
TwinReflectorProxy-MessageType	full, partial, delete	yes
TwinReflectorProxy-EventTimestamp	e.g. 2018-09-20T07:35:00.8587882-07:00	no	timestamp as UTC as ISO 8601 formatted date string that identifies the sampling time of the payload. For example, 2018-09-20T07:35:00.8587882-07:00 (optional)
TwinReflectorProxy-CorrelationId		no	UUID (optional)

Send event to create or update topology element

The create and update message contains the full state (all attributes, all properties and all relationships) of the topology element!

Event Hub Message

Headers:

• TwinReflectorProxy-MessageType: ingest

Message Payload (Example):

{
  "id": "F3",                               // external Id
  "entity-type": "Space",
  "attributes": {
     "friendlyName": "Office"
  },
  "properties": [
      {
          "entity-type": "propertykeys",
          "name": "title",
          "value": "Office London"
      }
  ],
  "relationships": [
      {
          "entity-type":     "space",
          "name":            "parent",
          "target-id":       "C1"           // external Id of the campus
      }
  ]
}

Twin Reflector Proxy Message Processing

• Query for the topology element by the external id (via ADT REST Management API as GET)
• Apply the attributes and properties from the message (Unit of Work Pattern)
• Creates or update the element in ADT (via ADT REST Management API as POST or PATCH)
• Updates the properties of the element in ADT

ADT State (ADT Model V1)

# Space
- id: "7487abd8-a5b9-acab-5fdb-7a74ac1674f5"                  # ADT ID
 - name: "C1"

- id: "5d3600bc-5475-11e9-8647-d663bd873d93"                  # ADT ID
 - name: "F3"
 - friendlyName: "Office"
 - parentId: "7487abd8-a5b9-acab-5fdb-7a74ac1674f5"           # ADT Id parent
 - properties:
    title: "Office London"

Send event to delete a topology element

Event Hub Message

Headers:

• TwinReflectorProxy-MessageType: delete

Message Payload (Example):

{
  "id": "F3",                               // external Id
}

Twin Reflector Message Processing

• Query (search) for the topology element by the external Id
• Delete the topology element
• Dont't delete other topology elements

Write integration tests for Management Client

Twin Reflector Proxy - Provide AKS deployment support

• Helm templates
• ARM templates

Twin Reflector Proxy - Integrate Application insights

Include Sonarcloud check into master build

Currently sonar is generally disabled as it will not work on PRs (Sonar service connection disabled in branch builds).

We should create an extra build for master not based on the pipeline Yaml that is checked in.

Evaluate spring-cloud-azure-eventhubs-stream-binder as alternative to Kafka binder

https://github.com/microsoft/spring-cloud-azure/tree/master/spring-cloud-azure-stream-binder/spring-cloud-azure-eventhubs-stream-binder

Twin Reflector Proxy - Concept for feedback channel

How should TRP report back successful as well as unsuccessful events.

Twin Reflector Proxy - Provide PoC implementation

...that verifies that we can sync inbound TRP events into ADT by management API v1.

PoC should proof that create/update/delete calls on a space are ingested into ADT. A framework following UnitOfWork pattern for property and attribute changes as well as parent relationships needs to be defined.

Rename dispatcher to Twin Reflector Proxy

Migrate to newest Spring, Java and other 3rd party versions

• Java 11
• Docker: Zulu
• Spring 2.1.5
• Maven: latest
• JUnit 5 (includes test migration)
• Spring Cache: caffeine
• Jackson for Json/object mapping

Move all dependency versions into parent as properties