eclipse-ditto / ditto-clients

Eclipse Ditto™: Digital Twin framework - Client SDKs for Java and JavaScript

Eclipse Ditto™ Client SDKs

This repository contains client SDKs in different languages for Eclipse Ditto.

Java

Implemented in Java, targeted for use in Java and all JVM based languages (source and target level is Java 8).

JavaScript

Implemented in TypeScript, this SDK contains a JavaScript library for usage of Ditto in both browser and Node.JS based environments.

Go

Implemented in Go, target for use in Go. Source is contained in separate GitHub repository because of Go specific GitHub based versioning/dependency mechanisms.

Python

This repository contains the Python client SDK for Ditto. Source is contained in separate GitHub repository because of specific GitHub based versioning/dependency mechanisms.

ditto-clients's Issues

[JS] Unravel the WebSocket implementation

The WebSocket implementation of the client API currently is very difficult to read, understand and extend. When reading the code (or the tests for it), it is not possible to understand if everything works as expected or if it doesn't.

Therefore the implementation definitely requires some work (and probably a rewrite?).

Additionally to the existing provided functionality, there should also be the possibility to manually connect and disconnect the WebSocket client. Currently it will connect instantly after building it and there is no way to disconnect it.

[JS] Connection error

Hi! For some reason I keep getting a connection error.

Error: Error: getaddrinfo ENOTFOUND localhost:8080
    at ClientRequest.<anonymous> (/home/user/Desktop/dtwin/twin/node_modules/@eclipse-ditto/ditto-javascript-client-node/dist/node/src/node-http.js:67:27)
    at ClientRequest.emit (node:events:365:28)
    at TLSSocket.socketErrorListener (node:_http_client:447:9)
    at TLSSocket.emit (node:events:365:28)
    at emitErrorNT (node:internal/streams/destroy:193:8)
    at emitErrorCloseNT (node:internal/streams/destroy:158:3)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)
error Command failed with exit code 1.

Am I doing anything wrong?

Here is the code:

import {                                                                                                                                                                                                   
    DittoNodeClient,                                                                                                                                                                                       
    NodeHttpBasicAuth,                                                                                                                                                                                     
    Thing,                                                                                                                                                                                                 
} from "@eclipse-ditto/ditto-javascript-client-node";                                                                                                                                                      
                                                                                                                                                                                                           
const domain = "localhost:8080";                                                                                                                                                                           
const username = "ditto";                                                                                                                                                                                  
const password = "ditto";                                                                                                                                                                                                                                                                                                                                                             
                                                                                                                                                                                                           
const client = DittoNodeClient.newHttpClient()                                                                                                                                                             
    .withoutTls()                                                                                                                                                                                          
    .withDomain(domain)                                                                                                                                                                                    
    .withAuthProvider(NodeHttpBasicAuth.newInstance(username, password))                                                                                                                                   
    .build();                                                                                                                                                                                              
                                                                                                                                                                                                           
const thingsHandle = client.getThingsHandle();                                                                                                                                                             
                                                                                                                                                                                                           
const thing = new Thing("the:thing");                                                                                                                                                                      
thingsHandle                                                                                                                                                                                               
    .putThing(thing)                                                                                                                                                                                       
    .then((result) =>                                                                                                                                                                                      
        console.log(
            `Finished putting thing with result: ${JSON.stringify(result)}`
        )
    );

Also if I execute the following command:

curl -u devops:foobar http://localhost:8080/status/health

I get the following output

{"label":"roles","status":"UP","children":[{"label":"expected-roles","status":"UP","details":[{"INFO":{"missing-roles":[],"extra-roles":[]}}]},{"label":"concierge","status":"UP","children":[{"label":"172.20.0.5:2551","status":"UP","children":[{"label":"persistence","status":"UP"},{"label":"SingletonStatusReporter","status":"UP","details":[{"INFO":{"enabled":true,"events":[],"credit-decisions":[],"actions":[]}}]}]}]},{"label":"things","status":"UP","children":[{"label":"172.20.0.7:2551","status":"UP","children":[{"label":"persistence","status":"UP"},{"label":"MongoMetricsReporter","status":"UP","details":[{"INFO":{"reporter":"/user/thingsRoot/healthCheckingActor/MongoMetricsReporter","resolution":"PT5S","maxTimerNanos":[0,0,0,0,0]}}]}]}]},{"label":"connectivity","status":"UP","children":[{"label":"172.20.0.9:2551","status":"UP","children":[{"label":"persistence","status":"UP"}]}]},{"label":"policies","status":"UP","children":[{"label":"172.20.0.3:2551","status":"UP","children":[{"label":"persistence","status":"UP"}]}]},{"label":"things-search","status":"UP","children":[{"label":"172.20.0.6:2551","status":"UP","children":[{"label":"persistence","status":"UP"},{"label":"backgroundSync","status":"UP","details":[{"INFO":{"enabled":true,"events":[],"progressPersisted":":_","progressIndexed":":_"}}]}]}]},{"label":"gateway","status":"UP","children":[{"label":"172.20.0.8:2551","status":"UP"}]}]}

Java Websocket client disconnecting

As discussed on gitter, I'm getting following exception when using the java websocket connection to ditto:

2020-04-22 11:05:58,389 DEBUG [org.ecl.dit.cli.mes.int.WebSocketMessagingProvider] (ditto-client-c4e0b023-00c1-439d-a1da-a4c6efbcce00-104) Client <ditto:68ca9e13-63df-4fe3-a43d-59a1b8c9bb10>: Got TWIN ThingErrorResponse: <GatewayWebsocketSessionExpiredException: The websocket session expired because the JWT used for authentication has expired. - You need to periodically refresh your websocket session with a new JWT.>
2020-04-22 11:05:58,389 TRACE [org.ecl.dit.cli.int.ResponseForwarder] (ditto-client-c4e0b023-00c1-439d-a1da-a4c6efbcce00-104) Received response for correlation-id <e44741df-a847-4f45-989d-67f6ef376d22>.
2020-04-22 11:05:58,389 DEBUG [org.ecl.dit.cli.int.ResponseForwarder] (ditto-client-c4e0b023-00c1-439d-a1da-a4c6efbcce00-104) No promise found for response with correlation-id <e44741df-a847-4f45-989d-67f6ef376d22>!
2020-04-22 11:05:58,387 INFO  [org.ecl.dit.cli.mes.int.WebSocketMessagingProvider] (ditto-client-c4e0b023-00c1-439d-a1da-a4c6efbcce00-106) Client <ditto:68ca9e13-63df-4fe3-a43d-59a1b8c9bb10>: WebSocket connection to endpoint <ws://ditto-nginx:8081/ws/2> was closed by Server with code <1000> and reason <null>.
2020-04-22 11:05:58,389 INFO  [org.ecl.dit.cli.mes.int.WebSocketMessagingProvider] (ditto-client-c4e0b023-00c1-439d-a1da-a4c6efbcce00-106) Client <ditto:68ca9e13-63df-4fe3-a43d-59a1b8c9bb10>: Reconnection is enabled. Reconnecting in <5> seconds ...

We are using the following snippet to setup the connection:

        final AuthenticationProvider authenticationProvider =
                AuthenticationProviders.clientCredentials(ClientCredentialsAuthenticationConfiguration.newBuilder()
                        .clientId(dittoConfiguration.keycloakClientId())
                        .clientSecret(dittoConfiguration.keycloakClientSecret())
                        .scopes(Collections.singleton("*"))
                        .tokenEndpoint(dittoConfiguration.keycloakEndpoint())
                        .build());

The ditto SDK client version we are using, is version 1.0.0. We tested this with both Ditto 1.0.0 and 1.1.0 and are encoutering the same issues.

Release latest build on npm

When I look in the npm repository the latest release is from 8 months ago, are these build not going to npm? How do I get a hold of build in that case?

Java client: Let user configure all thread pools

Ditto java client 1.1.2 allocates and shuts down its own thread pools, which lead to performance problems when many instances of the java client run in the same JVM. It would be better to let the user provide every ExecutorService the client uses so that the threads are shared across all instances.

The client should not shutdown any user-provided ExecutorService when destroyed. Instead, it should cancel tasks it scheduled and refrain from submitting new tasks. For example, Retry won't be able to rely on the client terminating its executor service and must become aware when its owning client was destroyed.

Client always connecting on localhost

I'm trying to connect to a ditto instance running on another host with this configuration.

        const httpClient = DittoNodeClient
            .newHttpClient()
            .withTls()
            .withDomain(this.domain)
            .withAuthProvider(NodeHttpBasicAuth.newInstance(this.username, this.password))
            .apiVersion2()
            .build();

        const thingsHandle = httpClient.getThingsHandle();

        thingsHandle.getThing('org.example:device_1')
            .then(returnedThing => {
                console.log(`Get returned ${JSON.stringify(returnedThing)}`);
            })
            .catch((err) => {
                console.log('error', err);
            });

suing this library version

    "@eclipse-ditto/ditto-javascript-client-api_1.0": "^2.0.0",
    "@eclipse-ditto/ditto-javascript-client-node_1.0": "^2.0.0",

But I get this error

Error: connect ECONNREFUSED 127.0.0.1:80

It seems it's connecting to localhost even though domain is set.

[JS] Add Server Sent Events (SSE) implementation for Eclipse Dittos events

Currently the JS client only allows receiving events via WebSocket. We should also provide an implementation that uses the existing SSE API that is already provided by Eclipse Ditto.

Note: Main reason for this not being in the initial commit is, that the event-source-polyfill seemed to not work in web components that used the client. When implementing SSE, definitely test if the implementation works inside web components (e.g. a StencilJS web component)

Sometimes DittoClients.newInstance(...) blocks forever

Creating ditto client sometimes blocks forever. Here is the thread that is blocked and never ends.

"ditto-client--adaptable-bus-8335710b-34bb-4263-8294-207f2e63bfcc-2" #396 prio=5 os_prio=0 tid=0x000000001f745000 nid=0x338c waiting on condition [0x000000002b6ec000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000e2bfc658> (a java.util.concurrent.CompletableFuture$Signaller)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693)
        at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
        at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729)
        at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1934)
        at org.eclipse.ditto.client.messaging.internal.WebSocketMessagingProvider.initialize(WebSocketMessagingProvider.java:171)
        - locked <0x00000000e2c0c448> (a java.util.concurrent.atomic.AtomicBoolean)
        - locked <0x00000000e2c0b990> (a org.eclipse.ditto.client.messaging.internal.WebSocketMessagingProvider)
        at org.eclipse.ditto.client.internal.DefaultDittoClient.init(DefaultDittoClient.java:240)
        at org.eclipse.ditto.client.internal.DefaultDittoClient.configureTwin(DefaultDittoClient.java:197)
        at org.eclipse.ditto.client.internal.DefaultDittoClient.newInstance(DefaultDittoClient.java:123)
        at org.eclipse.ditto.client.DittoClients.newInstance(DittoClients.java:132)
        at org.eclipse.ditto.client.DittoClients.newInstance(DittoClients.java:90)
        at org.eclipse.ditto.client.DittoClients.newInstance(DittoClients.java:68)
        at org.eclipse.ditto.client.DittoClients.newInstance(DittoClients.java:51)
        at com.bosch.iot.dm.things.access.DittoClientFactory.lambda$createClient$9(DittoClientFactory.java:305)
        at com.bosch.iot.dm.things.access.DittoClientFactory$$Lambda$799/26418538.call(Unknown Source)
        at com.bosch.iot.dm.vertx.Utils.outOfEventLoop(Utils.java:410)
        at com.bosch.iot.dm.vertx.Utils.outOfEventLoop(Utils.java:368)
        at com.bosch.iot.dm.things.access.DittoClientFactory.createClient(DittoClientFactory.java:305)
        at com.bosch.iot.dm.things.access.DittoClientFactory.getInstance(DittoClientFactory.java:286)
        - locked <0x00000000e169b460> (a java.util.HashMap)
        at com.bosch.iot.dm.things.access.DittoClientFactory$$Lambda$1359/687663372.apply(Unknown Source)
        at com.bosch.iot.dm.async.CompletableFuture.lambda$null$24(CompletableFuture.java:982)
        at com.bosch.iot.dm.async.CompletableFuture$$Lambda$778/1637673631.call(Unknown Source)
        at com.bosch.iot.dm.auth.AuthCtx.callWithMDCSubscrId(AuthCtx.java:300)
        at com.bosch.iot.dm.auth.AuthCtx.callWithMDCAuthCtx(AuthCtx.java:280)
        at com.bosch.iot.dm.auth.AuthCtx.callAs(AuthCtx.java:239)
        at com.bosch.iot.dm.auth.AuthCtx.callAs(AuthCtx.java:216)
        at com.bosch.iot.dm.async.CompletableFuture.lambda$authCtxAware$25(CompletableFuture.java:982)
        at com.bosch.iot.dm.async.CompletableFuture$$Lambda$777/2135287206.apply(Unknown Source)
        at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981)
        at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124)
        at com.bosch.iot.dm.async.CompletableFuture.thenCompose(CompletableFuture.java:758)
        at com.bosch.iot.dm.async.CompletableFuture.thenCompose(CompletableFuture.java:61)
        at com.bosch.iot.dm.things.access.DittoClientFactory.apply(DittoClientFactory.java:230)
        at com.bosch.iot.dm.devices.events.impl.FeatureEventSource.lambda$definitionMatch$45(FeatureEventSource.java:256)
        at com.bosch.iot.dm.devices.events.impl.FeatureEventSource$$Lambda$2075/234707026.call(Unknown Source)
        at com.bosch.iot.dm.auth.AuthCtx.callWithMDCSubscrId(AuthCtx.java:304)
        at com.bosch.iot.dm.auth.AuthCtx.callWithMDCAuthCtx(AuthCtx.java:284)
        at com.bosch.iot.dm.auth.AuthCtx.callAs(AuthCtx.java:243)
        at com.bosch.iot.dm.auth.AuthCtx.callAs(AuthCtx.java:216)
        at com.bosch.iot.dm.devices.events.impl.FeatureEventSource.definitionMatch(FeatureEventSource.java:256)
        at com.bosch.iot.dm.devices.events.impl.FeatureEventSource.lambda$null$33(FeatureEventSource.java:242)
        at com.bosch.iot.dm.devices.events.impl.FeatureEventSource$$Lambda$2074/1300308500.apply(Unknown Source)
        at java.util.Optional.map(Optional.java:215)
        at com.bosch.iot.dm.devices.events.impl.FeatureEventSource.lambda$definitionMatch$35(FeatureEventSource.java:242)
        at com.bosch.iot.dm.devices.events.impl.FeatureEventSource$$Lambda$2073/1013156636.apply(Unknown Source)
        at java.util.Optional.map(Optional.java:215)
        at com.bosch.iot.dm.devices.events.impl.FeatureEventSource.definitionMatch(FeatureEventSource.java:241)
        at com.bosch.iot.dm.devices.events.impl.FeatureEventSource.featureChangeMatch(FeatureEventSource.java:233)
        at com.bosch.iot.dm.devices.events.impl.FeatureEventSource.lambda$match$27(FeatureEventSource.java:221)
        at com.bosch.iot.dm.devices.events.impl.FeatureEventSource$$Lambda$2064/1538987199.apply(Unknown Source)
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
        at com.bosch.iot.dm.devices.events.impl.FeatureEventSource.match(FeatureEventSource.java:222)
        at com.bosch.iot.dm.devices.events.impl.DeviceStatusEventSource.lambda$match$11(DeviceStatusEventSource.java:163)
        at com.bosch.iot.dm.devices.events.impl.DeviceStatusEventSource$$Lambda$2056/1116002950.apply(Unknown Source)
        at java.util.Optional.map(Optional.java:215)
        at com.bosch.iot.dm.devices.events.impl.DeviceStatusEventSource.match(DeviceStatusEventSource.java:160)
        at com.bosch.iot.dm.devices.events.impl.DeviceStatusEventSource.lambda$subscribe$17(DeviceStatusEventSource.java:182)
        at com.bosch.iot.dm.devices.events.impl.DeviceStatusEventSource$$Lambda$805/1003587043.accept(Unknown Source)
        at org.eclipse.ditto.client.internal.bus.SelectorUtil.lambda$registerForChanges$9(SelectorUtil.java:236)
        at org.eclipse.ditto.client.internal.bus.SelectorUtil$$Lambda$1284/2117200343.accept(Unknown Source)
        at org.eclipse.ditto.client.internal.bus.DefaultPointerBus.lambda$notify$1(DefaultPointerBus.java:49)
        at org.eclipse.ditto.client.internal.bus.DefaultPointerBus$$Lambda$1493/1048547345.accept(Unknown Source)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
        at org.eclipse.ditto.client.internal.bus.DefaultPointerBus.notify(DefaultPointerBus.java:49)
        at org.eclipse.ditto.client.internal.bus.PointerBus.notify(PointerBus.java:49)
        at org.eclipse.ditto.client.internal.bus.SelectorUtil.lambda$addHandlerForThingEvent$4(SelectorUtil.java:149)
        at org.eclipse.ditto.client.internal.bus.SelectorUtil$$Lambda$1032/588256040.accept(Unknown Source)
        at org.eclipse.ditto.client.internal.bus.DefaultPointerBus.lambda$notify$1(DefaultPointerBus.java:49)
        at org.eclipse.ditto.client.internal.bus.DefaultPointerBus$$Lambda$1493/1048547345.accept(Unknown Source)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
        at org.eclipse.ditto.client.internal.bus.DefaultPointerBus.notify(DefaultPointerBus.java:49)
        at org.eclipse.ditto.client.internal.bus.PointerBus.notify(PointerBus.java:49)
        at org.eclipse.ditto.client.internal.bus.PointerBus.notify(PointerBus.java:37)
        at org.eclipse.ditto.client.internal.CommonManagementImpl.lambda$null$17(CommonManagementImpl.java:630)
        at org.eclipse.ditto.client.internal.CommonManagementImpl$$Lambda$1450/170622711.accept(Unknown Source)
        at org.eclipse.ditto.client.internal.CommonManagementImpl.lambda$subscribeAndPublishMessage$19(CommonManagementImpl.java:650)
        at org.eclipse.ditto.client.internal.CommonManagementImpl$$Lambda$1292/680998013.accept(Unknown Source)
        at org.eclipse.ditto.client.internal.bus.DefaultAdaptableBus.lambda$runConsumerAsync$6(DefaultAdaptableBus.java:214)
        at org.eclipse.ditto.client.internal.bus.DefaultAdaptableBus$$Lambda$1340/2080062198.run(Unknown Source)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

does things-search work? or how can I retrieve all the things from the ditto

I am new to ditto. From the HTTP API, it shows there are endpoints for things-search. How could I use it through ditto-client? Thanks

[JS] ReferenceError: EclipseDittoJavascriptClientApi is not defined

Hi,

I want to use the client in a VueJs environment (with Webpack).
When I want to use the client I get the following error:

ReferenceError: EclipseDittoJavascriptClientApi is not defined

my code:

    static getDittoClient() {
        return DittoDomClient.newHttpClient()
            .withTls()
            .withDomain('example.com')
            .withAuthProvider(DomHttpBasicAuth.newInstance("username", 'password'))
            .apiVersion2()
            .build();
    }

reported source file:

var EclipseDittoJavascriptClient = (function (exports, dittoJavascriptClientApi_1_0Pre) {
'use strict';

[....]

}({}, EclipseDittoJavascriptClientApi));

//# sourceURL=webpack-internal:///./node_modules/@eclipse-ditto/ditto-javascript-client-dom_1.0-pre/dist/index.bundle.js

versions:

"@eclipse-ditto/ditto-javascript-client-api_1.0-pre": "^1.0.0-pre2",
"@eclipse-ditto/ditto-javascript-client-dom_1.0-pre": "^1.0.0-pre2",

[JS] Eq filter not working for boolean values

When crating a filter in the JS client via the Eq filter interface and passing a boolean value, the filter is not correctly parsed by ditto.
Example:

Eq('attributes/active', true)

The problem seems to be that the value is always put in quotes, but ditto then treats it as a string and the type comparision fails (by design).
When the same filter is manually sent to ditto but with the quotes removed (i.e. eq('attributes/active', true) instead of eq('attributes/active', "true")) the filter works as expected.

I only tested & verified this for the Eq filter and boolean values, but I believe other filters and primitive types (number, object) will have the same problem.

Line at fault:
https://github.com/eclipse/ditto-clients/blob/a5735ed1aafad7f743f4a3cca501c9d98e1bc277/javascript/lib/api/src/options/filter.options.ts#L205

Add example for a device connected to ditto via Azure IoT Hub

Write an example which shows the possibilities connecting a device via Azure IoT Hub to Ditto.

This can be used to evaluate using Ditto with Azure IoT Hub as a broker.

[JS] Buffered WebSocket client sends messages before being connected

The buffered implementation for WebSockets has bug that it will try to send the buffered messages to the WebSocket before the WebSocket has finished connecting.

From what I can observe now, the problematic code can be found in StandardResilienceHandler#addToOutstandingBuffer. In this function, an outstanding request is added to the buffer and a timeout is started for calling #poll(). The #poll() function however does not verify if this.webSocket is already initialized.

In case the connection takes longer than the initial timeout of 500ms, this.requestBuffer#sendNectOutstanding will be called with an undefined WebSocket and crash.

Add support for top-level policy management

With eclipse-ditto/ditto#554 policies are supported in the ditto protocol. The Ditto Java Client will also be extended to support top-level policy management. I.e. there will be no no Java API to manipulate subjects/entries, but only the whole policy (nevertheless the respective subject/entry commands can be built using the ProcotolAdapter directly).

Looking for metadata retrieval support in Java client

Hi guys!
After reading the source and searching around the Java codebase, I finally can use DittoHeaders putMetadata to add metadata to a thing. However, I tried several apis and failed to retrieve the metadata. Am I missing something?

[JS] putThing throws error when updating an existing Thing

According to the docs, a PUT request for an existing Thing will have ditto respond with status 204 and an empty body.
The JS client SDK, however, always expects body: https://github.com/eclipse/ditto-clients/blob/master/javascript/lib/api/src/client/request-factory/request-sender.ts#L48

Thing object not fully populated on change

After the following (example taken from documentation):

client.twin().startConsumption().toCompletableFuture().get(); 
System.out.println("Subscribed for Twin events");
client.twin().registerForThingChanges("my-changes", change -> {
     System.out.println("An existing Thing was modified: " + change.getThing());

The Thing returned from change.getThing() has only fields populated that changed. Other fields like the id, policy or other are null.
There is another method:

change.getValue()

that can be used to find out what changed. I believe the idea for the getThing() was to return a fully populated object that was affected by the change.

[java] Incorrect test for DITTO_DUMMY_AUTH_USER in DittoClientUsageExamples

In the file java/src/test/java/org/eclipse/ditto/client/DittoClientUsageExamples.java at the line 764, the test should be DITTO_DUMMY_AUTH_USER != null && !DITTO_DUMMY_AUTH_USER.isEmpty(), and not just DITTO_DUMMY_AUTH_USER != null.

Currently, if ditto.dummy-auth-user is empty in the config file DITTO_DUMMY_AUTH_USER will be set to "", causing the test to pass when it should fail, which then causes the program to behave incorrectly.

[JS]: incompatible with typscript 4.7.4

We're currently using the ditto js sdk v2.4 with typescript 4.7.4.
Since 3.0 is out, we wanted to upgrade but our build fails with the following errors:

Error: node_modules/@eclipse-ditto/ditto-javascript-client-dom/dist/api/src/model/policies.model.d.ts:30:9 - error TS2611: 'id' is defined as a property in class 'EntityWithId', but is overridden here in 'Policy' as an accessor.
30     get id(): string;
           ~~
Error: node_modules/@eclipse-ditto/ditto-javascript-client-dom/dist/api/src/model/policies.model.d.ts:62:9 - error TS2611: 'id' is defined as a property in class 'EntityWithId', but is overridden here in 'Entry' as an accessor.
62     get id(): string;
           ~~
Error: node_modules/@eclipse-ditto/ditto-javascript-client-dom/dist/api/src/model/policies.model.d.ts:119:9 - error TS2611: 'id' is defined as a property in class 'EntityWithId', but is overridden here in 'Subject' as an accessor.
119     get id(): string;
            ~~
Error: node_modules/@eclipse-ditto/ditto-javascript-client-dom/dist/api/src/model/policies.model.d.ts:143:9 - error TS2611: 'id' is defined as a property in class 'EntityWithId', but is overridden here in 'Resource' as an accessor.
143     get id(): string;
            ~~
Error: node_modules/@eclipse-ditto/ditto-javascript-client-dom/dist/api/src/model/things.model.d.ts:39:9 - error TS2611: 'id' is defined as a property in class 'EntityWithId', but is overridden here in 'Thing' as an accessor.
39     get id(): string;
           ~~
Error: node_modules/@eclipse-ditto/ditto-javascript-client-dom/dist/api/src/model/things.model.d.ts:91:9 - error TS2611: 'id' is defined as a property in class 'EntityWithId', but is overridden here in 'Feature' as an accessor.
91     get id(): string;

Apparently you can only override fields when they are declared as an abstract field to begin with.

[JS] Provide initial Ditto JavaScript client

We would like to have a client published to npm in order to interact with digital twins via a Eclipse Ditto backend.

[JS] CORS error in Firefox when POST is answered with redirect

Hello everyone,
yesterday I encountered a strange bug where the HTTP POST request for sending a message to a feature.

The initial request is

POST /cloakedapi/2/things/c0001:CE6C2B4F3681A948/features/GridFlowTemperature/inbox/messages/history HTTP/1.1
Host: twin.othermo.de
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0
Accept: */*
Accept-Language: de,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate, br
authorization: Bearer <token>
content-type: application/json
Origin: https://localhost
Content-Length: 182
Connection: keep-alive

Which is answered with a HTTP 303 redirect as follows:

HTTP/2 303 See Other
access-control-allow-credentials: true
access-control-allow-headers: Accept,Authorization,Cache-Control,Content-Type,Content-Length,DNT,If-Match,If-Modified-Since,If-None-Match,Keep-Alive,Origin,User-Agent,X-Requested-With
access-control-allow-methods: GET, POST, PUT, DELETE, OPTIONS
access-control-allow-origin: https://localhost
content-type: application/json
correlation-id: c225de5f-88eb-49e9-b64d-fd27e8f560b0
date: Wed, 13 Jan 2021 10:36:14 GMT
location: https://proxy.domain/c9239733-313b-4759-a3bf-39c47ab0338a
referrer-policy: no-referrer
response-required: false
server: nginx/1.13.12
vary: Accept-Encoding
version: 2
content-length: 2
X-Firefox-Spdy: h2

Firefox now prints a warning in the console that CORS Allow-Origin and Origin do not match, even though they are the exact same string.

Firefox even follows the redirect, and the pre-flight request for the next domain works, but for the actual GET of the URL, the same error occurs. The request promise is rejected and the request appears to be failed to the client.

After some digging, this appears to be a bug in the Firefox fetch implementation, because:

• The exact same code works in Chrome & Edge
• The exact same request worked before through axios which I believe uses XHR
• In current Firefox nightly and dev builds, the code works

There are even bugs in the Firefox Bugzilla describing this problem:

• https://bugzilla.mozilla.org/show_bug.cgi?id=1444278
• https://bugzilla.mozilla.org/show_bug.cgi?id=1482309

Now the question is wether it is worth investing time and effort into a fix for this problem.

Contra:

• This bug is quite niche, it seems to only happen
  • for POST requests that are answered with a redirect
  • In one specific browser with a rather small market share
• It seems to be fixed in Firefox 85, which is going to be released in about 14 days

Pro:

• This behavior seems to be present in firefox since atleast version 58.
• The error will still be present for people using an older verion of Firefox
• There might be other cases where CORS break requests that I just haven't discovered yet.

[JS] Add Bearer-Token based authentication

Ditto currently support basic auth, google JWT and OpenID Connect tokens. In order for the last two to work a Bearer token class is required.

Open Questions

Is it possible in the websocket-implementation of this project, to have custom headers in the websocket connection request? This is required to enable header-based token authentication on websocket connections. Ditto itself supports this mechanism, browser implementations of the websocket protocol don't. This is a requirement to add full bearer-token support for websocket and http APIs.

Implementation

I propose to add a simple first bearer-token class with basic PoC-level functionality similar to the basic auth classes.

I can do this and file a PR.

[JS] Add realtime local clone

It'd be cool if the Ditto SDK could have a local clone of the remote digital twin, that is updated with websocket/sse. That way, the number of http requests could be a lot lower.

One example of this is the firebase firestore. The JS SDK can be set up to work like this, too.

What's your opinion on this? I'm willing to work on it, but it might take a while; it's quite a bit of work, and I don't have much time to spare.

The websocket session expired because the JWT used for authentication has expired

Im am using ditto-client 1.0.0-M3 and OAuth based authentication (client-id, client-secret).
I open a websocket connection to get notified about twin events.
Every 60 minutes, I get the following message:

2019-11-10T09:04:43.028+01:00 Client <e68d7238-32bb-444f-9bd6-dc04dd7e1433:1db28b98-2e94-4b83-a5ef-65580566e28f>: Reconnection is enabled. Reconnecting in <5> seconds ...
2019-11-10T09:04:43.028+01:00 Client <e68d7238-32bb-444f-9bd6-dc04dd7e1433:1db28b98-2e94-4b83-a5ef-65580566e28f>: WebSocket connection to endpoint wss://things.eu-1.bosch-iot-suite.com/ws/1 was closed by Server with code <1000> and reason .
2019-11-10T09:04:43.028+01:00 Client <e68d7238-32bb-444f-9bd6-dc04dd7e1433:1db28b98-2e94-4b83-a5ef-65580566e28f>: Got TWIN ThingErrorResponse: <GatewayWebsocketSessionExpiredException: The websocket session expired because the JWT used for authentication has expired. - You need to periodically refresh your websocket session with a new JWT.>
2019-11-10T09:04:48.029+01:00 Recreating Websocket..
2019-11-10T09:04:48.290+01:00 Connecting WebSocket on endpoint wss://things.eu-1.bosch-iot-suite.com/ws/1
2019-11-10T09:04:48.505+01:00 Client <e68d7238-32bb-444f-9bd6-dc04dd7e1433:1db28b98-2e94-4b83-a5ef-65580566e28f>: WebSocket connection is established
2019-11-10T09:04:48.506+01:00 Client <e68d7238-32bb-444f-9bd6-dc04dd7e1433:1db28b98-2e94-4b83-a5ef-65580566e28f>: Requesting at backend that this client wants to with params <{}>
2019-11-10T09:04:48.506+01:00 Client <e68d7238-32bb-444f-9bd6-dc04dd7e1433:1db28b98-2e94-4b83-a5ef-65580566e28f>: Subscribing again for messages from backend after reconnection

While this is not a big issue, as the ditto-client reconnects automatically after 5 seconds, there is still the risk that I loose incoming events within this 5 seconds interval.
Are there plans to implement the "refreshing" of the JWT ?

[JS]: Thing is missing the definition field

It is returned in the response, but it is never mapped.

Remove Thread.sleep from DittoClientUsageExamples.java ?

I think we should set users on the right path of asynchronous programming by not calling Thread.sleep in the client usage examples.

Add support for Async JsonWebTokenSupplier

Ditto currently supports access token authentication using only synchronous JsonWebTokenSupplier. Refresh token operation is supposed to be asynchronous one and it will be good if a support for asynchronous JsonWebTokenSupplier is provided, e.g. something like

public interface AsyncJsonWebTokenSupplier extends Supplier<CompletionStage<JsonWebToken>> {
  CompletionStage<JsonWebToken> get();
}

Question on usage on Gateway

This is rather a question than an Issue but nonetheless I just post it here.
We are using Apache PLC4X in our java based application (plc4x.apache.org) which could, at the same time serve as Gateway in Eclipse IoT Terminology. So this would be the "client" which identifies with the "thing".
Is the Java Client right for this use case?
And if so, how do I correctly use the "client.live()" functionality? I did not fully get it from the documentation.

Thanks!
Julian

[JS] Acknowledge NO_PROXY env variable in node implementation

The proxy implementation in the NodeJS implementation of the JS client is currently capable of using the HTTPS_PROXY and HTTP_PROXY environment variables and will automatically proxy requests if those are set. However, it ignores the NO_PROXY variable, which e.g. might cause problems in localhost scenarios.

[JS] Responding to messages

I'm not sure if I'm doing something wrong, or if this is a problem in the api.

I'm trying to respond to a message that is sent to a thing.

As the documentation states out here, I have to set the correlation-id header to the same value as the message that I've received.

I currently do not have chance to get the correlation-id of the message.

  const client = DittoNodeClient.newWebSocketClient()
    .withTls()
    .withDomain(domain)
    .withAuthProvider(NodeHttpBasicAuth.newInstance(username, password))
    .withBuffer(1024 * 10)
    .liveChannel()
    .build();

  const messages = client.getMessagesHandle();
  await messages.requestMessages();
  messages.subscribeToThing(thingId, (msg) => {
    console.log(msg.headers); // always undefined
    console.log(msg);
  });

In my example the msg.headers attribute is always undefined. I've had a look into the code and the headers are present but will not get copied into the message:

websocket-request-handler.js:174-196

/**
 * A subscription to messages matching a certain pattern.
 */
class Subscription {
    constructor(_callback) {
        this._callback = _callback;
    }
    /**
     * Calls the callback function with the message provided.
     *
     * @param message - The message to send.
     */
    callback(message) {
        const splittedTopic = message.topic.split('/');
        const action = splittedTopic.length > 0 ? splittedTopic.pop() : '';
        this._callback({
            action,
            topic: message.topic,
            path: message.path,
            value: message.value
        });
    }
}

The debugger states out that the headers are present in message but won't get forwarded to _callback.

Do I need another api to answer messages?

[JS] Missing support for _metadata

As the title implies, the js client is currently lacking support for the _metadta fields.

I am working on a PR to resolve this, but I am currently not 100% sure about the structure of the _metadata field and couldn't find any resource on it.
My working assumption is that _metadata only contains a features property that is structured like the features property of a thing. Is this correct or can there be additional fields inside the _metadata property of a thing?

I'd be great if someone could clarify that for me or point me to a complete description of the ditto schema if one exists!

Add support in Java client for notification/monitoring of connection problems

The Java client establishes a WebSocket connection to the Ditto back end and automatically re-connects this connection in case of problems. This is fine, but it is done invisible to my application that uses the client. This means, that my application cannot monitor such connection problems and alert if they are critical. During re-connects some (optionally) important thing events could also be missed.

It would be good to have a possibility to configure a callback/event handler to get informed about such connectivity problems - even so they are compensated by automatic re-connects.

Things sorting with JS Client

Hi,
I use your Javascript binding (@eclipse-ditto/ditto-javascript-client-dom) for HTTP access to Ditto.

load5ThingsAsc() {
    var options = DefaultSearchOptions.getInstance().withLimit(0,5);
    options = options.withSort("+thingId");
    const p = this.searchHandle.search(options).then(result => console.log("returned",result.items))
  }

does fire a:

http://localhost:8080/api/2/search/things?option=limit(0%2C5)%2Csort(%252BthingId)

this results to a Error 400 with the message "Invalid input '%', expected Asc or Desc". A

...
options = options.withSort("-thingId");
...

works fine.

feels like a bug

Thomas

[JS] error handling in @eclipse-ditto/ditto-javascript-client-dom

Hi,

when I open a ditto-connection with a wrong PW:

this.restclient = DittoDomClient.newHttpClient()
            .withoutTls()
            .withDomain(domain)
            .withAuthProvider(DomHttpBasicAuth.newInstance("ditto","wrongpw))
            .build();

and call the count method:

this.searchHandle = this.restclient.getSearchHandle();
this.searchHandle.count().then(r => {
     console.log(r);
    },e => {
      console.log(e); // this returns undefined
    })

... HTTP returns properly 401 - since I want to distinguish between HTTP errorcodes, I need to interpret the errorcode. But e is returned as undefined.

What do I miss?

Thomas

[Java] Analyze and fix raising memory consumption in client

The DefaultRegistry.pointerCache seems to build up a lot over time.
First guess is this line: https://github.com/eclipse/ditto-clients/blob/master/java/src/main/java/org/eclipse/ditto/client/internal/bus/DefaultRegistry.java#L105

That will cache any "not found" selector in the pointerCache - the backing ConcurrentHashMap however is unlimited which would lead eventually always to memory issues.

Maybe just get rid of the "not found caching"?

[Java] Can't add second consumer on same JsonPointer after events were consumed

Currently, the client doesn't allow a second consumer for the same selection after already events were consumed. Here's how to reproduce the problem:

1. Register consumer A with id "a-id" (+ startConsumption() for the client)
2. Generate events -> A consumes the events
3. Register consumer B with id "b-id"
4. Still only A consumes events
5. Cancel consumer A -> B will now consume events.

Desired functionality:
In step 4, A and B should both consume the same events.

Current implementation:
The client uses the DefaultRegistry class for storing for which JsonPointerSelection which consumers get called. The first time the registry is asked if a JsonPointer matches any of the known selections, it will add the result to a cache. If it is asked again for the same JsonPointer, it will simply return the result of the cache. Since the cache isn't cleared on new registrations, consumer B will never show up in the cached result.

Possible Fix:
This can be fixed by clearing the cache when a new consumer is registered.

[JS] Typings of the thing model

When accessing the properties of a feature in typescript, the compiler complains that

TS2339: Property 'xxx' does not exist on type 'object'.

The quickest workaround here is to add an //@ts-ignore above the line, but that can get bloated quickly.

Another quick solution would be to change the return type of get properies() in the Feature class to any.
A more "proper" approach could be to set the return type to a dictonary, (I assume the properties object is always a dictionary with string keys and arbitrary entries?), but that only moves the problem down one layer in the hierarchy.

What are your thoughts on this?

[JS] Add junit test reports

Add JUnit test reports for the javascript tests.
Can be done using the jest-junit reporter.

[Java] Javadoc of `register...` methods should refer to `startConsumption` methods

Without calling startConsumption the register... methods have no effect.

[java] add desiredProperties CRUD support

In order to use the ditto-client to manage desired feature properties, the operations have to be executed via creating ditto-protocol messages manually in the client link
In communication pattern when properties are set by device only and desiredProperties are set by server - special CRUD operations for desired feature properties are missing in the client.

Would be great to add desiredProperties support for java client to keep it in sync with ditto-protocol.

java ditto-client threads leakage

Steps executed:

• Open java ditto-client
• call registerForThingChanges
• destroy the client

Every DittoClients seems to occupy ~10 threads in this scenario
When all clients are destroyed, the following threads are still open

Number of opened clients	Number of opened threads	Number of ditto-client--adaptable-bus threads after destroy	Number of pool-X-thread-Y threads after destroy
10	102	22	10
20	202	42	20
100	1002	202	100

The district thread stacks are as follow (if needed I can upload the whole stack trace)

"ditto-client--adaptable-bus-74ef6b92-2a09-4b4f-ad24-c3b3689880c7-1" Id=1406 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2f5ac102
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2f5ac102
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	...

"ditto-client--adaptable-bus-0d61b4fa-984c-4bdf-abf0-f06f951e78f8-2" Id=1396 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@895416d
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@895416d
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	...

"pool-99-thread-1" Id=1394 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@71a06021
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@71a06021
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

[Java] Support token based authentication

Currently the java client SDK only supports basic authentication against a Ditto backend, but the backend supports authentication via OpenID Connect.
We should provide an implementation for token based authentication in the client SDK.