fiware / ngsi-ld_testsuite

:bar_chart: This repository contains tests to verify compliance of NGSI-LD implementations

NGSI-LD Test Suite

This project is intended to define a test suite for NGSI-LD, the evolution of NGSIv2 for supporting Linked Data based on JSON-LD.

Test Summary

• contextProvision. Section 5.6
• contextConsumption. Section 5.7
• contextSubscription. Section 5.8
• contextSourceRegistration. Section 5.9
• batchOperations. Section 5.6

How to execute the tests

Prerequisites

• node version 10.17, docker-compose version 1.26

npm install
# Example: http://localhost:1026 for Orion, http://localhost:9090 for Scorpio
export TEST_ENDPOINT=<Your NGSI-LD Broker endpoint>
# Example: http://localhost:3000
export ACC_ENDPOINT=<The accumulator endpoint>
# Example: http://host.docker.internal:3000/acc
export NOTIFY_ENDPOINT=<The notification endpoint>

# Start the accumulator and services under test - e.g. to start the docker-compose file
./services [orion|scorpio]

# Run the test suite
npm test

• the test endpoint is http://localhost:1026 By default
• the accumulator endpoint is http://localhost:8085 By default
• the notification endpoint is http://accumulator:8085/acc By default

Please note, that if you are running the NGSI-LD accumulator endpoint locally, it can be referred to as http://host.docker.internal:<port>/acc, so that other Docker containers can get access to the accumulator server running on the host.

Note: To run the accumulator locally you can start the service using

export WEB_APP_PORT=8085
node accumulator/accumulator.js

See also

• https://forge.etsi.org/gitlab/NGSI-LD/NGSI-LD/tree/master/schema
• https://forge.etsi.org/gitlab/NGSI-LD/NGSI-LD/tree/master/examples
• https://forge.etsi.org/gitlab/NGSI-LD/NGSI-LD/tree/master/spec/

ngsi-ld_testsuite's Issues

Test Number 1 and 3 are not implemented

It expected 200 but received 400 because the Code not implemented

notification_test problem

Hi,

the notification tests are somehow not properly working as you can see here
https://travis-ci.org/github/ScorpioBroker/ScorpioBroker

There notifications send and received by the accumulator but when the test wants to access them it fails with undefined.

Regards
Benni

Notification_not_sent_test fails on remote scorpio setup

Hi!

So I have a remote scorpio server and the rest is local on my machine. The notification_not_sent_test.js test can't pass...

I tried to set jest.setTimeout(30000); but the problem is the same.
Is there something I need to tweak for a remote server?

Thanks.

The following tests will be excluded: 999
The Accumulator endpoint listens on http://localhost:8085
Subscriptions will notify the accumulator using http://localhost:8085/acc
NGSI-LD Broker endpoint ... at http://X.X.X.X.156:9090
 FAIL  notifications/notification_not_sent_test.js (37.465s)
  Subscription yields to no Notification. JSON
    ✕ should not send a notification. Subscription to specific attribute. Update other 149 (5455ms)
    ✕ should not send a notification. Subscription to entity which id does not match 150 (5183ms)
    ✕ should not send a notification. Subscription to idPattern does not match 151 (5167ms)
    ✕ should not send a notification. Subscription to entity which type does not match 152 (5109ms)
    ✕ should not send a notification. Watched attribute does not exist 153 (5154ms)
    ✕ should not send a notification. Subscription is not active 154 (5137ms)
    ✕ should not send a notification. Subscription has expired 155 (5364ms)

  ● Subscription yields to no Notification. JSON › should not send a notification. Subscription to specific attribute. Update other 149

    Timeout - Async callback was not invoked within the 5000ms timeout specified by jest.setTimeout.Error: Timeout - Async callback was not invoked within the 5000ms timeout specified by jest.setTimeout.

      at mapper (node_modules/jest-jasmine2/build/queueRunner.js:25:45)

....
....

Test Suites: 5 failed, 32 passed, 37 total
Tests:       16 failed, 153 passed, 169 total
Snapshots:   0 total
Time:        94.196s, estimated 100s

test 109 is not a valid relationship

Hi,

test 109 in entity creation tries something which is not valid. Relationships can only have one object. MultiEntries would be
AR1: [{
type: 'Relationship',
object: 'urn:ngsi-ld:T:1234'
},
{
type: 'Relationship',
object: 'urn:ngsi-ld:T:5678'
}
]

Subscription tests have intermittent failures

it seems there are race conditions between the tests and the Accumulator server

test 005 is also a bad request

Hi,
in query_entities_error_tests in test 005 you are checking for invalid request because of the bad accept header. the way you are doing it by get /ngsi-ld/entities is also a bad request because it's missing at least a type.
Since there is not really an order in which errors should appear i would suggest to add some ?type to make it a valid request from this pov.

Benni

Full resource URI on HTTP Response (entity creation)

Hi
when creating a new entity, the spec says that upon success, the HTTP response shall include a "Location" HTTP header that contains the resource URI of the created entity resource.
When I run my tests, I get many errors because the test suite expects only the path segment of the URI and not the full resource URI. Could you please check if this is ok? Thank you

Expected the object:
      {"connection": "close", "content-type": "application/json;charset=UTF-8", "date": "Wed, 28 Nov 2018 12:21:42 GMT", "location": "http://10.0.2.15:9090/ngsi-ld/v1/entities/urn:ngsi-ld:T:1543407698215", "transfer-encoding": "chunked"}
    To have a nested property:
      "location"
    With a value of:
      "/ngsi-ld/v1/entities/urn:ngsi-ld:T:1543407698215"
    Received:
      "http://10.0.2.15:9090/ngsi-ld/v1/entities/urn:ngsi-ld:T:1543407698215"

Regards

Update reame

The instructions in the readme (./services start) do not work.

They should be updated to ./services orion or ./services scorpio. (Other errors might also be in there)

Furthermore, a minimum version of docker-compose is needed to be able to run the script. It does not work on 1.17.1 but does work with 1.25.4 so the minimum version is somewhere between those 2 versions.

Async behaviour of the test suite

Hello,

We are currently trying a custom implementation of NGSI-LD and try to validate it with this test suite. However, we're suspecting that some tests are failing because our implementation is async in nature (a POST and a GET directly following each other will not necessarily result in you seeing the change, there might be a small delay).

Is this assumption valid? Will this test suite basically always fail when there is an inherent delay to the NGSI-LD implementation? And a follow-up question: if it always fails, does it make sense to make the test suite 'async aware', by building in a predetermined delay for each HTTP call for example?

All the best, thanks!

Question on expected attributes when retrieving a subscription (test 165)

Hi,

When I run the test 165, I get this error on the returned payload for the subscription:

FAIL contextSubscription/create_subscription_test.js
  Create Subscription. JSON
    ✕ should create subscription with Q Parameter 165 (105ms)

  ● Create Subscription. JSON › should create subscription with Q Parameter 165

    expect(received).toEqual(expected) // deep equality

    - Expected
    + Received

    @@ -3,10 +3,11 @@
          Object {
            "type": "Vehicle",
          },
        ],
        "id": "urn:ngsi-ld:Subscription:mySubscription1609080232571",
    +   "isActive": true,
        "notification": Object {
          "attributes": Array [
            "numberOfItems",
            "stocks",
            "locatedIn",
    @@ -14,10 +15,11 @@
          "endpoint": Object {
            "accept": "application/json",
            "uri": "http://tutorial:3000/subscription/low-stock-store001",
          },
          "format": "keyValues",
    +     "timesSent": 0,
        },
        "q": "numberOfItems<10;locatedIn==urn:ngsi-ld:Store:001",
        "type": "Subscription",
        "watchedAttributes": Array [
          "speed",

I was wondering why those 2 attributes (isActive and timesSent) are considered to be faulty. In the specification, I can't see something saying they should not be included. For timesSent, it is even mentioned that it is provided by the system when querying the details of a subscription

Expected and Received label inverted

Hi, I just noticed that the label "Received" and "Expected" are inverted in the TestSuite output.
For example, I m looking for the test number "057" and its output is that:

Thank you