Coder Social home page Coder Social logo

tutorial-resources's Introduction

Tractus-X Tutorial

1. MXD

Wanna deploy your own Tractus-X Connector? Or even your own dataspace? Go check out this tutorial to learn how!

2. EDC Policy Playground

Wanna play with Tractus-X EDC Policies? Run locally the Playground or go to the hosted version here

tutorial-resources's People

Contributors

ciprian-nicuta avatar dependabot[bot] avatar eclipse-tractusx-bot avatar hemantxpatel avatar ieuna avatar jkbquabeck avatar moiarayam avatar ndr-brt avatar paullatzelsperger avatar ravinderkumarsap avatar s3bb1 avatar tomaszbarwicki avatar wadahiro avatar wolf4ood avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

tutorial-resources's Issues

chore: Organize Tutorials in mxd/docs directory

Description

Organize all tutorials into mxd/docs directory and provide tutorial link in the mxd/README.md.

  • Move Access policy tutorial from mxd to mxd/docs dir.
  • Move EDR tutorial from main mxd/README.md to a new tutorial file in mxd/docs.
  • Move Trudy tutorial from main mxd/README.md to a new tutorial file in mxd/docs.
  • Add a tutorial section in mxd/README.md and add links to all tutorials.
  • Cleanup mxd/README.md sections which are not needed.

Data transfer: extend samples to utilize S3 (minio)

WHAT

The MXD walkthrough should be extended so that one provider connector, e.g. "Alice" hosts an asset, that is actually stored in S3 (using the MinIO container image).
Bob then transfers the data into some arbitrary location, e.g. his own S3 store.

In the course of this issue, it should also be tested, whether a transfer HTTP -> S3 can be achieved.

WHY

Highlighting standard Tx-EDC use cases and making them accessible for users of MXD.

HOW

  • add terraform resources for MinIO (suggestion: use a kubernetes_deployment)
  • create a bucket on provider side and put some test data into that bucket. This needs to be done automatically, without manual commands. (suggestion: could use a job)
  • add relevant documentation to mxd/README.md outlining the process, including the correct curl command and transfer request JSON
  • repeat for transfers HTTP -> S3
  • Suggestion: after each transfer, show the contents of the relevant database tables on provider and consumer side, to further illustrate what's going on.

mxd: pin the `tractusx-edc` version

Description

Currently the version of the EDC relies on the latest branch, this is not optimal because after every tx release the main branch of the mxd could be not working anymore.

The proposal is to:

  • pin the version to the latest (0.6.0)
  • set up dependabot for getting a PR whenever a new tx version gets released
  • make the Run MXD locally run also on PRs, so then every PR must ensure that the changes made don't break the CI

Acceptance Criteria

  • [criteria 1]
  • [criteria 2]
  • [criteria 3]

Additional Information

Integrate Digital Twin Registry into MXD

Why

The Digital Twin Registry (DTR) is a solution which lists all digital twins and references their aspects including information about the underlying asset, asset manufacturer, and access options, like aspect endpoints. Moreover, the DTR is used to register and find data related to DTs. In general, every partner must proactively register their DTs / Aspects to this service in order to reveal their "order" to share respective data sets.

Hence, the DTR represents an enabling component for Catena-X. Within the interdisciplinary E2E User Journey work stream, the integration of the DTR into the MXD was requested.

The major contribution expected from this integration will be the demonstration of how a consumer actually derives its information demand in terms of relevant providers and data assets.

How

  • The DTR shall be integrated based on Helm charts.
  • It needs to be investigated if/what persistence service the DTR requires and if the existing PostgreSQL can be reused.
  • The DTR can be seeded during the setup phase or could be populated with data by the tutorial user to learn more out it.

What

  • Attached are Postman collections demonstrating relevant API calls.
  • These DT KIT docs provide detailed descriptions on API calls.

Threats

  • The DTR could bloat the MXD cluster.
  • The API interaction with the DTR could bloat the tutorial description.

References:

Postman_collections.zip

Feat: Enhance E2E Tests - Http to Http Transfer

Description

Enahance E2E tests to validate a successful http to http transfer via backend-service.

  • Create a dummy content in backend service and get an url.
  • Use above url as a http source while creating asset.
  • Create requests in postman to initiate negotiation.
  • Validate Negotiation state as COMPLETED.
  • Initiate transfer with http destination as backend service.
  • Validate Transfer State as COMPLETED.
  • Validate http transfer via backend service.

Related to eclipse-tractusx/sig-release#316

CI: improve the MXD test

WHAT

Currently, during CI runs, we only test that both bob and alice are up and running, check out this workflow job. That is not very thorough and we should improve it.

At least the following aspects should be tested:

  • create an asset, policy, contract def
  • make a catalog request (bob -> alice, alice -> bob)
  • [optional] perform contract negotiation
  • [optional] perform data transfer

For that, we could use another postman collection and leverage the test feature, or even just plain curl commands.

WHY

Testing should be improved, MXD does indeed serve as a testing ground of sorts.

Bug: Fix Existing E2E Tests

Description

Current E2E tests validation in postman are incorrect.

  • Requests Create Policy BOB_MANAGEMENT_URL / Create Policy Alice_MANAGEMENT_URL use invalid BPN number.
  • Requests Create Contract Definition BOB_MANAGEMENT_URL / Create Contract Definition ALICE_MANAGEMENT_URL use invalid access policy id / contract policy id / asset selector.
  • For requests Get all Policies / Get all Assets, response validation is incorrect. It validates unrelated ids.
  • Query Catalog Response validation script validated unrelated assets.
  • Remove unused postman collection variables.

Related to eclipse-tractusx/sig-release#316

Tutorial for Azure Storage to Azure Storage Transfer

Description

  • Add a tutorial for Azure storage to Azure Storage Transfer
  • Add a postman collection for the same.

Create a common postan collection MXD Blob Transfer API.json (or something similar) and add a folder inside the collection for azure to azure.
So at the end, there will be four folder inside that collection S3 to S3, S3 to Azure, Azure to S3 and Azure to Azure.

Related to eclipse-tractusx/sig-release#315

Enhance S3 to S3 Transfer Tutorial using AWS Temp Credentials (AWS STS Token)

Current Setup

Assuming Alice as a provider connector and Bob being consumer connector.
Bob's shares its actual AWS credential with Alice to initiate the file transfer. Bob should generate an AWS Temporary Token with limited access and expiry. Alice can misuse Bob's credentials.

Issue

MinIO has limited support of STS token. We can generate a STS token using AWS CLI

aws --endpoint-url http://localhost:9000 sts assume-role --policy '{"Version":"2012-10-17","Statement":[{"Sid":"S3Access","Effect":"Allow","Action":"s3:*","Resource":"arn:aws:s3:::*"}]}' --role-arn arn:xxx:xxx:xxx:xxxx --role-session-name anything --duration-seconds $((365 * 86400))

This command returns a response similar to:

{
    "Credentials": {
        "AccessKeyId": "6V6DRU6086RMH7D9LEN0",
        "SecretAccessKey": "PH8eyvr+5G7iydON8t2mgaTXgjH8JOFXlBQsphPa",
        "SessionToken": "eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NLZXkiOiI2VjZEUlU2MDg2Uk1IN0Q5TEVOMCIsImV4cCI6MTczMjk2ODI0MiwicGFyZW50IjoiYWxpY2Vhd3NjbGllbnQiLCJzZXNzaW9uUG9saWN5IjoiZXlKV1pYSnphVzl1SWpvaU1qQXhNaTB4TUMweE55SXNJbE4wWVhSbGJXVnVkQ0k2VzNzaVUybGtJam9pVXpOQlkyTmxjM01pTENKRlptWmxZM1FpT2lKQmJHeHZkeUlzSWtGamRHbHZiaUk2SW5Nek9pb2lMQ0pTWlhOdmRYSmpaU0k2SW1GeWJqcGhkM002Y3pNNk9qb3FJbjFkZlE9PSJ9.9WvwmfUyeES0VxbPPzVH0fOdlh_Y394BcxCscZp4EszliIFqgpwNKaUe-OeE8SKyKHN3QM5IGCrjMPQAv0FAig",
        "Expiration": "2024-11-30T12:04:02+00:00"
    },
    "AssumedRoleUser": {
        "Arn": ""
    }
}

Now this new token is not accepted by MinIO. Alice`s connector-dataplane throws an error:

software.amazon.awssdk.services.s3.model.S3Exception: The Access Key Id you provided does not exist in our records.

Bug: Query Catalog request has invalid request body (querySpec section)

Description

Query Catalog request body is incorrect in tutorials.
Existing:

"querySpec": {
        "offset": 0,
        "limit": 100,
        "filter": "",
        "range": {
            "from": 0,
            "to": 100
        },
        "criterion": ""
    }

filter, range, criterion are unknown fields.

A correct querySpec example could be:

"querySpec": {
        "offset": 0,
        "limit": 50
}

MIW Service fails to start in the cluster

Description

MIW is failing to start while running mxd.

How to reproduce

  • Follow steps mentioned in readme upto terraform apply
  • Check console output for failure.

Additional Info

MIW Pod Startup Logs

Caused by: java.lang.RuntimeException: JsonLdError[code=There was a problem encountered loading a remote context [code=LOADING_REMOTE_CONTEXT_FAILED]., message=There was a problem encountered loading a remote context [code=LOADING_REMOTE_CONTEXT_FAILED].]
	at org.eclipse.tractusx.ssi.lib.proof.transform.LinkedDataTransformer.toDataset(LinkedDataTransformer.java:82) ~[cx-ssi-lib-0.0.16.jar!/:na]
	at org.eclipse.tractusx.ssi.lib.proof.transform.LinkedDataTransformer.transform(LinkedDataTransformer.java:50) ~[cx-ssi-lib-0.0.16.jar!/:na


Caused by: com.apicatalog.jsonld.JsonLdError: Unexpected response code [524]
	at com.apicatalog.jsonld.loader.DefaultHttpLoader.loadDocument(DefaultHttpLoader.java:95) ~[titanium-json-ld-1.3.2.jar!/:na]
	at com.apicatalog.jsonld.loader.HttpLoader.loadDocument(HttpLoader.java:6) ~[titanium-json-ld-1.3.2.jar!/:na]
	at org.eclipse.tractusx.ssi.lib.model.RemoteDocumentLoader.loadDocument(RemoteDocumentLoader.java:95) ~[cx-ssi-lib-0.0.16.jar!/:na]
	at com.apicatalog.jsonld.context.ActiveContextBuilder.fetch(ActiveContextBuilder.java:535) ~[titanium-json-ld-1.3.2.jar!/:na]
	... 81 common frames omitted

MXD: MIW responded with a 404

MIW not working after running the MXD terraform installation
MIW responds with 404
No error logs generated on MIW side
Error seems not to happen on every installation

image

OS: Mac OS M1

After manually redeploying MIW module everything worked
terraform taint kubernetes_deployment.miw

Contract agreement received. Validation failed: Policy in the contract agreement is not equal to the one in the contract offer

Current Behavior

With the lastest mxd version (main 01.03.2024) the contract negotiation request produces an error message in the logfile

negotiate-contract.json

{
	"@context": {
		"odrl": "http://www.w3.org/ns/odrl/2/"
	},
	"@type": "NegotiationInitiateRequestDto",
    "counterPartyAddress": "http://alice-controlplane:8084/api/v1/dsp",
	"protocol": "dataspace-protocol-http",
	"connectorId": "BPNL000000000001",
	"providerId": "BPNL000000000001",
	"offer": {
		"offerId": "OTk5OQ==:OTk5OQ==:MmNhODkwNmMtZTdjZi00ZmU1LTlkNWUtZmUzNzkzYmY3ZGM1",
		"assetId": "9999",
		"policy": {
			"@type": "odrl:Set",
			"odrl:permission": [],
			"odrl:prohibition": [],
			"odrl:obligation": [],
			"odrl:target": "9999"
		}
	}
}

curl -d @negotiate-contract.json -X POST -H "x-api-key: password" -H "content-type: application/json" http://localhost:29193/management/v2/contractnegotiations -s | jq

WARNING 2024-03-01T09:37:43.062145068 The attribute https://w3id.org/edc/v0.0.1/ns/offer has been deprecated in type https://w3id.org/edc/v0.0.1/ns/ContractRequest, please use https://w3id.org/edc/v0.0.1/ns/policy
DEBUG 2024-03-01T09:37:43.063631112 [ConsumerContractNegotiationManagerImpl] ContractNegotiation 206cc3e7-9239-4dc3-8cd4-63aa348d529e is now in state INITIAL
DEBUG 2024-03-01T09:37:44.042691051 [ConsumerContractNegotiationManagerImpl] ContractNegotiation 206cc3e7-9239-4dc3-8cd4-63aa348d529e is now in state REQUESTING
DEBUG 2024-03-01T09:37:44.044702999 ContractNegotiation: ID 206cc3e7-9239-4dc3-8cd4-63aa348d529e. [Consumer] send request
DEBUG 2024-03-01T09:37:44.223446098 ContractNegotiation: ID 206cc3e7-9239-4dc3-8cd4-63aa348d529e. [Consumer] send request
DEBUG 2024-03-01T09:37:44.234796334 [ConsumerContractNegotiationManagerImpl] ContractNegotiation 206cc3e7-9239-4dc3-8cd4-63aa348d529e is now in state REQUESTED
DEBUG 2024-03-01T09:37:44.651674738 DSP: Incoming ContractAgreementMessage for class org.eclipse.edc.connector.contract.spi.types.negotiation.ContractNegotiation process: 206cc3e7-9239-4dc3-8cd4-63aa348d529e
**DEBUG 2024-03-01T09:37:44.716826739 [Consumer] Contract agreement received. Validation failed: Policy in the contract agreement is not equal to the one in the contract offer
**DEBUG 2024-03-01T09:37:44.717233287 DSP: Service call failed: Contract agreement received. Validation failed: Policy in the contract agreement is not equal to the one in the contract offer****

Expected Behavior

In the version from 23.02.2024 the same request works

DEBUG 2024-03-01T09:15:58.366822896 [TransferProcessManagerImpl] TransferProcess c8d529b0-6ef6-4421-aa3d-e096ba17da3e is now in state INITIAL
DEBUG 2024-03-01T09:15:58.373030391 Process c8d529b0-6ef6-4421-aa3d-e096ba17da3e is now INITIAL
DEBUG 2024-03-01T09:15:58.417321893 Transfer Process created c8d529b0-6ef6-4421-aa3d-e096ba17da3e
DEBUG 2024-03-01T09:15:59.489493191 [TransferProcessManagerImpl] TransferProcess c8d529b0-6ef6-4421-aa3d-e096ba17da3e is now in state PROVISIONING
DEBUG 2024-03-01T09:15:59.517926534 TransferProcess: ID c8d529b0-6ef6-4421-aa3d-e096ba17da3e. Provisioning
DEBUG 2024-03-01T09:15:59.529742438 [TransferProcessManagerImpl] TransferProcess c8d529b0-6ef6-4421-aa3d-e096ba17da3e is now in state PROVISIONED
DEBUG 2024-03-01T09:15:59.559497548 [TransferProcessManagerImpl] TransferProcess c8d529b0-6ef6-4421-aa3d-e096ba17da3e is now in state REQUESTING
DEBUG 2024-03-01T09:15:59.570513999 TransferProcess: ID c8d529b0-6ef6-4421-aa3d-e096ba17da3e. send transfer request to http://alice-controlplane:8084/api/v1/dsp
DEBUG 2024-03-01T09:16:01.471093801 TransferProcess: ID c8d529b0-6ef6-4421-aa3d-e096ba17da3e. send transfer request to http://alice-controlplane:8084/api/v1/dsp
DEBUG 2024-03-01T09:16:01.530304848 [TransferProcessManagerImpl] TransferProcess c8d529b0-6ef6-4421-aa3d-e096ba17da3e is now in state REQUESTED
DEBUG 2024-03-01T09:16:02.866885159 DSP: Incoming TransferStartMessage for class org.eclipse.edc.connector.transfer.spi.types.TransferProcess process: c8d529b0-6ef6-4421-aa3d-e096ba17da3e
DEBUG 2024-03-01T09:16:05.04394783 TransferProcess c8d529b0-6ef6-4421-aa3d-e096ba17da3e is now in state STARTED

Steps To Reproduce

Create Asset

create-asset.json

{
    "@context": {},
    "@id": "9999", 
    "properties": {
        "description": "AssetAdministrationShell"
    },
    "dataAddress": {
        "@type": "DataAddress",
        "type": "HttpData",
        "baseUrl": "http://host.minikube.internal:4001/aasServer/shells",
		"proxyPath": "true"
    }
}

curl -d @create-asset.json -H "x-api-key: password" -H "content-type: application/json" http://localhost:19193/management/v3/assets -s | jq

Create Policy
create-policy.json

{
  "@context": {
    "@vocab": "https://w3id.org/edc/v0.0.1/ns/",
    "odrl": "http://www.w3.org/ns/odrl/2/"
  },
  "@id": "9999",
  "policy": {
    "@type": "odrl:Set",
    "odrl:permission": [],
    "odrl:prohibition": [],
    "odrl:obligation": []
  }
}

curl -d @create-policy.json -H "x-api-key: password" -H "content-type: application/json" http://localhost:19193/management/v2/policydefinitions -s | jq

Create Contract Definition
create-contract-definition.json

{
    "@context": {},
    "@id": "9999",
    "@type": "ContractDefinition",
    "accessPolicyId": "9999",
    "contractPolicyId": "9999",
    "assetsSelector" : []
    }
}

curl -d @create-contract-definition.json -H "x-api-key: password" -H "content-type: application/json" http://localhost:19193/management/v2/contractdefinitions -s | jq

Fetch Catalog
fetch-catalog.json

{
    "@context": {
        "edc": "https://w3id.org/edc/v0.0.1/ns/"
    },
    "@type": "CatalogRequest",
    "counterPartyAddress":"http://alice-controlplane:8084/api/v1/dsp",
    "protocol": "dataspace-protocol-http",
    "querySpec": {
        "offset": 0,
        "limit": 50
    }
}

curl -d @fetch-catalog.json -H "x-api-key: password" -H "content-type: application/json" http://localhost:29193/management/v2/catalog/request -s | jq

Negotiate Contract
negotiate-contract.json

{
	"@context": {
		"odrl": "http://www.w3.org/ns/odrl/2/"
	},
	"@type": "NegotiationInitiateRequestDto",
    "counterPartyAddress": "http://alice-controlplane:8084/api/v1/dsp",
	"protocol": "dataspace-protocol-http",
	"connectorId": "BPNL000000000001",
	"providerId": "BPNL000000000001",
	"offer": {
		"offerId": "OTk5OQ==:OTk5OQ==:MmNhODkwNmMtZTdjZi00ZmU1LTlkNWUtZmUzNzkzYmY3ZGM1",
		"assetId": "9999",
		"policy": {
			"@type": "odrl:Set",
			"odrl:permission": [],
			"odrl:prohibition": [],
			"odrl:obligation": [],
			"odrl:target": "9999"
		}
	}
}

curl -d @negotiate-contract.json -X POST -H "x-api-key: password" -H "content-type: application/json" http://localhost:29193/management/v2/contractnegotiations -s | jq

Policy output check

The main structure of the policy playground is there. We should ensure that the JSON-LD generated is compliant and working with the Tx-EDC version 0.5.1

Playground: introduce checks on PRs

Even though the Playground is in a WIP state, we should introduce some checks on PRs for contributions like

  • running test (not many at this time)
  • running linter/checkstyle

Remove /v2/assets API Usage from MXD

Description

/v2/assets API has been deprecated and it has been removed in latest tractus-connector release.
Need to upgrade to /v3/assets API.

Changes Needed:

  • Update all Postman APIs to use latest /v3/assets API.
  • Update all examples in README.md / tutorials.

Use different BPNs for all participants

WHAT

Currently, all participants in the example dataspace have the same BPN, and by extension, the same client-id in Keycloak (BPNL000000000000).

While that is not technically a problem, it is does not really reflect an actual use case. Every participant (Bob, Alice, the MIW) should have their own client-id/BPN.

WHY

We cannot credibly illustrate any real scenario unless all participants are actually separated, auth-wise. We'll need this to show and explore how policies work, what effect they have, etc.

HOW

  1. update Keycloak: update the Keycloak realm file (keycloak/miw_test_realm.json) manually or through the Keycloak admin UI.
  2. seed MIW data: every participant should have their wallet seeded in the MIW database upon launch.

MXD initial `terraform apply` throws error

Current Behavior

Error while running Terraform:

kubernetes_job.seed_connectors_via_mgmt_api: Still creating... [40s elapsed]
kubernetes_job.seed_connectors_via_mgmt_api: Still creating... [50s elapsed]
kubernetes_job.seed_connectors_via_mgmt_api: Still creating... [1m0s elapsed]
╷
│ Error: job: default/seed-connectors is not in complete state
│ 
│   with kubernetes_job.seed_connectors_via_mgmt_api,
│   on seed_data.tf line 24, in resource "kubernetes_job" "seed_connectors_via_mgmt_api":
│   24: resource "kubernetes_job" "seed_connectors_via_mgmt_api" {

Expected Behavior

Kubernetes cluster gets fully initialized

Steps To Reproduce

Follow the instructions in https://github.com/eclipse-tractusx/tutorial-resources/blob/main/mxd/README.md until terraform apply throws the error.
Running terraform apply again results in the same error.

Running `kubectl logs job/seed-connectors` while Terraform is deploying:
Defaulted container "newman-bob" out of: newman-bob, newman-alice, newman-miw
newman

MXD Management API Seed

❏ SeedData
↳ Create Business Partner Group
  POST http://10.96.144.238:8081/management/business-partner-groups [409 Conflict, 234B, 44ms]
  ✓  Status code is 204 (if new BPN) or 409 Conflict (if BPN already exists)

↳ Create Asset 1
  POST http://10.96.144.238:8081/management/v3/assets [409 Conflict, 217B, 8ms]
  ✓  Status code is 204 No Content (if new asset) or 409 Conflict (if asset already exists)

↳ Create Asset 2
  POST http://10.96.144.238:8081/management/v3/assets [409 Conflict, 217B, 8ms]
  ✓  Status code is 204 No Content (if new asset) or 409 Conflict (if asset already exists)

↳ Create Policy
  POST http://10.96.144.238:8081/management/v2/policydefinitions [400 Bad Request, 1.01kB, 7ms]
  1. Status code is 204 No Content (if new policy) or 409 Conflict (if policy already exists)

↳ Create Contract Definition 1
  POST http://10.96.144.238:8081/management/v2/contractdefinitions [409 Conflict, 231B, 7ms]
  ✓  Status code is 204 No Content (if new contract definition) or 409 Conflict (if contract deifinition already exists)

↳ Create Contract Definition 2
  POST http://10.96.144.238:8081/management/v2/contractdefinitions [409 Conflict, 231B, 7ms]
  ✓  Status code is 204 No Content (if new contract definition) or 409 Conflict (if contract deifinition already exists)

┌─────────────────────────┬──────────────────┬──────────────────┐
│                         │         executed │           failed │
├─────────────────────────┼──────────────────┼──────────────────┤
│              iterations │                1 │                0 │
├─────────────────────────┼──────────────────┼──────────────────┤
│                requests │                6 │                0 │
├─────────────────────────┼──────────────────┼──────────────────┤
│            test-scripts │               18 │                0 │
├─────────────────────────┼──────────────────┼──────────────────┤
│      prerequest-scripts │               12 │                0 │
├─────────────────────────┼──────────────────┼──────────────────┤
│              assertions │                6 │                1 │
├─────────────────────────┴──────────────────┴──────────────────┤
│ total run duration: 299ms                                     │
├───────────────────────────────────────────────────────────────┤
│ total data received: 1.45kB (approx)                          │
├───────────────────────────────────────────────────────────────┤
│ average response time: 13ms [min: 7ms, max: 44ms, s.d.: 13ms] │
└───────────────────────────────────────────────────────────────┘

  #  failure         detail                                                                                   
                                                                                                              
 1.  AssertionError  Status code is 204 No Content (if new policy) or 409 Conflict (if policy already exists) 
                     expected 400 to be one of [ 200, 204, 409 ]                                              
                     at assertion:0 in test-script                                                            
                     inside "SeedData / Create Policy"

Transfer fails

Current Behavior

Transfer doesn't complete.

Expected Behavior

Transfer should be completed successfully.

Steps To Reproduce

  • Initiate negotiation.
  • Validate Negotiation state FINALIZED.
  • Initiate transfer.
  • Check Transfer State.

Related to: eclipse-edc/Connector#3810

Possible Fix

  • Generate a private / public key pair and store it in connector's vault.
  • Define vault keys as connector config. Check config details here.

Update Initiate Negotiation Request - offer field has been deprecated

Description

  • Switch to policy field instead of offer while initiating negotiation and EDR.
  • Switch to counterPartyAddress instead of connectorAddress while initiating negotiation, EDR and transfer.

Acceptance Criteria

  • Postman Collections should be updated [API, E2E, Seed]
  • Tutorials should be updated.

Additional Information

Deprecation Logs:

WARNING The attribute https://w3id.org/edc/v0.0.1/ns/connectorAddress has been deprecated in type https://w3id.org/edc/v0.0.1/ns/ContractRequest, please use https://w3id.org/edc/v0.0.1/ns/counterPartyAddress
WARNING The attribute https://w3id.org/edc/v0.0.1/ns/offer has been deprecated in type https://w3id.org/edc/v0.0.1/ns/ContractRequest, please use https://w3id.org/edc/v0.0.1/ns/policy
WARNING The attribute https://w3id.org/edc/v0.0.1/ns/connectorAddress has been deprecated in type https://w3id.org/edc/v0.0.1/ns/TransferRequest, please use https://w3id.org/edc/v0.0.1/ns/counterPartyAddress

Seed Connector and E2E Test Fails

Current Behavior

  1. Seed Connector Job fails while deploying MXD.
  2. E2E Test fails.

Expected Behavior

  1. Both shouldn't fail.

Steps To Reproduce

  • Run terraform apply
  • Montior output and Seed Connector logs.

Reason

Create Policy Request fails with the below error:

[
  {
    "message": "Property '@type' was invalid: unknown. Error was: Invalid type https://w3id.org/edc/v0.0.1/ns/Policy for ODRL policy, should be one of [http://www.w3.org/ns/odrl/2/Set, http://www.w3.org/ns/odrl/2/Offer, http://www.w3.org/ns/odrl/2/Agreement]",
    "type": "InvalidRequest",
    "path": null,
    "invalidValue": null
  },
  {
    "message": "ARRAY must be org.eclipse.edc.policy.model.Policy but was: unknown",
    "type": "InvalidRequest",
    "path": null,
    "invalidValue": null
  }
]

Fix

Use "@type": "policy" "@type": "odrl:Set" inside policy object of the request body.

Wrong port for postgres data base

Description

The port, that is forwarded for accessing the postgres data base in the tutorial seems to be wrong.
From my testing, the port opened inside the postgres container is 5432 and not 5423, what is implied by the docs.
Therefore the command:

kubectl port-forward postgres-5b788f6bdd-bvt9b 5432:5423

should be changed to:

kubectl port-forward postgres-5b788f6bdd-bvt9b 5432:5432

Update Trudy Tutorial to include S3 / Azurite Keys

Description

Trudy Tutorial currently fails due to MinIO / Azurite configs not provided to connector.
Its terraform file should be updated to add these configs.

Acceptance Criteria

  • [] Terraform configs should be updated.

Additional Information

Policy JSON-LD output: Supports multiple formats

The JSON-LD output of the playground could be formatted in multiple ways depending on the context or
alias usage. For now we should have at least two output format

  • plain (without usage of prefix)
  • prefix (eg. odrl: )

EDC doesn't come up with terraform

Current Behavior

I tried to deploy (local) the MXD as described in the tutorial (basically the same as in your readme here). Everything went fine. But terraform apply doesn't work.

Both control planes dont start
image

For the pod logs, you can find more information
image

Expected Behavior

Both control planes should come up :) @paullatzelsperger already provided an interim fix for this problem, but its just a local fix and needs to be adapted here in this repository. main.tf file

image

Steps To Reproduce

Just follow the tutorial, or the readme within in the MXD folder. What i additional did, I changed the uses EDC Version (control plane and data plane) to version 0.5.3 instead of the latest.

Bug: Fix Bob Connector MinIO Config

Description

MinIO credentials are not getting passed to bob connector, hence it picks the default values for MinIO credentials.

Context

MinIO credentials get passed correctly for Alice connector but it got mixed up while resolving merge conflict in PR #147

Use services names instead of IPs

Some configuration rely directly on IPs instead of service names. It would be cool to switch to names especially
for connector to connector communication, where the URL it's also used in our postman collection

Example:

ssi-config = {
miw-url = "http://${local.miw-ip}:${var.miw-api-port}"
miw-authorityId = "BPNL000000000000"
oauth-tokenUrl = "http://${kubernetes_service.keycloak.spec.0.cluster_ip}:${var.keycloak-port}/realms/miw_test/protocol/openid-connect/token"
oauth-clientid = "miw_private_client"
oauth-secretalias = "client_secret_alias"
oauth-clientsecret = "miw_private_client"

env : {
"TX_SSI_ENDPOINT_AUDIENCE" : "http://${kubernetes_service.controlplane-service.spec.0.cluster_ip}:8084/api/v1/dsp"
"EDC_DSP_CALLBACK_ADDRESS" : "http://${kubernetes_service.controlplane-service.spec.0.cluster_ip}:8084/api/v1/dsp"
}

Enable readiness / liveness probe for keycloak

WHAT

Keycloak deployment takes time to start the server, but Kubernetes assumes it is ready / live once container gets created.

WHY

Seed container job fails since it tries to access keycloak which is not yet ready to serve traffic. terraform apply command throws error sometime.

kubernetes_job.seed_connectors_via_mgmt_api: Creating...
kubernetes_job.seed_connectors_via_mgmt_api: Still creating... [10s elapsed]
kubernetes_job.seed_connectors_via_mgmt_api: Still creating... [20s elapsed]
kubernetes_job.seed_connectors_via_mgmt_api: Still creating... [30s elapsed]
kubernetes_job.seed_connectors_via_mgmt_api: Still creating... [40s elapsed]
kubernetes_job.seed_connectors_via_mgmt_api: Still creating... [50s elapsed]
kubernetes_job.seed_connectors_via_mgmt_api: Still creating... [1m0s elapsed]
╷
│ Error: job: default/seed-connectors is not in complete state
│
│   with kubernetes_job.seed_connectors_via_mgmt_api,
│   on seed_data.tf line 24, in resource "kubernetes_job" "seed_connectors_via_mgmt_api":
│   24: resource "kubernetes_job" "seed_connectors_via_mgmt_api" {
│
╵

HOW

Enable keycloak deployment liveness / readiness probe using keycloak health check APIs (/health/ready, /health/live).
https://www.keycloak.org/server/health

Create a http backend service to validate the transfer

Description

Create a REST API Application to complete and validate a successful asset transfer.

  • Create a POST API /api/v1/contents to store an assets. It should return a url / id of this content in the response.
  • Create a GET API /api/v1/contents/{id} to fetch the content. This URL will be used as a DataAddress in the assets API.
  • Create a POST API /api/v1/transfers to accept the transfer data from the connector. Connector will push data similar to this:
{
  "id": "<transfer id>",
  "endpoint": "http://alice-tractusx-connector-dataplane:8081/api/public",
  "authKey": "Authorization",
  "authCode": "<Auth Code>",
  "properties": {}
}

Persist above data with transfer id as primary key.

  • Create a GET API /api/v1/transfers/{id} to return the above json pushed by connector.
  • Create a GET API /api/v1/transfers/{id}/contents to return the actual assets content (i.e. the data which is stored at the above endpoint http://alice-tractusx-connector-dataplane:8081/api/public)

Common Points:

  • No flavour of spring or other related framework to be used. Take Connector's Data Management APIs as a reference.
  • Use Postgres as a persistent storage.

Connector to Connector Transfer Flow:

  • Store a dummy content into http backend service and get an url for this content.e.g http://backend-service:8080/api/v1/contents/1.
  • Create an asset with url: http://backend-service:8080/api/v1/contents/1 on provider connector.
  • Create a policy and contract definition on provider connector.
  • Initiate negotiation using consumer connector.
  • Initiate Transfer with backend service url as http://backend-service:8080/api/v1/transfers
  • Provider Connector transfers the data.
  • Call backend service API /api/v1/transfers/{id} to check if transfer is successful or not. (Optional Step)
  • Call backend service API /api/v1/transfers/{id}/contents to fetch the assets data and compare with the dummy data stored in first step. It should match.
  • Update Backend Service Urls in existings postman collection.

Related to eclipse-tractusx/sig-release#314

Get Asset Data via Consumer Proxy Request fails

Current Behavior

Proxy call to consumer dataplane fails, as consumer dataplane tries to reach out to the provider's public url http://localhost/alice/api/public which doesn't exist.

curl --location 'http://localhost/bob/proxy/aas/request' \
--header 'Content-Type: application/json' \
--header 'X-Api-Key: password' \
--data '{
    "assetId": "1",
    "providerId": "BPNL000000000001"
}'

Above curl returns

'java.net.ConnectException: Failed to connect to localhost/[
  0: 0: 0: 0: 0: 0: 0: 1
]: 80'

Expected Behavior

Consumer Proxy Url should return the asset data.

Steps To Reproduce

  • Initiate an EDR Negotiation
  • Make above Proxy Request

Cause

This PR #233 adds the local ingress url as dataplane public url, due to this an EDR Response looks like this.

{
  "@type": "DataAddress",
  "type": "EDR",
  "authCode": "auth code",
  "endpoint": "http://localhost/alice/api/public",
  "id": "2a17a657-479a-454c-ab42-52078c2824b8",
  "authKey": "Authorization",
  "@context": {}
}

When Consumer Dataplane tries to reach out to the provider's public url http://localhost/alice/api/public, it fails.

Proposed Solution

PR #233 should be reverted.

bug: MXD on Cloud - KeyCloak / MIW has static IPs

Current Behavior

MIW and KeyCloak require static IP which are provided as a constant in terraform config.
Current IPs are in range 10.96.. which works on a kubernetes deployed on local system, but kubernetes deployed on some clouds
doesn't allow IPs in range. 10.96.*.*.
Some providers allow IPs in range 100.104.*.*.

Expected Behavior

MXD should work on any kubernetes server irrespective of where it is deployed.

Steps To Reproduce

  • Run terraform init
  • Run terrafrom apply
  • Observe MIW / KeyCloak Deployment failures.

Proposed Solution

Make static IPs as variable instead of constant, so that its value can be overridden while running terraform apply.

terraform plan -var-file="override.tfvars" # override via a property file
# or
terraform apply -var miw-ip="x.x.x.x" -var keycloak-ip="x.x.x.x" 

doc: Tutorial For Business Partner Group Policy

Description

Tutorial for how to use Business Partner Group Policy

Acceptance Criteria

  • A tutorial for asset transfer including CRUD for business partner group.
  • Update Postman collection to use business partner group

Additional Information

BusinessPartnerPermissionFunction is marked as deprecated and is scheduled to be removed in 0.6.x version.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.