Coder Social home page Coder Social logo

xmidt-org / xmidt Goto Github PK

View Code? Open in Web Editor NEW
17.0 22.0 19.0 122 KB

Highly scalable pipes for communicating with devices all over the place.

License: Apache License 2.0

Makefile 9.97% Go 29.54% Dockerfile 24.93% Shell 35.57%
xmidt hacktoberfest

xmidt's Introduction

XMiDT

Xmidt (pronounced "transmit") is a combination of a server cluster and client that provide a highly available data path to devices deployed all over the world.

Summary

A device connects to the XMiDT cloud via the XMiDT client. Cloud based services can use the XMiDT cloud api to send and receive messages from the devices without needing to worry about where the device is in the world or in the network. Push and pull data from the device with ease.

Table of Contents

Code of Conduct

This project and everyone participating in it are governed by the XMiDT Code Of Conduct. By participating, you agree to this Code.

XMiDT Servers

Caduceus provides the pub-sub message delivery mechanism for XMiDT.

Build Status codecov.io Go Report Card Code Climate Issue Count Apache V2 License Release

Scytale accepts the inbound requests, fans out across data centers and delivers the messages to the Talaria machines that could be hosting the device connection.

Build Status codecov.io Go Report Card Code Climate Issue Count Apache V2 License Release

Talaria maintains the secure websocket connections from the device and passes the messages from or to the device.

Build Status codecov.io Go Report Card Code Climate Issue Count Apache V2 License Release

Petasos helps reduce the load on the Talaria machines during mass reboot cases by calculating which specific Talaria a device should connect to & redirecting the incoming request.

Build Status codecov.io Go Report Card Code Climate Issue Count Apache V2 License Release

Mnemosyne provides a mapping between names of devices.

XMiDT Client

Parodus is the light weight client that reaches out to the XMiDT cloud to establish the connection.

Build Status codecov.io Coverity Apache V2 License Release

Server Support Libraries

Build Status codecov.io Go Report Card Release

Build Status codecov.io Go Report Card Release

Build Status codecov.io Go Report Card Release

Build Status codecov.io Go Report Card Release

Build Status codecov.io Go Report Card Release

Client Support Libraries

Build Status codecov.io Coverity Apache V2 License Release

Build Status codecov.io Coverity Apache V2 License Release

Build Status codecov.io Coverity Apache V2 License Release

Build Status codecov.io Coverity Apache V2 License Release

Build Status codecov.io coverity Release

xmidt's People

Contributors

equanox avatar florianbachmann avatar joe94 avatar kcajmagic avatar kraj avatar kristinapathak avatar mtrinh11 avatar sadhyama avatar schmidtw avatar selvamkrish avatar ulan08 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

xmidt's Issues

Unable to run deploy.sh for docker-compose

We are not able successfully run the deploy.sh as the following error is seen:

[root@osboxes docker-compose]# /xmidt/deploy/docker-compose/deploy.sh
/xmidt /xmidt/deploy/docker-compose
Building Simulator...
/xmidt/deploy/docker-compose
Running services...
unknown shorthand flag: 'f' in -f
See 'docker --help'.

Usage: docker [OPTIONS] COMMAND

It is coming from line 22:
docker compose -f $ROOT_DIR/deploy/docker-compose/docker-compose.yml up -d $@

It works fine after changing it to :
docker-compose -f $ROOT_DIR/deploy/docker-compose/docker-compose.yml up -d $@

xmidt/simulator error for docker-compose setup

Hi ,
We are setting up WEBPA using docker-compose ,using the reference from here https://github.com/xmidt-org/xmidt/tree/master/deploy/docker-compose

But the recent setup of the same is giving the below error.We were not facing the below error when we had tried out the same few months back!

The command '/bin/sh -c cd /build && git clone https://github.com/Comcast/parodus2mockTr181.git && cd parodus2mockTr181 && mkdir build && cd build && cmake .. && make && cd /build && git clone https://github.com/Comcast/parodus.git && cd parodus && git checkout ad2d43b4f6e980a6cc1c1340fc82564104eb1dd8 && mkdir build && cd build && cmake .. && make && cd /build && git clone https://github.com/Comcast/aker.git && cd aker && git checkout cfb54022fa6e0ba70040e419d34655da955637b5 && mkdir build && cd build && cmake .. && make' returned a non-zero code: 2
/home/ubuntu/WEBPA_DOCKER/xmidt
Running services...
Pulling simulator (xmidt/simulator:latest)...
ERROR: The image for the service you're trying to recreate has been removed. If you continue, volume data could be lost. Consider backing up your data before continuing.

Continue with the new image? [yN]y
Pulling simulator (xmidt/simulator:latest)...
ERROR: pull access denied for xmidt/simulator, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

Can someone help

Need help with a WebPA server installation

Suffered same situation with issue #3. But seems no resolve guide there.
I had issued talaria installation command, "GO111MODULE=on go install github.com/xmidt-org/talaria@latest", after install is finished, O can launch talaria program. But it showed with below error message, is it expected?

[LOG]
$ talaria 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x178 pc=0xa405f7]

goroutine 1 [running]:
github.com/spf13/viper.(*Viper).AllKeys(0x0)
	/home/wnc/go/pkg/mod/github.com/spf13/[email protected]/viper.go:2069 +0x97
github.com/spf13/viper.(*Viper).Unmarshal(0x418f54?, {0xd97940, 0xc000072080}, {0x0, 0x0, 0x0})
	/home/wnc/go/pkg/mod/github.com/spf13/[email protected]/viper.go:1118 +0x31
github.com/xmidt-org/webpa-common/v2/service/servicecfg.NewEnvironment(0xc000138540?, {0x10ca860, 0x0}, {0x0, 0x0, 0x1?})
	/home/wnc/go/pkg/mod/github.com/xmidt-org/webpa-common/[email protected]/service/servicecfg/environment.go:28 +0x104
main.talaria({0xc0000400b0, 0x1, 0x1})
	/home/wnc/go/pkg/mod/github.com/xmidt-org/[email protected]/main.go:151 +0x7d8
main.main.func1()
	/home/wnc/go/pkg/mod/github.com/xmidt-org/[email protected]/main.go:271 +0x28
main.main()
	/home/wnc/go/pkg/mod/github.com/xmidt-org/[email protected]/main.go:274 +0x13

Any I use go v1.21.6

$ go version
go version go1.21.6 linux/amd64

$ talaria -v
talaria:
  version: 	undefined
  go version: 	go1.21.6
  built time: 	undefined
  git commit: 	undefined
  os/arch: 	linux/amd64

Current diagram is out of date.

Diagram [https://github.com/Comcast/xmidt/blob/master/diagrams/xmidt-architecture.ditaa] is out of date.

  1. ZooKeep Ensemble is no longer a component
  2. Scytale no longer communicates with communicate with a SNS.
  3. Include how caduceus subscribes to a SNS for the purpose of a upstream service publisher.

Facing version issue when trying to run docker-compose deploy script

Hi ,
We are setting up WEBPA using docker-compose using the steps mentioned from https://github.com/xmidt-org/xmidt/tree/master/deploy/docker-compose.

When running the script, facing the below error.
``deploy/docker-compose/deploy.sh scytale talaria
~/webpa/WEBPA_DOCKER/xmidt ~/webpa/WEBPA_DOCKER/xmidt
Building Simulator...
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&t=xmidt%2Fsimulator%3Alatest&target=&ulimits=null&version=1": dial unix /var/run/docker.sock: connect: permission denied
~/webpa/WEBPA_DOCKER/xmidt
Running services...
ERROR: Version in "webpa/WEBPA_DOCKER/xmidt/deploy/docker-compose/../..//deploy/docker-compose/docker-compose.yml" is unsupported. You might be seeing this error because you're using the wrong Compose file version. Either specify a supported version (e.g "2.2" or "3.3") and place your service definitions under the services key, or omit the `version` key and place your service definitions at the root of the file to use version 1.
For more on the Compose file format versions, see https://docs.docker.com/compose/compose-file/
~/webpa/WEBPA_DOCKER/xmidt$ sudo deploy/docker-compose/deploy.sh scytale talaria

Could you please help us on this issue.

Docker compose yaml file is having the version 3.8. Is there any version restriction?

Dynamodb failure observed when launching deploy.sh script

ERROR: for dynamodb Cannot start service dynamodb: driver failed programming external connectivity on endpoint dynamo-xmidt (564854992e1d558c7b54b7c985e982fa1eafe97e10c99aacc79359ae819512c4): Error starting userland proxy: listen tcp4 0.0.0.0:8000: bind: address already in use
ERROR: Encountered errors while bringing up the project.

Unable to build docker image for simulator

Facing issue on trying to build docker image of simulator . Getting below error on building the image

-- Checking for module 'libcurl'
-- Package 'libcurl', required by 'virtual:world', not found
CMake Error at /usr/share/cmake/Modules/FindPkgConfig.cmake:412 (message):
A required package was not found
Call Stack (most recent call first):
/usr/share/cmake/Modules/FindPkgConfig.cmake:588 (_pkg_check_modules_internal)
CMakeLists.txt:58 (pkg_check_modules)

-- Configuring incomplete, errors occurred!
See also "/build/aker/build/CMakeFiles/CMakeOutput.log".
The command '/bin/sh -c cd /build && git clone https://github.com/Comcast/parodus2mockTr181.git && cd parodus2mockTr181 && mkdir build && cd build && cmake .. && make && cd /build && git clone https://github.com/Comcast/parodus.git && cd parodus && git checkout ad2d43b4f6e980a6cc1c1340fc82564104eb1dd8 && mkdir build && cd build && cmake .. && make && cd /build && git clone https://github.com/Comcast/aker.git && cd aker && mkdir build && cd build && cmake .. && make' returned a non-zero code: 1

What are the compatible CentOS versions

We want to deploy the Xmidt server components for RDK Tech summit on a AWS VM and amazon currently gives option to create instances using CentOS 6.4 & 7.0 versions.
What CentOS versions are currently compatible for deploying Xmidt server components?.

Regards
Ganesh

Guide to common server configuration

Today, each server repo provides a sample configuration file with useful comments on each config. While this works, it creates a lot of repeated effort and takes away focus from configuration values that are specific to the service.

One approach to

  • reduce repeated effort: is having a guide to configuration values that are common across the different XMIDT servers. This could take the form of a yaml file or wiki page with all the common config values and sufficiently thorough explanation similar to the yaml config files we already have in each server package.

  • regain focus: transform the existing config files in each server repo by placing server-specific values with comments at the top and placing all common values (already explained in the common guide) at the bottom

How to handle redirection outside docker for mutiple talaria nodes

Hi,
We are trying to bring up webpa using docker-compose with two talaria nodes.The setup works fine inside the docker and the get operations when tested with the simulator are all a success.But connecting to a parodus client outside the docker setup is giving 404 error.On inspecting the parodus logs,It is seen that the redirection URL is giving the docker-container id to the client and we are suspecting that might be the reason for the 404 error code.


`2021 Aug 03 07:56:16.653209 "parodus"[2719]: 210803-07:56:16.650252 [mod=PARODUS, lvl=INFO] [tid=2719] PARODUS: ../../git/src/nopoll_conn.c:331 IPv4 address of 192.168.1.9 is 192.168.1.9
2021 Aug 03 07:56:16.653824 "parodus"[2719]:
2021 Aug 03 07:56:16.654436 "parodus"[2719]: 210803-07:56:16.650360 [mod=PARODUS, lvl=INFO] [tid=2719] PARODUS: ../../git/src/nopoll_conn.c:377 Create socket with non blocking-mode
2021 Aug 03 07:56:16.794267 "parodus"[2719]: 210803-07:56:16.793505 [mod=PARODUS, lvl=INFO] [tid=2719] PARODUS: ../../git/src/nopoll_conn.c:264 Result of wait after connect EINPROGRESS = 0
2021 Aug 03 07:56:16.795028 "parodus"[2719]:
2021 Aug 03 07:56:17.042720 "parodus"[2719]: 210803-07:56:17.042124 [mod=PARODUS, lvl=INFO] [tid=2719] PARODUS: ../../git/src/nopoll_conn.c:3068 websocket server denied connection with: 307
2021 Aug 03 07:56:17.043565 "parodus"[2719]:
2021 Aug 03 07:56:17.044498 "parodus"[2719]: 210803-07:56:17.043017 [mod=PARODUS, lvl=INFO] [tid=2719] PARODUS: ../../git/src/nopoll_conn.c:2915 Received uncomplete listener handshake reply
2021 Aug 03 07:56:17.045241 "parodus"[2719]: 210803-07:56:17.043237 [mod=PARODUS, lvl=INFO] [tid=2719] PARODUS: ../../git/src/nopoll_conn.c:5230 nopoll_conn_wait_for_status_until_connection
2021 Aug 03 07:56:17.045924 "parodus"[2719]: 210803-07:56:17.043273 [mod=PARODUS, lvl=INFO] [tid=2719] PARODUS: Received temporary redirection response message Redirect:http://22af9d543209:
2021 Aug 03 07:56:17.046673 "parodus"[2719]: 210803-07:56:17.043303 [mod=PARODUS, lvl=INFO] [tid=2719] PARODUS: full url: http://22af9d543209:6200/api/v2/device
2021 Aug 03 07:56:17.047379 "parodus"[2719]: 210803-07:56:17.043329 [mod=PARODUS, lvl=INFO] [tid=2719] PARODUS: server address copied from url
2021 Aug 03 07:56:17.048004 "parodus"[2719]: 210803-07:56:17.043357 [mod=PARODUS, lvl=INFO] [tid=2719] PARODUS: server 22af9d543209, port 6200, http_match 1
2021 Aug 03 07:56:17.048622 "parodus"[2719]: 210803-07:56:17.043428 [mod=PARODUS, lvl=INFO] [tid=2719] PARODUS: ../../git/src/nopoll_ctx.c:338 Unregistered connection id 2
2021 Aug 03 07:56:17.049235 "parodus"[2719]: 210803-07:56:17.043496 [mod=PARODUS, lvl=INFO] [tid=2719] PARODUS: cloud_status set as offline after connection close
2021 Aug 03 07:56:17.049903 "parodus"[2719]: 210803-07:56:17.046318 [mod=PARODUS, lvl=INFO] [tid=2719] PARODUS: Waiting max delay 3 backoffRetryTime 3 secs 0 usecs`

Saw the mention of this possible issue here :https://github.com/xmidt-org/xmidt/tree/master/deploy/docker-compose#outside-docker
But some more details on how to proceed on the above would be really helpful!!

Regards
Karthika

Need help with a WebPA server installation

I am new to WebPA and am trying to install WebPA components on a single VM running CentOS 6.9. I have installed the following versions of components:

talaria-0.1.1-169
caduceus-0.1.1-294
petasos-0.1.1-113
scytale-0.1.1-91
tr1d1um-0.1.1-272

When I start the services using their default configurations, all services except tr1d1um fail to start. I don't have access to AWS services (SNS, SQS) in my lab. My goal is to experiment with WebPA REST API (configure, query) and notifications using devices simulated using parodus. I have referred to xmidt Wiki and have clarity on the architecture. Since the configurations for each of components is not documented, I am unable to progress with the installation. Any help to bring up the services is highly appreciated.

Following are the errors observed in the log:

Talaria:
[signal SIGSEGV: segmentation violation code=0x1 addr=0xd0 pc=0x7af3c5]

goroutine 1 [running]:
vendor/github.com/spf13/viper.(*Viper).AllKeys(0x0, 0x7, 0xc4201f9a20, 0x411f59)
/root/rpmbuild/BUILD/talaria-0.1.1/src/vendor/github.com/spf13/viper/viper.go:1378 +0x45
vendor/github.com/spf13/viper.(*Viper).AllSettings(0x0, 0x4127a8)
/root/rpmbuild/BUILD/talaria-0.1.1/src/vendor/github.com/spf13/viper/viper.go:1458 +0x51
vendor/github.com/spf13/viper.(*Viper).Unmarshal(0x0, 0xad86a0, 0xc4201ae840, 0xc42024a680, 0x13)
/root/rpmbuild/BUILD/talaria-0.1.1/src/vendor/github.com/spf13/viper/viper.go:738 +0x2f
vendor/github.com/Comcast/webpa-common/service/servicecfg.NewEnvironment(0xd16ae0, 0xc4201cfac0, 0xd16dc0, 0x0, 0x0, 0x0, 0xd16b60, 0xc42026a4d0)
/root/rpmbuild/BUILD/talaria-0.1.1/src/vendor/github.com/Comcast/webpa-common/service/servicecfg/environment.go:25 +0xd1
main.talaria(0xc42001e1c0, 0x1, 0x1, 0x0)
/root/rpmbuild/BUILD/talaria-0.1.1/src/talaria/talaria.go:120 +0xb28
main.main.func1(0xc420090058)
/root/rpmbuild/BUILD/talaria-0.1.1/src/talaria/talaria.go:170 +0x49
main.main()
/root/rpmbuild/BUILD/talaria-0.1.1/src/talaria/talaria.go:173 +0x22

Caduceus:
Unable to initialize new caduceus sender wrapper: Linger must be positive.

Petasos:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xd0 pc=0x7cc245]

goroutine 1 [running]:
vendor/github.com/spf13/viper.(*Viper).AllKeys(0x0, 0x7, 0xc420225af0, 0x411f59)
/root/rpmbuild/BUILD/petasos-0.1.1/src/vendor/github.com/spf13/viper/viper.go:1378 +0x45
vendor/github.com/spf13/viper.(*Viper).AllSettings(0x0, 0x4127a8)
/root/rpmbuild/BUILD/petasos-0.1.1/src/vendor/github.com/spf13/viper/viper.go:1458 +0x51
vendor/github.com/spf13/viper.(*Viper).Unmarshal(0x0, 0xa93fe0, 0xc4201646a0, 0xc42025b440, 0xc420225c28)
/root/rpmbuild/BUILD/petasos-0.1.1/src/vendor/github.com/spf13/viper/viper.go:738 +0x2f
vendor/github.com/Comcast/webpa-common/service/servicecfg.NewEnvironment(0xcb8600, 0xc4201579c0, 0xcb8880, 0x0, 0x0, 0xcb8560, 0xc42025b440, 0xc420225eb8)
/root/rpmbuild/BUILD/petasos-0.1.1/src/vendor/github.com/Comcast/webpa-common/service/servicecfg/environment.go:25 +0xd1
main.petasos(0xc42001e200, 0x1, 0x1, 0xc420090058)
/root/rpmbuild/BUILD/petasos-0.1.1/src/petasos/petasos.go:67 +0x5ed
main.main()
/root/rpmbuild/BUILD/petasos-0.1.1/src/petasos/petasos.go:125 +0x45

Scytale:
...
{"error":"missing 'aws' key","level":"error","msg":"unable to create primary handler","ts":"2018-05-23T07:31:23.349334053Z"}

Rename master branch to main

Also have to change references to the branch in .travis.yml, README, and CONTRIBUTING. Double check any other markdown files as well - sometimes links have the branch name in them.

Need Xmidt architecture details

We are setting up a minimal webPA cluster for RDK-M, as per our understanding we are setting up the webPA cluster having 4 nodes with following node allocation:
1x Talaria, 1x Cadeceus, 1x Petasos, 1x Scytale

  • As Xmidt and tr1d1um are introduced recently, we wanted to know how these new components will fit in. i.e. whether these services will sit with any of the above nodes or need dedicated servers to run.
  • Also any architecture detail or diagram pointing out communication flow will be good for us to understand the whole ecosystem & implementation.

Regards
Ganesh

Need help with webpa docker-compose setup

Hi ,
We are setting up a minimal webpa cluster with tr1d1um,talaria,petasos,scytale and prometheus using docker-compose following below documentation https://github.com/xmidt-org/xmidt/tree/master/deploy/docker-compose.The simulator tests given in the above link are all pass.

But we are not able to perform get and set operations from the parodus client (RDKB rpi) which is outside the docker environment as it is giving a HTTP 403 response.

root@RaspberryPi-Gateway:~# curl -i -H "Authorization: Basic dXNlcjpwYXNz" "http://192.168.1.11:6100/api/v2/device/mac:b827ebbed712/config?names=Device.DeviceInfo.Manufacturer" -H "X-Xmidt-Partner-ID: comcast,nbc,sky"
HTTP/1.1 403 Forbidden
X-Scytale-Build: unkown
X-Scytale-Flavor: mint
X-Scytale-Region: east
X-Scytale-Server: 29a2b5ddfbdc
X-Scytale-Start-Time: 01 Jul 21 10:46 UTC
X-Talaria-Build: unkown
X-Talaria-Flavor: mint
X-Talaria-Region: east
X-Talaria-Server: 779c0c610c6e
X-Talaria-Start-Time: 01 Jul 21 10:46 UTC
X-Tr1d1um-Build: unkown
X-Tr1d1um-Flavor: mint
X-Tr1d1um-Region: east
X-Tr1d1um-Server: 996fb098e8f9
X-Tr1d1um-Start-Time: 01 Jul 21 10:46 UTC
X-Webpa-Transaction-Id: nR0fW8v7WdJUV5LgWrsC1g
X-Xmidt-Span: "http://779c0c610c6e:6200/api/v2/device/send","2021-07-01T11:00:50Z","2.172427ms"
Date: Thu, 01 Jul 2021 11:00:50 GMT
Content-Length: 0

The talaria logs have the following errors seen:

{"caller":"manager.go:215","id":"mac:b827ebbed712","level":"error","msg":"**missing security information**","ts":"2021-07-01T10:52:08.904603746Z"} {"caller":"manager.go:219","convey":{"boot-time":1625475738,"fw-name":"rdkb-generic-broadband-image_rdk-next_20210531050507","hw-manufacturer":"Raspberry","hw-model":"RPI","hw-serial-number":"000000002dbed712","webpa-interface-used":"erouter0","webpa-last-reconnect-reason":"webpa_process_starts","webpa-protocol":"PARODUS-2.0-1.1.4-22-gbdc2733"},"id":"mac:b827ebbed712","level":"info","ts":"2021-07-01T10:52:08.904651036Z"}

tr1d1um logs:

{"caller":"utils.go:83","duration":"3.174742ms","level":"info","msg":"record","request":{"address":"192.168.1.6:49682","path":"/api/v2/device/mac:b827ebbed712/config","query":"names=Device.DeviceInfo.Manufacturer","method":"GET"},"response":{"code":403,"headers":{"X-Scytale-Build":["unkown"],"X-Scytale-Flavor":["mint"],"X-Scytale-Region":["east"],"X-Scytale-Server":["29a2b5ddfbdc"],"X-Scytale-Start-Time":["01 Jul 21 10:46 UTC"],"X-Talaria-Build":["unkown"],"X-Talaria-Flavor":["mint"],"X-Talaria-Region":["east"],"X-Talaria-Server":["779c0c610c6e"],"X-Talaria-Start-Time":["01 Jul 21 10:46 UTC"],"X-Tr1d1um-Build":["unkown"],"X-Tr1d1um-Flavor":["mint"],"X-Tr1d1um-Region":["east"],"X-Tr1d1um-Server":["996fb098e8f9"],"X-Tr1d1um-Start-Time":["01 Jul 21 10:46 UTC"],"X-Webpa-Transaction-Id":["2P5vZmNk-1lcVS-9PHpuFw"],"X-Xmidt-Span":["\"http://779c0c610c6e:6200/api/v2/device/send\",\"2021-07-01T10:54:18Z\",\"1.357226ms\""]}},"satClientID":"user","tid":"2P5vZmNk-1lcVS-9PHpuFw","ts":"2021-07-01T10:54:18.605668225Z"}

scytale logs

Entrypoint script for scytale Server started. {"configurationFile":"/etc/scytale/scytale.yaml","level":"info","msg":"initialized Viper environment","ts":"2021-07-01T11:20:37.818348785Z"} {"could not create CPU profile: ":"open cpuprofile: permission denied","ts":"2021-07-01T11:20:37.818814738Z"} {"configurationFile":"/etc/scytale/scytale.yaml","level":"info","ts":"2021-07-01T11:20:38.022351022Z"} {"enabled":false,"level":"info","msg":"tracing status","ts":"2021-07-01T11:20:38.022516749Z"} {"level":"info","msg":"using consul for service discovery","ts":"2021-07-01T11:20:38.023459546Z"} {"datacenter":"dc0","instances":1,"level":"info","passingOnly":true,"service":"talaria","tags":"[stage=dev flavor=mint]","ts":"2021-07-01T11:20:38.202480577Z"} {"caller":"primaryHandler.go:238","level":"error","msg":"creating primary handler","ts":"2021-07-01T11:20:38.202852938Z"} {"level":"info","msg":"using service discovery for fanout","ts":"2021-07-01T11:20:38.202875616Z"} {"datacenter":"dc0","eventCount":0,"level":"info","msg":"subscription monitor starting","passingOnly":true,"service":"talaria","tags":["stage=dev","flavor=mint"],"ts":"2021-07-01T11:20:38.204161659Z"} {"datacenter":"dc0","eventCount":1,"instances":["http://779c0c610c6e:6200"],"level":"error","msg":"service discovery update","passingOnly":true,"service":"talaria","tags":["stage=dev","flavor=mint"],"ts":"2021-07-01T11:20:38.204208127Z"} {"bindAddress":":6300","level":"error","msg":"starting server","serverName":"scytale","ts":"2021-07-01T11:20:38.204524009Z"} {"bindAddress":":6301","level":"error","msg":"starting server","serverName":"scytale.health","ts":"2021-07-01T11:20:38.204574339Z"} {"bindAddress":":6302","level":"error","msg":"starting server","serverName":"scytale.pprof","ts":"2021-07-01T11:20:38.204652786Z"} {"bindAddress":":6303","level":"error","msg":"starting server","serverName":"scytale.metrics","ts":"2021-07-01T11:20:38.204719483Z"} {"X-Webpa-Device-Name":"mac:b827ebbed712/config","level":"error","msg":"all fanout requests failed","remoteAddr":"172.29.0.9:41396","requestMethod":"POST","requestURI":"/api/v2/device","statusCode":403,"ts":"2021-07-01T11:21:29.729039113Z","url":"/api/v2/device"}

It would be of great help if someone can point out what I am missing in the configuration.

Regards
Karthika

Talaria jwt token clarification

Not an issue, but a request for help!
Its not clear to me how to force Talaria to only accept registrations using jwt tokens (assuming I have understood correctly how xmidt works!).

I have set up xmidt using the docker-compose example and was expecting this config

jwtValidators:
    -
      keys:
        Factory:
            uri: "http://themis:6500/keys/{keyId}"
        purpose: 0
        updateInterval: 604800000000000

to force the use of tokens to register to Talaria.

However, if I rebuild the docker-simulator and modify "token-server" to something that doesn't exist and rebuild, or simply shutdown themis, it seems I can still register to Talaria.

Here are my rdk simulator logs using themis:6501 as the token server:

# docker run --rm --network=e5ac3216e7d1 -e CMAC=998877665544 xmidt/rdkb-simulator            
[1588029390][PARODUS][Info]: RAND_MAX is 2147483647 (0x7fffffff)
[1588029390][PARODUS][Info]: ********** Starting component: Parodus **********
 [1588029390][PARODUS][Info]: Setting default values to parodusCfg
[1588029390][PARODUS][Info]:  cfg->webpa_protocol is PARODUS-2.0-1.1.3-37-g1d85742
[1588029390][PARODUS][Info]: Default cloud_status is offline
[1588029390][PARODUS][Info]: Parsing parodus command line arguments..
[1588029390][PARODUS][Info]: hw-model is aker-testing
[1588029390][PARODUS][Info]: cert_path is /etc/ssl/certs/ca-certificates.crt
[1588029390][PARODUS][Info]: client_cert_path is /etc/ssl/certs/ca-certificates.crt
[1588029390][PARODUS][Info]: hw_serial_number is mock-rdkb-simulator
[1588029390][PARODUS][Info]: hw_manufacturer is Example
[1588029390][PARODUS][Info]: hw_mac is 998877665544
[1588029390][PARODUS][Info]: hw_last_reboot_reason is unknown
[1588029390][PARODUS][Info]: fw_name is mock-rdkb-firmware
[1588029390][PARODUS][Info]: boot_time is 1588029390
[1588029390][PARODUS][Info]: partner_id is comcast
[1588029390][PARODUS][Info]: parodus local_url is tcp://127.0.0.1:16014
[1588029390][PARODUS][Info]: webpa_ping_timeout is 60
[1588029390][PARODUS][Info]: token_server_url is http://themis:6501/issue
[1588029390][PARODUS][Info]: webpa_backoff_max is 2
[1588029390][PARODUS][Info]: webpa_interface_used is eth0
[1588029390][PARODUS][Info]: webpa_url is http://petasos:6400
[1588029390][PARODUS][Info]: Force IPv4
[1588029390][PARODUS][Info]: Received reboot_reason as:unknown
[1588029390][PARODUS][Info]: Received reconnect_reason as:webpa_process_starts
[1588029390][PARODUS][Info]: User-Agent: PARODUS-2.0-1.1.3-37-g1d85742 (mock-rdkb-firmware; aker-testing/Example;)
[1588029390][PARODUS][Info]: X-WebPA-Convey Header: [316]{"hw-model":"aker-testing","hw-serial-number":"mock-rdkb-simulator","hw-manufacturer":"Example","fw-name":"mock-rdkb-firmware","boot-time":1588029390,"webpa-protocol":"PARODUS-2.0-1.1.3-37-g1d85742","webpa-interface-used":"eth0","hw-last-reboot-reason":"unknown","webpa-last-reconnect-reason":"webpa_process_starts"}
[1588029390][PARODUS][Info]: Device_id mac:998877665544
[1588029390][PARODUS][Info]: full url: http://petasos:6400
[1588029390][PARODUS][Info]: server address copied from url
[1588029390][PARODUS][Info]: server petasos, port 6400, http_match 1
[1588029390][PARODUS][Info]: default server_Address petasos
[1588029390][PARODUS][Info]: default port 6400
[1588029390][PARODUS][Info]: uuid_header formed X-Midt-Uuid: 19d59317-02c3-4014-a870-de67c3ac1620
[1588029390][PARODUS][Info]: curl Ip resolve option set as default mode
[1588029390][PARODUS][Info]: themis curl response 0 http_code 200
[1588029390][PARODUS][Info]: curl response Time: 0.0 seconds
[1588029390][PARODUS][Info]: cURL success
[1588029390][PARODUS][Info]: cfg->webpa_auth_token created successfully
[1588029390][PARODUS][Info]: nopoll_conn.c:331 IPv4 address of petasos is 172.25.0.6 
[1588029390][PARODUS][Info]: nopoll_conn.c:377 Create socket with non blocking-mode 
[1588029390][PARODUS][Info]: nopoll_conn.c:264 Result of wait after connect EINPROGRESS = 0
[1588029390][PARODUS][Error]: nopoll_conn.c:3067 websocket server denied connection with: 307 Temporary Redirect
[1588029390][PARODUS][Error]: nopoll_conn.c:2914 Received uncomplete listener handshake reply (0 0 0) 
[1588029390][PARODUS][Info]: nopoll_conn.c:5229 nopoll_conn_wait_for_status_until_connection_ready() response: message: Redirect:http://talaria-0:6200/api/v2/device 
[1588029390][PARODUS][Info]: Received temporary redirection response message Redirect:http://talaria-0:6200/api/v2/device
[1588029390][PARODUS][Info]: full url: http://talaria-0:6200/api/v2/device
[1588029390][PARODUS][Info]: server address copied from url
[1588029390][PARODUS][Info]: server talaria-0, port 6200, http_match 1
[1588029390][PARODUS][Info]: nopoll_ctx.c:338 Unregistered connection id 2 
[1588029390][PARODUS][Info]: cloud_status set as offline after connection close
[1588029390][PARODUS][Info]: nopoll_conn.c:331 IPv4 address of talaria-0 is 172.25.0.8 
[1588029390][PARODUS][Info]: nopoll_conn.c:377 Create socket with non blocking-mode 
[1588029390][PARODUS][Info]: nopoll_conn.c:264 Result of wait after connect EINPROGRESS = 0
 
[1588029390][PARODUS][Info]: nopoll_conn.c:5246 *****End nopoll_conn_wait_for_status_until_connection_ready **** 
[1588029390][PARODUS][Info]: Connected to server
[1588029390][PARODUS][Info]: cloud_status set as online after successful connection
[1588029390][PARODUS][Info]: connect_time-diff-boot_time=0
[1588029390][PARODUS][Info]: libseshat disabled, Hence proceeding without registration
[1588029390][PARODUS][Info]: nanomsg server gone into the listening mode...
[1588029390][PARODUS][Info]: No clients are registered, waiting ..
[1588029391][PARODUS][Info]: Upstream message received from nanomsg client
[1588029391][PARODUS][Info]: 
 Nanomsg client Registration for Upstream
[1588029391][PARODUS][Info]: Adding first client to list
[1588029391][PARODUS][Info]: client service aker is added to list with url: tcp://127.0.0.1:16015
[1588029391][PARODUS][Info]: sending auth status to reg client
[1588029391][PARODUS][Info]: Client aker Registered successfully. Sending Acknowledgement... 
 [1588029391][PARODUS][Info]: Sending ack:new_node->sock 1 service:aker

And as expected, I can see the device in the devices api:

# curl -s  -H "Authorization: Basic dXNlcjpwYXNz" http://localhost:6200/api/v2/devices |jq
{
  "devices": [
    {
      "id": "mac:998877665544",
      "pending": 0,
      "statistics": {
        "bytesSent": 0,
        "messagesSent": 0,
        "bytesReceived": 0,
        "messagesReceived": 0,
        "duplications": 0,
        "connectedAt": "2020-04-27T23:16:30.937257801Z",
        "upTime": "16.626220638s"
      }
    }
  ]
}

and here is my rdk simulator having run "docker stop themis-image-id"

# docker run --rm --network=e5ac3216e7d1 -e CMAC=998877665544 xmidt/rdkb-simulator
[1588029729][PARODUS][Info]: RAND_MAX is 2147483647 (0x7fffffff)
[1588029729][PARODUS][Info]: ********** Starting component: Parodus **********
 [1588029729][PARODUS][Info]: Setting default values to parodusCfg
[1588029729][PARODUS][Info]:  cfg->webpa_protocol is PARODUS-2.0-1.1.3-37-g1d85742
[1588029729][PARODUS][Info]: Default cloud_status is offline
[1588029729][PARODUS][Info]: Parsing parodus command line arguments..
[1588029729][PARODUS][Info]: hw-model is aker-testing
[1588029729][PARODUS][Info]: cert_path is /etc/ssl/certs/ca-certificates.crt
[1588029729][PARODUS][Info]: client_cert_path is /etc/ssl/certs/ca-certificates.crt
[1588029729][PARODUS][Info]: hw_serial_number is mock-rdkb-simulator
[1588029729][PARODUS][Info]: hw_manufacturer is Example
[1588029729][PARODUS][Info]: hw_mac is 998877665544
[1588029729][PARODUS][Info]: hw_last_reboot_reason is unknown
[1588029729][PARODUS][Info]: fw_name is mock-rdkb-firmware
[1588029729][PARODUS][Info]: boot_time is 1588029729
[1588029729][PARODUS][Info]: partner_id is comcast
[1588029729][PARODUS][Info]: parodus local_url is tcp://127.0.0.1:16014
[1588029729][PARODUS][Info]: webpa_ping_timeout is 60
[1588029729][PARODUS][Info]: token_server_url is http://themis:6501/issue
[1588029729][PARODUS][Info]: webpa_backoff_max is 2
[1588029729][PARODUS][Info]: webpa_interface_used is eth0
[1588029729][PARODUS][Info]: webpa_url is http://petasos:6400
[1588029729][PARODUS][Info]: Force IPv4

[1588029729][PARODUS][Info]: Received reboot_reason as:unknown
[1588029729][PARODUS][Info]: Received reconnect_reason as:webpa_process_starts
[1588029729][PARODUS][Info]: User-Agent: PARODUS-2.0-1.1.3-37-g1d85742 (mock-rdkb-firmware; aker-testing/Example;)
[1588029729][PARODUS][Info]: X-WebPA-Convey Header: [316]{"hw-model":"aker-testing","hw-serial-number":"mock-rdkb-simulator","hw-manufacturer":"Example","fw-name":"mock-rdkb-firmware","boot-time":1588029729,"webpa-protocol":"PARODUS-2.0-1.1.3-37-g1d85742","webpa-interface-used":"eth0","hw-last-reboot-reason":"unknown","webpa-last-reconnect-reason":"webpa_process_starts"}
[1588029729][PARODUS][Info]: Device_id mac:998877665544
[1588029729][PARODUS][Info]: full url: http://petasos:6400
[1588029729][PARODUS][Info]: server address copied from url
[1588029729][PARODUS][Info]: server petasos, port 6400, http_match 1
[1588029729][PARODUS][Info]: default server_Address petasos
[1588029729][PARODUS][Info]: default port 6400
[1588029729][PARODUS][Info]: uuid_header formed X-Midt-Uuid: a0c60c1c-77e3-4961-93c2-d4c657b677f2
[1588029729][PARODUS][Info]: curl Ip resolve option set as default mode
[1588029735][PARODUS][Info]: themis curl response 6 http_code 0
[1588029735][PARODUS][Info]: curl response Time: 4.9 seconds
[1588029735][PARODUS][Error]: curl_easy_perform() failed: Couldn't resolve host name
[1588029735][PARODUS][Error]: Failed to create new token
[1588029735][PARODUS][Error]: Curl execution is failed, retry attempt: 1
[1588029735][PARODUS][Info]: uuid_header formed X-Midt-Uuid: afc5acd7-badb-40dd-bd64-08f012b936da
[1588029735][PARODUS][Info]: curl Ip resolve option set as V4 mode
[1588029740][PARODUS][Info]: themis curl response 6 http_code 0
[1588029740][PARODUS][Info]: curl response Time: 4.9 seconds
[1588029740][PARODUS][Error]: curl_easy_perform() failed: Couldn't resolve host name
[1588029740][PARODUS][Error]: Failed to create new token
[1588029740][PARODUS][Error]: Curl execution is failed, retry attempt: 2
[1588029740][PARODUS][Info]: uuid_header formed X-Midt-Uuid: d86b7998-cbc3-436a-a38a-4535c1700ba0
[1588029740][PARODUS][Info]: curl Ip resolve option set as V6 mode
[1588029745][PARODUS][Info]: themis curl response 6 http_code 0
[1588029745][PARODUS][Info]: curl response Time: 4.9 seconds
[1588029745][PARODUS][Error]: curl_easy_perform() failed: Couldn't resolve host name
[1588029745][PARODUS][Error]: Failed to create new token
[1588029745][PARODUS][Error]: Curl execution is failed, retry attempt: 3
[1588029745][PARODUS][Error]: Curl retry is reached to max 3 attempts, proceeding without token
[1588029745][PARODUS][Info]: nopoll_conn.c:331 IPv4 address of petasos is 172.25.0.6 
 
[1588029745][PARODUS][Info]: nopoll_conn.c:377 Create socket with non blocking-mode 
[1588029745][PARODUS][Info]: nopoll_conn.c:264 Result of wait after connect EINPROGRESS = 0
 
[1588029745][PARODUS][Error]: nopoll_conn.c:3067 websocket server denied connection with: 307 Temporary Redirect
 
[1588029745][PARODUS][Error]: nopoll_conn.c:2914 Received uncomplete listener handshake reply (0 0 0) 
[1588029745][PARODUS][Info]: nopoll_conn.c:5229 nopoll_conn_wait_for_status_until_connection_ready() response: message: Redirect:http://talaria-0:6200/api/v2/device 
[1588029745][PARODUS][Info]: Received temporary redirection response message Redirect:http://talaria-0:6200/api/v2/device
[1588029745][PARODUS][Info]: full url: http://talaria-0:6200/api/v2/device
[1588029745][PARODUS][Info]: server address copied from url
[1588029745][PARODUS][Info]: server talaria-0, port 6200, http_match 1
[1588029745][PARODUS][Info]: nopoll_ctx.c:338 Unregistered connection id 2 
[1588029745][PARODUS][Info]: cloud_status set as offline after connection close
[1588029745][PARODUS][Info]: uuid_header formed X-Midt-Uuid: 71747b8a-5ae5-4928-8aea-48e867dcbb53

I can still see the device when I curl Talaria devices api.
I'm expecting somewhere in Talaria configuration to be able to block clients that "proceed without token"?

[1588029745][PARODUS][Error]: Curl retry is reached to max 3 attempts, proceeding without token

Simplify docker-compose process

Currently, the deploy package in this repo contains a detailed readme with the instructions on how to bring up a docker-compose cluster for XMiDT. While there may not be functional problems with it, we'd like to change the following:

  1. Instead of requesting the user to build service images (Talaria, Scytale, Petasos, Caduceus, Tr1d1um) locally, simply pull existing images from our new docker hub repo: https://hub.docker.com/u/xmidt

  2. For the components where we need to build an image locally, clone repos inside the /tmp folder to keeps things tidy.

Important notes:
Remember to leverage the fact that both image pulling at step 1 and docker-compose leverage the {service}_VERSION environment variable (i.e. TALARIA_VERSION=0.1.3) to know which service image tag to use.
Make this obvious in the way you setup your scripts such that when I want to upgrade the version of any service, it'll be dead simple to do it.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.