Environment variables not found about azure-iot-pcs-remote-monitoring-dotnet

[raamanihilent]

Thanks Let me inspect and get back.

[raamanihilent]

Thanks for the update Parvez.
@sushilraje

I am listing a few more items, which i encountered during my debugging by starting the services separately in a command prompt by using the run.cmd inside docker directory.

1. The container APIs are accessible from the browser. However they are not reached by the containers.
2. Some of the expected collections and values are not present in the Cosmos DB.
   http://localhost:9022/v1/collections/device-twin-properties/values/cache
   http://localhost:9022/v1/collections/solution-settings/values/seedMutex
   http://127.0.0.1:9004/v1/rules
3. If there is no seed data injection, please provide steps to call the REST APIs(headers/body)

Hope this helps.
Thanks

[sushilraje]

@raamanihilent: I believe you need to run the pcs -s local command after pcs login but before docker-compose up. Please run it and let me know if it works.

[raamanihilent]

@sushilraje
Thanks.
It did not help.
when i run the pcs -s local command, the command prompts to create Azure resources, which was already done by executing the start.cmd command from the documentation link provided earlier.

The following steps were followed and i could verify that the azure resources were created.
Run the start.cmd script. The script prompts you for the following information:
A solution name.
The Azure subscription to use.
The location of the Azure datacenter to use.

Will wait for your updates. Ty.

[sushilraje]

@raamanihilent:
It seems that start.cmd didn't stamp the environment variables on your machine. I will try to repro this issue. But a quick fix would be to create following environment variables on your machine.
PCS_KEYVAULT_NAME
PCS_AAD_APPID
PCS_AAD_APPSECRET
Values for these variables would be
Go to your azure portal go to resource group (same as solution name)
keyvault-****
go to keyvault > secrets
aadAppId
aadAppSecret

[raamanihilent]

@sushilraje
Thanks for the information.
It was set correctly by start.cmd command. I have verified this before creating the issue.
Here is my env snapshot.

Will wait for your updates.

[sushilraje]

@raamanihilent Which docker compose yaml file are you using?
azure-iot-pcs-remote-monitoring-dotnet\services\scripts\local OR
azure-iot-pcs-remote-monitoring-dotnet\scripts\local

You should be using the one underneath the services folder.

[raamanihilent]

yes I ran the script from
azure-iot-pcs-remote-monitoring-dotnet\services\scripts\local

[sushilraje]

Can you check foll,?
Does your docker-compose.yml file have foll structure,

especially the highlighted environment variables.

:
image: azureiotpcs/-dotnet:testing
.........
environment:
- PCS_KEYVAULT_NAME
- PCS_AAD_APPID
- PCS_AAD_APPSECRET

Also, could you try deleting all the images from your local machine and try docker-compose up?

[raamanihilent]

Yes and i deleted all the images, ran docker-compose up.
Did not work .
Will wait for your updates.

[sushilraje]

Can you docker inspect on any of the containers and ensure that the variables are correct values are passed through?

[sushilraje]

I tried it out locally, it did work for me.
Please do docker inspect and check if the required three variables
PCS_AAD_APPID,
PCS_AAD_APPSECRET,
KEYVAULT_NAME
are getting passed along correctly as environment variables to the container.
Also, what is the tag of your containers?

[sushilraje]

@raamanihilent: Were you able to get any data around the issue?

[raamanihilent]

@sushilraje Since you mentioned it worked for you locally, I am planning to redo the steps. Let me get back to you with my updates.

[raamanihilent]

@sushilraje
As mentioned i cleaned up my environment.
Followed the steps again and tried docker-compose up. The local proxy was failing with error -

ERROR: for reverseproxy Cannot start service reverseproxy: OCI runtime create failed: container_linux.go:344: starting container process caused "process_linux.go:424: container init caused "rootfs_linux.go:58: mounting \"/host_mnt/c/MyStuff/remotemonitoring/azure-iot-pcs-remote-monitoring-dotnet/services/scripts/local/config/nginx.conf\" to rootfs \"/var/lib/docker/overlay2/f9531ebe0054b7d0436008af3945903e9f311f3052d029ab202d0ef3dba0cbd7/merged\" at \"/var/lib/docker/overlay2/f9531ebe0054b7d0436008af3945903e9f311f3052d029ab202d0ef3dba0cbd7/merged/app/config/nginx.conf\" caused \"not a directory\""": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
ERROR: Encountered errors while bringing up the project.

I then stopped all my containers, started each container in a separate window and inspected them.
I could verify the following environment variables set via docker inspect
"PCS_KEYVAULT_NAME=",
"PCS_AAD_APPID=",
"PCS_AAD_APPSECRET=",

However these are my errors in console -
Config container

StorageAdapter returns 0 for request http://localhost:9022/v1/collections/device-twin-properties/values/cache, {"Uri":"http://localhost:9022/v1/collections/device-twin-properties/values/cache","StatusCode":0,"Content":"An error occurred while sending the request. - Couldn't connect to server"}

ASA

Received error updating device to device group mapping, {"e":{"ExceptionFullName":"Microsoft.Azure.IoTSolutions.AsaManager.Services.Exceptions.ExternalDependencyException","ExceptionMessage":"Unable to get list of device groups","StackTrace":" at Microsoft.Azure.IoTSolutions.AsaManager.DeviceGroupsAgent.DeviceGroupsClient.d__8.MoveNext() in /home/azureiotpcs/vsts-agent/_work/20/s/asa-manager/DeviceGroupsAgent/DeviceGroupsClient.cs:line 83\n--- End of stack trace from previous location where exception was thrown ---\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\n at Microsoft.Azure.IoTSolutions.AsaManager.DeviceGroupsAgent.Agent.d__13.MoveNext() in /home/azureiotpcs/vsts-agent/_work/20/s/asa-manager/DeviceGroupsAgent/Agent.cs:line 85","Source":"Microsoft.Azure.IoTSolutions.AsaManager.DeviceGroupsAgent","Data":{},"InnerException":{"ExceptionFullName":"Microsoft.Azure.IoTSolutions.AsaManager.Services.Exceptions.ExternalDependencyException","ExceptionMessage":"Unable to load get device groups","StackTrace":" at Microsoft.Azure.IoTSolutions.AsaManager.Services.Http.HttpClient.d__9`1.MoveNext() in /home/azureiotpcs/vsts-agent/_work/20/s/asa-manager/Services/Http/HttpClient.cs:line 104\n--- End of stack trace from previous location where exception was thrown ---\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\n at Microsoft.Azure.IoTSolutions.AsaManager.DeviceGroupsAgent.DeviceGroupsClient.d__8.MoveNext() in /home/azureiotpcs/vsts-agent/_work/20/s/asa-manager/DeviceGroupsAgent/DeviceGroupsClient.cs:line 78","Source":"Microsoft.Azure.IoTSolutions.AsaManager.Services","Data":{},"InnerException":null}}}

iot_hub_manager

[StorageAdapterClient:CheckStatusCode] StorageAdapter returns 0 for request http://localhost:9022/v1/collections/device-twin-properties/values/cache, {"Uri":"http://localhost:9022/v1/collections/device-twin-properties/values/cache","StatusCode":0,"Content":"An error occurred while sending the request. - Couldn't connect to server"}

Will wait for your updates.
Thanks

[ac2707]

deploy locally with Visual studio - cannot start the microservices as well - any leads on this ?

environment variables set correctly -

any more changes here ? please advise, thanks

[ac2707]

Also can you provide more details for the deploy with VS Code -
Deploy all other microservices on local machine
Open the azure-iot-pcs-remote-monitoring-dotnet module from your local copy in VS Code.
Copy the files launch.json and tasks.json from scripts\local\launch\idesettings\vscode. Create a new folder azure-iot-pcs-remote-monitoring-dotnet.vscode and paste the files there.
Open the Debug panel in VS Code and run the Run all microservices configuration. This configuration runs the device simulation microservice in Docker and runs the other microservices in the debugger.
don't see an option in debug panel for running all microservices... can you please provide detailed steps for this. Thanks

[ac2707]

The microservices don't start when doing a docker compose up either - screenshot below
Can you please add on more steps here to have it running. Thanks.

[sushilraje]

@raamanihilent @ac2707 I am extremely sorry for the delayed response.

[sushilraje]

@raamanihilent: We do need to mount a directory containing nginx.conf file on to the docker container.
If you are using Windows, you will need to give permissions to docker daemon.
Open your docker console > settings > Shared Drives > Select your drives > Apply

[sushilraje]

@ac2707 sn

deploy locally with Visual studio - cannot start the microservices as well - any leads on this ?

environment variables set correctly -

any more changes here ? please advise, thanks

This is actually working. The logs are not Error level but warnings. Here microservices are trying to inherit env variables. As a first step, the services try to inherit files from local env, if not found then the services go to key-vault. Since you have given key-vault credentials, the services will get the variables/secrets from key-vault.

[sushilraje]

Also can you provide more details for the deploy with VS Code -
Deploy all other microservices on local machine
Open the azure-iot-pcs-remote-monitoring-dotnet module from your local copy in VS Code.
Copy the files launch.json and tasks.json from scripts\local\launch\idesettings\vscode. Create a new folder azure-iot-pcs-remote-monitoring-dotnet.vscode and paste the files there.
Open the Debug panel in VS Code and run the Run all microservices configuration. This configuration runs the device simulation microservice in Docker and runs the other microservices in the debugger.
don't see an option in debug panel for running all microservices... can you please provide detailed steps for this. Thanks

In your file launch.json from scripts\local\launch\idesettings\vscode, do you see "compound" section with name "Run all Microservices" as here?

[sushilraje]

The microservices don't start when doing a docker compose up either - screenshot below
Can you please add on more steps here to have it running. Thanks.

This is a different issue than just provisioning. Can you open up a new issue? In that please add foll. details

1. o/p of docker ps
2. o/p of docker ps -a
3. Complete logs of
   1. UiConfig/Config
   2. Device Simulation
   3. Storage Adapter microservices,
      preferably as files

[sushilraje]

@raamanihilent & @ac2707 Do let me know if you face any further issues.

[ac2707]

The microservices don't start when doing a docker compose up either - screenshot below
Can you please add on more steps here to have it running. Thanks.

This is a different issue than just provisioning. Can you open up a new issue? In that please add foll. details

1. o/p of docker ps

2. o/p of docker ps -a

3. Complete logs of

   1. UiConfig/Config
   2. Device Simulation
   3. Storage Adapter microservices,
      preferably as files

Device simulation --

[ac2707]

@ac2707 sn

deploy locally with Visual studio - cannot start the microservices as well - any leads on this ?

environment variables set correctly -

any more changes here ? please advise, thanks

This is actually working. The logs are not Error level but warnings. Here microservices are trying to inherit env variables. As a first step, the services try to inherit files from local env, if not found then the services go to key-vault. Since you have given key-vault credentials, the services will get the variables/secrets from key-vault.

The command prompts and the web browser windows close, unable to run the microservices from Visual studio..
debug output log in Visual studio below -

[fahadashrafi]

Any Solution Found for starting microservices ???