Stuck in env.reset() until RAM runs out about macad-gym

[ZhizhenQin]

Might not be super helpful, but env.reset() stuck on my computer and the whole system is frozen (might be a sign of running out of memory) when I used Carla 0.9.9.3. After I switched to Carla 0.9.4 it was resolved and the test script proceeded perfectly. (I was using Ubuntu 18.04 + Python 3.7.3)

[praveen-palanisamy]

Hi @tbienhoff , my apologies for the delayed response. This past week was a bit crazy.
Can you share which version of CARLA server and client you are using?

Listed below are the steps you could try to identify/debug the issue:

1. To eliminate other sources of errors, the first thing to try will be to see if you are able to launch the CARLA server binary on the P100 node without a display. This will make sure there are no OpenGL/Vulkan related rendering issues on the headless node.

2. If the CARLA server binary runs without issues, the next thing to would be to launch the CARLA server binary from a singularity container on the node. This will make sure there are no permission issues while accessing the GPU and running CARLA.

3. If 1. & 2. passed without issues, the next thing to check will be the CARLA client and server compatibility. Once you get to this step, please post your CARLA client and server versions.

4. I all the above goes well, there could be some previously unknown bug in MACAD-Gym which I can fix to resolve the whole issue.

Feel free to reach out if you need any assistance and I will try to respond soon.

[tbienhoff]

I am able to complete steps 1, 2, and 3. I can run a CARLA server in singularity using the provided Docker images from CARLA (https://hub.docker.com/r/carlasim/carla/), and I can connect a CARLA client to the server. I have successfully done this for CARLA versions 0.8.2, 0.9.4, 0.9.5, and 0.9.6.

I am using version 0.9.5 with MACAD-Gym within a docker image built from the Dockerfile provided in Macad-Agent.

I believe the error might have something to do with not having a display for the server to output to.

[praveen-palanisamy]

Hmmm. From the last few lines of the error log you posted which contains the following lines:

sh: 1: xdg-user-dir: not found
error: XDG_RUNTIME_DIR not set in the environment.

The first hunch is that the user account inside the Singularity container doesn't seem to have the correct permissions or access to the X11/Xorg display server or to the video card (in case of SDL offscreen rendering) on the host.

[praveen-palanisamy]

Since you are able to launch a CARLA server inside the container and connect to it from a client (on the host?), I believe, this issue can be sorted out.

From the MACAD-Gym log you posted, it looks like, both multi-GPU mode and headless mode failed for launching the CARLA server failed on your node and therefore the CARLA server process was launched with rendering enabled which also eventually failed

Can you post the CARLA server log generated by MACAD-Gym (can be found in LOG_DIR/server_*) when you run into this issue?

If you don't have the server log , you can also get the stdout console log from running the following commands inside the container and post it:

1. export CARLA_SERVER=FULL_PATH_TO_CARLA_SERVER_BINARY_ON_THE_CONTAINER
2. $(CARLA_SERVER) /Game/Carla/Maps/Town05 -windowed -ResX=600 -ResY=800 -benchmark -fps=20 -carla-server -carla-world-port=8376 -carla-streaming-port=0

[no-response]

This issue has been automatically closed because there has been no response to the request for more information from the original author. With only the information that is currently in the issue, there is not enough information to take action. Please reach out if you have or find the answers/request-information needed to investigate further.