Linux local docker dev workspace not working about prairielearn

[SethPoulsen]

Got it working!

My /etc/docker/daemon.json looks like:

{
  "default-address-pools":
  [
    {"base":"10.254.0.0/16","size":24}
  ]
}

So I instead had to do --add-host host.docker.internal:10.254.0.1

[SethPoulsen]

done, thanks

[trombonekenny]

@SethPoulsen Did you see something similar?

[echuber2]

@trombonekenny Try adding back --add-host=host.docker.internal:172.17.0.1. This is the issue I was referring to with #9802

[SethPoulsen]

That did not work for me. I've got the same OS setup as Dave, Ubuntu 22.04. My terminal:

(base) seth@seth-t14s:~/git/PrairieLearn$ docker run -it --rm -p 3000:3000     -e NODEMON=true    -v "$HOME/git/PrairieLearn:/PrairieLearn"     -v "$HOME/pl_ag_jobs:/jobs"    -e HOST_JOBS_DIR="$HOME/pl_ag_jobs"     -v /var/run/docker.sock:/var/run/docker.sock  --add-host=host.docker.internal:172.17.0.1   prairielearn/prairielearn
Starting PrairieLearn...
[nodemon] 3.1.0
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,cjs,json
[nodemon] starting `yarn dev:no-watch`
[nodemon] 3.1.0
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,ts,json,sql
[nodemon] starting `yarn dev:no-watch`
info: Workspace server listening on port 8081
info: Workspace host ready
info: PrairieLearn server ready, press Control-C to quit
info: Go to http://localhost:3000
info: Launching workspace-1-1 (useInitialZip=true)
info: Pulling docker image: codercom/code-server:3.4.1
error: Error starting container for workspace 1 Max startup time exceeded for workspace 1 (version 1, launch uuid 7b046115-bf29-4876-984f-1a8ec6ba4450) {"stack":"Error: Max startup time exceeded for workspace 1 (version 1, launch uuid 7b046115-bf29-4876-984f-1a8ec6ba4450)\n    at <anonymous> (/PrairieLearn/apps/workspace-host/src/interface.js:543:13)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"}

[SethPoulsen]

strangely, my error actually looks different than Dave's, I don't have the Error looking up container for launch_uuid, only the Max startup time exceeded error. (This could be due to different logging configs though, I don't know.

[echuber2]

For the "time exceeded" error, try removing your mount of the local /PrairieLearn source and using what's already in the image for that.

[SethPoulsen]

that did not solve the problem, I still get the same error output

[trombonekenny]

Give it a little bit of time. Does the second error show up?

[SethPoulsen]

oh, you're right Dave, the second error did show up eventually after a while:

error: Error looking up container for launch_uuid 7b046115-bf29-4876-984f-1a8ec6ba4450 Cannot read properties of undefined (reading 'Id') {"stack":"TypeError: Cannot read properties of undefined (reading 'Id')\n    at _getDockerContainerByLaunchUuid (/PrairieLearn/apps/workspace-host/src/interface.js:354:46)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at <anonymous> (/PrairieLearn/apps/workspace-host/src/interface.js:294:19)"}

[echuber2]

The outdated codercom/code-server:3.4.1 image may also be chowning lots of stuff slowly and causing the launch to time out. This is a long-standing issue that motivated our other derived VS Code images, like the one currently in PR. (The "other" images being ad-hoc ones that course staff made for themselves.)

[trombonekenny]

The outdated codercom/code-server:3.4.1 image may also be chowning lots of stuff slowly and causing the launch to time out. This is a long-standing issue that motivated our other derived VS Code images, like the one currently in PR. (The "other" images being ad-hoc ones that course staff made for themselves.)

I was using the xterm image / example course question. I don't think it does any chown commands: https://github.com/PrairieLearn/PrairieLearn/blob/master/workspaces/xtermjs/Dockerfile

[echuber2]

I was referring to Seth's output:

info: Pulling docker image: codercom/code-server:3.4.1

With the xtermjs one, in contrast, I was able to reproduce your issue locally by removing the --add-host and fix the issue by putting it back.

[trombonekenny]

Thanks for confirming, @echuber2. I'm not sure why it fails for @SethPoulsen. The VSCode example question (with the outdated image you mentioned above) worked fine too.

And I learned the make -j 2 dev dev-workspace-host trick, which is nice.

[trombonekenny]

It's a little unsatisfying that we need to do the docker host configuration for workspaces when we don't need it for external graders, but I understand that the workspace is doing different dockerode commands and engaging with the host docker differently. The documentation change resolves this issue for me (and makes me want to revisit docker-compose as a more supported natural way to run non-native dev).

[echuber2]

Seth's case might need another solution then... It doesn't look like he has a custom course mounted in the PL invocation, but the usual trigger for the code-server timeout would be having a lot of pre-installed files in /home/coder for the particular question, or in a modified copy of the image.

[SethPoulsen]

The only other difference I can see between me and Dave is that I start the server when I start the container, but Dave does those two steps seperately. Dave, what is your command for starting the container by itself without starting the server? is it some variation of docker start? (rather than docker run which is what I do).

[jonatanschroeder]

The only other difference I can see between me and Dave is that I start the server when I start the container, but Dave does those two steps seperately. Dave, what is your command for starting the container by itself without starting the server? is it some variation of docker start? (rather than docker run which is what I do).

Just add bash at the end of the docker run command.

[echuber2]

If you have custom options in a PrairieLearn/config.json, that might play a role too.

[SethPoulsen]

Still not working if I start the container then start the server after.

My config is empty.

At some point I had to reroute some docker ports because it was interfering with something on my University networks. This could have something to do with it.

[SethPoulsen]

(in case anyone cares, I had to move the docker ports because it interfered with my machine communicating with other devices on my department network, because of how my IT has those IP addresses mapped)

[echuber2]

Would you like to add a note to #9802 about that? I don't know if it's too much of a special case to mention, but there are probably other faculty who might run into it.

[trombonekenny]

(in case anyone cares, I had to move the docker ports because it interfered with my machine communicating with other devices on my department network, because of how my IT has those IP addresses mapped)

That used to be true for IllinoisNet back in the early days of docker (using the same 10.17.x range)! I think a lot of institutional IT spaces, including ours, have moved away from using that range to try to minimize the conflicts and configuration -- but I understand it's still an issue.

Closing this one out, thanks for the troubleshooting!