Graceful Shutdown of Windows Server Containers about windows-containers

[jsturtevant]

I believe graceful shutdown now works with Containerd via HCS v2 api. Don't know the status for Docker. A couple sample apps: https://github.com/marosset/windows-container-graceful-shutdown

[awakecoding]

@TBBle @immuzz if we restrict the scope to Windows Server 2019 specifically, is there a way one could grab and run an upstream version of the Docker CLI and engine that uses HCS v2 + containerd? I currently instruct my customers to grab the official msftdocker, but I would be fine with giving them instructions to migrate to another Docker distribution that is not affected by ungraceful shutdown issues.

I already require Windows Server 2019 instead of Windows Server 2016 because 2016 has limited docker networking support among other things, so we decided not to bother with it. In most cases, customers just buy a new Windows Server 2019 Standard license to create a dedicated VM to run our multi-container product.

What worries me is to learn that while containerd is much better, there appears to be a lot of features that would require "manual assembling" like networking, etc. Maybe it could help to point out what I actually use and need from the Docker CLI:

• Docker networks
• Docker volumes
• Graceful shutdown

There is a lot I can handle outside of the Docker CLI because I wrap everything in a PowerShell module. If I have to inject 10 additional parameters to each container because an upstream Docker CLI + engine wouldn't provide as much helper functions, it would be annoying, but I can still do it. For instance, I recall doing some experimentation with podman on RHEL8, which lacked proper docker networking. It can work if I manually specify a complete list of IPs and hostnames to all containers: annoying, but still doable.

I can meet HCS v2 + containerd halfway, as long as I can figure out a way to get started using it on Windows Server 2019. If it means losing all a lot of helpful features that are automatic in the Docker CLI then so be it, I'll just add a whole lot of PowerShell scripting on top of it to inject a lot more parameters to each container.

My question is, how would I get started running containers on Windows Server 2019 using containerd, knowing that it is supposed to support graceful shutdowns, unlike HCS v1?

Also, just pointing out the obvious elephant in the room: why not fix HCS v1 even if it apparently has no future? After all, 2016/2019 are long-term support releases, and I can't believe that things like a lack of graceful shutdown would not be considered a critical bug that needs a backported fix. Either backport fixes to HCS v1, or bring HCS v2 to 2019, all that matters really is having containers that can be shutdown gracefully.

[awakecoding]

@TBBle so let me summarize what I understand of the situation:

• Docker on Windows Server 2016 / 2019 currently uses HCS v1
• HCS v2 + containerd is affected by critical issues like this one
• HCS v2 + containerd could work for 2019, but not for 2016
• containerd is not complete on its own, we need CNI plugins
• containerd is not really dealt with "directly" like the Docker CLI
• while we're figuring this out, Docker is plain broken on Windows?

Is it me or Kubernetes is slowly becoming the new Docker? What appears to matter now is the container runtime which is way too complex to deal with directly unless you use something on top of it like the Docker CLI or Kubernetes, and the world is slowly moving away from the Docker CLI. That would be a nightmare vision of the future because Kubernetes is absolutely overkill for a lot of use cases and I wouldn't recommend it to anyone unless they really need it.

[msftbot]

This issue has been open for 30 days with no updates.
@brasmith-ms, please provide an update or close this issue.

[msftbot]

This issue has been open for 30 days with no updates.
@brasmith-ms, please provide an update or close this issue.

[msftbot]

This issue has been open for 30 days with no updates.
@brasmith-ms, please provide an update or close this issue.

[awakecoding]

@brasmith-ms could this issue be related to #37 and possibly docker-library/mongo#435 that prevent proper MongoDB operation inside Windows containers? If containers are not gracefully shut down, that would explain a lot.

Can you give an update on this? It is a critical issue that has been lingering on for years, and we really, really need to get working containerized MongoDB databases on Windows Server. A lot of customers wish to use Windows and won't accept switching to Linux just to run containers.

[awakecoding]

@jsturtevant that's very interesting, it would mean there are some upstream fixes already, I just have no idea if there is a way to get an up to date version of Docker that would use it. Is it even possible to rebuild Docker for Windows with the latest upstream code?

[immuzz]

@brasmith-ms can confirm, but I dont think there are plans to fix it for docker container runtime. Since docker uses HCS v1, it probably wont be possible. Is there something stopping you from moving to containerd?

[awakecoding]

@immuzz to be honest I only learned yesterday that it was possible to run Windows containers without Docker. I don't mind switching to containerd as long as it is officially supported on Windows Server 2019? My software basically just calls the Docker CLI, if you can point me to more information on how to switch to containerd for Windows containers, I'd be more than happy to take a closer look at it.

[TBBle]

Per moby/moby#41455, it's likely that Docker will continue to use HCS v1 for the 20.x series (as it's too late to switch), and perhaps will drop it (by moving to use containerd on Windows, as they have on Linux) in a 21.x series, which brings in the HCS v2 API.

One issue for Docker is that HCS v2 requires RS5 (LTSC 2019), and they still support RS1 (LTSC 2016) in the 20.x series, so they cannot move to containerd. There doesn't seem to be much appetite for supporting both direct HCS calls and containerd at the same time.

As far as official support, containerd themselves support Windows, but containerd by itself is not the full solution, e.g., you need to bring your own CNI plugin for networking or otherwise configure HNS directly (as Docker does), and that's a surprising hassle to set up by hand, if you need anything more complex than NAT. Even NAT was a minor hassle to work out how to do it. (It's possible that there was an easy way to do this that I overlooked, of course. And that was a while ago, maybe this has been streamlined since.)

Containerd itself is designed to be used by other services, the direct CLI (ctr) is more of a debugging tool, I understand. So normally you'd consumer it via Docker (there used to be an env-var for this, but it's probably bitrotted, hence the above-mentioned issue) or as a CRI plugin for Kubernetes. In both cases, the wrapping software is taking care of things like setting up the CNI networking integration, and generally bundles or supplies the appropriate plugins.

So Microsoft would formally support Docker (using containerd) via the existing msftdocker install source once Docker includes containerd. As of Kubernetes 1.20, containerd/CRI is an officially supported kubernetes-on-Windows runtime, and since the Docker shim is deprecated, that makes it the only supported option.

[TBBle]

HCS v2 + containerd is affected by critical issues like this one

I assume you meant "HCS v2 + containerd is not affected by critical issues like this one".

---

The most recent instructions on using Containerd in Windows with Docker is the "How to use" on moby/moby#38541 from January 2019.

I checked and the env-var still exists, but a recent enabling of this config on the Docker Jenkins CI instance (moby/moby#41479) saw lots of test failures, so it's quite possible that your use-case will also hit issues.

You can grab Win64 containerd builds from https://github.com/containerd/containerd/releases. Helpfully, if you grab the cri-containerd-cni Win64 package as well, that contains the binaries you need from hcsshim, so you won't need to download that hcsshim's CI system. You shouldn't need the CNI binaries from that package, because Docker takes care of the networking setup itself, but if you want to run containerd without Docker, then those're the CNI binaries you need.

[awakecoding]

@TBBle installing the containerd service on Windows wasn't too hard, but I didn't get a chance to try using the experimental Docker containerd support. At a bare minimum, it is good news to hear that there is such experimental support meant to facilitate the eventual migration to containerd. However, it looks like there is still no clear plan as to what is going to happen with regards to HCSv1, HCSv2 + containerd, and the future of Docker for Windows.

My understanding is Azure Kubernetes Services supports Windows containers through containerd, and I assume all the focus is on Azure, but that doesn't mean Docker is suddenly dead outside of Kubernetes.

[TBBle]

The Docker for Windows plan seems reasonably clear to me: Per moby/moby#41455, it seems the current thinking is that Docker 21.x (or later, if not done in-time) will use containerd on Windows (as it has done on Linux for years) and consequently drop RS1 (Windows Server 2016) support.

[awakecoding]

@TBBle while Docker for Windows has a plan, what I'm not so sure about is the plan for the Microsoft Docker distribution. Will they simply go along with the plan described in moby/moby#41455 and leave Windows Server 2016 with an older version of Docker that uses HCSv1? I'm all for it, as I really only care about Windows Server 2019.

[TBBle]

Ah, I see. Since the person who opened moby/moby#41455 is a Microsoft employee, I am assuming that this reflects the MS plan for Docker on Windows Server. But it's only an assumption. ^_^

[msftbot]

This issue has been open for 30 days with no updates.
@brasmith-ms, please provide an update or close this issue.

[msftbot]

This issue has been open for 30 days with no updates.
@brasmith-ms, please provide an update or close this issue.

[msftbot]

This issue has been open for 30 days with no updates.
@brasmith-ms, please provide an update or close this issue.

[awakecoding]

The bot has already poked @brasmith-ms 3 times for an update... So, can we get a status update on this? Is a switch to containerd in Docker for Windows the chosen solution to the problem?

[brasmith-ms]

Hi all, @awakecoding, apologies for the delay in my response here! As mentioned previously, we have confirmed that containers operating over HCSv2 do not exhibit the graceful shutdown issues described here. Our focus is centered around the development of HCSv2 so we do not have plans to fix this in HCSv1. This will require a switch to containerd in Docker for Windows going forward. HCSv2 fully supports Windows Server 2019 (and later) containers, so today you can use WS2019 containers on HCSv2 with the ability to gracefully shutdown the container. Per moby/moby#41455 it is expected that docker will add HCSv2 support in 21.x

I'm going to close this, but if you have any additional questions please feel free to reach out to me.