microsoft / windows-containers

Welcome to our Windows Containers GitHub community! Ask questions, report bugs, and suggest features -- let's work together.

Welcome to the Windows Containers GitHub Repository

This repository is managed by the Windows Containers platform team and is dedicated to tracking features and issues related to Windows Containers. We monitor this repository to engage with our community, address questions, discuss customer scenarios, and consider feature requests.

Getting Started

Are you new to Windows Containers?

1. Start by setting up your local environment and running your first Windows container. Follow our documentation here for a step-by-step guide.
2. Next, explore our training module to learn how to deploy, manage, monitor, and operate Windows containers.

Product Updates

Stay informed about the latest features and improvements by visiting the Windows Container Blog.

For open-source tools that enhance the Windows Containers experience, visit the Windows Container Tools repository. We encourage you to test the tools, provide feedback, and contribute to the development of Windows Container Tools.

Troubleshooting

Encountering issues with Windows Containers? Microsoft offers a comprehensive set of resources for troubleshooting Windows Containers:

1. Windows Containers Troubleshooting Guide: This guide provides troubleshooting information for common issues encountered while working with Windows Containers.
2. Windows Containers Troubleshooting Training Modules: This training module offers a guided troubleshooting learning path for various aspects of Windows Containers.
3. Windows Containers Troubleshooting Workshop: This lab teaches advanced techniques for configuring, troubleshooting, and debugging Windows containers.
4. Windows Containers Networking Troubleshooting This article highlights step-by-step instructions for addressing common networking issues.

Issues

For specific bugs or issues related to the Windows Container platform, please use the Issues tracker on this GitHub repository. Issues without clear reproduction steps or that remain inactive for more than two weeks will be closed.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

windows-containers's Issues

Virtualized Time Zone in Windows Server Containers

Currently Windows containers use the host time zone by default. If a user wants to change the time zone they must change the time zone of the host. Running the tzutil executable in a Windows Server container will simply throw a lack of privileges error, and registry time zone settings are ignored upon starting the container. The only way of ensuring that the time zone is set properly is through configuration of the host – which defeats the package-and-ship paradigm of containers.

This issue has been discussed in detail here.

Upon the release of the fix for this issue users will be able to use the tzutil command, Set-TimeZone powershell cmdlet, or set the virtual registry key.

Networking - HNS crashing repeatedly on newer builds

Fix is ready. Needs to go through servicing pipeline.

[Feature] gMSA support in AKS

Support of gMSA so that Windows applications using gMSA can run as Windows Containers on AKS.

Networking - Enable Session Affinity in Kubeproxy for Windows

WAC-Push container images to OCI compliant repos

The containers extension on Windows Admin Center should allow users to push existing container images to OCI compliant repositories.

WAC-Manage Docker Swarm from WAC

Request from user on User Voice: https://windowsserver.uservoice.com/forums/295071-management-tools/suggestions/38866393-manage-windows-docker-swarm-from-container-extensi

Ability to manage Windows Docker Swarm from Container Extension. The current container extension only allows management of containers on an individual server. It would be good to manage containers in a Docker Swarm, services, secrets etc.

GMSA with Windows Containers doesnt work with ContainerD

When using the ContainerD runtime (Feature State: Kubernetes v1.19 [beta]) accessing restricted network shares via the GMSA domain identity fails. The container will receive the identity of and calls from nltest.exe /query will work. It is recommended to use the Docker EE runtime if access to network shares is required.

Support for Edge on Windows Containers for automated UI testing

Request from User Voice: https://windowsserver.uservoice.com/forums/304624-containers/suggestions/34081195-edge

Need Edge in a container for automated UI testing

Windows Server Containers hard link limit issue

Error: An attempt was made to create more links on a file than the file system supports. More details : moby/moby#40511

WAC-Create new image - Web applications

The new Create Image functionality should support basic web applications to be containerized:

• IIS web applications from folder for static web apps.
• IIS web applications from Visual Studio solution for ASP.Net applications.
• Any application from an existing dockerfile.

Latest Windows OS SAC and LTSC support in K8s

From SIG-Windows K8s Community : kubernetes/kubernetes#90142

Domain Join as computer account

Allow containers to be domain joined even if the host is not or even joined to a different domain.
Containers are meant to be a stripped down userland that contains everything the application needs to run without interacting with "the world outside the container".
Therefore the current approach of integrating containers into a domain by using service accounts is not great.

If I'm maintaining e.g. a web app that has ldap auth, the container should be able to perform this without requiring anything from the host, so that the container can be moved to another host without needing preparation.
Also I consider the secret that is used to connect to the AD "state" of the container and therefore the container should be able to store it on a volume itself controls. There is just no need for the host to treat these kind of information special.

Also as a service provider I want to be able to have the hosts within my activedirectoy, so that my admins and my automation can log into it, but also to offer the applications to customers. Now if the application is an domain integrated one it would (currently) require the host to be joint to the customer domain and an gMSA to be configured. I'd like it to be split. The container should be able to be joined to the customer domain (appear with a normal computer account) and be treated indistinguishable from normal computers but with stripped down services and functionality.
Such a setup would allow to make a service cut between providing the application and managing the host server. The service provider could than have the application running within a k8s cluster and providing automated fault tolerance and uptime monitoring while requiring no change to the containerized application (even applications not compatible with gMSA will work).

Networking - Improve HNS orphaned NIC cleanup logic

Windows Server Container cross-version host compatibility

Windows containers are currently more or less strictly version bound to the same version as the host (process isolation).
I do not count Hyper-V isolation, as that is in my opinion just another kind of VM hosting and has nothing to do with containers.

Dependencies can be viewed here: https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/version-compatibility?tabs=windows-server-2004%2Cwindows-10-2004

It would be very useful if windows userland could work independent of the kernel version.
E.g. a 2004 kernel being able to start a container with 1903 userland and vice verse.

As a developer bundling applications I want to be able to have a single container being able to run on multiple windows host (kernel) versions, so that I do not have to create the container for all windows versions that a customer might be running (maybe even on outdated versions because of other containers not being packaged against newer ones).

As a operator I do not want to consider containers when updating the host operating system so that I can install updates (maybe even automatically) when I want to.

As a user I do not want engage with the server operator to deploy an older host for some application container so that I can just "docker run" the application without having to consider the windows version it was build against.

As a security engineer I do not want operators holding back with installing windows updates (e.g. because of compatibility issues or the container not being available for the new version) so that all systems can be kept up to date without service impact.

WAC-Run new containers

The containers extension on Windows Admin Center should enable users to run containers locally (single-node) with Docker arguments.

docs for Host Compute Service (HCS) API + bindings ?

I wonder if there's a reference documentation for the Host Compute Service (HCS) C API and/or the Golang/.NET bindings for it? Both are mentioned here, but I lack to find any docs about it?

https://docs.microsoft.com/en-us/virtualization/community/team-blog/2017/20170127-introducing-the-host-compute-service-hcs#feedback

(Just to say, I'm not looking to replicate Docker/Moby functionality but am - as a Docker on Windows user - interested in debugging/troubleshooting beyond what's currently possible with hcsdiag.exe).

UI automation

Support UI (and UI automation) the windows ecosystem sadly is often "UI first" when it comes to software design. And the powershell/command line interfaces are not available or not documented at all. It would be great for sysadmins to be able to encapsulate these applications in containers, so that they are no longer bound to a single host and can be put into a k8s cluster without requiring something developers and Ranorex and a lot of time.

Provide support for Office for Windows Containers

Request from User Voice: https://windowsserver.uservoice.com/forums/304624-containers/suggestions/19686220-provide-office-support-for-containers

Many Business Application are dependent on Office to edit files or import/export of ppt or word formats. So adding office support for Containers would be a good idea.

Test Networking issue

This is a test issue

go-winio fails to import layers with a file larger than 8gB

Mirroring moby/moby#40444 as it appears to be a HCS-level issue.

---

Description

When trying to create a container on Windows with a file larger than 8gB (8,589,934,592 bytes of random data is my test-case), Docker fails to commit the layer, giving an error along the lines of re-exec error: exit status 1: output: write \\?\C:\ProgramData\docker\tmp\hcs816749287\Files\UnrealEngine\LocalBuilds\Engine\Windows\Engine\Plugins\Experimental\BlastPlugin\Binaries\Win64\UE4Editor-BlastAuthoring.pdb: There is not enough space on the disk. or hcsshim::ImportLayer - failed failed in Win32: The system cannot find the path specified. (0x3).

---

Reproduced on Windows Server LTSC 2019, and Windows 10 1903, 1909, and 2004.

See moby/moby#40444 for more details and reproduction steps, and a log of the hcsshim calls made by Docker in the reproduction case.

If necessary, I could try and produce a reproduction case calling https://github.com/microsoft/hcsshim directly.

WAC-Create new Image - From MSI installer

For customers trying to containerize applications that are deployed based on MSI files, the containers extension on Windows Admin Center should support MSI based applications. The new Create New Image functionality should support inputs for MSI files.

There are two scenarios for containers that should be supported:

• Applications that are deployed via MSI doesn't need an entry point.
• Applications that are deployed via MSI and run on the background and need an entry point.

Networking - Enable DSR mode in RS5 LTSC builds

Servicing is in progress.

Docker log rotation stops after reaching limit on Docker EE 19.3.05

Add for visibility on this project. The details are in Azure/aks-engine#3573

Create base container with full GUI support

Request from User Voice: https://windowsserver.uservoice.com/forums/304624-containers/suggestions/13785024-create-base-container-with-full-gui-support

Networking - Set custom VxLAN port during VxLAN setup

kubelet/dockerd consume a lot of CPU on Windows node

More details on AKS repo : Azure/AKS#1482

Push for stateless software design

Currently stateless software is more the exception than the norm within the windows ecosystem. For windows containers this is quite a challenge to overcome, as containers are not designed for stateful software. They are basically a tool to package an application without it's data. Therefore by design the application needs to be stateless and being able to resume from just it's database/files. For an upgrade it needs to be able to work with the files of the previous version, as the old version is just terminated and the application in the newer version is pointed towards the old files and started.
If one tries this with most windows software currently it just fails to start.
Therefore more in terms of educating developers and shifting best practices towards stateless software design within the windows ecosystem needs to be done. Compared with linux where every application works (or can be made working using a shart startup script), windows has still to catch up.
As an example, a container with ubuntu can run a whole desktop environment, one is running xrdp and another a custom web application that allows users to spawn there container/desktop environment and finally another one is running guacamole for html5 based rdp. Now the customer has a full vdi infrastructure within 4 containers (working on e.g. 2gb of ram and 2 cpu cores on any cpu architecture) and is able to login either via html5 or native rdp. Where as Windows requires a hypervisor and much much much more resources in general. In the linux example the root filesystem within the container can be mounted read only with the actual /home of the user being persistent. Now if the base container updates everything still continues to work for the user. Where as in the windows world applications start to bail out because they expect being able to do the resource/configuration version update within the installer/updater and not upon startup.

Therefore I'd like to see the following design goals/principals to be pushed more by microsoft:

• Push the PackageManager WinGet more
• Push steless software design concepts more. Most developers on windows don't know about steless design it seams.
• This should be more popular among windows developers https://12factor.net
• Get rid of the pet design within the windows ecosystem. And drive towards much more cattles based workloads (And finally drop the last pets).

WAC-Install Container feature from Containers extension in WAC

Request from user from User Voice: https://windowsserver.uservoice.com/forums/295071-management-tools/suggestions/38999869-containers-host-install-option

Provide an "Install" option for containers to make a server a container host

WAC-Run containers on Azure Container Instance from WAC

From the Containers extension on Windows Admin Center, users should be able to select an image and run that image on Azure Container Instance.

WAC-Pull container images

The containers extension on Windows Admin Center should enable users to pull container images from OCI compliant repositories into the targeted container host.

Signed Docker Images for Windows Images

Are there any plans to sign windows images so that we can take advantage of ACR Content Trust / image signing in Azure? There was a forum question over a year ago regarding this but it would appear this still hasn't happened.

As a Developer I want to be able to build Windows Containers with BuildKit

The windows BuildKit is only supported on Linux

WAC-Integrate with Kubernetes

Feedback from user at User Voice: https://windowsserver.uservoice.com/forums/295071-management-tools/suggestions/40627096--kubernetes-integrate-windows-kubernetes-in-wac

Integrate the Kubernetes Windows solution in the WAC.

Have the status from Kubelet ( like cpu/memory usage) in the WAC

Have the ability to easy check HSN networks and IPS

Smaller, compactor and dependable codebase

Windows is currently a big blob of proprietary software that changes between version without any way of referencing that your application needs either at least some specific library version or specific feature to be enabled. Even though there are manifest files for programmers, there is no way to provide this in a comparable way like package managers do. And the lack of a good package manager is just ineligible for windows containers.

Ideally only the libraries actually needed by the application are present within the container and no unnecessary parts of windows. And for that to work also system components need to be served using the same package manager as the software is (not necessarily from the same repository though). Windows system components need to be additive and within a package manager. E.g. Start with nothing and add some features it's a nano server, add some more it's a core server and add a few libraries it's a "full" server.
But if you do not need one of the features e.g. dhcp client, cmd or notepad you do not need to install it.

As this feature is great by its own, what is the benefit for containers one might ask? Well, you just need to have a base container that has only a working package manager and nothing else. Than you ran e.g. winget install PowerShell to get a powershell and also all system libraries powershell needs to work.

This will provide:

• lower resource utilization
• faster container pull time
• less security vulnerabilities, something that does not exist cannot have a vulnerability.
• Much simpler to identify and troubleshoot applications (even if the bug is within a native library)
• Much simpler to identify applications that potentially break upon updates of os components (and rollback just that one component / library)
• Reduce the amount of system updates. Only installed components need to be updated and intermediate library versions can be skipped instead of all patches being applied sequentially and the same library being overwritten several times.
• (for the microsoft developers) allows to separate out the os components into separate projects, focus on the individual components. The base windows is currently too cluttered with features/stuff that on most machines will never be used (e.g. Modem libraries or Map provider on a file and database servers).
• (for the microsoft developers) will simplify writing patches as them themselves know which what things they touch depend on.
• (for the microsoft developers) allow them to better (and also automatically) test updates, estimate test coverage and identify libraries that do not have a maintainer (for some things in windows it looks like there is not a single developer taking care of it).
• (for the microsoft developers) Allows to move stuff out of the windows mono repository into separate projects. The announcement how big it is and that it is by far the biggest git repository was shocking.
• (for application developers) allow to identify breaking os patches much quicker, as only patches to used libraries need to be considered.
• (for application developers) allows to do automated testing with different patch levels and library versions quickly.
• (for system administrators) needing to install less updates and needing to care about less security vulnerabilities.
• (for system administrators) knowing exactly what runs on a system. No unnecessary background services.

WAC-Better real-time monitoring data on WAC

Request from user on User Voice: https://windowsserver.uservoice.com/forums/295071-management-tools/suggestions/40531138--container-managing-windows-server-inside-the-con

We have now the Container Feature in Windows Admin Center, but is it possible to manage this Windows Server Container like a Windows Server VM with WAC Features when the Container is running?

Here is some more information what I mean by Azure Monitoring Containers :

https://docs.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-livedata-overview

This would be an awesome feature for in WAC too and just see what's running inside the container and if it's healthy or not.

Add support for MSDTC on Windows Containers

Request from User Voice: https://windowsserver.uservoice.com/forums/304624-containers/suggestions/30982237-support-for-msdtc

We have legacy .NET based business applications making extensive use of MSDTC. It will be good to have support for this role in containers to enable Lift & Shift scenarios.

testing triage both

[Feature] Calico Network Policy Support for Windows Containers on AKS

Calico Network policy support for Windows Containers on AKS

WAC-Create new container images

The containers extension on Windows Admin Center should allow users to create new container images directly from the WAC UI. The goal is to have a wizard like experience on which the users can provide the configuration of the application they'd like to containerize and WAC would:

• Create the preview of the dockerfile.
• Allow edits to the dockerfile preview.
• Run the Docker Build process to create a new container image locally.
• Store the new dockerfile on the application folder.

Windows container monitoring events and logging improvement ask

Due to the nature of docker, all console logs are pushed out thru stdout and stderr. Since Windows application places their given windows format, customer as coming back asking how can they push their logs thru docker format but able to use to troubleshoot. Some of the customer responses we see are IIS logs.

• length of the log should fit stdout/stderr
• log format should fit the common standard which is close to linux so vendors can consume

Performance Issue on Windows servercore container

In prepping for general availability of Windows Server, version 2004 release, we identified a potential performance issue. This issue is noticeable compared to using Windows Server, version 1909 images.

The symptoms that we observed are:

If you use the Server Core container image to build your own image and upload to a remote container registry such as Azure Container Registry, and then you pull that image from the registry and run it, you will see a slower performance of the container. However, if you build the image and run the image locally, you will not observe that performance difference.

This issue is now root caused and we are working on the fix.
microsoft/hcsshim#830
moby/moby#41066
containerd/containerd#4301

We are also actively with our partner Mirantis getting that fix in Docker EE.

WAC-Create new image - Web applications from WebDeploy

To support customers who currently have a web application deployed on an IIS web server, the Create New Image functionality on the containers extension on WAC should allow customers to containerize an application that was exported from IIS with WebDeploy.

Windows containers do not have entries in the hosts file (Single File Mapping)

File Mounting support bug reported in K8s SIG-Windows : kubernetes/kubernetes#70189

Windows Server 2019: LockFile lock is not released by OS when process exits

Combining: moby/moby#39088 and docker-library/mongo#385 and StefanScherer/dockerfiles-windows#349

Description from @drnybble on (moby/moby#39088) - thank you!

Description

This problem is exhibited by running Mongo 4.0.8 with a named volume. If you restart Mongo it will not start because the WiredTiger.lock file remains locked. This is a general problem demonstrated by a sample Windows CLI program shown below.

Steps to reproduce the issue:

Using Visual Studio 2017, compile the Windows CLI application shown below. I used the C++ code generation option "Multi-threaded" so the VS redistributables did not need to be included in the Dockerfile.

#include "pch.h"
#include "Windows.h"
#include <iostream>
int main() {
	const wchar_t* FILE_NAME = L"c:\\data\\Test.lock";
	HANDLE h = ::CreateFile(FILE_NAME, GENERIC_WRITE, 0, NULL, CREATE_NEW, FILE_ATTRIBUTE_NORMAL, NULL);
	if (h == INVALID_HANDLE_VALUE) {
		std::cout << "CreateFile failed";
		exit(1);
	}
	std::cout << "Created file " << FILE_NAME << std::endl;

	const char* buf = "Test";
	DWORD written;
	if (!::WriteFile(h, (LPCVOID)buf, sizeof(buf), &written, NULL)) {
		std::cout << "WriteFile failed" << std::endl;
		exit(1);
	}
	std::cout << "Wrote content to file " << FILE_NAME << std::endl;

	if (!::LockFile(h, 0, 0, 1, 0)) {
		std::cout << "LockFile failed" << std::endl;
		exit(1);
	}
	std::cout << "Locked first byte of file " << FILE_NAME << std::endl;

	std::cout << "Exiting without closing handle, OS must unlock" << std::endl;
}

Use the following Dockerfile:

FROM mcr.microsoft.com/windows/servercore:ltsc2019
COPY LockFileTest.exe /
CMD ["C:\\LockFileTest.exe"]

Build it:

docker build -t locktest . 

Create a named volume:

docker volume create locktest 

Run it:

docker run -v locktest:c:\data locktest 

Describe the results you received:

A file called Test.lock is located in the named volume C:\ProgramData\docker\volumes\locktest_data. Try to open it with Visual Studio Code, it fails with the error EBUSY. The file remains locked even though the owning process has exited.

Describe the results you expected:

As described by the documentation for LockFile (https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-lockfile) the OS should release the lock when the process exits. If you run the LockFileTest.exe on the desktop you will find that the file is not locked after the process exits.

Additional information you deem important (e.g. issue happens only occasionally):

Workaround is that you can delete the file even though it is locked. So for Mongo you can delete the WiredTiger.lock file on startup if it exists.

Graceful Shutdown of Windows Server Containers

The ability to stop and start containerized workloads is critical for minimizing cost and unnecessary resource usage. Once a container is deployed there is a high probability that it will eventually need to be shut down for servicing, migration, or some other domain specific reasoning. Thus, a container must not be shut down forcibly without notifying the containerized application first. Failing to do so can cause unwanted side effects and allow for the accumulation of corrupted and useless data.

Windows Server containers currently have limited support for graceful shutdown. The -t parameter passed to docker stop is effectively ignored, and the user has little control over when the container is forcibly terminated. The fix for this issue will give the user a greater range of control over this scenario

docs for Host Compute Service (HCS) API

I wonder if there's a reference documentation for the Host Compute Service (HCS) C API and/or the Golang/.NET bindings for it? Both are mentioned here, but I lack to find any docs about it?

https://docs.microsoft.com/en-us/virtualization/community/team-blog/2017/20170127-introducing-the-host-compute-service-hcs#feedback

(Just to say, I'm not looking to replicate Docker/Moby functionality but am - as a Docker on Windows user - interested in debugging/troubleshooting beyond what's currently possible with hcsdiag.exe).

Azure Spot support for Windows Containers on AKS

Azure/AKS#982

Support for MSMQ on Windows Containers

In order to containerize existing applications, MSMQ (a popular feature in Windows Server) needs to be supported on Windows Containers.

This request came from User Voice: https://windowsserver.uservoice.com/forums/304624-containers/suggestions/15719031-create-base-container-image-with-msmq-server

The MSMQ-Server role is a prerequisite for Particular NServiceBus installation. As NServiceBus handlers are potentially a good fit for containerization, it was disappointing to find that it is not possible to install MSMQ-Server in a container. Please create a base image with this support!

WAC-Make Windows Admin Center as a container

Request from user from User Voice: https://windowsserver.uservoice.com/forums/295071-management-tools/suggestions/32408029--deployment-container-containerize

Make Honolulu available as a container image. This would be helpful for creation on semi-annual servers.

Memory leak on Windows nodes, svchost.exe consumed a lot of memory

Reported by AKS users : Azure/AKS#1397