Virtualized Time Zone in Windows Server Containers about windows-containers

[brasmith-ms]

Hi everyone, apologies for the delayed update! Our team was unfortunately hit with some high sev issues that entirely consumed our development cycles and delayed the completion of this feature.

Nonetheless the feature is complete and we are working through validation and getting the feature through OS servicing. Right now we are on track to hit the 4C servicing deadline, which translates to a release in late April. I understand this is later than expected, but again we are working to get this feature out ASAP. I will be sure to update this thread again once the feature is on its way through servicing. Feel free to reach out to me if you have concerns - I am more than happy to work something out if needed.

For more info on servicing:
https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/update-containers
https://techcommunity.microsoft.com/t5/windows-it-pro-blog/windows-10-update-servicing-cadence/ba-p/222376

@SachinL9 the workaround there simply creates a K8s DaemonSet that sets your preferred time zone of each node in your cluster. The current behavior of Windows server containers is that the container shares the time zone settings of the host, so if you set the time zone of a host it will be reflected by the container. So with the DaemonSet workaround every pod deployed should share the same configured time zone.

With the change we are introducing in this feature the time zone will be completely virtualized within the container. When you launch a container it will mirror the TZ settings of the host, but you can configure the container's time zone from within the container using tzutil or Set-Timezone in powershell. Doing this requires a significant number of kernel changes, which is why it's been taking time to get this feature out.

Again if you have other questions, don't hesitate to reach out to me!

[brasmith-ms]

Hi all, thank you for your patience! The feature work is in its final stages and we are currently preparing it for the servicing pipeline. Taking the holiday servicing gap into account, our aim is to get this feature out to the public for the February/March servicing timeframe. I will continue to keep this page updated with status as we go, and again feel free to reach out to me if you have specific concerns.

Rest assured we are working on getting this out as soon as we possibly can, as we do recognize the degree of its importance.

An additional note, if you are running containers in Kubernetes there is the option to set the time zone of the Kubernetes pod which can act as a temporary workaround:

https://mohammedomar.net/2020/08/18/changing-timezone-for-kubernetes-windows-based-nodes/

https://github.com/MikkelHegn/k8-deployments/tree/main/other_demos/set_timezone

[brasmith-ms]

@GerryWilko good points! This feature will be available on 1809 (server 2019 LTSC), 2004, and 20h1 (for reference). It will not be available on 1903 or 1909 since both will be EOL by the time this is released. It will also be available on Windows server 2022 once it GAs. Basically, all mainstream supported versions will have this feature 🙂

And you are correct, at the moment insider releases cannot be run on AKS so use of this version would purely be for dev / testing purposes. If there are any specific concerns on this please reach out to me, happy to work something out.

[yylai]

Hi,

We are also interested in any updates & the progress of this issue as it might impact our design/plan for migrating legacy applications into AKS windows containers.

Thanks!

[brasmith-ms]

Hi folks! Jumping in here with another update. The feature is complete and was checked in and tested for the 4C release. An important thing to note, however, is that this feature requires both the host and guest to be updated to function properly. Per the Windows servicing process container images are only released during B releases, so the feature was released for 4C but guest images won't be available publicly until 5B (May patch Tuesday, the 11th). I appreciate your patience in waiting for this and am anxious to get this feature into your hands.

We will be releasing documentation for the feature over the next week or so as well, so stay tuned!

[GerryWilko]

That great @brasmith-ms. Congrats to the team!

What does this mean for AKS? Do we need to wait for an Kubernetes version bump? I don't think we have options to select node pool OS version.

Cheers!

[GerryWilko]

Might have found the answer here actually. Azure/AKS#2295

Seems to be rolling out already.

[brasmith-ms]

Hey everyone! We have published the documentation for this feature. Thanks again to everyone for your patience, please let me know if you hit any problems or have additional suggestions!

https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/virtual-time-zone

[giangpham05]

I'd like to see this feature coming out as well. There are a number of use cases we want to be able to change the time zone. It would be ideal if we could do that through docker/AKS arguments, leaving the host's timezone untouched.
Something like docker run -e TZ=Australia/Brisbane my-windows-image as we would with Linux's.

[brasmith-ms]

Hi all, thank you for raising your concerns here. The fix for this is actively in development and I will keep you all in the loop once it is ready for release. We recognize the importance of this issue so we're working to get the solution out as fast as possible.

Again if you have specific concerns that you would like to mention I would be happy to hear it. Always helps me to hear customer use cases. Feel free to reach out to me via [email protected], or on Twitter https://twitter.com/brandon8675_309.

[GerryWilko]

Yeah I just found this help article. https://docs.microsoft.com/en-us/azure/aks/node-image-upgrade

Running the upgrade now to see if this resolves the issue.

Many thanks!

[jayeekayoh]

I agree. The inability to set a container timezone independent from the host is a significant limitation. In scenarios where applications need different timezones, the only solution is to sort them onto corresponding hosts with matching timezones. This is not the most convenient workaround in platforms where time consistency across hosts and infrastructure is critical for logging and troubleshooting.

[msftbot]

This issue has been open for 30 days with no updates.
@brasmith-ms, please provide an update or close this issue.

[MatheusLFreitas]

Hello @brasmith-ms Do you have any updates on this issue?

[thovipe]

Hi, @brasmith-ms and @vrapolinario
Do you guys have an update?

[jcjwent]

I am also eager to use this feature. Any update?

[msftbot]

This issue has been open for 30 days with no updates.
@brasmith-ms, please provide an update or close this issue.

[msftbot]

This issue has been open for 30 days with no updates.
@brasmith-ms, please provide an update or close this issue.

[SachinL9]

Hi all, thank you for your patience! The feature work is in its final stages and we are currently preparing it for the servicing pipeline. Taking the holiday servicing gap into account, our aim is to get this feature out to the public for the February/March servicing timeframe. I will continue to keep this page updated with status as we go, and again feel free to reach out to me if you have specific concerns.

Rest assured we are working on getting this out as soon as we possibly can, as we do recognize the degree of its importance.

An additional note, if you are running containers in Kubernetes there is the option to set the time zone of the Kubernetes pod which can act as a temporary workaround:

https://mohammedomar.net/2020/08/18/changing-timezone-for-kubernetes-windows-based-nodes/

https://github.com/MikkelHegn/k8-deployments/tree/main/other_demos/set_timezone

May we get an update on when this will be available?

We are desperately looking for ability to change windows container timezone to what's desired.

Regarding the workaround provided by you, we tried this in a dev environment and it seems to be working fine. Can you confirm if this is safe to be done in a production environment too?
Also, I am not sure what is timezone of k8s master, would different timezones in master and windows node pool cause any issues?

[yylai]

Hi,

I read that virtualized timezones will be available in the upcoming Windows Server 2022, but I am assuming the feature being worked on in this issue will not require Windows Server 2022, is that correct? Sounds like the work done here will be brought into 2022 :)

https://techcommunity.microsoft.com/t5/containers/what-s-new-for-windows-containers-on-windows-server-2022/ba-p/2167036

Thanks again!

[brasmith-ms]

Hey everybody, sending another quick update that we are still on track to hit the 4C release! The feature required a significant number of modifications to both the Windows kernel and guest OS so we've been running it through rigorous testing to ensure the feature works properly.

It should also be noted that the serviced container images are not produced externally until the following B release, which in this case will be about mid-May. This feature requires both an updated host and guest to function properly. I will be putting together detailed documentation on this in the next few weeks.

@yylai that is correct! We developed this feature as part of the mainline OS and then backported it to WS2019 and other intermediary releases. If you'd like to test it out the feature is available to insiders today. And again, more documentation will follow over the next few weeks.

[GerryWilko]

Hey @brasmith-ms, thats great to hear!

Not sure if I'm the only one who gets confused about this but could you (or anyone else) clarify will this feature be released into the LTSC channel or SAC channel? If LTSC will MSFT be updating the tag from ltsc2019?

Also am I correct in thinking that we cannot run Insider releases on AKS? Windows container support requires the host OS to be updated to the same level IE our nodepools would need to also be on the insider track which we cannot enable in AKS currently.

[sicklittlemonkey]

This seems to be working now. From a colleague's testing, no luck with tzutil but Set-TimeZone works. Thanks!

[remenaker]

@sicklittlemonkey what node image are you using? I just upgraded to the latest version and still got a Access Denied error.

[sicklittlemonkey]

@remenaker - Our node OS image reference from the portal is:

Publisher: MicrosoftWindowsServer
Offer: WindowsServer
SKU: 2019-Datacenter-with-Containers
Version: latest

We use the following base image and tag in our Dockerfile:
FROM mcr.microsoft.com/dotnet/framework/aspnet:4.8-windowsservercore-ltsc2019

[cwiederspan]

@sicklittlemonkey - Just to make sure I'm barking up the right tree, can you confirm that you've seen this work in AKS or Kubernetes? Just trying to confirm all the pieces that I need to worry about between the application's base image and the node's OS image. Any details would be great.

[sicklittlemonkey]

@cwiederspan - We use Docker with Service Fabric, but yes, Set-TimeZone seems to be working (but not tzutil).

PS C:\> date
Friday, May 14, 2021 11:01:45 PM
PS C:\> Get-TimeZone
Id                         : AUS Eastern Standard Time
DisplayName                : (UTC+10:00) Canberra, Melbourne, Sydney
StandardName               : AUS Eastern Standard Time
DaylightName               : AUS Eastern Daylight Time
BaseUtcOffset              : 10:00:00
SupportsDaylightSavingTime : True
PS C:\> Set-TimeZone -Name "New Zealand Standard Time"
PS C:\> date
Saturday, May 15, 2021 1:02:09 AM

Note that tzutil /s appears to change the time zone without error, but the time doesn't change:

PS C:\> tzutil /g
New Zealand Standard Time
PS C:\> tzutil /s "AUS Eastern Standard Time"
PS C:\> tzutil /g
AUS Eastern Standard Time
PS C:\> date
Saturday, May 15, 2021 1:04:32 AM

[mattjohnsonpint]

I may be mistaken, but it seems that Set-TimeZone broadcasts a WM_SETTINGCHANGE message after it changes the time zone, but tzutil /s does not. IMHO, that should be fixed.

[cwiederspan]

FWIW, I still cannot get this to work in Azure Kubernetes (AKS). My application image using the latest base images now builds fine, it works locally, but in AKS the time zone in the container is not set/persisted. I suspect it has to do with the node image being out of date. I think @brasmith-ms is also working this angle, so maybe he'll provide an update if/when he's got something.

[mattjohnsonpint]

Actually, I was mistaken. The Windows message is sent either way.

The issue @sicklittlemonkey reported is that PowerShell relies on the cached data in TimeZoneInfo to output the date command. Set-TimeZone is a PowerShell cmdlet implemented in .NET, and calls TimeZoneInfo.ClearCachedData when it's done setting the time zone. tzutil /s on the other hand is implemented in native code, and thus doesn't have the ability to clear the TimeZoneInfo cache.

You can see this yourself if after the example @sicklittlemonkey showed, call [System.TimeZoneInfo]::ClearCachedData(), then call date again and it will pick up the change.

That happens regardless of whether running in a container or not. My recommendation is that if you are using PowerShell and continuing with other commands in the same session, that you use Set-TimeZoneInfo. Otherwise using tzutil /s is safe. (They both set the system time zone in the same way, through the SetDynamicTimeZoneInformation Win32 API.)

[brasmith-ms]

Hi everyone! Yes the V-TZ feature is now available on MCR. For Windows server 2019 the image version is 10.0.17763.1935, and for the latest SAC release it is 10.0.19042.985. Note that both the host and container image must be at least this version to work properly as the feature had both kernel and user space modifications.

I'll be publishing more detailed documentation on MSDN in the next week or so to help detail the differences. The gist of this is that the best way to set the time zone in the container is through tzutil or powershell Set-TimeZone. For images without either of these functions and utility that calls the SetDynamicTimeZoneInformation Win32 API will be necessary.

For the folks running on AKS I apologize for the confusion here. AKS takes some time to ingest new servicing releases and they are currently working on building a new AKS Windows VHD based on the 5B servicing release which should be included in next week's AKS RP release (5/20 target). Once this is available the V-TZ feature will be fully functional using the latest MCR images.

[AbelHu]

It seems like that both Set-TimeZone and tzutil work without errors in the Windows containers in the Windows host with 5B. But as @sicklittlemonkey said, tzutil does not change the datetime.

[GerryWilko]

Hi, @brasmith-ms,

Do you have an update for us on this feature? I am still unable to get this feature to work on AKS and without the docs for this feature its unclear what we need to do to get it to work.

Currently running the Set-Timezone as a RUN command in a Dockerfile for context:

RUN Set-Timezone -Name 'GMT Standard Time'

At the moment as far as I can tell I have patched our build agents to latest available version (currently 10.0.17763 Windows Server 2019 Datacenter). AKS is showing AKSWindows-2019-17763.1879.210414 with latest release even after a full node recycle.

One thing that would be great to clear up is does this Timezone patch need to be run each time a container starts? Does the built image retain this virtualised setting or does this need to be executed in an ENTRYPOINT powershell script for example?

[AbelHu]

@GerryWilko, which region are you using? The latest AKS Windows VHD should be 17763.1935.210513 and Set-TimeZone only works in this VHD with 5B.

[GerryWilko]

Hi @AbelHu,

I am in North Europe for my AKS.

Cheers

[AbelHu]

@GerryWilko

{
  "automaticOsUpgradeProperties": {
    "automaticOsUpgradeSupported": false
  },
  "dataDiskImages": [],
  "disallowed": {
    "vmDiskType": "None"
  },
  "extendedLocation": null,
  "features": null,
  "hyperVGeneration": "V1",
  "id": "/Subscriptions/8ecadfc9-d1a3-4ea4-b844-0d9f87e4d7c8/Providers/Microsoft.Compute/Locations/northeurope/Publishers/microsoft-aks/ArtifactTypes/VMImage/Offers/aks-windows/Skus/aks-2019-datacenter-core-smalldisk-2105/Versions/17763.1935.210513",
  "location": "northeurope",
  "name": "17763.1935.210513",
  "osDiskImage": {
    "operatingSystem": "Windows",
    "sizeInBytes": 32214352384,
    "sizeInGb": 31
  },
  "plan": null,
  "tags": null
}

[GerryWilko]

So after upgrading my nodes to AKSWindows-2019-17763.1935.210513. This feature still doesnt seem to work. This is a container built on 10.0.17763 Windows Server 2019 Datacenter. And deployed to the cluster. Running Get-Timezone I get the following:

I can manually set the timezone with Set-Timezone but it doesnt respect the Timezone the image's Dockerfile set at build time. Is the only option to run Set-Timezone at run time in an ENTRYPOINT script?

[cwiederspan]

@GerryWilko - With a little help from @brasmith-ms, I was able to confirm on Friday that this was all working for me on AKS. My approach was this...

• Create a custom Dockerfile that simply hard-codes the new time zone.

• I had to manually run an az aks nodepool upgrade command to get my node images updated to the >= 1935 builds.

• Once I deployed the new app/container, I exec into the pod's container to prove out that it's running as the desired time zone.

As I said, this all worked for me. In reality, I wouldn't want to hard-code the time zone into the Dockerfile, and would instead want to set it at container creating time based on an env or config variable, otherwise I'd need a different image for every time zone (duh!).

[GerryWilko]

Hi @cwiederspan,

Thanks! Strange then, I'm starting to think that I have not patched the build agents to the right level to make this work. Can you confirm what OS version you built your Dockerfile on?

Powershell Get-SystemInfo should do the trick. I'm wondering if the patch hasnt been released for 2019 ltsc just yet.

[cwiederspan]

I built the container image on Azure Container Registry using an az acr build command - see this README.

The one thing that I did struggle with was ensuring that the AKS node's OS image was up to date with the latest version. To do so, I ran the aforementioned az aks nodepool upgrade command (which took about 10+ mins to run). Once I did that, I ran...

az aks nodepool show \
--resource-group $NAME \
--cluster-name $NAME \
--name win001

That then shows me some output like this...

...
  "mode": "User",
  "name": "win001",
  "nodeImageVersion": "AKSWindows-2019-17763.1935.210513",
  "nodeLabels": null,
  "nodePublicIpPrefixId": null,
  "nodeTaints": null,
  "orchestratorVersion": "1.20.5",
...

Once I saw that nodeImageVersion showing the new .1935. version, then things started to work.

[GerryWilko]

Thanks for this @cwiederspan!

Ok so it looks like it should all be working but we just need to know when to expect this 5B servicing release in normal windows updates channels so that we can patch our on-premise build agents. @brasmith-ms @AbelHu would that be right? Is there any way I can similarly force upgrade of my Windows Server 2019 LTSC build agents to pull the 5B patch?

[brasmith-ms]

@GerryWilko the 5B update is already available and live via the normal Windows Server Update Service / Windows update channels so you should just be able to follow the regular process. Let me know if this doesn't work for you. https://support.microsoft.com/en-us/topic/may-11-2021-kb5003171-os-build-17763-1935-3f03e74b-4759-4ca3-b9f1-4bc0d5ab5d27