Packages not updating? about unitynuget

[xoofx]

I could see the zip of the new versions for SonarAnalyzer, but another package (I think it is MessagePack) is failing to update, and to the whole process fails to update.
I restarted the server and the server is now offline because of this. I'm gonna try to fix this locally and push a fix.

[xoofx]

I can think of exposing a /status, /healthcheck or similar

@bdovaz I like the idea!

If we define the mechanism, I offer to create the PR.

We can start with the minimum required to provide diagnostics for the problem described above, and later add more features if necessary.

So I would think that:

1. The RegistryCacheUpdater everytime there is a progress in the registry + a log error, we create a new reporting instance e.g RegistryCacheReport and store it along the _currentRegistryCache. As for the RegistryCache, this instance is created from scratch after each update (e.g can be a clone of the actual report being build)
2. We can add a small html endpoint e.g /status to the existing controller that will get this report from the cache singleton and display a simple HTML page (no need to be fancy, we don't need razor or anything complicated, that can be done with a simple HTML scriban template).
3. The RegistryCacheReport can store anything that is relevant to investigate the state of the registry: e.g if there is an update in progress or not. The current log of errors. The package listed with their versions...etc.

We might want also to improve the error recovery model today, as it is a "all or nothing". For example, if a new package version is not supported, it should log an error, but it should not make the entire registry unusable.

That's my first thoughts on this.

This way at least we can quickly see who is "the culprit" and create a PR as soon as possible.

Go ahead! 🙂

[xoofx]

Thanks, unfortunately I'm on holidays and will be only able to check this in August

[bdovaz]

@xoofx a simple reminder, now that I see that you are active

[xoofx]

For me the definitive solution to the lack of space is to generate them in a lazy way.... Until a package is requested for the first time, it is not downloaded and upm package generated.

Afair, I couldn't do that for the reason that you need to generate the -/all request with all the information of the packages so that UPM can present them, and that information includes the link to the tar.gz, their hash and dependencies...etc. Since then, I haven't investigated what exactly is required in the -/all that would make it working (with the dependencies...etc.)

So if you want to investigate this further, you can, but it might not be that straightforward.

[xoofx]

So I think that what is happening is that a new version of a package added a new dependency, and the update is failing.

[bdovaz]

Knowing all of us that this problem may return in the future and there is no mechanism to control it.

@xoofx can you think of something that can be developed to at least have some mechanism that can help us identify this problem from outside? And from the outside I mean without having access to the service logs in Azure.

I can think of exposing a /status, /healthcheck or similar that exposes the result of:

https://github.com/xoofx/UnityNuGet/blob/master/src/UnityNuGet.Server/RegistryCacheUpdater.cs#L63

And if you want filtering the exception if you don't want to expose the whole stacktrace for security.

If we define the mechanism, I offer to create the PR.

This way at least we can quickly see who is "the culprit" and create a PR as soon as possible.

[xoofx]

So the registry is still down, I checked the logs and the error is

Unexpected error System.IO.IOException: There is not enough space on the disk. : 'D:\local\LocalAppData\NuGet\v3-cache\670c1461c29885f9aa22c281d8b7da90845b38e4$ps_api.nuget.org_v3_index.json\list_messagepack_index.dat-new'

Gonna need to see what to do with that.

[xoofx]

Looking at issue like this don't see a workaround, hm... 😒

[xoofx]

Getting more details on the admin page about the situation:

D:\home usage: 51,200 MB total; 26,189 MB free
D:\local usage: 11,264 MB total; 0 MB free

So I could move the NuGet folder to D:\home instead but 26GB might be not enough in the long term. Gonna check the disk space configured and If I can change it.

[bdovaz]

For me the definitive solution to the lack of space is to generate them in a lazy way.... Until a package is requested for the first time, it is not downloaded and upm package generated.

In order not to lose the initial validation and the validation updates every x time, we continue doing them but without generating the packages.

This way we also gain in initial startup time and GitHub workflow time when validating PR.

If we want, we can leave it configurable whether it is lazy or not so as not to change the operation for those who still prefer it to work in the previous way.

[bdovaz]

I can think of exposing a /status, /healthcheck or similar

@bdovaz I like the idea!

If we define the mechanism, I offer to create the PR.

We can start with the minimum required to provide diagnostics for the problem described above, and later add more features if necessary.

So I would think that:

1. The RegistryCacheUpdater everytime there is a progress in the registry + a log error, we create a new reporting instance e.g RegistryCacheReport and store it along the _currentRegistryCache. As for the RegistryCache, this instance is created from scratch after each update (e.g can be a clone of the actual report being build)
2. We can add a small html endpoint e.g /status to the existing controller that will get this report from the cache singleton and display a simple HTML page (no need to be fancy, we don't need razor or anything complicated, that can be done with a simple HTML scriban template).
3. The RegistryCacheReport can store anything that is relevant to investigate the state of the registry: e.g if there is an update in progress or not. The current log of errors. The package listed with their versions...etc.

We might want also to improve the error recovery model today, as it is a "all or nothing". For example, if a new package version is not supported, it should log an error, but it should not make the entire registry unusable.

That's my first thoughts on this.

This way at least we can quickly see who is "the culprit" and create a PR as soon as possible.

Go ahead! 🙂

Created a PR #129