[Bug]: stuart_setup etc. report ERRORs if Linux-x86, Windows-x86 etc. directories are deleted, and doesn't recover about edk2-pytool-extensions

[Javagedes]

Hi @bcran, What situation brought about an external dependency being half-way deleted? I can probably make some changes to catch this when calling stuart update (which is responsible for updating the external dependnecies) by re-downloading any external dependency that throws the error mentioned, but I'm hoping to figure out the situation that this happens in and try and prevent that instead.

The expectation for external dependencies is that they are to remain untouched as they are dependencies that are used, not co-developed (think of them as pip packages, we download them, and use them, but we don't really ever edit them)

Thanks,
Joey

[bcran]

I can't remember why I deleted them, but it's possible I was trying to clean up the clone and hadn't realized the extdep_state.yaml wasn't part of the repo.

[Javagedes]

There are two reasons that this error can occur:

1. The extdep has the "host_specific" flag which signals that the top-level directory of the external dependency contains folders specific to combination of os/arch/bit (see here) and that particular host combination is not supported.
2. the extdep has the "host_specific" flag and that particular host combination was supported, but the folder was deleted.

My original intent was to add sha256 hashing to all dependencies as default by hashing all files in the dependency as follows:

1. Download the external dependency
2. Hash all files inside the external dependency and set the commit hash in the extdep_state.yaml file
3. When environment verification is completing, re-hash the files and compare the commit hash.
4. If the hashes don't match the external dependency is redownloaded

Unfortunately, this breaks down when the extdep has an extdep inside of it. In this situation, the old hash becomes incorrect when the internal extdep is downloaded. There are a few ways we could work around this, but honestly it is extra complexity for an extremely uncommon situation.

In light of keeping the detection of this issue simple, this will be the new approach:

If the external dependency is "host_specific", the supported host combinations will be recorded in the extdep_state.yaml file and verified all directories still exist. While this won't catch changes internal to those files, it will at least be able to:

1. keep the current error, updating it slightly to say that the detected host combination is unsupported by this external dependency
2. Detect when a particular host combination was deleted, and re-download the external dependency.