guekka / bethutil Goto Github PK

It appears vcpkg caching does not work, wasting many CI minutes

Not only this lib is experimental, it is now abandoned in favor of flux. It has several bugs that will never be fixed, and likely undiscovered others

Crunch is a state of the art library for compressing with the standard DXTn. DirectXTex, while fine by itself, doesn't produce the best quality.

https://github.com/FrozenStormInteractive/Crunch2

Hi, as discussed, there are several things that must be handled to have full compatibility with FNV (and other older titles I assume).

Crucial

• Archive flags - As per Geckwiki, for a BSA to be usable by the game, it must be flagged correctly. There are two sets of bits, archive and file flags. The relevant (for FNV) are the following:

Archive flags:
00000000000000001 - Named Directory - always set
00000000000000010 - Named Files - always set
00000000000000100 - Compressed - use zlib compression
00000000000010000 - Retain Names - set with sound effects
00000000010000000 - Startup Strings - set with meshes
00000000100000000 - Embed Name - set with textures

The first three are self-explanatory. The other three are important for the given asset type. That is, if the archive in question contains either of the noted types, it must have the given flag set.

The file tags are self-explanatory. The important thing is that for the BSA to be usable, it must have the correct file flags set depending on the content - otherwise, the game will never check the archive for the given type. Full mapping of file extensions to file flags (pulled from engine) in comments.

• Disable BSA compression completely (CAO) - Compressed BSAs load very slowly due to old zLib version. Updated zLib version is still slower than using uncompressed BSAs and there is no real benefit to compressing them in terms of game performance/memory.

• Disable mesh and animation optimizations (CAO) - Animation optimizations are irrelevant and mesh optimizations cause breakage from preliminary tests. As per discussion with a mesh optimizer, it's not really worth it to try to figure out an automatic process. Best choice is to disable both completely.

• Ignore packing the following files - A list of extensions to ignore for BSA packing, as they could have trouble loading correctly/working in game.

KF - This is a priority. Most animations are build with an engine extension that currently does not support BSA packed KF files.
MP3
XML
JSON
INI

Necessary improvements

• Dummy plugins generated by CAO are unusable by the game - Dummy plugins must have at least FalloutNV.esm set as master. Even when added manually the attached BSA failed to load, so something is probably wrong with how they are generated, must debug further.

• Do not process LOD textures at all (CAO) - LOD textures are processed elsewhere and compression and mipmaps are handled there. These are files with the suffix _lod.dds or _lod_n.dds (potentially others, but I am not sure if such are ever used).

• Do not recompress files with different algorithm - Mostly connected to the next point.

• Use correct compression algorithms - bethutils currently uses the wrong compression algorithms for older Beth games (defined here).

The game supports DXT1, DXT3, and DXT5. DXT3 is useless. The current logic causes problems - textures without any alpha will be compressed with BC5 which is uncompatible with Dx9 the game uses (as far as I can tell). Textures using a single bit of alpha get processed with BC3 which fills out the alpha (for some reason), which can for example mess up the specular on normal textures. The correct logic is:

• if texture has up to 1 bit of alpha use DXT1 (BC1),
• otherwise use DXT5 (BC3).

In fact, using DXGI_FORMAT_R8G8B8A8_UNORM when not compressing does break the specular as well (specular - alpha component of normal texture). Wonder if there is some explicit alpha adjustment being done by bethutil itself or it's due to the format.

Mipmap generation - Mipmaps should be generated for everything but UI textures (textures/interface/*).

Important note: Gamma correction should be applied (and only ever applied) to diffuse textures. Assuming standard conventions (described here) this means gamma correction should be applied to any texture not ending with _n/_m/_em/_g/_sk/_hl/_e.

Scratched. Turns out DirectXTex has no direct support for gamma correction of mipmaps. It could probably be done manually, but not worth the effort.

Nice to have

• Automatic .override generation (CAO) - .override files are empty text files used by an engine extension to load the corresponding BSA at the same "load order" as the plugin (similar to newer Bethesda titles). If we have a plugin test.esp and archive test.BSA, this means generating test.override. Similarly, if splitting archives, this would mean having test 0.bsa, test 0.override, test 1.bsa, test 1.override.

Not crucial, but would be very helpful.

For example, if a file is converted to dds from tga, its extension should change

Most components use expected for error handling: nif, tex...
But common sometimes uses exceptions, in filesystem and strings
And bsa is the worst of all, sometimes using both or just raw errors (beware of the dreaded std::vector<std::pair<Path, std::string>>. Can you guess that represents an error?)

This archive has nothing special compared to the standard one. I'm wondering if we could use it to store all types of file, not only textures

If the files get bigger, the archive might get bigger than the original size. Worse, they can exceed the game limit, which would result in crash
We need to handle this case

Path comparison is dangerous because a != b does not implies !equivalents(a, b)
Maybe we should ban path comparison using == and provide an explicit function

Currently, the CI checks for invalid formatting and autocommits fixes. This has several issues:

• breaks with branch protection rules
• creates useless commits

Instead, it would be better to reject PRs with invalid formatting, letting the author fix it. We could also add pre-commit hooks

Restore listLandscapeTextures and listHeadparts from CAO

According to some people, it is possible to compress partially a BSA. This would remove the need for a different, uncompressed archive

It would be nice to be able to compare performance after changes. Could even be used in CI