stephen-bunn / bethesda-structs Goto Github PK

A wrapper for Bethesda's popular plugin/archive file formats

License: MIT License

Python 99.94% Shell 0.06%

bethesda filetype structures archive python36 construct

bethesda-structs's Introduction

Hey 👋 I'm Stephen

I'm a "full-stack" dev who mainly uses free time to create small tools for myself. Most of my professional time has been spent working with Python, Rust, or various JS frontends. Mainly my experience has been focused on web development and internal engineering tools.

If you would like to view more information about my experience, you can skim my resume at bunn.io. If you are interested in contacting me about open opportunities, please reach out with an email to [email protected].

Otherwise, you can find me on Mastodon.

bethesda-structs's People

Contributors

Stargazers

Watchers

Forkers

zhengc1 clayne cornytrace sw4t

bethesda-structs's Issues

Save Parsing

I would like to have the ability to parse Bethesda's save format.
Should be a contrib type parser

Haven't looked into it yet

TES3 Plugin Parsing

TES5 Plugin Parsing

Plugin Subrecord Parsing

The most critical and logically difficult problem for parsing plugins is determining the subrecord structure to use for parsing the subrecord. This is difficult due to how nested subrecord collections are allowed including the options to be optional and multiple.

Handle flat subrecord structures
Handle "optional" subrecord structures
Handle "multiple" subrecord structures
Handle infinitely nested subrecord collections
Handle "optional" subrecord collections
Handle "multiple" subrecord collections

A quickie post about the issue: http://stephen.bunn.io/blog/2018/07/02/parsing-subrecords/

BSL Workspaces

FNV Plugin Parsing

Fallout: New Vegas support for parsing subrecord structures for various record types in .esp, .esm, and .esl plugins.

Records can be built mostly from fopdoc.

Support for Building / Writing

As a hobby project, i'm trying to write a python script to automate some transformations on esm / esp files. Your library looks excellent, however after a bit of reading I figured out that there is no write support (or build support in construct terms). I figured I could do something like plugin.plugin_struct.build_file(plugin.container, "Test.esm"). This does produce a valid file, and if you change something like plugin.container.header.version it does show in Test.esm. However, if something in a subrecord is changed (e.g. the CNAM author name in the header), this does not propagate to Test.esm.
What changes would be necessary to enable writing esm's?
Is this a feature you are planning / would be willing to implement?

fails to extract Cathedral Weathers - Textures.bsa

Hi,

Just wanted to report a problem. I tried to use bethesda-structs to extract Cathedral Weathers - Textures.bsa from the Cathedral Weathers and Seasons mod, and ran into an lz4-related crash.

My python:

(venv) PS D:\git\skyrim-builder> python
Python 3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23 2018, 23:09:28) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

My virtual environment:

(venv) PS D:\git\skyrim-builder> pip list
Package                Version    Location
---------------------- ---------- -----------------------------
atomicwrites           1.3.0
attrs                  19.1.0
bethesda-structs       0.1.4
cached-property        1.5.1
certifi                2020.4.5.1
chardet                3.0.4
Click                  7.0
colorama               0.4.1
construct              2.9.45     
flake8                 3.6.0
gitdb2                 2.0.5
GitPython              2.1.11
idna                   2.9
Jinja2                 2.10
lxml                   4.3.0
lz4                    3.0.2
MarkupSafe             1.1.1
mccabe                 0.6.1
more-itertools         7.0.0
multidict              4.5.2
pip                    18.1
pluggy                 0.6.0
py                     1.8.0
pycodestyle            2.4.0
pyflakes               2.0.0
pyfomod                1.2.1      d:\git\pyfomod\src
pynxm                  0.1.0
pytest                 3.6.1
pywin32                227
PyYAML                 5.1.1
requests               2.23.0
setuptools             40.6.2
six                    1.12.0
skyrim-builder         0.1        d:\git\skyrim-builder
skyrim-package-manager 0.1        d:\git\skyrim-package-manager
skyrimbuilder          0.1
smmap2                 2.0.5
spm                    0.1
tabulate               0.8.6
tqdm                   4.46.0
urllib3                1.25.9
websocket-client       0.57.0

My sample repro script:

import click
from pathlib import Path
from bethesda_structs.archive import BSAArchive


@click.command()
@click.argument('bsa')
@click.argument('dest')
def main(bsa, dest):
    bsa = Path(bsa)
    dest = Path(dest)
    assert bsa.exists()
    assert dest.exists()

    print(bsa)
    print(dest)

    archive = BSAArchive.parse_file(str(bsa))
    print(archive.container.header)
    print(archive.container.directory_records)
    archive.extract(str(dest))


if __name__ == '__main__':
    main()

The output:

(venv) PS D:\git\skyrim-builder> python .\test.py 'C:\users\{user redacted}\desktop\Cathedral Weathers - Textures.bsa' C:\users\{user redacted}\desktop\dest       
C:\users\{user redacted}\desktop\Cathedral Weathers - Textures.bsa
C:\users\{user redacted}\desktop\dest
Container: 
    magic = b'BSA\x00' (total 4)
    version = 105
    directory_offset = 36
    archive_flags = Container:
        directories_named = True
        files_named = True
        files_compressed = True
        files_prefixed = True
    directory_count = 2
    file_count = 24
    directory_names_length = 30
    file_names_length = 368
    file_flags = Container:
        dds = True
ListContainer:
    Container:
        hash = 7911102381315617907
        file_count = 8
        name_offset = 452
    Container:
        hash = 13447612162718329721
        file_count = 16
        name_offset = 598
Traceback (most recent call last):
  File ".\test.py", line 25, in <module>
    main()
  File "D:\git\skyrim-builder\venv\lib\site-packages\click\core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "D:\git\skyrim-builder\venv\lib\site-packages\click\core.py", line 717, in main
    rv = self.invoke(ctx)
  File "D:\git\skyrim-builder\venv\lib\site-packages\click\core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "D:\git\skyrim-builder\venv\lib\site-packages\click\core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File ".\test.py", line 21, in main
    archive.extract(str(dest))
  File "D:\git\skyrim-builder\venv\lib\site-packages\bethesda_structs\archive\_common.py", line 170, in extract
    archive_files = list(self.iter_files())
  File "D:\git\skyrim-builder\venv\lib\site-packages\bethesda_structs\archive\bsa.py", line 256, in iter_files
    file_record.offset + (file_record.size & self.SIZE_MASK)
  File "D:\git\skyrim-builder\venv\lib\site-packages\construct\core.py", line 304, in parse
    return self.parse_stream(io.BytesIO(data), **contextkw)
  File "D:\git\skyrim-builder\venv\lib\site-packages\construct\core.py", line 316, in parse_stream
    return self._parsereport(stream, context, "(parsing)")
  File "D:\git\skyrim-builder\venv\lib\site-packages\construct\core.py", line 328, in _parsereport
    obj = self._parse(stream, context, path)
  File "D:\git\skyrim-builder\venv\lib\site-packages\construct\core.py", line 1979, in _parse
    subobj = sc._parsereport(stream, context, path)
  File "D:\git\skyrim-builder\venv\lib\site-packages\construct\core.py", line 328, in _parsereport
    obj = self._parse(stream, context, path)
  File "D:\git\skyrim-builder\venv\lib\site-packages\construct\core.py", line 2468, in _parse
    return self.subcon._parsereport(stream, context, path)
  File "D:\git\skyrim-builder\venv\lib\site-packages\construct\core.py", line 328, in _parsereport
    obj = self._parse(stream, context, path)
  File "D:\git\skyrim-builder\venv\lib\site-packages\construct\core.py", line 3593, in _parse
    return sc._parsereport(stream, context, path)
  File "D:\git\skyrim-builder\venv\lib\site-packages\construct\core.py", line 328, in _parsereport
    obj = self._parse(stream, context, path)
  File "D:\git\skyrim-builder\venv\lib\site-packages\construct\core.py", line 715, in _parse
    return self._decode(obj, context, path)
  File "D:\git\skyrim-builder\venv\lib\site-packages\bethesda_structs\archive\bsa.py", line 48, in _decode
    return lz4.frame.decompress(obj)
RuntimeError: LZ4F_getFrameInfo failed with code: ERROR_frameType_unknown

The lz4 stuff are beyond me to understand. Looking at other BSA extractors, the Bethesda Archive Extractor page has in its version history a line that says "0.07 - Corrected the size sent to LZ4 for decompression, which affected only a very small number of files." Maybe this is related?

Confirmed that the same bsa file can be successfully extracted through the Bethesda Archive Extractor (above mentioned GUI tool) and the BSA Browser, which comes with a cli utility that I can use for the time being. Still though, it would be nice to be able to code this in native python at some point.

Anyway, thanks for your library!

PEX Decompilation

Support decommpilation of PEX compressed scripts back into the source Papyrus.
The best resource for this structure is from UESP.

Header struct
String table
Debug info
Support decompilation to Papyrus

construct.core.MappingError: building failed, unknown label: {'DDPF_ALPHA': True, 'DDPF_RBG': True}

Not sure if this project has been abandoned or not but I figured I might as well give it a shot.

Expected Behavior

Extract .BA2 archive to destination.

Current Behavior

Possible Solution

Might be my version of python? Not sure because i tried the same code with both 3.7.4 and 3.6.8. Might be windows 10? 64bit python?

Steps to Reproduce (for bugs)

I run the following code using my pipenv shell
Using fallout 4's "Fallout4 - Textures9.ba2" archive to test

Context

Trying to get all the .ba2 files in src folder and extract them to dst folder.

My Environment

Bethesda Structs Version: 0.1.3
Python Version: 3.6.8 and 3.7.4
Operating System and Version: Windows 10 Version 10.0.18362 Build 18362