jkanderson / soulsformats Goto Github PK
View Code? Open in Web Editor NEWA .NET library for reading and writing FromSoftware file formats.
A .NET library for reading and writing FromSoftware file formats.
Hello!
I recently had a problem with DS3 materials and mapped out layout of almost all GX## GXItems. I think, it would like be beneficial if this data was included in SoulsFormats for others to use.
All the results and claims I make below are based on Data from DS1, DS2, DS3, Bloodborne, Sekiro, and Elden Ring. I collected the material information (Material + GXLists) of all FLVER files (.flver, or .flv) in those games. All the raw data can be found here.
I want to thank Rayan(Dahoom) for providing me with the Bloodborne data.
The mapping includes:
The data bytes are especially useful. It's obvious that those bytes are just a list of either int32 or float32 values. Turns out, the number of those values + their type (int or float) is determined by the GXItem ID + Unk04. So I created a mapping of almost all GXItem data values to their corresponding DS3 debug menu entries. This mapping includes:
DS3, SK, and ER also have GXMD GXItems (items with "GXMD" as their ID). They are rare, and their bytes have a custom format. The bytes essentially contain a map<ID, Value>
where ID
is an int and Value
is either a float, float2, float3, or float5. Other value types are likely also possible, but I only found GXMD with those types. Unfortunately, I don't really understand GXMD. The strange thing about it is that the same ID can have different types of data. E.g. the ID 95 has a float2 value in some files and a float3 value in others. No idea what that is about.
So I don't have a mapping for GXMD, but I did write a parser for it.
Here is a JSON with descriptors for almost all GX## (where ## are digits) items. The descriptors follow this format, so you can use JsonSerializer.Deserialize
to parse the JSON.
Here is a JSON list of all Unk04 values I found for GXMD items in DS3, SK, and ER.
This is the descriptor (simplified) for GX03 v1.0:
{
"ID": "GX03",
"Unk04": 100,
"Items": [
{
"Name": "[Emissive] EmissiveIntensity",
"Type": 2,
"Min": 0,
"Max": 8
},
{
"Name": "Unknown",
"Type": 0
},
{
"Name": "Unknown",
"Type": 0
},
{
"Name": "Unknown",
"Type": 0
}
]
}
This data layout (Items) of the GX item. The data of every GX03 item always contains 4 values. The first value is a float between 0 and 8 that is called "[Emissive] EmissiveIntensity" in the debug menu. The 3 remaining values are unused.
I said "almost" a few times. That's because a few GX## items are not mapped or only partially mapped.
Apart from that, everything is mapped. All remaining "Unknown" items (example) are unused values. They do not appear in the debug menu and all files in DS3 set them to 0. It's likely that From added a few vacant items for future extensions. I believe that this can already be seen in GX07 where the ScrollSpeeds for Normal3 are suspiciously last.
While the parser also knows the format, I wanted to write it down here. The GXMD bytes have the following format:
The bytes are either int32 or float32 values (meaning that the total number of bytes is a multiple of 4).
The stream of values starts with an int32 that is the number of items. This number is always present. After this, only items follow. Each item start with 2 int32s. The first int is the id of the item. The second int is the data type. Depending on the data type, some number of floats follow. The stream ends after all items have been read. There is no ending 0 or similar.
It's important to note that item ids are unique, but may appear in any order. That's why I said that GXMD data is essentially a map.
Correction: What I said above is correct for DS3. In SK and ER, From extended the GXMD format. Each item can now be optionally followed by a "flag". This flag is a boolean value that is kinda like a second value for that item. While an unflagged item has the format id, dataType, data
, a flagged item has the format id, dataType, data, id, dataType=0, data=0|1
. I have no idea what those flags do in SK and ER. They are also somewhat rare.
This data is applicable to Bloodborne, Dark Souls 3, Sekiro, and Elden Ring. DS1 PTDE does not use GXItems. DS2 has a weird GXItem format that I don't really understand, but it seems to be similar. I have not tested DS1 Remastered and Demon Souls.
I do not plan on making a PR, but feel free to ask me anything if you have questions.
All my code and data is licensed under CC0, so take whatever you want.
Firstly, great job, loving the Elden Ring support!
So, most of the binder files inside item.msgbnd.dcx seem fine, but for some reason, MagicName.fmg, MagicInfo.fmg, MagicCaption.fmg are empty?
string gameRoot = "C:\\Program Files (x86)\\Steam\\steamapps\\common\\ELDEN RING\\Game";
BND4 bnd = BND4.Read(gameRoot + "\\msg\\engus\\item.msgbnd.dcx");
foreach (BinderFile binderFile in bnd.Files)
{
FMG fmg = FMG.Read(binderFile.Bytes);
Console.WriteLine(binderFile.Name + " has " + fmg.Entries.Count + " entries.");
}
Output:
N:\GR\data\INTERROOT_win64\msg\engUS\GoodsName.fmg has 4207 entries.
N:\GR\data\INTERROOT_win64\msg\engUS\WeaponName.fmg has 3142 entries.
N:\GR\data\INTERROOT_win64\msg\engUS\ProtectorName.fmg has 629 entries.
N:\GR\data\INTERROOT_win64\msg\engUS\AccessoryName.fmg has 764 entries.
N:\GR\data\INTERROOT_win64\msg\engUS\MagicName.fmg has 1 entries.
N:\GR\data\INTERROOT_win64\msg\engUS\NpcName.fmg has 531 entries.
N:\GR\data\INTERROOT_win64\msg\engUS\PlaceName.fmg has 1045 entries.
N:\GR\data\INTERROOT_win64\msg\engUS\GoodsInfo.fmg has 4215 entries.
N:\GR\data\INTERROOT_win64\msg\engUS\WeaponInfo.fmg has 39 entries.
N:\GR\data\INTERROOT_win64\msg\engUS\ProtectorInfo.fmg has 625 entries.
N:\GR\data\INTERROOT_win64\msg\engUS\AccessoryInfo.fmg has 764 entries.
N:\GR\data\INTERROOT_win64\msg\engUS\GoodsCaption.fmg has 4225 entries.
N:\GR\data\INTERROOT_win64\msg\engUS\WeaponCaption.fmg has 3143 entries.
N:\GR\data\INTERROOT_win64\msg\engUS\ProtectorCaption.fmg has 629 entries.
N:\GR\data\INTERROOT_win64\msg\engUS\AccessoryCaption.fmg has 764 entries.
N:\GR\data\INTERROOT_win64\msg\engUS\MagicInfo.fmg has 1 entries.
N:\GR\data\INTERROOT_win64\msg\engUS\MagicCaption.fmg has 1 entries.
N:\GR\data\INTERROOT_win64\msg\engUS\GemName.fmg has 263 entries.
N:\GR\data\INTERROOT_win64\msg\engUS\GemInfo.fmg has 252 entries.
N:\GR\data\INTERROOT_win64\msg\engUS\GemCaption.fmg has 252 entries.
N:\GR\data\INTERROOT_win64\msg\engUS\GoodsDialog.fmg has 122 entries.
N:\GR\data\INTERROOT_win64\msg\engUS\ArtsName.fmg has 201 entries.
N:\GR\data\INTERROOT_win64\msg\engUS\ArtsCaption.fmg has 201 entries.
N:\GR\data\INTERROOT_win64\msg\engUS\WeaponEffect.fmg has 83 entries.
N:\GR\data\INTERROOT_win64\msg\engUS\GemEffect.fmg has 1 entries.
N:\GR\data\INTERROOT_win64\msg\engUS\GoodsInfo2.fmg has 1081 entries.
KF1 doesn't use this file format, it's from the Sword of Moonlight King's Field maker tool which was used to remake KF1 by FromSoft themselves.
I know King's Field formats aren't your primary target, but a few rough specs for them are in my repository 'Psycpros'. Most of the formats are either just standard PsyQ formats or modified variants, like .MIM and .MO which use 'MIMe' routines.
Sorry for being a pain :)
Firstly, thanks for this awesome set of tools! I'm using this with Dark Souls Remastered. I know that some file formats have changed between DS1 and DSR, but I'm only focusing on map models.
I can successfully load character models and map piece models, but I'm having trouble loading map collision models. Here's what I'm trying:
Logic:
string ROOT_INPUT_FOLDER = "C:\\Program Files (x86)\\Steam\\steamapps\\common\\DARK SOULS REMASTERED\\";
string inputBhdFile = ROOT_INPUT_FOLDER + "map\\m10_02_00_00\\h10_02_00_00.hkxbhd";
string inputBdtFile = ROOT_INPUT_FOLDER + "map\\m10_02_00_00\\h10_02_00_00.hkxbdt";
BXF3 bxf = BXF3.Read(inputBhdFile, inputBdtFile); // this works
BinderFile file = bxf.Files[0]; // file.Name is "m10_02_00_00\\h0000B2A10.hkx.dcx"
byte[] bytes = DCX.Decompress(file.Bytes); // this works
SoulsFormats.HKX hkx = SoulsFormats.HKX.Read(bytes, HKX.HKXVariation.HKXDS1); // this throws InvalidDataException
Output Exception message:
Read UInt32: 0xAC230300 | Expected: 0x57E0E057 | Ending position: 0x4
I propose to add new function to PARAM.Cell class that takes ReadOnlySpan as first argument and it converts it to type of its parafdef and assigns it to value
property just as setter of Value does.
I don't want to make allocations (new string(...)) just to set cell to a value so I can relief some pressure from GC and yield better performance.
If this proposal is accepted I can make a PR with this change included.
as the title says, TA3.Animation lacks a constructor. Even a copy constructor.
TAE3 itself also lacks a constructor.
The chunk offsets at 0x74
, 0x84
, etc. are relative to the beginning of the file when they should be relative to the beginning of the compressed data (I think the value at 0x74
is always 0
). The resulting files cause the game to hang indefinitely when loading. Subtracting the length of the headers from each of these offset values in a hex editor fixed the load issue entirely.
I actually encountered this using Yabber.DCX, but I'm pretty sure the issue is here.
Any plans to add support for Elden Ring? At minimum, maybe just add comments in the README and FORMATS. For example, "some formats appear to be the same as DS3, but not planning to enhance this anytime soon." ?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.