valveresourceformat / valvekeyvalue

📃 Next-generation Valve's key value framework for .NET

Source 2 Viewer

* The library component of Source 2 Viewer is called ValveResourceFormat (VRF).

Valve's Source 2 resource file format parser, decompiler, and exporter. Source 2 files usually end with _c, for example .vmdl_c.

This repository is split into three components:

• CLI Decompiler - File data viewer, decompiler and a playground for testing new formats and features.
• GUI Viewer - A vpk archive viewer and extractor. Also supports viewing resources such as sounds, textures, models, maps, and much more.
• Library - Provides public API to parse resource files and some helpers.

⚒ View the official website for downloads.

Join our Discord

Eye catchy screenshots

What's supported?

• VPK viewer which supports opening and exporting files
• Creating new vpk archives
• Model viewer and decompiler to glTF and modeldoc
• Map viewer and decompiler to glTF and vmap
• Material decompiler to vmat
• Sound player
• Binary KeyValues3 parser
• NTRO support

Limitations

This tool is based entirely on a reverse engineered effort because Valve does not provide any documentation or Source 2 code (SDK or engine code), while the Source 1 SDK and leaked engine code are helpful, a lot of systems and formats have changed.

The code contained in this repository is based on countless hours of reverse engineering Source 2 games and not all intricate details have been figured out.

If you are interested in helping, take a look at the open issues and join our Discord.

Not all formats are 100% supported, some parameters are still unknown and not fully understood.

Supported resource types

Ext	Name	Support
vagrp	Animation Group	👍
vanim	Animation	👍
vanmgrph	Animation Graph	No
vcompmat	Composite Material	No
vcss	Panorama Style	👍
vdata	Data	👍
vents	EntityLump	👍
vjs	Panorama Script	👍
vmap	Map	👍
vmat	Material	👍
vmdl	Model	👍
vmesh	Mesh	👍
vmorf	MorphSet	👍
vpcf	Particle System	👍
vpdi	Panorama Dynamic Images	No
vphys	Physics Collision Mesh	👍
vpost	Postprocessing Settings	👍
vpsf	Particle Snapshot	No
vpulse	Pulse Graph Definition	No
vrman	ResourceManifest	👍
vrmap	Resource Remap Table	No
vrr	Response rules	👍
vseq	Sequence Group	No
vsmart	Smart Prop	Partially
vsnap	Particle Snapshot	👍
vsnd	Sound	👍
vsndevts	Sound Event Script	👍
vsndstck	Sound Stack Script	👍
vsurf	Surface Properties	No
vsvg	Panorama Vector Graphic	👍
vtex	Compiled Texture	👍
vts	Panorama TypeScript	👍
vvis	WorldVisibility	No
vwnod	WorldNode	👍
vwrld	World	👍
vxml	Panorama Layout	👍
vpk	Pak (package)	👍 Handled by ValvePak
vcs	Compiled Shader	👍 Handled by CompiledShader
vfont	Bitmap Font	👍 Decrypts VFONT1, supported in Source 1 and Source 2.
dat	Closed Captions	👍 Handled by ClosedCaptions
bin	Tools Asset Info	👍 Handled by ToolsAssetInfo
vdpn	Dota Patch Notes	👍
vdacdefs	DAC Game Defs Data	No
vfe	Flex Scene File	👍 Handled by FlexSceneFile
vcd	VCD	No
vcdlist	VCD list	👍

List of supported magics

Magic	Description
0x03564B56	VKV\x03 - First binary keyvalues 3 encoding with custom block compression
0x4B563301	KV3\x01 - Binary keyvalues 3 (version 1)
0x4B563302	KV3\x02 - Binary keyvalues 3 (version 2)
0x4B563303	KV3\x03 - Binary keyvalues 3 (version 3)
0x4B563304	KV3\x04 - Binary keyvalues 3 (version 4)
0x564B4256	VBKV - binary keyvalues 1 (handled by ValveKeyvalue)
0x55AA1234	VPK - valve package (handled by ValvePak)
0x44434356	VCCD - closed captions
0xC4CCACE8	tools asset info
0xC4CCACE9	tools asset info (newer version)
0x32736376	vcs2 - compiled shader
0x31415926	murmurhash2 seed used by StringToken
0xEDABCDEF	murmurhash64 seed used to encode resource IDs
VFONT1	"encrypted" font file
0x00564645	VFE - flex scene file

Command-line options

Option	Description
Input
--input (or -i)	Input file to be processed. With no additional arguments, a summary of the input(s) will be displayed.
--recursive	If specified and given input is a folder, all sub directories will be scanned too.
--recursive_vpk	If specified along with --recursive, will also recurse into VPK archives.
--vpk_extensions (or -e)	File extension(s) filter, example: "vcss_c,vjs_c,vxml_c".
--vpk_filepath (or -f)	File path filter, example: "panorama\" or "scripts/items/items_game.txt".
--vpk_cache	Use cached VPK manifest to keep track of updates. Only changed files will be written to disk.
--vpk_verify	Verify checksums and signatures.
Output
--output (or -o)	Output path to write to. If input is a folder (or a VPK), this should be a folder.
--all (or -a)	Print the content of each resource block in the file.
--block (or -b)	Print the content of a specific block, example: DATA, RERL, REDI, NTRO.
--vpk_decompile (or -d)	Decompile supported resource files.
--vpk_list (or -l)	Lists all resources in given VPK. File extension and path filters apply.
--vpk_dir	Print a list of files in given VPK and information about them.
Type specific export
--gltf_export_format	Exports meshes/models in given glTF format. Must be either 'gltf' (default) or 'glb'.
--gltf_export_materials	Whether to export materials during glTF exports.
--gltf_textures_adapt	Whether to perform any glTF spec adaptations on textures (e.g. split metallic map).
--gltf_export_extras	Export additional Mesh properties into glTF extras
--tools_asset_info_short	Whether to print only file paths for tools_asset_info files.
Other
--threads	If higher than 1, files will be processed concurrently.
--version	Show version information.
--help	Show help information.

There are also --stats related options, but they are not listed here as they are not relevant to most users.

Examples:

# List all files in the vpk
# Use `--vpk_dir` to also print file metadata
.\Decompiler.exe -i "core/pak01_dir.vpk" --vpk_list

# Export the entire vpk as is
.\Decompiler.exe -i "core/pak01_dir.vpk" --output "pak01_exported"

# Export only the "panorama/layout" folder
.\Decompiler.exe -i "core/pak01_dir.vpk" --output "pak01_exported" --vpk_filepath "panorama/layout"

# Decompile and export all Panorama files to a folder named "exported"
.\Decompiler.exe -i "core/pak01_dir.vpk" -e "vjs_c,vxml_c,vcss_c" -o "exported" -d

# Print resource blocks for a specific file similar to resourceinfo.exe in Source 2
# Use `--block DATA` to only print a specific block
.\Decompiler.exe -i "file.vtex_c" --all

# Decompile a specific file on disk
.\Decompiler.exe -i "file.vtex_c" -o exported.png

License

Contents of this repository are available under MIT license, except for Tests/Files folder contains files which have likely come from Valve's games.

valvekeyvalue's Issues

KV1 binary type 0x0A

SteamRE/SteamKit#376

A string where backwards slash is at the end of a a string does not parse

"Test"
{
  "BuildOutput" "..\output\"
}

It throws Found end of file while trying to read token. because it always thinks \ is an escape, even if HasEscapeSequences is false.

Example from: https://partner.steamgames.com/doc/sdk/uploading
Test case: https://github.com/SteamDatabase/ValveKeyValue/tree/backslash-no-escape

KV1TextReader: Parse 64-bit unsigned integer as `ulong` instead of `float`

Values like LastOwner in app manifest files are 64-bit unsigned integers. The current implementation of KV1TextReader.ParseValue only returns a KVObjectValue<ulong> when the text is formatted as a hexadecimal number:

https://github.com/SteamDatabase/ValveKeyValue/blob/2b650ead76d52e97caa463c6a83cc24fc3d92d38/ValveKeyValue/ValveKeyValue/Deserialization/KeyValues1/KV1TextReader.cs#L263-L275

Values like LastOwner often have values that look like this: 76561198110222274 which isn't a hex string, fails the int.TryParse because it's too large and ends up as a float. Parsing this 64-bit unsigned integer as a single-precision floating-point number results in data being lost:

// original: "76561198110222274"
var lastOwnerValue = data["LastOwner"]!;

// converted to ulong: 76561202255233024
ulong lastOwner = lastOwnerValue.ToUInt64(CultureInfo.InvariantCulture);

This can be fixed by adding a ulong.TryParse check after the int.TryParse and before the float.TryParse call:

1. check if the value is a 64-bit unsigned integer formatted as a hex string
2. check if the value is a 32-bit signed integer
3. check if the value is a 64-bit unsigned integer
4. check if the value is a single-precision floating-point number

Ideally, the second step should check for a 32-bit unsigned integer, but that can be part of another issue.

"Builder is not in a fully completed state." when there are multiple objects with discarded conditionals

I managed to boil to down to such an example:

"GameInfo"
{
	One
	{
		Key1 1 [$MOBILE]
	}

	Two
	{
		Key2 2 [$MOBILE]
	}

	Three
	{
		Key3 3 [$MOBILE]
	}
}

Removing one of these works, unsure why it happens yet.

Found in https://github.com/SteamDatabase/GameTracking-Underlords/blob/master/game/dac/gameinfo.gi

Implement KeyValues3 (Binary)

See #5 for text.

KV3 binary: https://github.com/SteamDatabase/ValveResourceFormat/blob/master/ValveResourceFormat/Resource/ResourceTypes/BinaryKV3.cs

Binary is complicated due to multiple compressions (Valve's own block compression, LZ4, ZSTD) and versions.

VKV can't parse KV when reading directly from HttpClient's stream using HttpCompletionOption.ResponseHeadersRead

HttpCompletionOption.ResponseHeadersRead is a special option we can pass to HttpClient which allows to return the response as soon as 200 OK is returned (as opposed to full data), which allows the user to start parsing the data while it's still being buffered.

A repro below:

private static async Task ReproNotWorking() {
    using HttpClient httpClient = new();

    using HttpRequestMessage request = new(HttpMethod.Get, new Uri("https://raw.githubusercontent.com/SteamDatabase/GameTracking-CSGO/master/csgo/resource/csgo_english.txt"));

    using HttpResponseMessage response = await httpClient.SendAsync(request, HttpCompletionOption.ResponseHeadersRead).ConfigureAwait(false);

    KVSerializer? serializer = KVSerializer.Create(KVSerializationFormat.KeyValues1Text);

    KVSerializerOptions options = new() {
        EnableValveNullByteBugBehavior = true,
        HasEscapeSequences = true
    };

    using Stream stream = await response.Content.ReadAsStreamAsync().ConfigureAwait(false);

    serializer.Deserialize(stream, options);
}

Throws:

ValveKeyValue.KeyValueException: Found end of file when another token type was expected.
 ---> System.InvalidOperationException: Attempted to finalize object while in state InObjectBetweenKeyAndValue.

The same repro using HttpCompletionOption.ResponseContentRead instead of HttpCompletionOption.ResponseHeadersRead fixes the issue.

For some reason VKV can't work in this scenario, my blind guess is that it doesn't wait for the end of the stream but rather stops parsing in the middle while more data is being buffered, in result crashing as if the data was cut somewhere in the middle.

The proper logic in this case would include reading stream to the very end, also when being buffered and not fully ready yet.

Could also be something entirely different, I don't know. However, it's nice to note that other deserializing libraries (Newtonsoft.Json tested) work in this scenario just fine.

Support the '=' as an assignment (CS:GO)

I was looking at the leaked CS:GO code, they added the following:

		// support the '=' as an assignment, makes multiple-keys-on-one-line easier to read in a keyvalues file
		if ( *value == '=' && !wasQuoted )
		{

Serialize dictionary values correctly

Currently, it calls ToString on the dictionary value and that produces unwanted results on dictionaries like Dictionary<string, float[]>.

Test code:

        public class SuperObjectTest
        {
            public Dictionary<string, float[]> test { get; } = new Dictionary<string, float[]>();
        }

        [Test]
        public void CreatesFloatArray()
        {
            var kv = new SuperObjectTest();
            kv.test.Add("test", new float[] {1.1234f, 2.2345f, 3.54677f});
            kv.test.Add("test2", new float[] {1.1234f, 2.2345f, 3.54677f});

            string text;
            using (var ms = new MemoryStream())
            {
                KVSerializer.Create(KVSerializationFormat.KeyValues1Text).Serialize(ms, kv, "test");

                ms.Seek(0, SeekOrigin.Begin);
                using (var reader = new StreamReader(ms))
                {
                    text = reader.ReadToEnd();
                }
            }
        }

Currently it produces this:

"test"
{
	"test"
	{
		"test"	"System.Single[]"
		"test2"	"System.Single[]"
	}
}

Without dictionary, it is able to serialize float[] just fine.

Reorganize files into logical folders

Currently all VKV files are in a single folder, and it's hard to follow.
Split it into readers, writers, helpers, serializers, etc.

Implement KeyValues3 (Text)

EDIT: Checklist move to #61

Booleans are serialized as empty objects

public bool Test { get; set; }

output:

"Test"
{
}

KVObject should implement IList<T>

I would like to be able to add/remove children to/from a KVObject. I would also like the ability to get the number of children in a KVObject without having to enumerate it.
I suggest having the KVObject class implement the System.Collections.Generic.ICollection<T> and System.Collections.Generic.IList<T> interfaces. This would also enable getting/setting children by index, thus allowing to enumerate via a for loop.

Thank you for considering my suggestion.

Implement deserializing into enums

ValveKeyValue/ValveKeyValue/ValveKeyValue.Test/Text/ObjectSerializationTypesTestCase.cs

Lines 34 to 36 in b61e276

	// TODO: Deserializing enums does not work yet
	//var deserialized = KVSerializer.Create(KVSerializationFormat.KeyValues1Text).Deserialize<DataObject[]>(text);
	//Assert.That(deserialized, Is.EqualTo(dataObject));

System.NotSupportedException : Converting to SomeEnum is not supported. (key = VEnum, type = Int32)

It probably just needs to cast as to the backing type.

Create compatibility tests with Valve's KeyValues implementation for consistency

The only problem with this is that they have differing implementations based on the project (and how old it is).

Steam, CS:GO and other Source games may have minor differences.

Support .NET trimming

ValveKeyValue.dll : warning IL2104: Assembly 'ValveKeyValue' produced trim warnings. For more information see https://aka.ms/dotnet-illink/libraries

Unable to deseralize into a dictionary

Unhandled Exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Reflection.TargetParameterCountException: Parameter count mismatch.
   at System.Reflection.RuntimeMethodInfo.InvokeArgumentsCheck(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.RuntimePropertyInfo.SetValue(Object obj, Object value, BindingFlags invokeAttr, Binder binder, Object[] index, CultureInfo culture)
   at System.Reflection.RuntimePropertyInfo.SetValue(Object obj, Object value, Object[] index)
   at System.Reflection.PropertyInfo.SetValue(Object obj, Object value)
   at ValveKeyValue.ObjectCopier.CopyValue[TObject](TObject obj, PropertyInfo property, KVValue value) in ValveKeyValue\ObjectCopier.cs:line 81
   at ValveKeyValue.ObjectCopier.CopyObject[TObject](KVObject kv, TObject obj, IPropertyMapper mapper) in ValveKeyValue\ObjectCopier.cs:line 57
   at ValveKeyValue.ObjectCopier.MakeObject[TObject](KVObject keyValueObject, IPropertyMapper mapper) in ValveKeyValue\ObjectCopier.cs:line 31
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
   at ValveKeyValue.ObjectCopier.CopyObject[TObject](KVObject kv, TObject obj, IPropertyMapper mapper) in ValveKeyValue\ObjectCopier.cs:line 68
   at ValveKeyValue.ObjectCopier.MakeObject[TObject](KVObject keyValueObject, IPropertyMapper mapper) in ValveKeyValue\ObjectCopier.cs:line 31
   at ValveKeyValue.ObjectCopier.MakeObject[TObject](KVObject keyValueObject) in ValveKeyValue\ObjectCopier.cs:line 13
   at ValveKeyValue.KVSerializer.Deserialize[TObject](Stream stream) in ValveKeyValue\KVSerializer.cs:line 48
   at kvtest.MainClass.Main(String[] args) in Program.cs:line 29

    class LanguageFile
    {
        public string Language { get; set; }
        public Dictionary<string, string> Tokens { get; set; }
    }

var test = KVSerializer.Deserialize<LanguageFile>(new StreamReader("tf_english.txt").BaseStream);
Console.WriteLine(test.Tokens.Count);

Implement the ToType and other methods on KVCollectionValue

We should probably implement the ToType method at least. Otherwise, it's impossible to deserialise a KeyValues1 document to a Dictionary<string, name> where name is replaced with your class name.

I'll see if I can work out a way to implement these later on.

Keep track of position (line/column) and add them to exceptions such as invalid syntax

Deserialize complex objects

We should support deserialization to:

• Array/T[]
• List<T>
• IEnumerable/IEnumerable<T>
• Dictionary<string, T>/IDictionary<string, T>/IDictionary
• user-defined complex objects (multiple levels deep)
• mixtures of the above

This may require exposing custom type conversion through ISerializable, TypeConverter, a new custom type, or something else entirely.

This may also require reworking the KVValue APIs, as it is a total pain to actually do anything useful with them after deserialization. In particular, we need to be able to say "hey this object is actually an array because it has numeric keys 0..N", and convert it.

Implement KeyValues1

• Parse KV1 text to KV types.
  • Quoted strings
  • Unquoted strings
• Parse KV1 text to object graph.
• Parse KV1 binary to KV types.
• Parse KV1 binary to object graph.
• Handle conditionals:
  • Provide defaults based on current execution environment ($WIN32/$OSX/$LINUX)
  • Allow user to specify conditional variables.
• Handle newlines, tabs and other text escaping.
• Write KV1 types to text.
• Write object graph to text.
• Write KV1 types to binary.
• Write object graph to binary.
• Handle circular object graphs.
• Create documentation
• Create compatibility tests with Valve's KeyValues from tier1 to ensure consistency.
• Parse and write properties
• Parse and write fields

Stress test: *_english.txt from recent Valve games (Dota 2, CS:GO, TF2).

VBKV

Appears to be a wrapper to normal Binary KeyValues, with a header and a changed end byte

see ValvePython/vdf#11

Provide an implementation of `IIncludedFileLoader`

For files opened with a path, we can use that path to get the folder.

And/or for streams we can make an argument to provide a path.

Text serializer writes utf8 bom

https://github.com/SteamDatabase/ValveKeyValue/blob/aeae7e44c29b78c14a6f2400854029b410d9d2b8/ValveKeyValue/ValveKeyValue/Serialization/KV1TextSerializer.cs#L16

Should probably replace that to new UTF8Encoding(false)

Invalid conditional syntax in Underlords gameinfo.gi

File: https://github.com/SteamDatabase/GameTracking-Underlords/blob/master/game/dac/gameinfo.gi

System.IO.InvalidDataException: Invalid conditional syntax "!$MOBILE && !$MOBILE_RSRC && !$BUILDALLSHADERS_RUN && !$CONTENTBUILDER_RUN && !$VMPI_GAMEINFO"
 ---> System.InvalidOperationException: Bad condition syntax.
   at ValveKeyValue.KVConditionEvaluator.ReadToken(TextReader reader)
   at ValveKeyValue.KVConditionEvaluator.Evalute(String expressionText)
   --- End of inner exception stack trace ---
   at ValveKeyValue.KVConditionEvaluator.Evalute(String expressionText)
   at ValveKeyValue.Deserialization.KV1TextReader.HandleCondition(String text)
   at ValveKeyValue.Deserialization.KV1TextReader.ReadObject()
   at ValveKeyValue.KVSerializer.Deserialize(Stream stream, KVSerializerOptions options)
   at GUI.Utils.AdvancedGuiFileLoader.FindAndLoadSearchPaths() in D:\GitHub\ValveResourceFormat\GUI\Utils\AdvancedGuiFileLoader.cs:line 160

Not all escape sequences are implemented

See the list here:
https://github.com/ValveSoftware/GameNetworkingSockets/blob/8348e9234aba1b034f51ceff406a715de9ef61ad/src/tier1/utlbuffer.cpp#L54

The only strange thing to me here is \?.

Bracketed value is being detected as a conditional

I had a branch with test case here: https://github.com/SteamDatabase/ValveKeyValue/compare/bracketed-value

System.IO.InvalidDataException : Found conditional while in state InObjectBetweenKeyAndValue.

Support merging of key values for same key

https://raw.githubusercontent.com/SteamDatabase/GameTracking-CSGO/master/csgo/scripts/items/items_game.txt contains structures like

"sticker_kits"
	{
		"0"
		{
			"name"		"default"
			"item_name"		"#StickerKit_Default"
			"description_string"		"#StickerKit_Desc_Default"
		}
		"1"
		{
			"name"		"dh_gologo1"
			"item_name"		"#StickerKit_dh_gologo1"
			"description_string"		"#StickerKit_desc_dh_gologo1"
			"sticker_material"		"dreamhack/dh_gologo1"
			"tournament_event_id"		"1"
		}
		.....
	}
	....
	"sticker_kits"
	{
		"1697"
		{
			"name"		"spray_std_axes_crossed"
			"item_name"		"#SprayKit_std_axes_crossed"
			"description_string"		"#SprayKit_desc_std_axes_crossed"
			"sticker_material"		"default/axes_crossed"
			"item_rarity"		"common"
		}
		"1698"
		{
			"name"		"spray_std_bubble_dead"
			"item_name"		"#SprayKit_std_bubble_dead"
			"description_string"		"#SprayKit_desc_std_bubble_dead"
			"sticker_material"		"default/bubble_dead"
			"item_rarity"		"common"
		}
                ...
 }

For this situation only the first set is taken and all others are omitted by the library

Implement KeyValues2 (DMX)

se2007 leak contains unit tests for dmx.

.NET reference we can use: https://github.com/kristiker/Datamodel.NET

KVObject[string] returns KVValue which is impossible to enumerate

KVObject["string"] returns a KVValue, but if it has children i can't access that
debuggers shows 'children', but i can only call ToXXX() on it
it gives me the this[string] accessor, but i can't enumerate the children if I dont know the keys
calling tostring on it would indicate its KVCollectionValue due to [Collection], but if I try to foreach() it says KVValue has no GetEnumerator

Example code for Steam's config.vdf:

            using var fs = new FileStream(filename, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
            var data = KVSerializer.Create(KVSerializationFormat.KeyValues1Text).Deserialize(fs);

            var depots = data["Software"]["Valve"]["Steam"]["depots"];

            foreach (var depot in depots) // KVValue has no GetEnumerator()
            {

            }

Fails to serialize `ValueTuple`

This line throws System.Reflection.TargetParameterCountException: 'Parameter count mismatch.'
https://github.com/SteamDatabase/ValveKeyValue/blob/ed10cc104681d33935fc2b7ea3d6701970a581f5/ValveKeyValue/ValveKeyValue/PropertyMember.cs#L29

public class Test
{
    public List<(string a, string b)> Something { get; } = new()
    {
        ( "hello", "world" )
    };
}

using var ms = new MemoryStream();
KVSerializer.Create(KVSerializationFormat.KeyValues1Text).Serialize(ms, new Test(), "Test");
Encoding.UTF8.GetString(ms.ToArray());

int64 values are being parsed as floats instead of strings

See #47 for context.

e.g. a value of "manifest" "6404082971767543753"

Optimize ReadNullTerminatedBytes

Take this code: https://github.com/SteamRE/SteamKit/blob/43ecd71e1069e4a136ef53acdb07bf96d79f46e5/SteamKit2/SteamKit2/Util/StreamHelpers.cs#L103

Why are object members sorted by name?

This loses developer defined order in classes when serializing.

https://github.com/SteamDatabase/ValveKeyValue/blob/ed10cc104681d33935fc2b7ea3d6701970a581f5/ValveKeyValue/ValveKeyValue/ObjectCopier.cs#L120

cc @yaakov-h

Add support for fields when deserializing strongly-typed objects

Not just properties.

This will require some abstraction over both.

"Found invalid data while decoding"

I'm currently working on a vmf-to-json program, and while reading a VMF file and attempting to deserialize it, I get "Found invalid data while decoding"

Here's the VMF file: https://hastebin.com/kojimugafo.pl

It's only a simple brush, and I don't see why it shouldn't work. I'm reading the file directly with a FileStream.

Casting to DateTime is not supported

There's this:
https://github.com/SteamDatabase/ValveKeyValue/blob/225ef38dca38bb83aefe3c43c65822d64c85b01c/ValveKeyValue/ValveKeyValue/KVObjectValue.cs#L44-L47

However TryConvertValueTo listed it as a supported conversion:
https://github.com/SteamDatabase/ValveKeyValue/blob/dfb96b52087fbd4f27e4128a5368cf318a04412b/ValveKeyValue/ValveKeyValue/ObjectCopier.cs#L407

I commented that out for now, was it simply never finished?

Comma separate integers being parsed as a single int

When parsing a value that is a list of comma-separated integers, the value returned is a single int.

ValveKeyValue version from NuGet: 0.8.0.139

Expected result: The original value as a string

Example (From dota npc_heroes)
"Rolelevels" "3,3,1" is parsed as an int32 with value: 331. Expected: "3,3,1"

Cannot deserialize into object if KeyValue Array starts with index > 0 or if there are gaps within the index.

Using your sample project SteamAppInfo to deserialize appinfo.vdf works perfectly fine.

In addition I created a class to deserialize the appinfo.vdf into an object and here I get the issue that InvalidOperationException is thrown in case the array index within appdata.vdf doesn't start with a 0 or if the array index has gaps in between.

The issue seems to be within ObjectCopier.IsArray where you actually do an index check within a for loop after converting the index. Since the index is starting with 1 or there might be a gap the function returns false due to no match with the index of the for loop:

https://github.com/SteamDatabase/ValveKeyValue/blob/41492b76968bf4fbfa88c2bf2726e61754241606/ValveKeyValue/ValveKeyValue/ObjectCopier.cs#L176-L187

Code looks like this:

using var stream = File.OpenRead(_cataloguePath);
using var reader = new BinaryReader(stream);

var magic = reader.ReadUInt32();

if (magic != Magic)
    throw new InvalidDataException($"Unknown magic header: {magic}");

_universe = (SteamUniverse)reader.ReadUInt32();

var deserializer = KVSerializer.Create(KVSerializationFormat.KeyValues1Binary);

List<DeserializedSteamCatalogue> deserializedSteamCatalogueList = new();
do
{
    var appId = reader.ReadUInt32();

    if (appId == 0)
        break;

    DeserializedSteamCatalogue item = new()
    {
        AppID = appId,
        Size = reader.ReadUInt32(),
        InfoState = reader.ReadUInt32(),
        LastUpdated = DateTime.UnixEpoch.AddSeconds(reader.ReadUInt32()),
        Token = reader.ReadUInt64(),
        Hash = new ReadOnlyCollection<byte>(reader.ReadBytes(20)),
        ChangeNumber = reader.ReadUInt32(),
        // !!!!! Problem is here -> Deserialize directly into object !!!!!
        Data = deserializer.Deserialize<DeserializedSteamCatalogue.DeserializedData>(stream),
        // !!!!! Problem is here -> Deserialize directly into object !!!!!
    };


    deserializedSteamCatalogueList.Add(item);
} while (true);


public class DeserializedSteamCatalogue
{
    public uint AppID { get; set; }
    public uint Size { get; set; }
    public uint InfoState { get; set; }
    public DateTime LastUpdated { get; set; }
    public ulong Token { get; set; }
    public ReadOnlyCollection<byte>? Hash { get; set; }
    public uint ChangeNumber { get; set; }
    public DeserializedData Data { get; set; } = new();

    public class DeserializedData
    {
        public class DscdConfig
        {
            public string ContentType { get; set; } = string.Empty;
            public string Installdir { get; set; } = string.Empty;
            public DscdLaunch[] Launch { get; set; } = Array.Empty<DscdLaunch>();
        }
        public class DscdLaunch
        {
            public string Executable { get; set; } = string.Empty;
            public string Type { get; set; } = string.Empty;
            public DscdLaunchConfig Config { get; set; } = new();
        }

        public class DscdLaunchConfig
        {
            public string OsList { get; set; } = string.Empty;
            public string OsArch { get; set; } = string.Empty;
        }

        public string AppID { get; set; } = string.Empty;
        public DscdConfig Config { get; set; } = new();
    }
}

The structured export of the appinfo looks like this (Recursive print of the KVObject):

Name: appinfo
	appid: 570
	Name: config
		installdir: dota 2 beta
		contenttype: 3
		Name: launch
			Name: 1
				executable: game\bin\win64\dota2.exe
				arguments: -steam +engine_experimental_drop_frame_ticks 1 +voice_fadeouttime 0 -prewarm_panorama
				Name: config
					oslist: windows
					osarch: 64
				Name: description_loc
					english: Play Dota 2
					sc_schinese: Play Dota 2
				description: Play Dota 2
			Name: 2
				executable: game/dota.sh
				arguments: +engine_experimental_drop_frame_ticks 1 +@panorama_min_comp_layer_dimension 0 -prewarm_panorama
				Name: config
					oslist: macos
				Name: description_loc
					english: Play Dota 2
					sc_schinese: Play Dota 2
				description: Play Dota 2

In this sample the "launch" array starts with Index 1 instead of 0 like for other games.
There are also games in appinfo.vdf which seem to have gaps in between the array index e.g. 0, 1, 3, 6 where it also fails.

For testing I have removed the for loop in the IsArray Method and it runs through without any problems. Not sure what the intention was for the check, at least in this use case it isn't needed.

Collapsed ints in binary kv1 implemented in CS:GO

	// Data type
	enum types_t
	{
		TYPE_NONE = 0,
		TYPE_STRING,
		TYPE_INT,
		TYPE_FLOAT,
		TYPE_PTR,
		TYPE_WSTRING,
		TYPE_COLOR,
		TYPE_UINT64,
		TYPE_COMPILED_INT_BYTE,			// hack to collapse 1 byte ints in the compiled format
		TYPE_COMPILED_INT_0,			// hack to collapse 0 in the compiled format
		TYPE_COMPILED_INT_1,			// hack to collapse 1 in the compiled format
		TYPE_NUMTYPES, 
	};

Looks like they ported this back from Source 2. This is not great beacuse the enum has conflicting values.

KV1BinaryNodeType.WideString

Was looking through steamapi msgs and found kv wstring type in lobby chat

In case one will need it, here is a short add for ValveKeyValue/Deserialization/KeyValues1/KV1BinaryReader.cs

replace

case KV1BinaryNodeType.WideString:
                    throw new NotSupportedException("Wide String is not supported, please create an issue saying where you found it: https://github.com/ValveResourceFormat/ValveKeyValue/issues");

with

case KV1BinaryNodeType.WideString:
    byte[] bytes = new byte[2];

    bytes = reader.ReadBytes(2);

    if (BitConverter.IsLittleEndian)
    {
        Array.Reverse(bytes, 0, 2);
    }

    int length = BitConverter.ToInt16(bytes, 0);

    List<byte> byteList = new List<byte>();

    for(int i = 0; i < length; i++)
    {
        bytes = reader.ReadBytes(2);
        
        byteList.AddRange(bytes);
    }

    string wstring_utf16 = Encoding.BigEndianUnicode.GetString(byteList.ToArray());
    //string wstring_utf8 = Encoding.UTF8.GetString(Encoding.UTF8.GetBytes(wstring_utf16));

    value = new KVObjectValue<string>(wstring_utf16, KVValueType.WString);
    //value = new KVObjectValue<string>(wstring_utf8, KVValueType.WString);
    break;

How can I sort KeyValues alphabetically?

I'd like to sort the following KeyValues alphabetically, specifically by the Orange, Apple & Banana keys. Is that something I can accomplish with this library? I've tried to deserialise them, turn them into a dictionary, use Linq but I can't get anything to work correctly.

"TopLevel"
{
	"Orange"
	{
		"Key1"	"Value1"
		"Key2"	"Value2"
	}
	"Apple"
	{
		"Key1"	"Value1"
		"Key2"	"Value2"
	}
	"Banana"
	{
		"Key1"	"Value1"
		"Key2"	"Value2"
	}
}

Implement different KV versions

Moved from ValveResourceFormat/ValveResourceFormat#37

The following file formats are required to have the Decompiler write the decompiled formats, and to read some of the compiled formats:

• Plaintext KeyValues(1)
• Binary KeyValues(1)
• dmx encoding keyvalues2_noids (not sure if this should be in the KeyValues family or in its own DMX family)
• KeyValues3 ValveResourceFormat/ValveResourceFormat#35
• Schema text (not sure if its strictly speaking in the KeyValues family, as its header implies otherwise)
• Unit tests for each keyvalues implementation

IntPtr does not serialize to text

        class DataObject
        {
            public IntPtr VPointer { get; set; }
        }

this serializes to: "VPointer"\n\t\t{\n\t\t}

FYI IntPtr is not IConvertible.

Simplify the api

12:21 <@Netshroud> [...] though I do want to revisit vkv because I built a very confusing api the first time around
12:22 <@Netshroud> though C# now has Records and withers so it might be easier

kvSerializer.Deserialize fails for latest csgo_english.txt

kvSerializer.Deserialize fails for latest vdf resouce file csgo_english.txt at https://raw.githubusercontent.com/SteamDatabase/GameTracking-CSGO/master/csgo/resource/csgo_english.txt

The issue seems to it cannot escape /x within the file

EOF when parsing backslash quote

"test case"
{
	"key"		"some value\""
	"foo"		"bar"
}

See https://github.com/ValveResourceFormat/ValveKeyValue/tree/eof-unescape-bug

This throws:

ValveKeyValue.KeyValueException : Found end of file while trying to read token.
  ----> System.IO.EndOfStreamException : Attempted to read past the end of the stream.

  Stack Trace: 
    KV1TextReader.ReadObject() line 50
KVSerializer.Deserialize(Stream stream, KVSerializerOptions options) line 44
EscapedQuoteNoEscapeTestCase.SetUp() line 21
RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
--EndOfStreamException
KV1TokenReader.Next() line 175
KV1TokenReader.ReadUntil(Char[] terminators) line 221
KV1TokenReader.ReadQuotedStringRaw() line 316
KV1TokenReader.ReadStringRaw() line 305
KV1TokenReader.ReadString() line 81
KV1TokenReader.ReadNextToken() line 64
KV1TextReader.ReadObject() line 42

\" causes it to trip up, it works with HasEscapeSequences: true. It's probably fine for this not to parse correctly, but I would at least expect the exception to be somewhat better than EOF

Implement schema text

Example file: https://github.com/MNoya/DotaCraft/blob/7791535a1977610defffb815aab58617235546da/content/dota_addons/dotacraft/models/elven_farm.vmdl

Create documentation

Can't deserialize shortcuts.vdf

First off, here are my shortcut item and container classes:

Lengthy-ish C# code block

[Serializable]
internal class Shortcut
{
	[KVProperty("appid")]
	public uint AppID { get; set; }

	public string? AppName { get; set; }
	public string? Exe { get; set; }
	public string? StartDir { get; set; }

	[KVProperty("icon")]
	public string? Icon { get; set; }

	public string? ShortcutPath { get; set; }
	public string? LaunchOptions { get; set; }
	public bool IsHidden { get; set; }
	public bool AllowDesktopConfig { get; set; } = true;
	public bool AllowOverlay { get; set; } = true;
	public bool OpenVR { get; set; }
	public bool Devkit { get; set; }
	public uint DevkitGameID { get; set; }
	public uint DevkitOverrideAppID { get; set; }
	public DateTime? LastPlayTime { get; set; }
	public string? FlatpakAppID { get; set; }

	[KVProperty("tags")]
	public List<string>? Tags { get; set; }

	public override string ToString()
	{
		return $"({this.AppID}) {this.AppName}";
	}
}

[Serializable]
internal class ShortcutsVDF : IEnumerable<Shortcut>
{
	[KVProperty("shortcuts")]
	public List<Shortcut> Shortcuts { get; set; }

	#region IEnumerable

	public IEnumerator<Shortcut> GetEnumerator()
	{
		return this.Shortcuts.GetEnumerator();
	}

	IEnumerator IEnumerable.GetEnumerator()
	{
		return this.Shortcuts.GetEnumerator();
	}

	#endregion

	public static ShortcutsVDF Load(string filename)
	{
		using var fs = File.OpenRead(filename);
		var kv = KVSerializer.Create(KVSerializationFormat.KeyValues1Binary);
		var kvObject = kv.Deserialize<List<Shortcut>>(fs);
		var vdf = new ShortcutsVDF
		{
			Shortcuts = kvObject,
		};
		return vdf;
	}
}

Trying to load a shortcuts.vdf file via ShortcutsVDF.Load() causes an OverflowException on the var kvObject = kv.Deserialize<List<Shortcut>>(fs); line saying "Value was either too large or too small for a UInt32". Changing AppID's type from uint to int (even though app IDs seem to be unsigned) changes the exception to a FormatException saying "Input string was not in a correct format".
What am I supposed to change about my code to make this work?

Here's the shortcuts.vdf I'm trying to deserialize. It was generated by Steam and contains only one item: shortcuts.vdf.zip

Implement WideString

CS:GO implements it like this:

// write
{
	int nLength = dat->m_wsValue ? V_wcslen( dat->m_wsValue ) : 0;
	buffer.PutShort( nLength );
	for( int k = 0; k < nLength; ++ k )
	{
		buffer.PutShort( ( unsigned short ) dat->m_wsValue[k] );
	}
	break;
}

// read
{
	int nLength = buffer.GetShort();

	dat->m_wsValue = new wchar_t[nLength + 1];

	for( int k = 0; k < nLength; ++ k )
	{
		dat->m_wsValue[k] = buffer.GetShort();
	}
	dat->m_wsValue[ nLength ] = 0;
	break;
}