offroaders123 / nbtify

A library to read and write NBT files on the web!

NBTify

Following in the footsteps of NBT.js and Prismarine-NBT, NBTify is a JavaScript library that allows for the parsing of NBT files on the web!

I started this project as a learning experience to try and make my own NBT parser from scratch. I didn't have much success in making it work reliably, so I've decided to make a brand-new fork of NBT.js that will support Bedrock Edition's little endian NBT format, one of my goals that spurred the idea for making a new library.

Prismarine-NBT seemed like a viable option to NBT.js, as it supports both Bedrock and Java formats. However, it doesn't support the browser out of the box, and bundling it seems fairly bloated just to support the browser. NBT.js is really compact, so I didn't want to take the option with more dependencies.

I really like the functionality of Prismarine-NBT and the simplicity of NBT.js, so I thought, why not meet somewhere in the middle?

NBTify has entered the chat!

Usage

Importing NBTify in the browser:

<script type="module">
  import * as NBT from "https://cdn.jsdelivr.net/npm/nbtify/dist/index.min.js";
</script>

Importing NBTify in Node:

import * as NBT from "nbtify";

Reading a file using the Fetch API in the browser:

const response: Response = await fetch("./bigtest.nbt");
const arrayBuffer: ArrayBuffer = await response.arrayBuffer();
const data: NBTData = await NBT.read(arrayBuffer);

Reading a file using the File System module in Node:

import { readFile } from "node:fs/promises";

const buffer: Buffer = await readFile("./bigtest.nbt");
const data: NBTData = await NBT.read(buffer);

Writing to a file using the File API in the browser:

const result: Uint8Array = await NBT.write(data);
const file: File = new File([result],"bigtest.nbt");

Writing to a file using the File System module in Node:

import { writeFile } from "node:fs/promises";

const result: Uint8Array = await NBT.write(data);
await writeFile("./bigtest.nbt",result);

nbtify's Issues

Bytes Read Success Checksum

Inspired from Prismarine NBT, I want to add the bytes successfully read checksum to the end of the NBTReader instantiater. It will protect against when part of the NBT stream is read, but there are still bytes to be read. If there is more to be read, it should throw.

zlib Reading + Writing

I have gzip working already, now I want to figure out what kind of zlib format Anvil chunks use and make that parseable from the compression functions too. I think it sounds like it's just Deflate, but I'm not sure if it's raw (no header) or full (with the header). And while it's not consistent with the compression functions, the compression option values will be "gzip" and "zlib" for the NBT.read() and NBT.write() functions, because I want those to follow the naming described in the NBT spec, like on wiki.vg and on the Minecraft Wiki.

Binary Empty ListTag Type Persistence

While looking into Schematic, Litematica, and Structure Block files, I wanted to see how NBTify's current support for them ranges. As of diffing the outputs from NBTify compared to the original source data, the tests didn't pass for the Schematic demo, and it ended up being because some of the ListTag values were types with a specific item type, which is something NBTify doesn't carry over during serialization if the array is empty. So if the ListTag in the file is typed for StringTag values, and the ListTag for them is empty, NBTify just uses EndTag as the item type, meaning the output diffs are inaccurate. This is not incorrect, and it's still properly accessible in terms of the spec and what the game can load, but it's not as accurate as being a perfect 1:1 deserialization-reserialization process, which is something I heavily strive for here with NBTify. If the exact same file isn't output, then NBTify is missing something, as I want it to be completely compatible with anything thrown at it.

This hasn't been a problem until now, because all of the other NBT files I have seen and tested against, all use EndTag for the empty array types. So I mostly haven't uncovered this just because I haven't come across a file that's implemented this way. Now that I have an example of one, I think it's easier to figure out how I should implement this.

I think this can actually work nicely with a custom Symbol() property on the array, which allows this persistence to be optional, and maybe even handled automatically by NBTify.

The other option I only recently found to be plausible, is that using Proxy objects to define custom NBTify primitive wrapper objects to limit/validate what kinds of properties can be assigned to a given wrapper primitive implementation. I didn't actually know about how Proxy objects worked, were implemented, or used, so this is a big hope to look into, and it gives me more ideas of ways to validate property assignment to these primitives, before passing them off to the write/serialization process.

My take for the longest time, since moving to using primitives as closely as possible, is to leave the run-time validation up to the time it is used, rather than when it is built/assigned to. Since JS doesn't have type validation when assigning to plain objects, arrays, or things like that, I didn't think it would be worth constructing full-blown custom structure primitives (objects, arrays) just to add type validation support, since that can be handled safely down the line anyways. Now that Proxy objects are in the picture, it might actually be feasible to validate types of things as they are assigned!

These were some links I used to explore this, I don't know too much as to what the best practices are for using Proxies with classes, say if you wanted to use them for these primitive wrappers, or for Web Components, something like that. My idea for using them with Web Components would help with not needing to use get() and set() accessors for everything, and in addition an internal private #field to store that value behind those public validators.

https://www.javascripttutorial.net/es6/javascript-proxy/
https://javascript.info/proxy
https://stackoverflow.com/questions/47779762/proxy-a-webcomponents-constructor-that-extends-htmlelement
https://grrr.tech/posts/2023/typescript-proxy-objects/

Listening to Devin Townsend Infinity for the first time, let's see how it goes! I'm already really liking it.
Wow, I'm already excited to listen to this again :)

Dedicated Test Subtests for Different File Types

Right now I'm running all of my tests in a single subtest, and things are getting very disorganized and chaotic. I want to split things apart so they can use individual logic for each kind of file. Maybe I can use a switch statement for the file extension of the current file to test against, then I can choose which extensions get which tests. That seems fairly organized. That can make things easier for defining which types get what tests as well, since one file type might want to use multiple of the tests I make available, rather than one or the other exclusively.

Modified UTF-8 (MUTF-8)

Tonight, we (Minecraft Manipulator) were discussing the NBT spec's use of the Java version of UTF-8, which is called MUTF-8, or Modified UTF-8. I have been meaning to look into the details about this and how it relates to NBT, so I'm glad that it came up in conversation again!

Found this JS implementation, which I will try to use for inspiration to add support for the format in NBTify, as it is inherent that strings will serialize correctly in the same fashion to how they do in the base game as well.

mutf-8 - GitHub
What does it mean to say "Java Modified UTF-8 Encoding"? - Stack Overflow
Modified UTF-8 - Wikipedia

Boolean Primitive Writing

Handling the JavaScript Boolean primitives as acceptable values when writing to raw NBT. They will be coerced to Byte tags first on the writing side, then also the reading side, so there will have to be a second layer to declare whether or not that key should be converted to a Boolean manually, or not. This is likely where the Minecraft type definitions and classes can handle that part of it.
*Edit: #6

SNBT LSP Support

This is more specific to NBT-Lang and Tnze's NBT language extension, but it's worth linking to here as well.

I want to enable first-class LSP support for SNBT files, possibly in combination with my type definitions from Region-Types. It would be similar to Svelte, in that it uses TypeScript's type checker, and HTML's language support, but in a different syntax that the extension would manage it's use of types on, compared to regular TS syntax. Not sure how much that is possible, but I have hopes for the best!

Global Shell Script

Running NBTify directly on your machine from the shell could be a really neat use case! It would be similar to the FFmpeg's ffmpeg command, where you work with your files right from the terminal.

SNBT Improved Error Parsing Messages

A reference to the discussion from Offroaders123/Dovetail#5:

I also think I need to look into improving the error messages with as to what part of the SNBT content isn't valid/parsing correctly. Say for this error, I could add a reference that it's in the array at key InvalidBecauseTheValuesArentOfTheSameType at item index [2], where that item is of type ShortTag, but the ListTag here can only accept ByteTag values. Something along the lines of that.

I think I can better accomplish this kind of handling if I bring the SNBT parser implementation back to a fully functional approach, rather than managing it with a class kind of thing. I can use callbacks of some sort which could allow the use of parameters that have the current key name and value or something like that, same with the current array index, item type, and other similar metadata-related things. It's like how you can use [].map((item,index,array) => {}) to get information about the iteration, as well as the source data you're working with, right from the callback.

Maybe I can look again into a more functional approach for the rest of the NBT reading and writing too, it doesn't have to be exclusively for SNBT handling. I do think the use of a class to manage the internal ArrayBuffer and DataView references is a little simpler to understand than passing them around everywhere too though. So, we'll see about that one. I feel SNBT's work with string manipulation and appending is a bit more akin to simple return types though, hence why it works naturally with a functional approach. It's not as straightforward to do that for the NBT parsing/writing counterparts.

Unnamed Compound Tag

Support for unnamed root Compound tags.

Why does this use top-level `await`?

It seems unnecessary...

And my framework doesn't seem to support top-level await.

Strict Flag: NBTData Option + Auto-Parsing

Since some NBT file data terminates before the end of the full file itself, this has brought up the use for adding the strict flag to the reading process. I'm curious if this should also be symmetrical, in that the re-written file currently won't be the same as the input file. Should NBTify provide a symmetrical API which will give you a file with the same file length as the write output? This would require storing the strict flag somewhere in the NBTData process, and possibly the minimum file length. Should it instead be a fixed file length? I'm not sure why the nature of some files are longer than the data they actually hold, which is what would lead me to implement this in a certain way.

So if you were to open a file with the flag NBT.read(data,{ strict: false }), should this be reflected as part of the metadata for the resulting NBTData object?

While writing this, I just thought of another thing. I think this goes in hand with this feature idea (only if I do implement this symmetrical handling). If the strict-ness of the file length can be represented in the NBTData object, I think it would also be safe to start automatically reading files without strict enabled, only after parsing all of the other file types first, like it does now. So this would run after the newly-added deflate-raw handling, and it would simply pass all format options as undefined, only with strict set to false. I think that might sound a bit risky though, so I'm not sure about that one. I think a lot could go wrong there, having written that idea out. I think it might pick up on some weird data first, say if it was a non-strict file, and it was also compressed or something. The blocks might have NBT-looking data somewhere in there, and it might exit early with malformed, unexpected data.

So, having discussed that, I think auto-parsing for strict may not be viable because you can't ensure that what's coming out is indeed what it should be, unless there's another way to do this that I'm not thinking of. Oooh! This sounds horribly messy, and not worth it, but maybe if it opens the file successfully with a lot of bytes left, it will keep trying, and check against the (multiple?) read results to see which one has read the most of the file. I think that's fairly unlikely to happen, unless the data is magically synonymous with the different format options somehow, but maybe that can help ensure that it's the correct one? I feel like that could technically be incorrect in some circumstances too, though. Especially if the trailing data at the end of the file has random byte data there, it could incorrectly terminate the NBT in a different format, later in the file than where the real, shorter data is stored.

So, I will look into having a symmetrical API, see if that's viable (less likely), and secondly, go through with just letting the user know whether the resulting data was loaded non-strict-ly, through the NBTData object result. Thinking about it too, I think this could be necessary for the TypeScript-level of things too, as having two files that have the same structure, but one is loaded strict-ly, and the other non-strict-ly, they would appear the same on the type level, while they weren't actually loaded with the same format options. Saying it like that, it might not be an issue though, since they are same at the nominal-type-level anyways. I think this distinction should also probably happen at the reading level anyways, since that's where the types would differ when you pass them into the function. But then again, this would break up being able to use NBTData objects as FormatOptions objects directly, so I think this is probably necessary either way. Not having this currently limits you from being able to use them interchangeably like that.

Ok, nice rant. Also, I don't quite like the new GitHub UI layout yet. Why change it?? This time, I'm going to help myself learn to like it, because GitHub is just too darn useful to be annoyed by it. Why they gotta change it though, eeeeh. Hey, at least they're trying something? Wouldn't want the other option, GitHub to go under, and fall behind and not get 'any' updates. Ok, rant over hehe.

SNBT Float Types Formatting

Unit Tests

Create proper unit tests with Jest instead of using a browser test.

NBTData Content Default Unknown

Looked into setting the default of the NBTData<infer ThisOne> generic to unknown rather than any, but I don't know how to do that as to satisfy the generic's constraint T extends RootTag, since unknown doesn't satisfy it.

The ideal goal:

export class NBTData<T extends RootTag = unknown, const U extends FormatOptions = FormatOptions> { }

That way you don't get any as the default type for using a plain NBTData declaration. If you did want to unsafely traverse through the data without knowing it's shape, then you could pass in NBTData<any> explicitly. This would help prevent accidental usage of NBTData<any> when it wasn't intended as the result on purpose. It would allow you to catch instances where you aren't using types on the data's shape, resolving it by adding your own types for the data, or by manually casting it to any if you didn't have explicit types for it. Leaving it with the default T extends RootTag = any removes the catch step, so you yourself have to track if you are using the data through any, or by the shape you passed into the generic. It won't error for using any by default.

Ok, while writing that out, maybe the solution is to make the first generic non-optional! Then it would force you to specify a type, be it either an interface for the shape of the data, or any if you wanted to do it unsafely! I'll try that out.

Minified Build Support

I've been curious about looking into using esbuild instead of tsc to compile the source for use in Node and the browser. Things are still a little over my head in terms of implementing it nicely in conjunction with my existing tsconfig.json settings, so I'm going to wait before committing things yet. There were a bunch of resources I found while trying to look into this, but it's still just a little over my head which route is the right way to go for NBTify.

I think my trouble is that I want to allow for multiple different ways to be able to import NBTify. In the browser, I'd like to have a fully ESM-compatible bundled build (minified, with dependencies), where you can still do (essentially) import { feature } from "https://cdn.dev/nbtify". I also want a fully-ESM compatible non-bundled (dependency-wise), maybe minified, build for Node.js (not in CommonJS). I think my current hangup is whether or not the Node.js build should be minified, as it sounds like it's less helpful for tree-shaking when used with a bundler (since you would want to import the Node ESM version when used with something like Vite, rather than the one that already has the dependencies bundled in, as for the CDN ESM build). And for another part of this, how does this all work with that I also have a CLI script for NBTify, as well as needing to use tsc to provide type definitions for the library as a whole, for any of these combinations?

So maybe it's not that the setup for it is very hard, but that I haven't fully decided yet as to which is the best way to target each of these use-cases.

And yeah, what's the proper way to manage your "bin": "./dist/bin/index.js" setup, in conjunction with regular-old library exports? What folder structure do people usually use for that? And are these two separate esbuild entry points?

Should I still provide a CommonJS-specific build output? Or is it encourage towards moving away from the dual-package setup, for new packages? I haven't personally had really any issues with needing to use NBTify in CommonJS thus far, but maybe someone else needs it for something, and they don't want to/can't migrate to ESM yet. I'd like to be able to make NBTify work for them, rather than have them need to figure out all of this on their own. Using ESM in the Electron backend is a valid issue, but I think that sounds to be less of an issue as of the last year or two, since it supports ESM now.

Lots of links related to this one

https://souporserious.com/bundling-typescript-with-esbuild-for-npm/
https://github.com/souporserious/bundling-typescript-with-esbuild-for-npm
https://esbuild.github.io/getting-started/#build-scripts
https://esbuild.github.io/api/#outbase
https://bundlephobia.com/package/[email protected] (Seems to be fairly similar in size to what esbuild does in my current demos for this)
https://nodejs.org/api/packages.html#--input-type-flag (New find, didn't know -e had a flag for this! Awesome)
https://medium.com/outbrain-engineering/the-hidden-power-of-package-json-a93143ec0b7c (sciencesakura/mutf-8#17)
https://dev.to/andreasbergstrom/simplify-typescript-builds-with-esbuild-and-skip-tsctsx-2124
vitejs/vite#1585
https://news.ycombinator.com/item?id=28861433
https://www.youtube.com/watch?v=mSnDUMybZXk
https://blog.logrocket.com/getting-started-esbuild/
https://esbuild.github.io/content-types/#tsconfig-json
evanw/esbuild#1343 (I noticed this with TS a while back in setting up a proper ESM build as well, I like that esbuild followed TS with this one. It's the choice to stay consistent, rather than trying to fix things on it's own)
https://www.youtube.com/watch?v=z16rzIF5J40 (Unrelated, cool find)
https://www.youtube.com/watch?v=alrLzBTHFH8 (current song)

End of Buffer Error

Add an improved "end of buffer reached" error, as it's not quite apparent when that is the case, since the common errors that surround this issue mention things about the NBTReader's DataView and it's byteOffset range. This happens when the NBTReader hasn't reached a complete CompoundTag and it's equivalent EndTag, but it tries to read past the end of the buffer. This throws the error, and it doesn't make it quite clear that this is the case of what happened. Now that I have it figured out, it makes sense. But, it's not obvious that it means that, to the user. It took me a long time to get that's what it meant, and it shouldn't be difficult for the user to understand what went wrong with the reading process.

I think a nicer way to handle this could be to check if the NBTReader's internal #byteOffset property has passed the byteLength of the buffer itself, then throw an error before trying to read from the DataView. This could be inside each of the private #read tag calls inside of the NBTReader, and each error could describe what kind of tag it was attempting to read, at that byte offset.

I think this would be one of the last errors that isn't handled internally by the library with a custom message, but of course you're not sure until you find another, haha. Maybe this can help find other missing pieces that I haven't handled yet!

SNBT List Tag Float Parsing

Compression Streams API Polyfill

Forgot to mention this one too. Gotta add a polyfill of the Compression Streams API for browsers that don't support it yet (I think WebKit is the only odd one out, as of now!). I've been looking at Stardazed's implementation for a bit now. I will likely use theirs or make my own edit of it, since I'd ideally only want a polyfill for the Compression part of their Streams polyfill, as that's the only missing part in modern browsers.
https://github.com/stardazed/sd-streams

JSON Replacer Parity

Was thinking about how NBTify is meant to be fairly symmetrical to the JSON namespace, and remembered that neither the SNBT nor binary NBT modules support the replacer callback parameter.

I think I initially skipped over this because I didn't want to worry about supporting it on top of the already complex NBT process, and I didn't want to messify the codebase just to include something initially. I knew I could come back to it again later if it deemed worthy down the line. I think it's something helpful to have in general, for the serialization processes, and it's already in the JSON namespace's API, so I'm going to look into it now.

I was going to just get it all added tonight, because I thought it would be simple and quick to add. But having looked back at the standard lib type definitions, there's more overloads for JSON.stringify() than I remembered. So It's gonna take a little extra to get setup, and I think I'll want to do some additional refactors to the general NBTify API before I try and add these features into the existing stack.

// Standard lib typings for the JSON namespace

interface JSON {
  parse(text: string, reviver?: (this: any, key: string, value: any) => any): any;
  stringify(value: any, replacer?: (this: any, key: string, value: any) => any, space?: string | number): string;
  stringify(value: any, replacer?: (number | string)[] | null, space?: string | number): string;
}

The JSON.stringify() keys selector overload is the one that made me rethink quickly adding this, because I want it to be structurally sound, rather than just patched in real-quick. Anything to do with the serialization steps are definitely important, and even more-so when modifying the user's incoming data. You don't want to unexpectedly mess with that stuff! A lot could go wrong there.

Right now, NBTify just supports the space option with the stringify() function through an options object, which I used instead of a bare-parameter because I wanted to anticipate adding more possible options down the road. That's come to pass though, and I think I'll want to follow the JSON namespace's API instead.

// NBTify's current SNBT API

export declare function parse<T extends RootTag = any>(data: string): T;

export interface StringifyOptions {
    space?: string | number;
}
export declare function stringify<T extends RootTag = any>(data: T | NBTData<T>, options?: StringifyOptions): string;

List Tag Item Type Assertion

Semi-related to #1, List tags need to have a way to define their element type, since JavaScript arrays can have items with multiple types, unlike how other languages do. I already made it so you can define the item type assertion on the TypeScript side, but this doesn't help with how the resulting JavaScript should write the List to an NBT Uint8Array. I'm thinking that possibly using Symbols could work, I'll have to try a few different things out.

SNBT Escaped Characters

Discovered the original implementation of SNBT parsing and stringifying that I'm working with from Deno-Minecraft wasn't set up to handle special escaped characters. I tried implementing a fix for it tonight, but I don't know how I can add it to the parse side of things. I got it working on the stringifying side at least though!

Found some helpful docs about the SNBT format here. Also found a helpful Stack Overflow page that shows how to 'escape escape characters', haha.

quartz_nbt::snbt - Rust
Escape new lines with JS - Stack Overflow

And within the SNBTWriter.#writeString() method, this is the code that I wrote that appeared to consistently fix the stringifying portion:

declare let quotedString: string;

const escapedString = quotedString
  .replace(/\n/g,"\\n")
  .replace(/\r/g,"\\r")
  .replace(/\t/g,"\\t");

Less-Strict Write Handling

Inspired, related to, and continued from the later paragraphs of #19, I'm considering making the writing process less strict. This seems like a backwards step, but I think it could make it a bit more dynamic, and a little more flexible. It also make things more consistent from the JavaScript side of things.

One of my struggles with pushing x JavaScript data structures to be NBT serializable is that not all JavaScript features go hand in hand with the NBT format spec. Namely, how JavaScript allows multiple-type items in arrays, compared to allowing only a single type for all items in NBT List tags.

My thoughts on making things more dynamic, and less strict, are because I'm not sure if I can completely, and securely validate all of the data going into the library, and I'm not sure if I want to force what kinds of data structures people may be trying to write with the library. I can add checks to prevent things from going in, but that may cause unexpected things to happen, which I also don't want. Either way, something unexpected is probably going to happen, so I think I'd rather it work correctly if possible.

I'm not sure if I like that last paragraph exactly, as it's not quite in the same words to what I'm thinking. Essentially, I want NBTify to just handle any data you throw at it, without having to do anything custom just to make it not error out (That's a better mindset explanation :) ). If the data you try to put in to the library isn't correct, then you have to handle how the data should be serialized. NBTify will try it's best to put the data into the NBT format spec the best as it can.

I think the JSON module is really cool in that you can pretty much throw anything at it, and it will serialize everything that it can, and coalesce the other things it finds to what primitives it does support (JSON.stringify() documentation). This essentially happens to objects only, with it's accessible properties. If there aren't any properties to serialize, then I think that's where the empty { } value comes from in the stringified JSON output. This seems fairly extensible and easy to understand/work with. The only non-JSON serializable primitive is BigInt, and that one will throw an error if it is encountered as a property value with JSON.stringify().

Kind of wrote a lot there, and I'm not sure if I summed everything up completely, so I think I'll leave it at that :)

Minecraft Data Structure Type Definitions

Not sure if it will be in this project or not yet (or it could be a separate repo), but I want to write type definitions for the Minecraft file data structures. One example could be the Bedrock level.dat file. It may look something like this:

import { NBTData, ByteTag, IntTag, StringTag } from "nbtify";

export interface BedrockLevel extends NBTData {
  name: "";
  endian: "little";
  compression: "none";
  data: {
    BiomeOverride: StringTag;
    CenterMapsToOrigin: ByteTag;
    ConfirmPlatformLockedContent: ByteTag;
    Difficulty: IntTag;
    FlatWorldLayers: StringTag;
    ForceGameType: ByteTag;
    GameType: IntTag;
    Generator: IntTag;
    // More entries...
  };
}

Accepting ArrayBufferLike for Reading

Partially delayed because of this issue here.

Going to look more into different ways of implementing it first. Had to add a few extra unnecessary if checks that are only for type validation, and I wasn't fond of that. But, if that's the most straightforward way, then I'll probably just end up going with it.

README Writeup + Installation Docs

I gotta step up my game! Prismarine NBT and nbt-ts document everything much better than I have, haha.

Primitive Tag Value Type Assertions

Yet another game plan, this one expanding off of #6, the custom primitive types should allow for value type assertions. Essentially, I want to allow the Byte, Short, Int, and Float base classes to accept value assertions. Relating to that part, I want to add a shortcut to allow for better boolean descriptions, since there's not a boolean tag, only a ByteTag. It would essentially be a type shortcut that goes from ByteTag<0 | 1> | boolean to BooleanTag. It would then allow you to specify if a tag is just a plain ByteTag, a "boolean" one, or what it's possible values are.
Here's an example (not tested, not sure if this will work as I think it does, yet)

*edit: Thanks wiki!

import { NBTData, BooleanTag, ByteTag, IntTag, StringTag } from "nbtify";

export interface BedrockLevel extends NBTData {
  name: "";
  endian: "little";
  compression: "none";
  data: {
    BiomeOverride: StringTag;
    CenterMapsToOrigin: BooleanTag;
    ConfirmPlatformLockedContent: BooleanTag;
    Difficulty: IntTag<0 | 1 | 2 | 3>;
    FlatWorldLayers: StringTag;
    ForceGameType: BooleanTag;
    GameType: IntTag<0 | 1 | 2 | 3>;
    Generator: IntTag<0 | 1 | 2>;
    // More entries...
  };
}

List Item Type Coalescing

This extends from both #19 and #20. 19 was originally going to be both for this and the Less-String Writing, but these ended up feeling like separate issues, based on a common theme.

On par with making more object data structures accessibly writable, I think it could make sense to also coalesce array values to be a single primitive type, rather than throwing an error for it. Ideally, your data coming into NBTify should be NBT-spec compliant, but if not, I think it's fair for the data to coalesce into being spec-compliant.

This would leave the user to work with their data accordingly, and to fall back on type definitions, or manual error checking for the item types.

In writing that last sentence, the other side of my mind says that it also makes sense to keep the error, as this feels similar to the recursive object data structure error that JSON.stringify() will throw if it encounters an object with that being the case. Maybe the Less-Strict Writing idea (#20) should only apply to objects, and maybe arrays should still be strict.

Gonna experiment with all three of these ideas, and see how they feel together. Never know how you feel about an idea until you see it fleshed out!

Big Note:
Another thing too, is that List tag item type error handling feels the closest to the recursive object handling for the JSON module, and the Object prototype error handling feels like it's trying to be the BigInt error, which is kind of treating Compound tag objects as primitive-ish values, which might be part of what feels weird about it's error handling. Object prototype error handling is checking the object itself, while the List tag item type error handling is checking the structure of the data, just like the recursive object handling does. It's the shape of the data, vs the thing of data itself. I think I may be on to something! 😃

GitHub Pages Demo Site

Now that the library needs a build step, I may have to make a separate example page that's different from the test pages. At least now I can reference the latest builds from npm! That's a great bonus that will really help out!
Not sure if anyone's reading this, but does anyone know if you can associate TypeScript types for a CDN URL? I think I almost got it working by using declare module on the URL, and referencing the exports from an npm install of the library itself. Seems like a nice solution, but maybe there's a simpler one I don't know about yet.

Empty List Tags

Need to add support for reading/writing List tags with no entries inside.

SNBT Read/Write Support

Little Endian Varint

Going to add support for little endian varint encoding and decoding! This is inspired by the use-cases provided by Prismarine-NBT.

Bedrock Edition level format - wiki.vg
janispritzkau / mcproto
chrisdickinson / varint
keithamus / varint
CloudburstMC / NBT

Circular Data Structure Handling

Thinking about the behavior of JSON.stringify(), I remembered that it throws an error when encountering circular object references. I realized that I haven't specifically handled that in NBTify, so you simply get to a Maximum call stack exceeded error when writing a circular object structure to NBT. This isn't ideal, and I should detect this before the user runs into it, just like JSON.stringify() does.

Detecting and fixing circular references in JavaScript - Stack Overflow
TypeError: cyclic object value - MDN

Found an example of how to detect them on MDN, this one can be used as a JSON replacer callback, and it will remove any circular object references before serializing to JSON. While this essentially makes the circular object "fully" serializable, I will go with throwing an error instead, like JSON.stringify() does, since I want it to be something that should be handled specifically by the user, where they can choose what to do with the circular references, rather than implicitly removing them inside of NBTify.

const getCircularReplacer = () => {
  const seen = new WeakSet();
  return (key, value) => {
    if (typeof value === "object" && value !== null) {
      if (seen.has(value)) {
        return;
      }
      seen.add(value);
    }
    return value;
  };
};

JSON.stringify(circularReference, getCircularReplacer());
// {"otherData":123}

Because of that, it also further sways me to throw an error for inconsistent ListTag item types, rather than silently excluding them from the written NBT buffer (#21).

I think I will silently remove unsupported JavaScript types during the writing process though, like for null, undefined, Symbol, and function values (#20). That's what JSON.stringify() does for any function values on objects, and I think it keeps things straightforward to be able to simply pass most objects right into NBTify, but also still be explicit in throwing knowledgeable errors for object structure issues, like circular object references and array item type inconsistencies.

SNBT Array Tags Type Validation

Realized that currently I'm not validating the tag types present in ByteArray, IntArray, and LongArray tags when reading them from an SNBT string. I really need/want to rework my current SNBT parser, it's a bit haywire because I built it from a few different sources, and also made it my own. So it's kind of a monkey-patched mess a bit still hehe.

Streams API Support

I want to look into adding support for opening NBT files directly from ReadableStream, and writing back to WritableStream objects. This could eliminate the requirement of reading from a full buffer of the entire NBT structure, as well as writing back to an NBT structure. This could potentially take out some memory overhead when dealing with larger files. This hasn't been a concern for memory usage as of now, but I think it could make things slightly lighter, which is always a good thing. I feel like it could help with the speed of the library too, but I don't know for sure. That also hasn't been a problem yet, but it could always be a welcome improvement. I think the main benefit here is that you can start working with the NBT before it has even fully loaded in.

Interface Index Signature Validation with Satisfies

Run-Time List Item Type Check

At the NBT writing stage, add a check that ensures all list items are of the same type as the first item in the list. If they aren't, it will throw a writing error.

Class-Generated Compound Tags + Object Error Handling

Currently, only non-extended Object objects are allowed to validate as a Compound tag. I want to allow NBTify users (me included!) to be able to construct their own NBT-serializable objects, by using ES6 classes. This would allow you to create interface types for in-game NBT objects, and also write your own JavaScript/TypeScript class that will generate the exact same structure on the fly, without needing to have the game generate them.

At first, I couldn't figure out how I can distinguish serializable vs. non serializable objects, since your own custom classes won't directly have the Object prototype as the first parent, so that won't work to check the class type. This would invalidate your custom classes from being able to be read as a valid Compound tag structure. My current code is there for this, because I want to throw error handling messages when you try to serialize non-NBT kinds of objects, like RegExp, TextEncoder, or other ones like that. Essentially, any standard library objects that shouldn't be parseable down to NBT, unless you explicitly wanted to allow one to do that.

Similar to how toJSON and get [Symbol.toStringTag] work (Check out the Well-known Symbols section on MDN), I realized a really nice way to add a check for if any given object is NBT-serializable, I could check using both the original Object prototype method I am currently using, and with the use of a new property, which would be something along the lines of get [Symbol("toNBT")], or something like that. The new check would first check if the given object is a direct decendent of Object (current behavior). And if that's not the case, check if the object has the get [Symbol("toNBT")] present. If either of those are true, than attempt to parse that object as a Compound tag.

I'm not sure exactly what the Symbol will be called yet, and I'm also not sure what I want the return value of the property getter to be, either. I'm wondering if it should work like toJSON, in that you can add NBT-serializability (what a mouthful/typeful XD) to non-originally serializable objects.

In typing that last sentence, I'm curious if I should remove the forceful error handling, and make it work like how JSON.stringify() handles it, in the fact that it coalesces objects without the toJSON property as just an empty { } object. This will make the writing process less strict, but it does make it more in-line with what the JSON module does, and I'm trying to do that where applicable with NBTify too, since I want it to feel like it's part of the JS standard library.

If that last paragraph is what I go with, then I can simply serialize "unsupported" (or rather, unexpected) objects like how the JSON module does it. I would remove the error checking for this, and it would simply be enforced with user-defined get [Symbol("toNBT")] properties and TypeScript definitions. That's the middle ground I have been following with the rest of the library too, and it has made making data structures really nice too, since it's just JavaScript objects and primitives that you are working with, rather than NBT-related primitives.

Format Options Generic Merging

Reference: d5f3b44

I haven't quite figured out how to merge the values of a generic from it's default type, and with that of those provided by the caller. Here's a better demo for that:

const demo = new NBTData({},{ /*name: "",*/ endian: "little", compression: "deflate-raw", bedrockLevel: null });

const { name, endian, compression, bedrockLevel } = demo;
name; // eek, shouldn't be `unknown`, should be of type `Name`.
endian; // "little"
compression; // "deflate-raw"
bedrockLevel; // null

This has to do with how I'm implementing the generic types for the NBTData object, specifically the U generic which is of the type FormatOptions. See how that is implemented as of this issue, here.

SNBT Boolean Keywords

Export/Import from JSON Support

Similar to #4, but using just plain JSON. I'd like to be able to export standard JavaScript JSON to NBT, for things like a JavaScript game's save data. And kind of the other way around, it would be nice to export regular NBT, with all of it's custom non-JSON tag types, as JSON too. Ideally I think one would want to use SNBT to edit full NBT, better than my custom JSON edit, but I think the JSON version may also be a bit better supported by less complicated text editors, ones that might not detect .snbt files off the bat.
It will likely resolve the non Number based number values into strings with the type suffix following the value. Essentially, a string wrapper for the value, and it looks like how SNBT would.

Primitive Type Shape Checking

Relating to #6, my custom primitive type classes aren't typed with enough strength to differentiate each other, in TypeScript's terms. Since the shape of each class is the same, even if you declare the type of x as one of the primitive number types, it will still allow the value to be a different type. A better example, using code:

import { Byte, Short, Int, Float } from "nbtify";

interface LevelDat {
  Key: Byte
}

const data: LevelDat = {
  Key: 5 // Should be `new Byte(5)`, however it doesn't throw an error for this, while it should. It appears to be because the value has the same shape as the Byte object, nothing is unique between the two.
};

Root ListTag - Bedrock

Investigate Bedrock's use of having the ListTag type as the root element, as it appears to not always be the CompoundTag type.

NBT format - Minecraft Wiki

ArrayTag Generic Types

Looking into a few different ways to accomplish representing tuple types with TypedArray-based NBT key types.

Right now the standard library types don't provide anything like tuple support for TypedArray values. I'd like to be able to do something like Int8Array<[number, number]>, and only allow access of those two properties, just like the regular Array tuple type notation allows for ([number, number] on it's own).

I've encountered a few places where this would be a great help at type validation, namely things like player NBT UUID fields, which would be IntArrayTag<[number, number, number, number]>, as well as position tuple fields, which tend to be what would be described with IntArrayTag<[number, number, number]>.

Accept CompoundTag for Writing

Accept a CompoundTag compliant object as a value to write to the buffer, when using either NBT.write() or NBTWriter.write(). This makes it so you don't have to construct an NBTData object every time you simply want to write some data to a buffer. Currently, you have to wrap your NBT data CompoundTag object in an NBTData object in order to write it to a buffer. This seems like an extra step to need to do if you don't already have an NBTData object that you are working with. You can pass the same options in to the write methods that the NBTData object would define, so you don't lose anything by doing that either, as you can just manually add the configuration there instead of on the NBTData object. Functional and class-based programming! I think? Haha.

This will eventually make things really nice when I get to JSON and SNBT support (#5, #4), since you might just have a plain JavaScript object (JSON serializable) that you would want to save as NBT, and maybe you aren't using any of the custom NBT primitives at all.