jverkoey / binarycodable Goto Github PK

The encoding and decoding implementations presently use a byte-level cursor. If bit coding is desired, however, the byte granularity makes requires that bits are packed and unpacked into bytes before being coded. This creates a fair amount of overhead for tightly packed data formats.

A successful solution to this task will meet the following requirements:

• Both the encoder and decoder provide APIs for working with a bit-level cursor.
• There is a clear relationship between bit-level cursor and byte-level cursor.

One possible approach is to provide a bit container view.

Swift Codable is able to generate encoding and decoding implementations for free because of the availability of key/value containers. Binary Codable does not support key/value containers, so it is not presently possible to map properties to external representations without making some potentially scary assumptions about order.

A completely generalized solution to this problem is not desired due to the need for conditional encoding/decoding paths in complex cases. Solving conditional coding is considered a non-goal for this task; in such cases custom implementations of encoding/decoding are encouraged instead.

A successful solution to this task will meet the following requirements:

• It is possible to map properties to regions of memory bi-directionally (encoding + decoding).
• It is possible to map BinaryCodable entities to regions of memory bi-directionally (encoding + decoding).
• Such a definition only needs to be made once for a given object, similar to the CodingKeys pattern in Swift Codable.

Decoding a Data object, even one that's memory mapped, causes the entire file to be paged into memory which is costly for large files.

BufferedData currently pulls data into a buffer that will increase in size indefinitely. This implementation should be replaced with a ring buffer that only grows if blocks of data are requested above the current ring buffer size.

Hi @jverkoey, this is not an issue report but a question I'd like to ask. Why do you think we need an extra set of protocols to interact with binary data? I work with a bluetooth peripheral at my org and we created a naive implementation and mapping of the custom data layouts for our BLE API on the App side. My long term goal was it to explore custom decoder / encoder for Codable to unify the implementation by a battle tested functionality and also make use of code synthetisation as much as possible.

I also would love if you could bring up this topic about binary data to the Swift forums. I think some community member would love to elaborate with you on the general problem. I also think this problem area should be solved generally in the stdlib, because one day Swift will likely enter the embedded region where this type of functionality will be indispensable.

Prior to releasing 1.0 there should be a performance benchmark for encoding and decoding so that performance changes can be tracked over time.

It's currently possible for containers to be nested somewhat deeply as nested types are coded, potentially creating a non-trivial amount of cpu + memory overhead. Much of the container behavior has been added in order to support limiting container sizes, which has both practical and safety benefits. The goal of this issue is to find a way to reduce the number of nested containers and any associated overhead.

A .binarycookies file contains many pages of cookies, and the entire file contains a checksum near the end. The checksum is the sum of each page's checksum, which is the sum of every 4th byte in the page. I don't think it's possible to compute this with the current API, right?

I don't have a strong opinion about an ideal API for this, but if SequentialBinaryEncodingContainer.encode(_:) returned the encoded data then I think that would at least provide the necessary data. In regard to the recent addition of the ability to encode arrays of BinaryEncodable, if the API returned the encoded data for the entire array I don't think that would work in this case, because the "sum every 4th byte" behaviour needed for this checksum starts at the beginning of each page (Hopefully that explanation is clear). This only means that for this file format each page would need to be encoded individually instead of encoding the entire array at once.

In the current implementation, each page encodes itself to Data during the encoding process in order to compute its checksum, so each Page is encoded twice over the course of encoding a file. I would imagine that for larger files the inefficiency of this would be more noticeable.

In the .binarycookies format, there's a constant "footer" value and then the rest of the data in the file is binary plist data. I think it would be best to use PropertyListSerialization to decode this, so I need a Data to pass to it. I don't think there's a way to do this in the current BinaryCodable API, is that right? Perhaps this could look like container.decode(length: nil), similar to how you can create a nested container without a maximum length, or container.decodeToEnd(). I don't have a strong opinion about the naming though.

The current implementation reads a byte at a time (in a way that probably copies them, too?) until it hits the end of the container.