Signing Or Encrypting Part of a Message about aries-rfcs

[dhh1128]

@kdenhartog ^^

[dhh1128]

@SmithSamuelM Here is Kyle's doc that recommends a similar mechanism (the one that @TelegramSam suggested should be used as a stopgap). https://github.com/kdenhartog/indy-hipe/blob/d421fc77bae87c780aad346d15c0c49939adc281/text/digital-signatures/README.md

[SmithSamuelM]

these are essentially the same. Its signing a base64 serialization. My concern was that might be attempting to canonicalize JSON and sign that . Which is fraught with potential problems.

[kdenhartog]

I've heard that canonicalisation of JSON is a real problem, but I've never understood the depth of the problems it introduces. Do you have details into what are the tradeoffs with canonicalising JSON? FWIW, I just used base64 serialisation because it was used in JOSE and I didn't want to deviate.

[SmithSamuelM]

JSON and Javascript objects (data structures) are based on an underlying data structure called a hash table or dictionary. Historically hash tables entries store data as (key: value ) pairs. Each key has a value. The ordering of the keys in the dictionary is not ordered in any predictable way. (Not alphabetic nor in order of entry). The order is considered practically to be indeterminate. This underlying lack of ordering means that the serialization of a dictionary via JSON may have the order of the (key: value) pairs change when it converts from a serialization to a hash table and then back again resulting in a different serialization. A different serialization will have a different signature so any signatures may be invalidated by the conversion. Tooling for JSON does not preserve ordering when converting to from JavaScript objects. Similarly JSON only cares that the entries in the dictionary are preserved as in the set of (key: value) pairs not the ordering. Nor does JSONcare what white space is included in the serialization as long as it converts to the same set (dictionary) of key: value pairs. All of these characteristics mean that converting from a JSON serialization to a JavaScript Object (or the equivalent in some other computer language) and then back again into a JSON serialization (round tripping) is not guaranteed to produce the same JSON serialization. So digital signatures or digital hashes of the serialization also will not be guaranteed to remain valid after the round trip. So any scheme that unserializes and reserializes may break any digital security measures. Canonicalization is an attempt to enforce a strict round trippable JSON serialization that preserves the exact set of bytes so that signatures and hashes will also be valid. This means the off the self JSON tooling endemic in Web 2.0 cannot be used but special purpose tooling must be used that converts Javascript objects to ordered dictionaries and then to JSON with known white space. It’s doable but means off the shelf tooling won’t work. It’s swimming upstream. The point of using JSON in the first place is to leverage existing tooling, so adding the requirement for a canonical JSON is not leveraging existing tooling. One might as well use a custom serialization format as to use canonical JSON.

[dhh1128]

Agreeing with Sam, this is why we've chosen to say that what's signed is a string that contains JSON, rather than a JSON structure itself. This takes canonicalization off the list of concerns.

[kdenhartog]

@SmithSamuelM Thank you for the excellent detail. This helps quite a bit in my understanding and I definitely agree that this causes quite a bit of hassle.

I'm sure I'll be referencing people to this comment multiple times in the future.

As far as this issue goes, What would you like to see as the action in order to close this out?

[SmithSamuelM]

I think it can be closed

[kdenhartog]

Thanks for the information on here and I'm glad we're aligned on the general direction. I'll be sure to loop you in to the conversations as I'm proposing updates to DIDComm.