cyphrme / coze Goto Github PK
View Code? Open in Web Editor NEWCoze is a cryptographic JSON messaging specification.
Home Page: https://cyphr.me/coze
License: BSD 3-Clause "New" or "Revised" License
Coze is a cryptographic JSON messaging specification.
Home Page: https://cyphr.me/coze
License: BSD 3-Clause "New" or "Revised" License
Instead of making various issues for various JSON concerns. I'm going to use this issue to track any concerns with JSON.
Coze needs strictly defined JSON capabilities. Some of these capabilities are not provided by the standard
Go library, or misbehaves under certain circumstances. Currently, there are no known 3rd party libraries that we consider suitable for Coze. Our hope is that we can use JSONv2 when production ready and that will solve any JSON concerns.
A new JSON library should minimally include these characteristics in addition to
the existing behavior of the standard library:
&
, <
,>
. and while not adding an additional new line as the standard library currently does.)Three of the larger obstacles:
To resolve these obstacles, Coze implemented orderedmap, its own custom JSON unmarshaler, and a "checkDuplicate" helper function.
A search of other Go JSON issues that may be relevant to Coze:
https://github.com/golang/go/issues?q=is%3Aissue+is%3Aopen+encoding%2Fjson+in%3Atitle
See https://pkg.go.dev/github.com/go-json-experiment/json
There are other best practices in JSONv2.
Proposal here:
https://github.com/Cyphrme/CozeX/blob/master/proposal/unsigned_alg_tmb.md
It's possible to successfully unmarshal valid JSON into a MapSlice, but then fail to marshal that same MapSlice back into JSON.
I'm assuming this isn't desired behavior, but I could be wrong.
POC: https://go.dev/play/p/MlNV74p2cCY
package main
import (
"encoding/json"
"log"
"github.com/cyphrme/coze"
)
func main() {
b := []byte("{\"\x7f\":{}}") // discovered via fuzzing
if !json.Valid(b) {
log.Fatal("invalid JSON")
}
var ms coze.MapSlice
if err := json.Unmarshal(b, &ms); err != nil {
log.Fatalf("json.Unmarshal: %v", err)
}
if _, err := json.Marshal(ms); err != nil {
log.Fatalf("json.Marshal: %v", err) // fails here
}
}
During a conversation, I was sent this example message: https://cyphr.me/coze#?input={%22pay%22:{%22msg%22:%22Hello%20Retr0id!%22,%22alg%22:%22ES256%22,%22iat%22:1701603747,%22tmb%22:%221KGZzsqiFAYE5uDO3CCh3PMl9pqwlG1RrI8i5gZY94c%22,%22typ%22:%22cyphr.me/msg/create%22},%22sig%22:%22vZEALUQShRlLYyhJoGmkxpQhhEeUPlzbIZqyTsUyPBMlJMIqGClgSs3uXTDiwumsMivFQKU8K4z4Ec7WlxYIew%22}&dontSignRevoke&updateIat&selectedAlg=ES256&verify
By introducing a duplicate "msg" key, I was able to forge a new message that also passes signature verification according to the web UI: https://cyphr.me/coze#?input={%22pay%22:{%22msg%22:%22Hello,%20Zamicol.%20I%20believe%20you%20will%20find%20that%20this%20Coze%20message%20is%20also%20(supposedly)%20signed%20by%20your%20key!%20Coze%20could%20fix%20this%20issue%20with%20better%20UI,%20but%20I%20think%20this%20illustrates%20just%20how%20hard%20it%20is%20to%20canonicalize%20JSON.%20This%20wouldn't%20be%20possible%20in%20the%20first%20place%20if%20you%20were%20signing%20base64'd%20bytes.%22,%22msg%22:%22Hello%20Retr0id!%22,%22alg%22:%22ES256%22,%22iat%22:1701603747,%22tmb%22:%221KGZzsqiFAYE5uDO3CCh3PMl9pqwlG1RrI8i5gZY94c%22,%22typ%22:%22cyphr.me/msg/create%22},%22sig%22:%22vZEALUQShRlLYyhJoGmkxpQhhEeUPlzbIZqyTsUyPBMlJMIqGClgSs3uXTDiwumsMivFQKU8K4z4Ec7WlxYIew%22}&dontSignRevoke&updateIat&selectedAlg=ES256&verify
The web UI gives no indication that there's anything awry here, and proclaims that the message was verified.
This would be a non-issue in many use-cases, because any code reading the msg parameter will likely see only the real one - but an implementation in some other language may see only the first, causing breakage. But, for the purposes of the web UI, it's definitely misleading.
As an aside, I think Coze's general design/approach is fine, but dealing with JSON like this is error-prone, and here is one such error - one which is hopefully an easy fix.
Coze repository organization
- Coze ("Core"/main specification and the Go Coze reference implementation)
- Coze_x (Coze extended)
- Coze_go_x (Go implementation of extended features)
- Coze_js (Javascript implementation)
- Coze_js_x (Javascript implementation of extended)
- etc...
Coze
, aka "core".This repository should be split into a few different repositories. It is usually good practice to split out documents from code so that developers aren't burdened with reviewing changes to the repository.
I suggest the following repositories:
This unfortunately would probably require renaming "Coze" to "coze". "coze" would be for the main spec (README.md) which defines Coze Core, and also documents, discussion, proposals, best practices, and FAQ. Github appears to be a good place for discussion and document modification, but we don't want a large number of developers worrying about the changes happening in the repository due to simple document modification.
Experimental is for new algorithms and useful Coze related libraries not in the core spec (normal
would be one such library). New algorithms are allowed time to mature first in experimental before getting accepted into "Core".
The only planned expansion of "Coze Core", the implementation and specification that currently lives in this repo, is adding new algorithms. No changes are currently planned to the spec other than per algorithm adjustments. However, we wanted to give this more time and receive more feedback before we made a more final decision.
There will be minor tweaks on a per algorithm basis for Coze Core. For example, @LoupVaillant suggested doing the following when handling Ed25519:
My choice for Monocypher was to do the same as Zebra:
- Reject any S that equals or exceeds the order of the curve.
- Accept low-order A and R.
- Accept non-canonical A and R.
- Use the batch verification equation (it's the forgiving one).
This needs to be specified in Coze Core so that all implementations align.
We could do a less dramatic division. However, this is less normalized so I'd advocate for the aforementioned naming.
Playground demonstrating the issue:
There's an apparent problem with RFC 4648. There are three places base 64 representation may contain string variation:
What is "canonical encoding"? From the last three characters of the example tmb
, "cLj8vs...XNuhOk"
, the values hOk
and hOl
may both decode to the same byte value (in Hex, 84E9
) even though they are different UTF-8 values. (Example decoding hOk and hOl.) The canonical encoding is hOk
The RFC specifically addresses 1 and 2, but not really 3.
RFC 4648 advises to reject non-alphabet characters, which can include padding. I agree with this advice:
Implementations MUST reject the encoded data if it contains
characters outside the base alphabet when interpreting base-encoded
data, unless the specification referring to this document explicitly
states otherwise. [...] Furthermore, such specifications MAY ignore the pad
character, "=", treating it as non-alphabet data[.]
I don't see the RFC really address the to the third concern.
Obviously non-"strict"/non-canonical base 64 encoding is incorrect, and any encoder producing non-strict encoding should be fixed. However the question is what should Coze specify regarding non-strict encoding/decoding? Both Go and Javascript are permissive when decoding and do not throw errors.
Ultimately, the concern is different base 64 encoders/decoders may have different behavior. Ideally, Coze should specify the appropriate behavior for Coze. Section 3.5 mentions non-canonical encoding in the context of unpadded data and this issues is unrelated to padding (hOk=
and hOl=
, both padded, have the same issue as unpadded strings).
The concern is that if a Coze implementation used string comparison instead of byte comparison, this could result implementations disagreeing about valid messages. For example, with a non-strict tmb
encoded string, if a Coze implementation checks tmb
before cryptographic verification, it may check this based on the string value or the byte value, and comparing the string value or the byte value will result in different behavior.
Another note for any Coze restriction on encoding: JSON is base 64 unaware, any sort of Coze specified enforcement of base 64 encoding can only be applied to Coze known fields with type b64ut, and cannot be applied generally to any b64ut value.
There appears to be only two options to handle this:
2 is more conservative, but may require unnecessary checks that don't really add value. 1 has the potential to be more compatible if assuming that systems can decode permissively (other programming language's base 64 libraries decode permissively), which may be a bad assumption.
Regardless, I believe that 1 is the correct behavior here. Even if languages/system do no error on non-canonical encoding, implementing an encoding error can be implemented by re-encoding the decoded data and comparing strings.
This base 64 decoding bug doesn't appear to be a structural/architectural/security concern since Coze uses the UTF-8 encoding of the string for signing and verification, however it is a interesting problem that should be known when working with RFC base 64. Concerning specifically replay attacks, signatures are still not malleable as payloads are UTF-8 encoded and the signing operation is not base 64 aware.
If Coze used the base 64 representation directly, this would be a security concern and could result in reply attacks.
It should be obvious, but this situation also applies to the URI unsafe alphabet and messages with base 64 padding, which all are interpreted as the same bytes. (My conversion tool only has "base64 as an input and not the various permutations since all variations can be known (or is irrelevant) and results in the sames decoded binary payload.
I currently have errata open on one of the relevant sections.
I'm going to implement a non-canonical encoding check on Go and JS Coze.
See also the Go base64 package.
Go's base64 ignores carriage return and new line, so it is malleable, but JSON unmarshal does not, making Go Coze non-malleable. https://go.dev/play/p/X0J74F0zWVf See also the new line test in base64_test.go
Although the simple verifier is already in-browser, offline compatible, and is easily locally hosted, eventually we hope to provide the same ability with power verifier. It probably could be done programatically using a tool like gildas-lormeau/SingleFile
Coze Verifier
Simple Coze Verifier
Github Hosted Simple Verifier
Simple Verifier Codebase
indexCounter is accessed without synchronization, which produces data races that violate the memory model. As a proof of concept, build and run the following program with the -race
flag.
package main
import (
"encoding/json"
"sync"
"github.com/cyphrme/coze"
)
func main() {
n := 10
var wg sync.WaitGroup
for i := 0; i < n; i++ {
wg.Add(1)
go func() {
defer wg.Done()
var item coze.MapItem
json.Unmarshal([]byte(`{}`), &item)
}()
}
wg.Wait()
}
You'll see output like the following.
==================
WARNING: DATA RACE
Read at 0x00010041e838 by goroutine 13:
github.com/cyphrme/coze.nextIndex()
.../pkg/mod/github.com/cyphrme/[email protected]/mapslice.go:36 +0xf0
github.com/cyphrme/coze.(*MapItem).UnmarshalJSON()
.../pkg/mod/github.com/cyphrme/[email protected]/mapslice.go:118 +0x11c
encoding/json.(*decodeState).object()
...
Previous write at 0x00010041e838 by goroutine 12:
github.com/cyphrme/coze.nextIndex()
.../pkg/mod/github.com/cyphrme/[email protected]/mapslice.go:36 +0x108
github.com/cyphrme/coze.(*MapItem).UnmarshalJSON()
.../pkg/mod/github.com/cyphrme/[email protected]/mapslice.go:118 +0x11c
encoding/json.(*decodeState).object()
...
The word revoke may connotate that a key was never trusted/authorized.
The specification should better explain that rvk
is expected to be an expiry time.
Cozies signed before the rvk time should be considered valid and action signed after should be ignored.
As is currently noted by the specification, actionable events based on future expiration times are outside the scope of Coze, and revoke messages with future times should result in the signing key considered immediately expired.
Thanks @qbit for bringing up this concern.
A cli client would be fantastic.
JSON doesn't define "integer" but only the concept of a "number" which doesn't require any specific precision, leaving that up to the implementation. JSON numbers therefore can't be assumed to have any specific value, and fields iat
and rvk
would need to be specified as strings in order to provide deterministic and bijective encoding.
See issue golang/go#51082 (comment)
Once that is addressed, I'll update the docs with package links.
Is this project still active?
@LoupVaillant suggested doing the following when handling Ed25519:
In my opinion standardizing signatures and public keys is much more
important than worrying about anything related to the private key. And
just at that level you have to grapple with much more fundamental issues
than how to define your private key:
https://hdevalence.ca/blog/2020-10-04-its-25519am
So you have a public key A, and a signature R || S.
A and R are points on the curve, and S is just a number.
Thankfully, the main issues were dealt with from the beginning:
But there's still room for variation in the verifier:
When two verifiers disagree on any of the above, this can cause problems
when maliciously crafted signatures end up being accepted by some and
rejected by others, leading to problems like network partitions. Worse,
the RFC didn't clearly answer all of those questions, and allowed users
to chose which verification equation they would use. And it's difficult
in practice to find two implementations that behave identically. It's a
freaking nightmare.
My choice for Monocypher was to do the same as Zebra:
The reason I reject high S is because (i) everyone else does, and (ii)
accepting it would enable malleability attacks. For everything else I
chose to be as permissive as possible. This has the advantage of being
backwards compatible with any other implementation: no signature that
was previously accepted will be rejected.
The RFC on the other hand made the following choices:
I personally disagree with the last two items. Interoperability with
batch verification (which is twice as fast as regular verification)
should be mandatory, and rejecting non-canonical points makes the code
more complex for no benefit at all.
You'll have to make your own choice too if you want a complete
specification. I personally would recommend you imitate Zebra and
Monocypher, because many implementations can be made compatible with a
bit of pre-processing:
Ran into this great presentation on normalizing/standardizing Ed25519, "Taming the many EdDSAs"
https://csrc.nist.gov/csrc/media/Presentations/2023/crclub-2023-03-08/images-media/20230308-crypto-club-slides--taming-the-many-EdDSAs.pdf
Also consider the advice in:
https://github.com/Cyphrme/Coze/blob/01c154e4024b4e876b8d152166ce85cf2a945e22/README.md#coze-fields
Binary values are encoded as RFC 4648 base64 URI with padding truncated (b64ut).
https://www.rfc-editor.org/rfc/rfc4648#section-3.2
when assumptions about the size of transported data cannot be made, padding is required to yield correct decoded data.
As far as I can tell, the size of binary values is not communicated to recipients, and therefore padding should not be truncated. (The URI encoding is also non-standard.)
@LoupVaillant suggested to avoid semantical confusion by avoiding entirely the term "cryptographic agility". In its place we could say "provide loose primitive coupling."
The design goals would then become:
Edit: Thinking more on this, perhaps the fourth design goal is dropped altogether since Coze implementations inherently provide loose primitive coupling, and that phrase itself would need to be rigidly defined. It also doesn't capture what I was trying to convey, that Coze provides "versioning" via "alg". Perhaps even something along the lines of "provide defined cipher suites". Edit2: or "Specify cipher suite expectations"
post quantum crypto ftw?
if you can get python versions of this, there are entire ecosystems like django / flask that can adopt coze in place of JWT.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.