decentralized-identity / sidetree

Sidetree Specification and Reference Implementation

Sidetree

This repository contains both the blockchain-agnostic Sidetree specification. A reference implementation can be found here.

Specification

See the latest spec for the full Sidetree specification.

See the API spec for the full API specification to interact with a Sidetree node.

Specification Editing/Contributions:

1. Clone the repo.
2. Create a topic branch for your spec contributions.
3. run npm install
4. run npm run spec:edit
5. If you see errors like UnhandledPromiseRejectionWarning: Error: ENOENT: no such file or directory, open './www/spec/index.html'... create the missing directories and files and try again (they will be overwritten).
6. Type npx serve . in the root directory and open http://localhost:5000/www/spec.
7. Modify files in the spec/markdown/ directory to make changes, refresh to see changes.
8. Do not commit build assets.
9. Try and make blocks of text small so that changes can be suggested easily on specific lines.
10. When you are happy with your changes, commit to your topic branch and open a Pull Request on GitHub and reviewers will be alerted to review for a potential merge.
11. Make sure to tag people continuously to ensure your PR is reviewed in a timely manner.
12. PRs that sit open without comments / reviews, will be closed at the editors discretion.

sidetree's Issues

Create sidetree-specific DID Document constraints

Because storage size could potentially limit the ability for folks to run full nodes at scale, we need to create a more strict set of sidetree-specific DID Document constraints that limit the size and use of props/values. Here are a few possible constraints we may want to enforce:

1. Limit size/content of DID Fragment Identifiers
2. Limit key size/content to a whitelist of recognized schemes
3. Fail an op/batch for inclusion of non-standard props

Align with protocol spec to use operation request payload for operation hash calculation

Currently, only create operation hash is using the request payload for operation hash calculation, rest of the operation request types use the entire request buffer. Need to align with protocol spec.

Fix numerous Node crypto library deprecation warnings

There are numerous Node crypto library deprecation warnings when running the node:

(node:19808) [DEP0091] DeprecationWarning: crypto.DEFAULT_ENCODING is deprecated.
(node:19808) [DEP0010] DeprecationWarning: crypto.createCredentials is deprecated. Use tls.createSecureContext instead.
(node:19808) [DEP0011] DeprecationWarning: crypto.Credentials is deprecated. Use tls.SecureContext instead.

Persistent storage for processed operations

Currently all data regarding processed operations are in-memory, there needs to be a persistent storage layer for both desktop node and nodes running as a service.

Improve OperationProcessor tests

Improvements:

1. Create a chain of updates to a DID to more closely resemble real world scenarios.
2. Verify what content in the DidDocument, even if it's just DID and public keys.
3. Modify/update DidDcoument and submit PR to did-common-typescript as needed.

Consolidate DidPublicKey interface defined in both this project and in did-auth-jose

Probably best to use the definition defined in @decentralized-identity/did-auth-jose.

DID Cache - Address PR comments

1. Logic seems to be incorrect/incomplete to put signature verification in apply(). What happens if the operation to apply back-points to a unresolvable operation? Currently it doesn't look like there will be another chance/location where signature is verified again.

2. getHash() - Can't hard-code hash function to use, has to depend on protocol version ie. Protocol.getProtocol(...), this means either the operationHash or blockNumber needs to be given in the WriteOperation.

3. Decide if OperationInfo should contain operationHash or not.

4. resolve() should take in a DID string not a versionId as suggested by this issue

5. applyVersionChainUpdates() - Need to make sure timestamp of opHash is greater than timestamp of preVersionId, in case of an invalid operation pointing to an operation further down the chain from it.

6. Design: seems to me that with the approach of evaluating operations at the time of lookup, all the possible links needs to be preserved, not just the one with lower timestamp. For instance signature verification can't be performed if there is a missing parent operation while there is a fork.

Use the hash of the initial document as the DID instead of the hash of the create operation

Resolving a DID that had an invalid signature works but shouldn't

After discussions with multiple engineers, it has become a requirement that registration should not require a signature. We need to make the appropriate code changes and document the rationale.

Refactor code commonly used by all Sidetree microservices to lib/common

There are code commonly used by sidetree-core, sidetree-bitcoin and sidetree-ipfs such as:

Response
ResposeStatus
etc

Currently code are duplicated in three places, would be nice to refactor into a common repo.

Implement/use a faster Base58 encoder

The current popular bs58 npm package performs horribly, it takes milliseconds to encode buffer under 1KB.

This resulted in severe perf implication in my load testing. Need to find a better library or implement one.

Handle block reorganization

Need to rollback and reapply operations.

Parallelize downloading of anchor and batch files.

Parallelize downloading of anchor and batch files to increase ingestion throughput. This will be very useful mainly for initial bootstrapping/initialization phase.

Fix test code coverage execution

Currently getting the following error when running code coverage:

Error: Cannot find module './json/didDocumentTemplate.json'

Unit Tests for Blockchain, and CAS Clients

Use mocking libraries to mock the REST API calls to blockchain and CAS calls and validate the functionality

Enable runtime configuration of what the sidetree is persisting

There has been some discussion around extending the sidetree model to allow for storing more than DIDs. Example, storing attestations / claims.

The underlying requirements of batching transactions and persisting an anchor hash would be the same, but the processing of each item would be unique, eg for DIDs we'd need to validate the change, whereas for an attestation it'd probably be more of a straightforward save.

As far as implementation, this could be a run time config that swaps in the appropriate module based on what it is being instantiated for.

ID of public key returned by create needs to be prefixed by the created DID

{
"context": "https://w3id.org/did/v1",
"id": "did:sidetree:EiD3j1DfUzypfK8ZP",
"publicKey": [
{
"type": "RsaVerificationKey2018",
"id": "did:sidetree:EiD3j1DfUzypfK8ZP#master",
"publicKeyPem": "-----BEGIN PUBLIC KEY----- -----END PUBLIC KEY-----\n"
}
]
}

Update the trace API documentation

The /trace API exposed by the blockchain service layer has a different request/response semantics. Creating an issue to track this discrepancy. I'll go ahead and submit a PR.

Add REST API versioning in path as described in protocol.

Logger support

Requirements:

1. Each log will use the following schema:

{
    "correlationId": "Given as an optional request header.",
    "requestId": "Auto-generated globally unique ID",
    "highResolutionTime": "Timestamp",
    "level": "Log level",
    "message": "Log message",
    "callStack": "Will only log callstack when log level is error.",
    "additionalInfo": "Additional info to be logged.",
}

2. When logging, only level, message, and optional additionalInfo is needed, everything else is automatically populated for the developer. e.g:

log.error('Something bad happened', 'Additional info');

HINT: might need to use ths npm pacakge if koa doesn't have similar support for it.

3. Ability to pipe logs to different/multiple locations (files, console, etc) depending on environment/configuration.

4. Ability to format the output in format other than JSON strings.

Transaction Pagination on the Blockchain Sidetree

This is a question around pagination of the GET /v1.0/transactions endpoint.

https://github.com/decentralized-identity/sidetree-core/blob/master/docs/implementation.md#request-example

The sidetree protocol includes a afterBlockNumber field for paginating through GET transactions requests. This is specific to Bitcoin as the Ethereum Sidetree will be using other methods for specifying pagination location.

Could we rename that field to a more generic name? Fwiw, on Ethereum the anchor file hashes will all be saved in one smart contract and stored within an array. So the pagination value would point to an index within that array.

Revisit the relationship between ResolvedTransaction/OperationInfo/WriteOperation.

Consolidation opportunities:

ResolvedTransaction maybe able to replace OperationInfo.Timestamp.

OperationInfo is very similar to WriteOperation minus the operation buffer. There maybe opportunity there for refactoring/simplification.

Design strategy for discovering/handling/retrying unresolvable transactions

Need to design strategy for discovering/handling/retrying unresolvable transactions.

OperationProcessor - Document design in implementation.md and protocol steps in protocol.md

Use multihash throughout code as described by Sidetree protocol

Add anchor/batch file compression.

Investigate if WriteOperation.toDidDocument and WriteOperation.didDocument can be consolidated

Create TypeScript definitions for js-multihashes NPM package and use it.

Create a types/interfaces than `any` for public/privateKeyJwk objects.

Change DID Document Service Endpoint descriptors to match Identity Hub spec

Here is what the Service Endpoint descriptors should look like, per the Identity Hub spec: https://github.com/decentralized-identity/identity-hub/blob/master/explainer.md#did-document-service-endpoint-descriptors

CAS read interface needs to support undefined return type.

Change definition from:

read (address: string): Promise<Buffer>;

to:

read (address: string): Promise<Buffer | undefined>;

Ensure that the Sidetree node's IPFS component is setup to only replicate/store Sidetree data

Most IPFS installs allow the IPFS instance to replicate and store random data from around the network. This is something we need to block in our use. The Sidetree node should only be coded to replicate and store Sidetree-specific data.

Remove sig checks for CreateOperation and update protocol doc accordingly.

Defer operation processing until resolution time.

anchorFileJson contains unencoded buffer

{"batchFileHash":"EiBz-l5-pd0K9ZipVI_GMBaXmY1r9BCVNM5z18Sl7Jyx7Q","merkleRoot":{"type":"Buffer","data":[55,11,216,118,62,67,205,215,207,35,44,11,221,166,254,81,225,55,96,166,78,121,190,167,195,37,87,154,145,186,102,62]}}

merkleRoot should be string encoded as hex or base64.

Add signature verification code

Signature verification still yet to be added in code.

Add Secp256k1 signature verification support.

OperationProcessor - Change lookup() implementation to be iterative instead of recursive

To avoid stack overflow.

Consider JSON Stringify Deterministic

JSON.stringify() is not safe to use with CAS when the object in question is deep, it is possible to get a different hash for an object that is deep equivalent.

https://www.npmjs.com/package/json-stringify-deterministic

Implement a dynamic, high-accuracy means for estimating next-block txn inclusion fee

Can we store any DID method in any sidetree?

The way the spec is currently written it seems like each DID method will be persisted in a specific sidetree.

However, since all DIDs follow the same spec, shouldn't any sidetree be able to store any DID. eg, a bitcoin DID like did:btc:hashxyz could be included in a batch sent to the Ethereum sidetree, which would in turn know how to process that DID and its deltas, since it'd be the same way of parsing it for did:eth:hashxyx.

Create multiple batches when total operations count exceeds batch size limit.

Improve Rooter to anchor all pending operations in multiple batches in the case when the number of operation is greater than maximum number of allowed operations per batch.

DID Cache - Implement lastProcessedTransaction

CAS interface should return undefined for read if file is not found.

Change the current Cas interface from

read (address: string): Promise;

to:

read (address: string): Promise<Buffer | undefined>;

Implement recover operations

Recovery mechanism will dictate how signature verification will be performed for delete and recovery operations.

Document concepts of late-publishing & rollback

Need to design, document and test all possible late-publishing scenarios against the Operations Processor.

Implement Proof of Fee (batch size-relative fee over-payment scheme)

Implement proof-of-work according to Sidetree protocol.

Content of anchorFile differs from protocol spec

It looks like the sidetree-core implementation constructs the anchor file with the batchFileAddress whereas the sidetree protocol mentions it being the batchFileHash:

https://github.com/decentralized-identity/sidetree-core/blob/bc48eecb9d15637e5b14f7c938e77e3bdb5fb7ba/src/Rooter.ts#L79

It seems like the CAS address would be more helpful since we can also use that to verify its existence

Invesgitate using baseUrl in tsconfig and remove deep relative path imports in tests

Make sure Jasmine tests are able to resolve the import paths.

Generate requests in tests instead of hard-coded request files.

A good place to start is to generate requests used in RequestHandler tests.

Update the Rooter Pseudo Code in Implementation Doc

Update the pseudo code of Rooter, in the implementation.md file according to the implemented version