spruceid / kepler

Decentralized storage based on permissioned data overlays called orbits.

Kepler

Kepler is self-sovereign storage. It is architected as a decentralized storage system that uses DIDs and Authorization Capabilities to define Orbits, where your data lives and who has access. Any DID controller (e.g. people, applications, DAOs) can administer their own Kepler Orbit.

Quickstart

To run Kepler locally you will need the latest version of rust.

You will need to create a directory for Kepler to store data in:

mkdir kepler

Within this directory, create two more directories blocks and indexes:

mkdir kepler/blocks
mkdir kepler/indexes

You will then need to set the environment variables to point to those directories:

export KEPLER_STORAGE_BLOCKS_PATH="kepler/blocks"
export KEPLER_STORAGE_INDEXES_PATH="kepler/indexes"

Finally you can run Kepler using cargo:

cargo build
cargo run

Configuration

Kepler instances are configured by the kepler.toml configuration file, or via environment variables. You can either modify them in this file, or specify them through environment variable using the prefix KEPLER_.

The following common options are available:

Option	env var	description
log_level	KEPLER_LOG_LEVEL	Set the level of logging output, options are "normal", "debug"
address	KEPLER_ADDRESS	Set the listening address of the kepler instance
port	KEPLER_PORT	Set the listening TCP port for the kepler instance
storage.blocks.type	KEPLER_STORAGE_BLOCKS_TYPE	Set the mode of block storage, options are "Local" and "S3"
storage.indexes.type	KEPLER_STORAGE_INDEXES_TYPE	Set the type of the index store, options are "Local" and "DynamoDB"
orbits.allowlist	KEPLER_ORBITS_ALLOWLIST	Set the URL of an allowlist service for gating the creation of Orbit Peers

Storage Config

Storage can be configured for both Blocks and Indexes, depending on the type for each.

Local Storage

When storage.blocks.type and storage.indexes.type are Local, the local filesystem will be used for application storage. The following config options will become available:

Option	env var	description
storage.blocks.path	KEPLER_STORAGE_BLOCKS_PATH	Set the path of the block storage
storage.indexes.path	KEPLER_STORAGE_INDEXES_PATH	Set the path of the index store

AWS Storage

When storage.blocks.type is S3 and storage.indexes.type is DynamoDB, the instance will use the S3 and DynamoDB AWS services for application storage. The following config options will become available:

Option	env var	description
storage.blocks.type	KEPLER_STORAGE_BLOCKS_TYPE	Set the mode of block storage, options are "Local" and "S3"
storage.blocks.bucket	KEPLER_STORAGE_BLOCKS_BUCKET	Set the name of the S3 bucket
storage.blocks.endpoint	KEPLER_STORAGE_BLOCKS_ENDPOINT	Set the URL of the S3 store
storage.blocks.dynamodb_table	KEPLER_STORAGE_BLOCKS_DYNAMODB_TABLE	Set the name of the dynamodb table
storage.blocks.dynamodb_endpoint	KEPLER_STORAGE_BLOCKS_DYNAMODB_ENDPOINT	Set the URL of the dynamodb service
storage.indexes.path	KEPLER_STORAGE_INDEXES_PATH	Set the path of the index store

Additionally, the following environment variables must be present: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_DEFAULT_REGION.

Running

Kepler instances can be started via command line, e.g.:

KEPLER_PORT=8001 kepler

If the Kepler instance is not able to find or establish a connection to the configured storage, the instance will terminate.

Usage

Kepler is most easily used via the Kepler SDK. See the example DApps and tutorials for detailed information.

kepler's Issues

Implement DAG indexing service storage using S3/dynamoDB

PR #96 implements the rust-ipfs block and pin stores using a local-or-AWS enum to support a multi-instance deployment. The KV subsystem uses sled to store it's graph index stuff, which should also have an AWS-backed implementation to fully realise this requirement

IPFS API design

The currently accessible IPFS API is very limited and has missing features:

• only get, put and delete for single blocks at a time
• no built-in pinset replication/synchronisation
• the codec param is unused, all blocks are encoded as raw binary, disallowing pinning of full DAGs.

This API should at least:

• allow invoker-decidable block encodings based on the codec param (or similar)
• replicate the pin set based on puts and deletes

and preferably also enable the insertion of downstream DAG blocks when inserting/pinning a root block.

Use env var for host ID keys

Allow a host to set their host instance libp2p ID keys using an environment variable

Implement Kepler session keys

As a User, I want to be able to issue a capability with my Web3 wallet to delegate privileges to a temporary session key. The capability should expire within a short configurable window, support secure erasure if used within a browser, and also allow for "break glass" revocation within an orbit manifest. This should allow for enhanced user interfaces that still support standards such as ZCAPs along with proper consent display to the user, instead of relying upon diverse signing implementations across different blockchains.

Authorization for non-controllers

Authorization checks enforce that the requester is a member of the orbit controllers. Non-controllers with proper authorization should be able to access orbits as well. The existing TZ signed string authorization does not have a delegation model, so a way to handle or signal authorization of non-controllers in this case needs to be determined.

Dynamic peer creation should be a delegation, not an invocation.

A peer can be permitted to host an orbit explicitly by the orbit manifest, or the controller can delegate the #peer capability to a Kepler instance. This second "dynamic" peer creation currently works through the invocation of the peer capability by the controller (or delegate), but it should be a delegation. This way the peer can connect to other peers in the orbit and prove that it has the authority to host the orbit by sharing the delegation.

Discuss content identifiers

• Should we use content addressable identifiers (i.e., hash of the content)? What are the privacy implications here if other people may have the same file? Should we consider peppering it prior to hashing?
• What about S3 bucket-style key-value naming, so it's possible to address a content by that looks like a path, such as pictures/vacation/island3821.png?
• What other approaches exist? Should we support one or many?

@chunningham

Discuss types of Orbit Relationships

What kind of orbit relationships can exist? Some examples:

• Copy-on-Write orbits that are based on another orbit
• Composite orbits that are composed from one or more other orbits, possibly with a Copy-on-Write orbit serving as a sink for changes.
• Partial orbits which contain a subset of other orbits
• Projected orbits which are mapped from other orbits to change content identifiers or other aspects
• What else?

Questions:

• How do permissions work across orbit relationships?

Discuss Orbid Identifiers

What's in an Orbit ID? Is it just a uuidv4? Content-related hash, such as a Patricia Merkle trie root? Something else?

For now, we've defined what it could look like for blockchain accounts that use public key hashes, such as those represented by did-pkh, where HASH is some hash function to be determined:

orbitPepper = HASH("domain.org", ":", $pkhAddress, ":", $keplerOrbitSecretKey)
orbitId = HASH("domain.org", ":", $pkhAddress, ":", $orbitPepper)

This would allow us to in the future set default permissions by defining a verification method as part of this scheme which authenticates the keyholder as the orbit administrator during the authorization policy log inception event.

What other types of Orbit ID types should we support? Do we have a prefix for designating the different orbit types? Perhaps something like:

kepler://<orbit-id-type>:<orbit-id>/<content-id>

cc @chunningham

Upgrade to didkit/ssi v0.4.

Upgrading to didkit and ssi v0.4 leads to a compilation error for libp2p-noise:

error[E0282]: type annotations needed
   --> /home/jward/.cargo/registry/src/github.com-1ecc6299db9ec823/libp2p-noise-0.32.0/src/protocol/x25519.rs:221:45
    |
221 |         curve25519_sk.copy_from_slice(&hash.as_ref()[..32]);
    |                                        -----^^^^^^--
    |                                        |    |
    |                                        |    cannot infer type for type parameter `T` declared on the trait `AsRef`
    |                                        this method call resolves to `&T`
    |
    = note: type must be known at this point

For more information about this error, try `rustc --explain E0282`.
error: could not compile `libp2p-noise` due to previous error
warning: build failed, waiting for other jobs to finish...
error: build failed

Discuss Orbit Discovery and Management

Metadata

What data define an orbit? Here are some ideas:

• Patricia Merkle trie root hash of most current content updated by administrator
• Patricia Merkle trie root hash of most current authorization policy log updated by administrator
• List of IPv4/IPv6/etc. hosts that are in the orbit
• What else?

Discovery

How do you find an orbit? Here are some ideas:

Public Orbits - meant to be discovered and and accessed by anyone

• Smart contracts on public blockchains
• DHTs that are publicly accessible a la bittorrent

Private Orbits - meant to be selectively disclosed

• Smart contracts on private blockchains
• Smart contracts on public blockchains with encrypted contents
• Permissioned DHTs or distributed databases, possibility implemented using an Orbit
• FTP server

Management

How are orbits administered? TBD

cc @chunningham

Decompose `ipfs_embed` into separate services

Usages of Ipfs don't always need the full network behaviour and functionalities provided by ipfs_embed. Exposing the network and storage components of Ipfs separately would allow us more flexibility in resource consumption and maybe performance.

`get` invocation for a non-existent `kv` file returns an empty Blob instead of 404ing

Test to show this failing added here:
spruceid/kepler-sdk#43

Refactor `Action` handling to be less restrictive w.r.t. SIWE tokens

Currently a SIWE invocation will be interpreted as having only a single Action; the first in it's list of resources. This restricts SIWE clients and is implemented in quite a hacky way. This can take two forms:

• either refactor Action to be broader
• explicitly limit SIWE invocations to a single resource entry
  This refactor can also improve the extraction of authz tokens from the HTTP headers

CI: Cargo build getting stuck

PRs to the repo aren't building, cargo build is hanging. There are also clippy warns that don't appear when running it locally, for example:

error: field is never read: `task`
   --> src/orbit.rs:150:5
    |
150 |     task: Arc<AbortOnDrop<()>>,
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: `-D dead-code` implied by `-D warnings`

Rate limit IP addresses ops/sec or bw/sec

This provides important configurable rate limiting to fight spam.

Return 404 instead of 401 in release mode

401 messages give away quite a bit of information. In production, this should be sanitized or converted to a 404 message.

Refactor HTTP API around ZCAPs

Instead of having having individual endpoints for each action and subsystems, which are redundant with the action/target described in the ZCAP invocation and error-prone, we could have a single invoke endpoint (or maybe a few if we cannot handle all forms of upload with just POST) -- not unlike a GraphQL API.

Lock not acquirable for ipfs store.

After an orbit is evicted from the cache, the orbit cannot be reloaded as a lock cannot be acquired on one of the sqlite stores:

IO error: could not acquire lock on "/tmp/kepler/.../....ks3db/db": Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }

Implement Kepler as serverless infrastructure

As a user, I want to deploy my own instance of Kepler conveniently across the growing number of "serverless" application hosting platforms so that I can enjoy Internet scale infra that I control at low cost with extremely low (or zero) maintenance overhead; "set it and forget it."

For example, I am interested in deploying to:

• AWS Lambda + Amazon S3 or (equivalent on Azure/GCP/etc.)
• Cloudflare Workers + KV Store (bundled/paid okay to get 50 ms of execution time instead of 5 ms)
• DigitalOcean App Platform + Spaces
• Netlify Functions + FaunaDB (or equivalent store)
• Vercel + FaunaDB (or equivalent store)

Furthermore, I should be able to access the instance with my favorite Web3 wallet. Content replication is not required for this phase.

Hotload orbits

Add Prometheus

• Expose metrics on a different port from Rocket's (probably means using a little hyper server on the side).
• Desirable metrics
  • Enable process feature of prometheus
  • IPFS information

Implement comprehensive logging and tracing

• Ensure all logs are using tracing.
• Ensure a span is created per request and reuse the trace id from the load balancer.

Define Kepler Expression Language

kepex should be able to fully express Actions and also be evaluated as part of the cryptoscript runtime for predicates.