wnfs-wg / roadmap Goto Github PK

Summary

A short summary of this roadmap item (tip: fill this in last).

Problem / Need / Opportunity

When using multiple devices or when collaborating on a file system with multiple users, concurrent writes will happen. These writes may not actually modify the same file or directory at once, but nonetheless both change the root CID to two different values.
There is an opportunity to provide a fairly simple conflict reconciliation algorithm that works in the public and private file system for reconciling changes.

Success Scenario

WNFS is a form of a CRDT. Concurrent writes are merged automatically all peers eventually converge to the same state. Non-conflicting writes are intention-preserving.

Stretch goal: There is an API in developer-facing WNFS libraries for resolving conflicting concurrent writes in application-specific ways.

Children

• wnfs-wg/rs-wnfs#93
• wnfs-wg/rs-wnfs#7
• wnfs-wg/rs-wnfs#25
• wnfs-wg/rs-wnfs#74

ETA: 2023-09-30

Summary

A feature for attaching additional information to changes between revisions.

Problem / Need / Opportunity

• End users may care about being able to inspect the writes that happened, so they can audit them.
• Applications can use tombstones to detect when a file was key-rotated so they can raise a message to the user about being revoked read access.
• The WNFS conflict resolution algorithm may make use of move markers to improve expected outcome of concurrent changes when a directory was moved and modified at the same time.

Success Scenario

Each write in WNFS is annotated with additional information about it. The merge algorithm is improved in some cases. APIs to make use of the additional information exist and are used.

ETA: 2024-06-30

Summary

Verifying changes from peers with non-root write access.

Problem / Need / Opportunity

When non-root write access is delegated, verifying writes to the private file system is non-trivial. There should be an implementation that connects all the necessary components to realize this.

Success Scenario

There exists an implementation for developers to use, allowing them to delegate granular write-access and check changes and a write access level for rights escalations.

ETA: 2023-12-31

Summary

Widening the possible use cases to include bigger data sets.

Problem / Need / Opportunity

Due to practical IPFS block size restrictions, WNFS can only safely encode up to 1000-2000 directory entries in public and private directories.

Success Scenario

It's possible to create huge directories! We enable use cases like on-boarding bigger data sets comparable to all of wikipedia, the linux kernel or a nix cache.

Children

ETA: 2023-09-30

Notes

wnfs-wg/spec#8

Summary

Incorporate possible improvements into how the private history works so traversing history and conflict reconciliation can become better.

Problem / Need / Opportunity

To improve the file system conflict reconciliation, we need a very detailed view of file system history. There should not be any "interleaving anormalies" in the history when merging private file systems.

Success Scenario

Merging the private file system 'feels' pretty much like conflict resolution in the public file system.
It's possible to walk through the history of a single private file across revisions and maintain the same history 'track'.

ETA: 2023-03-31

Summary

WNFS contains all information to validate all writes contained in it relative to the owner's root DID.

Problem / Need / Opportunity

There is an opportunity to use peers that replicate WNFS for persistence and availability ("super-peers") also for broadcasting changes to WNFS. However, this requires them to prove to peers that the WNFS they store only contains valid writes. For this, they need to know about UCANs and WNFS-as-a-protocol.
Although these peers should already do this type of checking when accepting changes, we can make it easier for them by making WNFS self-authorizing. Thus, super-peers would only verify the authority of incoming changes, and after checking them - they'd just re-broadcast these changes without additional overhead.

Success Scenario

Peers can determine the validity of all writes in a WNFS given only the file system itself and the owner's root DID. Super-peers just check new roots to broadcast.

Children

Notes

wnfs-wg/spec#7

Summary

Storing more than a single byte array per file.

Problem / Need / Opportunity

File systems have evolved since their inception. It may make sense to associate bigger metadata like images with files, even if they point to some other data (a phone number, another image, a video, etc.).

Success Scenario

Files are associated with metadata about them, enriching the experience for end users. Operating systems like Capyloon use these interfaces.

Notes

wnfs-wg/spec#49
wnfs-wg/spec#18

ETA: 2023-09-30

Summary

Enable collaboration on WNFS files and directories across file systems/users.

Problem / Need / Opportunity

For collaboration purposes users want to share read and/or write access to parts of their file system asynchronously, but with a known recipient public key.

Success Scenario

We provide a specification and implementation for creating share payloads and receiving them such that end users can collaborate on WNFS files or directories.

ETA: 2023-03-31

Notes

• wnfs-wg/spec#43
• wnfs-wg/spec#51

Summary

Minimize future migrations by doing data format changes that we already know we'll want to do.

Problem / Need / Opportunity

Changes to WNFS's core data format mean migrations for users and support for both versions from implementations.
We should strive to crystallize the data format for WNFS as soon as possible, so only additional, optional features will be added in the future which don't require migrations.

Success Scenario

We have a stable data format that solves developer needs for a range of different use cases.

Children

• wnfs-wg/rs-wnfs#303
• wnfs-wg/spec#71
• wnfs-wg/spec#70
• wnfs-wg/spec#69
• wnfs-wg/spec#68
• wnfs-wg/spec#66

ETA: 2023-09-30

Summary

Possibly improve data transfer by switching to a different encoding.

Problem / Need

It may be easier for iroh (and in the future potentially more IPFS implementations) to support efficient data transfer using dag-pb or other encodings than dag-cbor.

Success Scenario

WNFS encodes into dag-pb or another encoding. This integrates well with iroh. In benchmarks, data transfer is proven to be faster than before.

Children

ETA: 2024-06-30

Summary

Selectively share access to files and directories to collaborate with other users.

Problem / Need / Opportunity

Users should be able to invite other people to collaborate on their files. It is possible to provide mechanisms to selectively share read access to files and directories, as well as to include links to 'foreign' file systems in your own for managing such access. This should be possible asynchronously, given that a user 'knows' the user they want to share with in advance.

Success Scenario

Users can link files that were shared with them across file system boundaries. There exists a mechanism to share access asynchronously.

Notes

wnfs-wg/spec#43
wnfs-wg/spec#24

ETA: 2023-09-30

Summary

Switch out the cryptographic accumulator used for private file system paths for bigger security margins.

Problem / Need / Opportunity

At the moment the write access & path obfuscation mechanisms in the private file system are built on bloom filters. These have a false-positive rate of 1 in a billion and a depth limit of 47 path elements.
Also, a path segment membership witness for a single file system block leaks information about which other private file system blocks also contain this path segment.
It's possible to improve this when switching to another accumulator scheme.

Success Scenario

There is no depth limit for paths. The security margin for brute-forcing is 2^112 or bigger. There is no correlation possible between labels of private file system blocks, membership witnesses leak no information about unrelated private file system blocks.

Notes

A possible solution could be RSA accumulators.

ETA: 2023-09-30

Summary

Implementation of compression built-in to WNFS.

Problem / Need / Opportunity

Compressing private data after encryption is ineffective, so this puts the burden of dealing with compression on users instead of services that store user WNFS data.
There is an opportunity to support compression on the protocol level, similar to ZFS compression.

Success Scenario

Implementations of WNFS can decide to transparently compress user data without them needing to know, leading to faster writes, faster reads and less memory and bandwidth usage.

Notes

To be considered is the security impact of compression. E.g. if an attacker effectively has an encryption oracle for the first half of a file and the data is compressed before it's encrypted, it may be possible for an attacker to "guess" words from the second half of the file and thus populate a compression dictionary and observe the impact that has on the ciphertext length.

Banyan.Computer is interested in this, please feel free to reach out if you are too.

ETA: 2024-06-30

Summary

Enable testing interfaces, integration points & performance with rs-wnfs and other systems.

Problem / Need

WNFS needs to be accessible on various platforms. Implementing it in rust allows everyone to integrate it in the browser and potentially in ADLs/IPVM with WASM and on servers or IPFS implementations on the CLI via native rust or a C library wrapper.
We need a first prototype to be implemented so we can experiment with this system design. Using this prototype we can take a look at how a single implementation may or may not solve multiple use cases at once and test & compare its performance characteristics with older implementations of WNFS.

Success Scenario

We can start

• integrating rs-wnfs into something like the webnative SDKs so web applications can use WNFS as the data format for writing encrypted data to IPFS
• integrating rs-wnfs into Iroh as a CLI, to experiment with rs-wnfs as a rust library for integration
• testing performance of various operations to identify bottlenecks early

ETA: 2022-12-31