Sampling and data extraction about celestia-core

[musalbas]

I believe both of the above features mentioned here are in fact the same feature: the ability to query specific chunks from blocks, from the P2P network (e.g. using a DHT).

The actual second feature we need is for nodes to be able to query messages from blocks based on their namespace, rather than their index in the block. The former is needed for clients to be able to download the messages for their application, and the latter is needed for clients to be able to do data availability checks and recover full blocks.

[musalbas]

It can be part of the same service/DHT, but my point is that the "two features" in your original post are actually the same feature, and the actual second feature we need is querying by namespace. The only difference is that you need to query chunks by namespace, not by index.

[liamsi]

The actual second feature we need is for nodes to be able to query messages from blocks based on their namespace

I think, a simple way to implement this is would be a RPC service for the Namespaced Merkle Tree (the already offers this feature) backed by a DB. The p2p layer could be used to find nodes that offer this capability. On the p2p-layer this would only require some form of "service discovery". Or are there any reasons this should work similarly to the other feature (querying chunks from block directly from a p2p network)?

[liamsi]

With #85 (comment), the sampling and data extraction can be achieved via querying the IPFS DAG:

The coordinates only need to be translated into a path that the ipld plugin can resolve: e.g. the coordinate (0,0) could correspond to CID/0/...0/0. A dag get request would return back the underlying share M_i. This can be used for both the light clients (or validator nodes that choose not to download message to validate the erasure coding) to validate the block (as in validate its data availability) as well as nodes that want to recover the whole block (which would need to traverse the whole tree).

We can later opimize this via graphsync instead of bitswap which aims to reduce the number of roundtrips when resolving a path.

[liamsi]

I think this issue is rather confusing and the ADR introduced in #170 as well as the issue #221 better capture the work that needs happen.