rollkit / celestia-da

Celestia Implementation of Modular Data Availability Interface

celestia-da

⚠️ celestia-da is deprecated. See the migration guide for details.

celestia-da is an implementation of the Generic DA interface for modular blockchains. It extends celestia-node and runs a gRPC service, which can be used by rollup clients to read and write blob data to a specific namespace on celestia.

Minimum requirements

Requirement	Notes
Go version	1.22 or higher

Installation

git clone https://github.com/rollkit/celestia-da.git
cd celestia-da
make build
sudo make install

Usage

celestia-da is a wrapper around celestia-node, so see celestia node documentation for details on configuring and running celestia-node.

celestia-da connects to celestia-node using JSON-RPC using the node rpc endpoint. See node rpc docs for details.

celestia-da exposes a gRPC service that can be used with any gRPC client to submit and retrieve blobs from a specific namespace on the celestia network.

Note that celestia-da version may differ from the bundled celestia-node version. Use the celestia-da version command to print the build information including the bundled celestia-node version.

To start a celestia-da instance, use the preferred node type with start command along with the gRPC specific flags as documented below.

Example

Run celestia-da light mainnet node with a default DA interface server accepting blobs on a randomly chosen namespace:

    celestia-da light start
        --core.ip <public ip>
        --da.grpc.namespace $(openssl rand -hex 10)

Note that the celestia-node RPC auth token is auto generated using the default celestia-node store. If passed, the da.grpc.token flag will override the default auth token.

Flags

Flag	Usage	Default
da.grpc.namespace	celestia namespace to use (hex encoded)	none; required
da.grpc.address	celestia-node RPC endpoint address	http://127.0.0.1:26658
da.grpc.listen	gRPC service listen address	127.0.0.1:0
da.grpc.network	gRPC service listen network type	tcp
da.grpc.token	celestia-node RPC auth token	--node.store auto generated
da.grpc.gasprice	gas price for estimating fee (utia/gas)	-1 celestia-node default

See celestia-da light/full/bridge start --help for details.

Tools

1. Install golangci-lint
2. Install markdownlint
3. Install hadolint
4. Install yamllint

Helpful commands

# Print celestia-da version build information, including bundled celestia-node version
celestia-da version

# Run unit tests
make test-unit

# Run all tests including integration tests
make test

# Run linters (requires golangci-lint, markdownlint, hadolint, and yamllint)
make lint

Contributing

We welcome your contributions! Everyone is welcome to contribute, whether it's in the form of code, documentation, bug reports, feature requests, or anything else.

If you're looking for issues to work on, try looking at the good first issue list. Issues with this tag are suitable for a new external contributor and is a great way to find something you can help with!

Please join our Community Discord to ask questions, discuss your ideas, and connect with other contributors.

Code of Conduct

See our Code of Conduct here.

celestia-da's Issues

[Feature Request]: NewCelestiaDA use consistent argument position for context

Implementation ideas

NewCelestiaDA should accept context as first param.

[Feature Request]: package docker image

Implementation ideas

It would be good to package celestia-da as a docker image, very similar to celestia-node, except that the binary wraps node and also runs the da grpc service. This can be used by local-celestia-devnet to provide a devnet which can directly be used by grpc clients to submit blobs to.

Unimplemented rpc error for MaxBlobSize

something wrong rpc error: code = Unimplemented desc = unknown method MaxBlobSize for service da.DAService
10:12AM ERR DA layer submission failed error="unable to get DA max blob size" attempt=11 module=BlockManager

this is happening because celestia-da (main branch and last release) used go-da 57bc36006772 which does not have MaxBlobSize.

we should update this to use 93600f2 which has the api MaxBlobSize.

v0.12.9 Does not have a install make command in the makefile.

Version of Git SHA

using tag - v0.12.9

System OS

Linux

Installed tools

No response

Others

No response

Steps to reproduce it

after running make build succcessfully
make install fails with

make: go: No such file or directory
make: *** No rule to make target 'install'.  Stop.

### Expected result

celestia-da should be installed and made executable 

### Actual result

`make install` fails with 

make: go: No such file or directory
make: *** No rule to make target 'install'. Stop.

Relevant log output

No response

Notes

No response

[Feature Request]: properly handle `ErrBlobNotFound`

Implementation ideas

After celestiaorg/celestia-node@a7a17e4 is released in celestia-node, we can improve error handling using errors.Is instead of comparing error strings in GetIDs.

[Feature Request]: allow configuring global gas price

Implementation ideas

Currently, clients need to pass gasPrice with every Submit request. Instead, we can allow a global gasPrice to be configured from cli options and allow clients to override it per request.

This makes rollout easier because it's easier to configure celestia-da than to update each integration to add the gas price flag.

[Feature Request]: bundle celestia-node as binary

Implementation ideas

While the Data Availability Service implementation can be run as a standalone service and connect to a remote celestia-node over RPC, it is convenient to combine both into the same binary so that whenever server is started, it also starts celestia-node and connects to the same instance.

As the manager for the celestia-node server, the new binary can access the node auth token directly from the store and no longer require the auth token flag. Only the namespace flag will be required.

Also, as we'll be combining the binaries, it will be better to have a namespace for the flags so that they don't clash with node flags. To make things consistent, we can rename the flags like so:

Old Key	New Key	Required	Default Value
rpc.address	da.grpc.address	false	http://127.0.0.1:26658
rpc.token	da.grpc.token	false	derived/generated from node store
namespace	da.grpc.namespace	required	n/a
listen.address	da.grpc.listen	false	127.0.0.1:0
listen.network	da.grpc.network	false	tcp

It would be better to wrap the celestia-node commands and implement this as a PreRun command so that no internals from node need to be exposed.

[Feature Request]: Add Tests to CelestiaDA Implemenation

Implementation ideas

self-explanatory

[Feature Request]: new release cycle and versioning

Implementation ideas

celestia-da versions were initially based on celestia-node and tightly coupled to it's releases.

As we have more breaking changes, this will be difficult to maintain. As such, starting from the next minor bump i.e. v0.13.x we should maintain celestia-da version independently from celestia-node version.

As we'll have different versions for celestia-da and celestia-node, there should be a version command that should output both versions.

[Feature Request]: allow configurable gas fee, rate options

Implementation ideas

We should be able to pass the gas fee, rate options from the client. This might need changes to the interface.

[Feature Request]: Handle big blobs

Implementation ideas

I like big blobs and I cannot lie

As pointed out in the code comments and review comments, there is high probability that bigger blobs can break the handling of Proofs.
The tests for this will be implemented in go-da (see: rollkit/go-da#17), but changes in the code will most probably be necessary.

[Feature Request]: package release binaries

Implementation ideas

We should create a workflow to build and package binaries with every release.

[Feature Request]: implement celestia-node's go-version handling in CI

Implementation ideas

REF: celestiaorg/celestia-node#2806

This use of a CI step that sets the go version as an output allows us to have a single go version defined for the entire CI.

[Feature Request][design]: DA error handling

Implementation ideas

This is a design discussion to coordinate error handling between celestia core/app and rollup clients like rollkit.

Some issues with the current model

• core/app accepts transactions which may not be mined see: celestiaorg/celestia-app#3081
• tx which are not mined return timed out waiting for tx to be included in block
  • should this be handled by core/node? see: rollkit/rollkit#1522
  • should core/node autoretry with higher fee
• if a tx is accepted by the mempool (e.g account being reused), rollup cannot do anything until it gets dropped
  • can core allow a faster ttl? or allow replace-by-fee?
• testing DA submission issues is dependent on mempool state (e.g. can be congested) which makes it hard to reproduce issues like rollkit/rollkit#1520

Overall this makes handling DA submission challenging because of the lack of congest control flow and lack of dynamic fee market.

[Feature Request]: Expose gRPC server

Implementation ideas

Create a binary so the DA interface can be served with gRPC.

[Feature Request]: upgrade to celestia-node v0.12.4

Implementation ideas

https://github.com/celestiaorg/celestia-node/releases/tag/v0.12.4

[Feature Request]: Remove Celestia OpenRPC dependecy.

Implementation ideas

Instead of importing the Celestia openRPC repository, we should import celestia-node directly. This was the main motivation behind rollkit/rollkit#1122

proof from method GetProof cannot be validated(unable to unmarshal proof) by method Validate

Version of Git SHA

v0.13.0

System OS

Ubuntu 22.04.3 LTS

Installed tools

docker image of celestia-node(v0.13.0)

Others

No response

Steps to reproduce it

1. launch a light celestia node: https://github.com/AnomalyFi/nodekit-relay/blob/with-streaming/scripts/start_celestia_light.sh
2. clone this repository: https://github.com/bianyuanop/celestiatest
3. type go run main.go in the folder of repository mentioned in step 2

Expected result

should validate without any error

Actual result

Cannot get validated because of cannot Unmarshal proof to pb.Proof, see image

Relevant log output

No response

Notes

No response

Investigate OOM leak

Container using version celestia-da:v0.12.1-rc0

Flags used: --gateway --metrics --metrics.tls=false

Getting OOM with 2GB memory

running otel-collector in same pod to export otel-metrics for prometheus

we should make sure this version is using the celestia node version that has the fix: celestiaorg/celestia-node#2905 (comment), and if so, then debug memory leaks in celestia-da