Comments (3)
it's probably not a bad idea to think about what should be the Tx that lazyledger-core includes in the mem-pool and later the block? Currently, from the PoV of tendermint they are just opaque blobs (Tx) with no inherent structure, besides what protobuf imposes (they are wrapped in a proto message - only the raw Tx end up in the block though): https://github.com/lazyledger/lazyledger-core/blob/7b84a4c74317453c6fd4192b1b82619328b1b97c/proto/tendermint/mempool/types.proto#L6-L14
So far we roughly discussed 4 ideas:
-
Do not modify current understanding of Tx: Tx remain opaque blobs for ll-core -> that means only the app knowns how to extract the namespace ID (and hence computation of the data root using the nmt will need to happen in the app)
-
Txs & Messages: Txs from the point of view tendermint are pairs of Tx and Messages
-
Txs & (namespaced) Shares: (similar to above) pair of Tx and optionally (namespaced) shares; only for
PayForMessage
transactions there will be shares present -
Only Messages: all Tx are messages -> change all Tx to contain a namespace ID (or implicitly assume the reserved ID for Tx if it isn't present)
This might not be the whole set of solutions. E.g., a "combinations" of (1.) and any other might also work: add a variant to the oneof
which adds to the existing Tx the particular type.
Pros & Cons
(1.) Pros:
- necessary changes only entail (re)moving logic from tendermint to the app
(1.) Cons:
- it seems wrong to move computing the root for the nmt to the app (as the data for data is in the block and the ordering is what is reached consensus on)
- the existing Tx types that we want to use from the Cosmos-SDK do not need to be modified at all (only the new type(s) needs to be implemented of course).
(2.) Pros:
- clean distinction between messages and Tx
- as they are submitted in together, e.g. CheckTx becomes very easy: validate the tx and see if the commitment in PayForMessage matches the Message
(2.) Cons:
- only a few Tx types will come with messages (all but one Tx type do not deal with LL-core logic but with staking, delegations etc) but the Tx format will be the same for all; although that seems odd it might not be a big issue as the message field would be skipped if it another Tx type
(3.) Pros:
- Tx format reflects what is in the current spec and we can simply put the Tx into the block (we can just extract it from the pair)
(3.) Cons:
- same as (2.)
- but additionally: before putting the message in the block the message has to be merged from the provided shares
- clients will have to compute the shares before submitting the Tx (maybe they have to this anyways? not sure cc @adlerjohn)
(4.) Pros:
- a very generic and simple approach
- introduces a minimal addition to the current Tx forma instead of a proto
message Message { oneof sum { Tx tx = 1; } }
, we'd add in a namespace field to each Tx, or, we could simply add a namespaced variant to theoneof
(4.) Cons:
- unclear how to do
CheckTx
for the particularPayForMessage
: e.g. if the corresponding message is passed in later CheckTx can't decide on the Tx (we can't simply accept Tx that pay for anything, right? for messages it might be easier to check if we've seen a Tx that payed for them)
from celestia-core.
To roughly visualize the above, I've summarized these 4 options in this pdf.
To summarize our discussions and the current conclusions: it seems like the best approach to leave Tx to be just blobs from the PoV of tendermint. At least to the point where they are checked against the app via CechkTx.
That means the Tx include their message (so both can be checked if they should enter the mem-pool and gossiped around as a single entity). This means there must be way to extract the message from the Tx though. How this could look like becomes clearer if we look at how Tx that are submitted should look like and to understand this best, I try to describe the lifecycle of a (LL) transaction from creation to inclusion in the mempool and later in the block:
Note: below we only consider Tx that pay for the inclusion of messages but keep in mind that existing Txs (other PoS-related Tx types from the SDK) should still work.
Sign-Data
Before signing the Tx, the client would need to compute potential share layouts, split the message it wants to include
into corresponding shares and commit to those.
The client then signs multiple such Tx (for each relevant share layout depending on potential square-sizes).
More concretely, signatures are computed on objects that look - from the PoV of the app's client -
sth like this (before encoding obviously):
message {
// ... non-LL specific fields skipped
bytes nid = 1;
bytes commit = 2; // commit(msg-shares)
};
Tx
The Tx that gets sent to the mem-pool is one Tx, before encoding, looks like this:
message {
// ... non-LL specific fields skipped
bytes nid = 1;
bytes msg-data = 2; // the actual message this Tx pays for (not split into shares)
repeated bytes signatures = 3; // signatures of above's sign-data
};
Note that, after encoding, this remains an opaque byte blob from the PoV of tendermint.
As mentioned above this is how the Tx could be passed to the mem-pool and how the Tx is
gossiped between nodes (as an opaque blob).
Further steps and processing
To verify if a Tx should enter the mem-pool, tendermint asks the app via CheckTx.
We noted that to decide if a Tx should enter the pool, the app (like the client above)
would already need to compute shares (corresponding to potential layouts) to verify the signatures in the Tx.
Hence, one option could be for CheckTx to already return multiple Tx (one for each sig) and
the corresponding (single) Message Data. That would mean some more changes to the mem-pool:
basically the pool would need to track multiple "things", and their relation,
instead of one blob. This needs further investigation!
Alternatively, CheckTx would at least need to return IDs that can be used to evict the pool.
The latter means that we could instead use a pre-process phase (because we can use these IDs
to evict the mem-pool after a block with the processed Tx was seen).
Further required clarifications
It's not yet entirely clear yet how in the above drafted approach, the square-size is decided and if that would 'live' entirely on the app-side of abci, or, in tendermint.
Furthermore, pre-processing looks promising but also turns out a bit more tricky regarding mem-pool eviction.
from celestia-core.
IMO, we can close this. Following discussions that emerged from the implementation, the spec now sufficiently describes Tx vs Messages. The remaining gaps are filled via what the usage of tendermint and the cosmos-sdk impose anyways.
There are still some considerations to be made here:
- reaping from the mempool depending on max square / block size (#77)
- making sure the mempool can actually deal with large tx (the combo of tx+message+signatures can become quite large - not sure the current logic will reach its limits with this quickly)
There might be more to consider. Either way it seems like we should rather open dedicated issues to resolve any remaining issues.
If you disagree, feel free to reopen this one! (cc @evan-forbes / @adlerjohn )
from celestia-core.
Related Issues (20)
- Benchmark the consensus reactor HOT 1
- Do we check this? i.e. rp.EndRow > rp.StartRow. If not, do we want to check this? HOT 2
- Consistent row proof range
- govulncheck fails on main branch HOT 1
- Expand TxStatus to include rejected transactions
- Remove v0 mempool HOT 2
- TestTxPool_ConcurrentlyAddingTx is flaky
- Port and test consensus reactor contention fixes
- Prototype using QUIC instead of tcp and proprietary handshakes HOT 1
- Compare duplicate messages with copy in receive
- Deprecate v0 mempool HOT 4
- Allow the user to request a proof of the tx's inclusion if it has been committed
- RequestPrepareProposal has unclear comments HOT 1
- Remove outdated TODO HOT 1
- Add ability to subscribe to block data via grpc HOT 2
- Add ability to request a block via its hash HOT 1
- Port not blocking on receives work from upstream HOT 1
- Expand `TxInfo` to save the resulting code of the transaction
- Proposal: Use CometBFT protos
- Investigate malformed data in JSONL traced files HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from celestia-core.