fuellabs / fuel-core Goto Github PK

Automate the deployment of helm charts via terraform:

• fuel core
• Cert Manager
• Prometheus
• Grafana

Update fuel-core repo to use Rust Edition 2021. Identify any improvements or refactors needed to our codebase to use the latest features and idiomatic syntax and implement them.

The current graphql connection for transactions just naively iterates over the main collection of transactions which are ordered by transaction id. Update this collection to return transactions in order by block height / time. This could be done by making a new sorted index of transaction id's by time, or using the existing index for blocks and then flattening the stored transactions ids stored on the sorted blocks.

closes FuelLabs/block-explorer-v2#3

Fuel Core needs to communicate with Layer 1 in order to publish blocks and bridge assets.

• Generic interface (not eth specific)
  • validator staking events
  • asset deposits / withdrawals
  • block publishing
• Mock layer 1 implementation (can be a simple rpc server with embedded db e.g. rocks or sqlite)
  • docker image
  • deployed via docker-compose & k8s
• L1 Relayer Adapters
  • in-memory (for unit testing)
  • Mock L1
  • Ethereum
• Service Configuration (e.g. cli args) allows choosing L1 adapter at runtime

blocked by FuelLabs/fuel-vm#46

Once fuel-vm supports gas consumption profiling, make these metrics available from the dry-run graphql mutation and fuel-client api.

Setup EKS:

• Cluster
• Node Groups
• IAM Roles (Cluster + Nodes)
• Coredns, VPC CNI, Kube-Proxy Add On

aes-soft has been merged into the aes crate

Please use the aes crate going forward. The new repository location is at:

<https://github.com/RustCrypto/block-ciphers/tree/master/aes>

AES-NI is now autodetected at runtime on i686/x86-64 platforms.
If AES-NI is not present, the aes crate will fallback to a constant-time
portable software implementation.

To force the use of a constant-time portable implementation on these platforms,
even if AES-NI is available, use the new force-soft feature of the aes
crate to disable autodetection.

See advisory page for additional details.

Potential segfault in localtime_r invocations

Impact

Unix-like operating systems may segfault due to dereferencing a dangling pointer in specific circumstances. This requires an environment variable to be set in a different thread than the affected functions. This may occur without the user's knowledge, notably in a third-party library.

Workarounds

No workarounds are known.

References

• time-rs/time#293

See advisory page for additional details.

aesni has been merged into the aes crate

Please use the aes crate going forward. The new repository location is at:

<https://github.com/RustCrypto/block-ciphers/tree/master/aes>

AES-NI is now autodetected at runtime on i686/x86-64 platforms.
If AES-NI is not present, the aes crate will fallback to a constant-time
portable software implementation.

To prevent this fallback (and have absence of AES-NI result in an illegal
instruction crash instead), continue to pass the same RUSTFLAGS which were
previously required for the aesni crate to compile:

RUSTFLAGS=-Ctarget-feature=+aes,+ssse3

See advisory page for additional details.

stdweb is unmaintained

The author of the stdweb crate is unresponsive.

Maintained alternatives:

• wasm-bindgen
• js-sys
• web-sys

See advisory page for additional details.

Client components

• GraphQL interface
  • Accept transactions #26
  • Read from state
  • Write to state
  • Misc (block height, block hash, get transaction, etc.)
• Validate transactions https://github.com/FuelLabs/fuel-specs/blob/master/specs/protocol/tx_validity.md
• Persistent database
• #780

Implement postconditions according to fuel-specs.

• Block executor has two modes, production vs validation.
  • Production will mutate the block into its final form after the VM modifies and sets the tx outputs
  • Validation will check that the resultant txs from the VM exactly match what's in the proposed block
• Ensure the inputs are consumed and appropriate change outputs are persisted even when a tx is reverted

Support the new data fields on logd and retd receipt types.

We have an API to get all coins associated with an owner. However, we'd like two additional filters:

• Color
• Spendable Status

cpuid-bool has been renamed to cpufeatures

Please use the `cpufeatures`` crate going forward:

<https://github.com/RustCrypto/utils/tree/master/cpufeatures>

There will be no further releases of cpuid-bool.

See advisory page for additional details.

Every client in space has these building blocks, and having this separation has a benefit on discussion:

• p2p layer: devp2p used by Ethereum but most of newer networks and eth2 uses libp2p it fixes some downsides of devp2p and it is more flexible protocol. Library made by tomaka: https://github.com/libp2p/rust-libp2p is I think only rust impl. Interface of lib seems complex when i looked at it few months ago and looking how lighthouse implemented it is maybe good way to go. ddos resistance.
• Sync layer. Handles p2p layer and handshake between peers. Here we need to define our own API on top of p2p. For example we need API for Initial sync and when client is activly working (broadcasting tx/block maybe consensus packets).
• Relayer: Thread is already started here: #52
• Execution layer: This mostly fall under vm additionaly we need to do checks on block/tx/signatures validity. I didnt see support for precompiles but this should be good to add, just for reusage of cryptography libs that are fast and audited (maybe this is more vm discussion).
• Consensus layer. PoS/PoA this can be tricky to integrate with other layers because it is used everywhere.
• Txpool: Should have ability to insert/remove transactions and sort them by gas_price. Creation of new block can be put into this layer but we can add new one and call it miner :). It needs to handle ddos and have mem/number limits.
• State/Database layer. In the beginning it will not have a big impact but as we grow this is pain problem for all blockchains. liveldb/rocskdb/mdbx
• web3/user API layer: It seems graphql is already started. ddos resistance.
• Bridge: bridge to ethereum network. Not exactly part of client but here for completeness sake.

In order to secure our network in sidechain mode, we'll need to operate sentries that monitor the consensus behaviors of participants for any foul play (e.g. vote withholding or voting for invalid blocks). This will also need to page the security team when problems are discovered.

The execution result of a transaction is available on the raw GQL API, but needs to be exposed via the Rust client.

Using the owner field from the inputs and outputs on transactions, we need to build up an index of transactions by owner sorted by age (block height).

Fuel-core needs a way to configure some initial state for a network. This is because we can't implement full state verification rules (e.g. input coins actually exist) until the state can be bootstrapped. Spendable coins are the main requirement, however being able to initialize contracts and contract state would be useful for contract developers using the SDK to write tests or even network regenesis (history flattening or performing major network upgrade).

Configurable state entities:

• spendable coins
• contracts
• contract state
• defaults for local dev (similar to how ganache bootstraps 10 accounts with eth)

While fuel-client doesn't depend on fuel-core at runtime, it depends on fuel-core in order to generate a graphql schema definition at build time.

As fuel-core grows with things like rocksdb, postgres, wasm etc, these dependencies will leak through the client and slow down the build times of any fuel-client users. This isn't sustainable as build times in sway have already hit 20+ minutes and fuel-core is going to keep getting heavier.

Come up with a strategy for publishing the graphql schema definitions from the fuel-core build as a static asset, which fuel-client consumes. This could involve using github packages or even simply just checking in the generated asset into version control and have a CI rule to verify the checked in asset matches the current fuel-core build.

In order to facilitate sharing state between graphql schema's, we currently have to inject it by overloading the GraphQL schema constructor. This approach feels kludgy so we should investigate if there's a better approach for doing this. Ideally we could put shared state into actix data, but when the graphql schema executes a request the actix data is no longer available in that scope.

Goal:

No longer require parameters to be passed into fn schema:

pub fn schema(state: Option<SharedDatabase>) -> TXSchema {

It looks like fuel-core no longer compiles on the latest fuel-vm:

error[E0432]: unresolved import `fuel_vm::data`
  --> fuel-core/src/database.rs:10:14
   |
10 | use fuel_vm::data::{DataError, InterpreterStorage, MerkleStorage};
   |              ^^^^ could not find `data` in `fuel_vm`

error[E0433]: failed to resolve: use of undeclared type `ExecuteError`
  --> fuel-core/src/schema/dap.rs:75:13
   |
75 |             ExecuteError::Io(io::Error::new(
   |             ^^^^^^^^^^^^ use of undeclared type `ExecuteError`

error[E0433]: failed to resolve: use of undeclared type `ExecuteError`
  --> fuel-core/src/schema/dap.rs:91:17
   |
91 |                 ExecuteError::Io(io::Error::new(
   |                 ^^^^^^^^^^^^ use of undeclared type `ExecuteError`

error[E0412]: cannot find type `ExecuteError` in this scope
  --> fuel-core/src/schema/dap.rs:38:21
   |
18 | impl ConcreteStorage {
   |     - help: you might be missing a type parameter: `<ExecuteError>`
...
38 |     ) -> Result<ID, ExecuteError> {
   |                     ^^^^^^^^^^^^ not found in this scope

error[E0412]: cannot find type `ExecuteError` in this scope
  --> fuel-core/src/schema/dap.rs:64:82
   |
18 | impl ConcreteStorage {
   |     - help: you might be missing a type parameter: `<ExecuteError>`
...
64 |     pub fn reset(&mut self, id: &ID, storage: DatabaseTransaction) -> Result<(), ExecuteError> {
   |                                                                                  ^^^^^^^^^^^^ not found in this scope

error[E0412]: cannot find type `ExecuteError` in this scope
  --> fuel-core/src/schema/dap.rs:84:63
   |
18 | impl ConcreteStorage {
   |     - help: you might be missing a type parameter: `<ExecuteError>`
...
84 |     pub fn exec(&mut self, id: &ID, op: Opcode) -> Result<(), ExecuteError> {
   |                                                               ^^^^^^^^^^^^ not found in this scope

This is unfortunately blocking me from running a debug session on the latest VM :(

be some kind of nightly github action that runs cargo-audit - https://crates.io/crates/cargo-audit - on our repos to alert us of any yanked deps or other security issues

This issue centers around adding a CI/CD workflow for fuel-core that will build & push image to ghcr.io.

Along the way when testing the build & push ghcr.io image, I found it was taking excess of 20 minutes to build the fuel-core image so I added a github action based caching solution.

there are several modifications made to prior ci/cd docker build + push workflows particularly using this Github docker project: https://github.com/docker/build-push-action with includes different ways to tag, label docker images and image GA based caching.

Given this is a novel approach, let the team analyze if this is the path we want to follow for other repos docker build + pushes + caching (there are possible other solutions but this approach is more Github actions friendly)

Difference between the first test of this new workflow: https://github.com/FuelLabs/fuel-core/actions/runs/1652067733 which took 26m 28s minutes to run.

The second run was 19m31s: https://github.com/FuelLabs/fuel-core/runs/4698274089?check_suite_focus=true

I believe us using a base docker image for repos will be a minor optimization (should be done still), but the chunk of the docker build time is coming from:

RUN --mount=type=ssh cargo build --release

I am not entirely sure how we can cache that step - need to discuss with @Voxelot

I also see @AlicanC is working on the gchr docker build & push too - the work there is revelant too- https://github.com/FuelLabs/fuel-core/tree/ghcr-publish - there is place to collaborate on these two approaches

The final result is the fuel-core image available on this page:

ghcr.io/fuellabs/fuelcore -> https://github.com/fuellabs/fuel-core/pkgs/container/fuelcore

@Voxelot @adlerjohn @AlicanC @ellioty - take a look and let me know (making the PR open for now just for review)- https://github.com/FuelLabs/fuel-core/pull/110/files

We're currently copying and pasting a lot of our CI configuration around. We should look into consolidating release or docker image publishing workflows into parameterizable templates: https://docs.github.com/en/actions/learn-github-actions/creating-starter-workflows-for-your-organization

txpool submit as Brandon said should return the result of tx inclusion inside txpool. When somebody wants to submit a signed transaction it needs to be sure that the transaction is already checked and with submit the user tell us it's the intention to commit, it is mostly a point of no return because tx is going to be broadcasted to the network.

There is an idea of the pending block, where the Block producer will every N second take best T transactions and with its best ability create pending block so that users know what to expect (this is just usability, it shouldn't have any effect on consensus) and can inspect executed transactions.

There should be the ability to dry run tx (or maybe even a list of tx if they are dependent) so that users can see gas used and calculate the gas limit. Dry run over the pending block or dry run over the best block. This can be even extended to: with this gas_price where is this transaction going to land inside the block and what amount of gas is going to be used, there are some MEV vibes here but seems like a topic for the future.

https://github.com/FuelLabs/fuel-specs/blob/master/specs/vm/opcodes.md

Changing to this NGINX Ingress Controller for routing to fuel-core backend

Add APIs for retrieving information about deployed contracts.

• Components
  • add to rust api in fuel-client
    • snapshot test
    • integration tests
  • add to GQL schema in fuel-core
• API
  • get contract by id
  • #395
• Data
  • id
  • salt
  • code
  • balances

The SDK should use a special API to select which utxos to use as inputs, instead of directly picking utxos from the full set. This is important for ensuring consistent behavior across SDK's and the ability to implement efficient dust mitigation strategies such as random-improve.

For inputs, the API will take:

owner: The Address of the utxo owner
spend_query: Vec<(AssetId, Amount)> The total amount of each asset type to spend
max_inputs: u8 The max number of utxos that can be used

Success:
Return a list of utxos <= max_inputs that will satisfy the amounts to be spent.

Errors:

• If the spend amount is higher than the available balance of the owner.
• If reaching the spend amount requires the list of utxos to be greater than max_inputs

This issue tracks the features we eventually want out of the interpreter

• Execute bytecode
• Return error codes in some standard (e.g. JSON)
• Return return values in some standard (e.g. JSON)
• Gas profiling (total gas, gas per op, gas per line)
• Bytecode coverage
• Debugging
  • Set breakpoints
  • Step through code, continue to next breakpoint, etc.
  • View entire memory space and registers

sccache is a tool that is used to speed up compilation times for rust or c++ dependencies. Investigate whether this will speed up fuel-core CI workflows and incorporate if there's a significant improvement.

On every merge to fuel-core master:

• Build a docker image of fuel-core tagged w/ the latest commit sha
• Publish to ECR
• Deploy to k8s cluster
  • use fresh PVC on every deployment until data layer is stable/upgrade-able
  • cleanup old resources

Add the ability to export the state of a node to a chain / genesis configuration file. This will allow for things like "forking main net", regenesis, or other testing utilities.

Related to #53

Potential segfault in the time crate

Impact

Unix-like operating systems may segfault due to dereferencing a dangling pointer in specific circumstances. This requires an environment variable to be set in a different thread than the affected functions. This may occur without the user's knowledge, notably in a third-party library.

The affected functions from time 0.2.7 through 0.2.22 are:

• time::UtcOffset::local_offset_at
• time::UtcOffset::try_local_offset_at
• time::UtcOffset::current_local_offset
• time::UtcOffset::try_current_local_offset
• time::OffsetDateTime::now_local
• time::OffsetDateTime::try_now_local

The affected functions in time 0.1 (all versions) are:

• at
• at_utc
• now

Non-Unix targets (including Windows and wasm) are unaffected.

Patches

Pending a proper fix, the internal method that determines the local offset has been modified to always return None on the affected operating systems. This has the effect of returning an Err on the try_* methods and UTC on the non-try_* methods.

Users and library authors with time in their dependency tree should perform cargo update, which will pull in the updated, unaffected code.

Users of time 0.1 do not have a patch and should upgrade to an unaffected version: time 0.2.23 or greater or the 0.3 series.

Workarounds

No workarounds are known.

References

time-rs/time#293

See advisory page for additional details.

In order to make the k8s deployment compatible with a variety of reigstries, we should use kustomize to make the full image path's configurable.

Currently, blocks are produced instantly when transactions are submitted. In order to support multi-tx blocks, we need a configuration mechanism to enable/disable different block production triggers on startup.

e.g. fuel-core run --block-production-mode interval or fuel-core run --block-production-mode instant

This task should cover the implementation of the following block production options:

interval Timer-based: For simulating consensus block delays, the timer trigger will fire a block production event on a configurable interval
instant Instant trigger: For integration testing contracts etc, make the tx pool trigger a block production event every time a tx is inserted.
none No block production: For passive nodes that simply listen for blocks

pos mode will be deferred to a later task.

term is looking for a new maintainer

The author of the term crate does not have time to maintain it and is looking
for a new maintainer.

Some maintained alternatives you can potentially switch to instead, depending
on your needs:

• crossterm
• termcolor
• yansi

See advisory page for additional details.

Implements BIP-114: MAST on Fuel VM

As seen in the image below, descriptions of fields in GQL Playground are uncapitalized and look weird.

We should note that some descriptions are derived from doc comments like in below:

For that reason, it might make sense to start capitalizing all doc comments (or even all comments) instead of just capitalizing the ones async-graphql read. (cc @Voxelot)

I think we could go even further and start capitalizing all comments on all repos.

While there are ESLint rules like capitalized-comments and spaced-comment to style comments a bit (demo) it doesn't seem like equivalents exist in the Rust ecosystem.

There are generic CI tools that could enforce this or we could contribute these rules to Clippy, but I don't think we have to go there at this time.

As of FuelLabs/fuel-vm#45, fuel-vm now supports error backtraces. Support vm backtraces in fuel-core.

• backtraces are turned off by default
• backtraces are enabled via cli argument
• use logging / tracing to report the error details (no graphql api changes)