jito-foundation / jito-solana

Jito Foundation MEV Solana Client

About

This repository contains Jito's fork of the Solana validator.

We recommend checking out our Gitbook for more detailed instructions on building and running Jito-Solana.

---

1. Install rustc, cargo and rustfmt.

$ curl https://sh.rustup.rs -sSf | sh
$ source $HOME/.cargo/env
$ rustup component add rustfmt

When building the master branch, please make sure you are using the latest stable rust version by running:

$ rustup update

When building a specific release branch, you should check the rust version in ci/rust-version.sh and if necessary, install that version by running:

$ rustup install VERSION

Note that if this is not the latest rust version on your machine, cargo commands may require an override in order to use the correct version.

On Linux systems you may need to install libssl-dev, pkg-config, zlib1g-dev, protobuf etc.

On Ubuntu:

$ sudo apt-get update
$ sudo apt-get install libssl-dev libudev-dev pkg-config zlib1g-dev llvm clang cmake make libprotobuf-dev protobuf-compiler

On Fedora:

$ sudo dnf install openssl-devel systemd-devel pkg-config zlib-devel llvm clang cmake make protobuf-devel protobuf-compiler perl-core

2. Download the source code.

$ git clone https://github.com/jito-foundation/jito-solana.git
$ cd solana

3. Build.

$ ./cargo build

Testing

Run the test suite:

$ ./cargo test

Starting a local testnet

Start your own testnet locally, instructions are in the online docs.

Accessing the remote development cluster

• devnet - stable public cluster for development accessible via devnet.solana.com. Runs 24/7. Learn more about the public clusters

Benchmarking

First, install the nightly build of rustc. cargo bench requires the use of the unstable features only available in the nightly build.

$ rustup install nightly

Run the benchmarks:

$ cargo +nightly bench

Release Process

The release process for this project is described here.

Code coverage

To generate code coverage statistics:

$ scripts/coverage.sh
$ open target/cov/lcov-local/index.html

Why coverage? While most see coverage as a code quality metric, we see it primarily as a developer productivity metric. When a developer makes a change to the codebase, presumably it's a solution to some problem. Our unit-test suite is how we encode the set of problems the codebase solves. Running the test suite should indicate that your change didn't infringe on anyone else's solutions. Adding a test protects your solution from future changes. Say you don't understand why a line of code exists, try deleting it and running the unit-tests. The nearest test failure should tell you what problem was solved by that code. If no test fails, go ahead and submit a Pull Request that asks, "what problem is solved by this code?" On the other hand, if a test does fail and you can think of a better way to solve the same problem, a Pull Request with your solution would most certainly be welcome! Likewise, if rewriting a test can better communicate what code it's protecting, please send us that patch!

Disclaimer

All claims, content, designs, algorithms, estimates, roadmaps, specifications, and performance measurements described in this project are done with the Solana Labs, Inc. (“SL”) good faith efforts. It is up to the reader to check and validate their accuracy and truthfulness. Furthermore, nothing in this project constitutes a solicitation for investment.

Any content produced by SL or developer resources that SL provides are for educational and inspirational purposes only. SL does not encourage, induce or sanction the deployment, integration or use of any such applications (including the code comprising the Solana blockchain protocol) in violation of applicable laws or regulations and hereby prohibits any such deployment, integration or use. This includes the use of any such applications by the reader (a) in violation of export control or sanctions laws of the United States or any other applicable jurisdiction, (b) if the reader is located in or ordinarily resident in a country or territory subject to comprehensive sanctions administered by the U.S. Office of Foreign Assets Control (OFAC), or (c) if the reader is or is working on behalf of a Specially Designated National (SDN) or a person subject to similar blocking or denied party prohibitions.

The reader should be aware that U.S. export control and sanctions laws prohibit U.S. persons (and other persons that are subject to such laws) from transacting with persons in certain countries and territories or that are on the SDN list. Accordingly, there is a risk to individuals that other persons using any of the code contained in this repo, or a derivation thereof, may be sanctioned persons and that transactions with such persons would be a violation of U.S. export controls and sanctions law.

jito-solana's Issues

Validator version 1.16.13 assert at core/src/unprocessed_transaction_storage.rs:819:9

Problem

Validator running 1.16.13 JITO crashed about half an hour ago. Logs show this immediately preceeding the solana-validator process exit and systemctl restart of the service:

[2023-09-13T04:25:34.298200991Z INFO  solana_metrics::metrics] datapoint: qos-service-errors id=1i bank_slot=217248296i retried_txs_per_block_limit_count=0i retried_txs_per_vote_limit_count=0i r
etried_txs_per_account_limit_count=0i retried_txs_per_account_data_block_limit_count=0i dropped_txs_per_account_data_total_limit_count=0i
thread 'solBanknStgTx03' panicked at 'assertion failed: `(left == right)`
  left: `0`,
 right: `1`', core/src/unprocessed_transaction_storage.rs:819:9
stack backtrace:
[2023-09-13T04:25:34.311850899Z INFO  solana_metrics::metrics] datapoint: qos-service-stats id=2i bank_slot=217248296i compute_cost_time=233i compute_cost_count=668i cost_tracking_time=280i sele
cted_txs_count=615i estimated_signature_cu=442800i estimated_write_lock_cu=669300i estimated_data_bytes_cu=16977i estimated_builtins_execute_cu=52050i estimated_bpf_execute_cu=139820000i actual_
bpf_execute_cu=859870i actual_execute_time_us=11069i
[2023-09-13T04:25:34.311878121Z INFO  solana_metrics::metrics] datapoint: qos-service-errors id=2i bank_slot=217248296i retried_txs_per_block_limit_count=53i retried_txs_per_vote_limit_count=0i 
retried_txs_per_account_limit_count=0i retried_txs_per_account_data_block_limit_count=0i dropped_txs_per_account_data_total_limit_count=0i
[2023-09-13T04:25:34.325990535Z INFO  solana_metrics::metrics] datapoint: qos-service-stats id=5i bank_slot=217248296i compute_cost_time=309i compute_cost_count=759i cost_tracking_time=264i sele
cted_txs_count=752i estimated_signature_cu=542160i estimated_write_lock_cu=856200i estimated_data_bytes_cu=18256i estimated_builtins_execute_cu=46950i estimated_bpf_execute_cu=186150000i actual_
bpf_execute_cu=2323406i actual_execute_time_us=26054i
[2023-09-13T04:25:34.326031883Z INFO  solana_metrics::metrics] datapoint: qos-service-errors id=5i bank_slot=217248296i retried_txs_per_block_limit_count=7i retried_txs_per_vote_limit_count=0i r
etried_txs_per_account_limit_count=0i retried_txs_per_account_data_block_limit_count=0i dropped_txs_per_account_data_total_limit_count=0i
   0: rust_begin_unwind
             at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:579:5
   1: core::panicking::panic_fmt
             at /rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/panicking.rs:64:14
   2: core::panicking::assert_failed_inner
   3: core::panicking::assert_failed
   4: solana_core::unprocessed_transaction_storage::UnprocessedTransactionStorage::process_packets
   5: solana_core::banking_stage::consumer::Consumer::consume_buffered_packets
   6: solana_core::banking_stage::BankingStage::process_loop
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
[2023-09-13T04:25:34.366116768Z ERROR solana_metrics::metrics] datapoint: panic program="validator" thread="solBanknStgTx03" one=1i message="panicked at 'assertion failed: `(left == right)`
      left: `0`,
     right: `1`', core/src/unprocessed_transaction_storage.rs:819:9" location="core/src/unprocessed_transaction_storage.rs:819:9" version="1.16.13 (src:00000000; feat:3949673676, client:JitoLabs)"

Proposed Solution

RPC dies after catching up with `--no-incremental-snapshots` with 7/BUS

Problem

Two Jito-Solana validators running v1.17.26-jito crashed after running with --no-incremental-snapshots with error SIGBUS.

Mar 18 03:37:48 amsterdam-mainnet-rpc-1 systemd[1]: solana-validator.service: Main process exited, code=killed, status=7/BUS

Update v-iface client authentication

Problem

We're changing the auth flow to require validators to sign a specific message. This breaks the client here since it signs a static string.

Proposed Solution

Write the client in such a way that it extracts the token from the error status message and retries with the token signed.

Prevent validator identity from being rugged through challenge-response

Problem

Block Engines and relays could rug validators by having them sign "challenge" messages which are actually transfers.

Proposed Solution

Determine a secure way to have validators sign messages with some random empty keypair, but also have it be associated with their identity keypair

QosService::update_or_remove_transaction_costs called with same tx_costs multiple times

Problem

• We pass the same tx qos results to QosService::update_or_remove_transaction_costs on multiple iterations

Proposed Solution

release v1.16.20-jito

Hello,

Do you plan to release v1.16.20-jito ?

We had issue yesterday using v1.16.19-jito with a fresh downloaded snapshot. I had to switch to the (non-jito) v1.16.20 release to make it work.

Each snapshot we tested finished with this kind of panic:

thread 'main' panicked at 'Snapshot bank for slot 232831868 failed to verify', /solana/runtime/src/snapshot_utils.rs:1627:9
stack backtrace:
   0: rust_begin_unwind
             at ./rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/std/src/panicking.rs:579:5
   1: core::panicking::panic_fmt
             at ./rustc/84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc/library/core/src/panicking.rs:64:14
   2: solana_runtime::snapshot_utils::bank_from_latest_snapshot_archives
   3: solana_ledger::bank_forks_utils::bank_forks_from_snapshot
   4: solana_ledger::bank_forks_utils::load_bank_forks
   5: solana_core::validator::Validator::new
   6: solana_validator::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
[2023-11-28T22:05:01.183338487Z ERROR solana_metrics::metrics] datapoint: panic program="validator" thread="main" one=1i message="panicked at 'Snapshot bank for slot 232831868 failed to verify', /solana/runtime/src/snapshot_utils.rs:1627:9" location="/solana/runtime/src/snapshot_utils.rs:1627:9" version="1.16.19 (src:00000000; feat:4033350765, client:JitoLabs)"
[2023-11-28T22:05:02.003073048Z INFO  solana_metrics::metrics] datapoint: accounts_db_active hash=1i
[2023-11-28T22:05:02.015604374Z INFO  solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=8i no_ports_delta=0i in_errors_delta=466i out_datagrams_delta=8i rcvbuf_errors_delta=466i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=2970962i rcvbuf_errors=2970962i sndbuf_errors=0i rx_bytes_delta=251694i rx_packets_delta=487i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=7827i tx_packets_delta=22i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i

Thanks

QosService::update_or_remove_transaction_costs incorrectly rolls back our transaction costs

Problem

• We're calling QosService::update_or_remove_transaction_costs incorrectly in a loop

Proposed Solution

Does BundleStage handle TxV2 properly?

Problem

What happens if banking stage processes a v2 transaction that modifies a lookup table in the middle of a bundle execution a transaction that loads from this lookup table?

Proposed Solution

Investigate this further

QoS in bundle_stage incorrectly rolls back QoS for uncommitted transactions

Problem

Proposed Solution

RPC crashes on simulate_transaction

Running v1.14.17-jito, the RPC periodically crashes with:

ERROR solana_metrics::metrics] datapoint: panic program="validator" thread="solRpcEl" one=1i message="panicked at 'simulation bank must be frozen', runtime/src/bank.rs:4077:9" location="runtime/src/bank.rs:4077:9" version="\"1.14.17 (src:devbuild; feat:3488713414)

todo:

• run 1.14.16-jito and v1.14.17 vanilla client to determine if issue is upstream

RPC simulateBundle response is not fully camelCase

in case of a failed bundle simulation the tx_signature field in the json response is in snake_case and not camelCase like all the other fields.

example response:

{
      "context": {
        "apiVersion": "1.13.7",
        "slot": 183287630
      },
      "value": {
        "summary": {
          "failed": {
            "error": {
              "TransactionFailure": {
                "InstructionError": [
                  2,
                  {
                    "Custom": 6001
                  }
                ]
              }
            },
            "tx_signature": "3RQxXBZ3V2vmAW8Nq17neRvBwB4DABp5bDCL156aom6bciUa4cGDSVZvHNPYGL3cMuJStv4VYwxKuMHWkEyJ394U"
          }
        },
        "transactionResults": [
          {
            "err": {
              "InstructionError": [
                2,
                {
                  "Custom": 6001
                }
              ]
            },
            "logs": [
              "Program ComputeBudget111111111111111111111111111111 invoke [1]",
              "Program ComputeBudget111111111111111111111111111111 success",
              "Program ComputeBudget111111111111111111111111111111 invoke [1]",
              "Program ComputeBudget111111111111111111111111111111 success",
            ],
            "postExecutionAccounts": null,
            "preExecutionAccounts": null,
            "unitsConsumed": 199577
          }
        ]
      }
    }
    ```

RpcClient will throw runtime error if mix of errors and success objects in batch jsonrpc responses.

See here: https://github.com/jito-labs/jito-solana/blob/01885b38faa327cdde93f52b9de8dd42ab8cc2f3/rpc-client/src/rpc_client.rs#L1157

If the response is a mix of errors as it can be with batch requests then it'll throw deserialization errors.

Whether there is a wss url?

Ws: / / bundles - API - rest. Jito. WTF/API/v1 / bundles/tip_stream this is ws Url address, but I need to use WSS, excuse me is there a WSS Url

Consider adding compute unit limit to bundles

Problem

Proposed Solution

QUIC packets aren't dropped if there's a relayer connected

Problem

Proposed Solution

Pass various constant / config values from backend on initial handshake

Problem

Once the validator is out in the wild, we can't really force clients to upgrade software (or we would need a mechanism to prevent auth based on version at least).

Proposed Solution

Let's parametrize the constants and configs we might want to change in the future and pass these when validator completes initial handshake.

Add an admin RPC endpoint to update block-engine and relayer addresses at runtime

tip-distro PDA not being init'd

Problem

The validator is expected to init a tip-distro PDA on the first bundle it receives; figure out why it's not doing this.

Drop Bundles At End of Leader Slot

Problem

Proposed Solution

Bank::collect_balances collects balances for transactions that fail to lock

Problem

In Bank::collect_balances, we collect balances on accounts that failed to lock

Proposed Solution

We should only collect balances for transactions that have a corresponding Ok(()) on lock_results

Check init PDA code paths.

Problem

From the runtime's perspective an account is "created" as soon as it has lamports. The initial owner of the program is the system_program and upon init-ing with anchor the owner is changed to be the program.

Regarding the tip_manager code paths, we check if the PDAs of interest exist in the bank and if it does then no "initialize" txs are created and propagated to the network. This might be a potential bug since the PDA may "exist" due to having lamports but not properly initialized (meaning it's rent-exempt and owned by the program used to derive the address).

Proposed Solution

Check if PDA exists and the owner is NOT the system_program.

Fix local multi-node cluster

Problem

Starting a local cluster using these commands does not work. The validator spin loops with the following error msg:
"Searching for an RPC service with shred version 41517 (Retrying: No snapshots available)..."

simulate_bundle RPC returns success on an empty list of transactions

This is misleading, return an error.

MEV Stage fails silently when address malformed

Let's fail fast and panic if the url is malformed so validators aren't stuck scratching their heads.

batch_simulate_bundle_with_config in RpcClient returning different number of accounts between pre-simulation and post-simulation for same config

Problem

I am using v1.14.16-jito branch for the sdk and the validator software

Example Tx

Signature - 47zGU9TNVmbu6ZT1RUgFrXvoewBqPkLMKXaRrk5wKVK4rVgc2KNZiF9SqMnnyYNfqa3gS1eAZ33TtNaccCn5DVMQ

Config

There are 17 accounts I want the before and after for.

RpcSimulateTransactionAccountsConfig {
    encoding: Some(
        Base64,
    ),
    addresses: [
        "2AEWSvUds1wsufnsDPCXjFsJCMJH5SNNm7fSF4kxys9a",
        "E2BcoCeJLTa27mAXDA4xwEq3pBUcyH6XXEHYk4KvKYTv",
        "4d35yC7C8zhCDec7JbPptL9SEb4NUddKHxURgmvD8hfo",
        "2QdhepnKRTLjjSqPL1PtKNwqrUkoLee5Gqs8bvZhRdMv",
        "4K4ayZdgL4RrY863WaBMCQcvjwogteAJRmr134mnUPop",
        "7gz9KepoX1NeD4KeUdpSL2rC4Mjvb4C4voBT4q7M29qd",
        "2cuyPh4GeWgC7prQwD4hmAsvA97uUY9dxVApx66tH4sS",
        "D3CDPQLoa9jY1LXCkpUqd3JQDWz8DX1LDE1dhmJt9fq4",
        "dwxR9YF7WwnJJu7bPC4UNcWFpcSsooH6fxbpoa3fTbJ",
        "83v8iPyZihDEjDdY8RdZddyZNyUtXngz69Lgo9Kt5d6d",
        "3yCQySKJ4gnvnPcHR62S8dHkEw55qGdzBBhFqPCxn7Ky",
        "8QubQgwwkWXzmK9k83UUFngRaXqWzZPaXFdFUx1fWw1n",
        "Bga4BYZEcpJMNNzeCxgALJ596MbZAPVPZM99PSyEzarB",
        "DdigEybG2begUfkpSUP63o5CKF2Q9yGCWktZ6Hnb1RxN",
        "AqFSS3hGnwPNZD7ZT2Tuyfp5CtPY9cj5BtjD3N8X5Dif",
        "2hSMtnF5Sxq4kuBkqf2gmWjcgJjKfWZP4daiN8p8sccM",
        "4KR1SCoQCbZgeQQv8awEwd6rTYKPd25PG5AZjbE2uoWi",
    ],
}

Simulation Response

The account in first position is 2AEWSvUds1wsufnsDPCXjFsJCMJH5SNNm7fSF4kxys9a and relates to a stSOL/SOL whirlpool swap pool. In the pre-execution accounts we can the data for this account whilst in the post-execution accounts this is skipped.

Note in the transaction this account is mutable and should update with every swap.

RpcSimulateBundleTransactionResult {
    err: None,
    logs: //EXCLUDED
    pre_execution_accounts: Some(
        [
            UiAccount {
                lamports: 5435781,
                data: Binary(
                    "P5XRDOGAYwkT5EH4ORPKaLBjT7Al/eqohzfoQRDRJV41ezN33e4czfwBAAEAZAAsAe/gitwZVBQEAAAAAAAAAACjJhxYq4Ks8gAAAAAAAAAA0vv//3BOAQAAAAAAzEIAAAAAAAAGm4hX/quBhPtof2NGGMA12sQ53BrrO1WYoPAAAAAAATcupyMWhev/jm806YQiUQh8olPQsaooX014ecIG/rQR0A6Yp+IFIgAAAAAAAAAAAGJxy3EZR2udzgDYFcj/MV/Iv30oSGM9NJQq39U18t7+WULT4sVshXcKatT1WLCJwf7YCp3vT7uUhMV9AJbAktHQqPV5k5wiAAAAAAAAAAAAaPKdZAAAAAAMANCv64YU2n8Zq6AtQPGMaSWF9lAg387T1eX5qcDE4VpjYnO1vcTu8y2sTj9U19V2NnAXBJuZXsd5J+U8dKZKvR0xrxfe/zwmhIFgCsr+SxQJjA/hQbf0oc34STRkRAMAAAAAAAAAAAAAAAAAAAAAdsktOmaAlAAAAAAAAAAAAPYI8k64mwlNiLNwXlpvLKkXSDwYXBGDByAn53lg2R4hFr3OQ+s1QQGZBdfww7X7wy1UOCikuo7DdyjIFrnkKPm9HTGvF97/PCaEgWAKyv5LFAmMD+FBt/ShzfhJNGREAwAAAAAAAAAAd6EAAAAAAABwgCqWs8t7SAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACOIlhm57zD46/6nljbIbVKuvxkxzkq8QZ/LwuyXPhySAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=",
                    Base64,
                ),
                owner: "whirLbMiicVdio4qvUfM5KAg6Ct8VwpYzGff3uctyCc",
                executable: false,
                rent_epoch: 361,
            },
           **...//TRUNCATED, 16 more accounts**
        ],
    ),
    post_execution_accounts: Some(
        [
            UiAccount {
                lamports: 30202988673470,
                data: Binary(
                    "BpuIV/6rgYT7aH9jRhjANdrEOdwa6ztVmKDwAAAAAAEU6YsaioAZz4hj3MC2cFsMjjcO1PbUr6r0SxlB/b8TTc4bSi54GwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQEAAADwHR8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
                    Base64,
                ),
                owner: "TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA",
                executable: false,
                rent_epoch: 0,
            },
            **//TRUNCATED, 15 more accounts**
        ],
    ),
    units_consumed: Some(
        605339,
    ),
    return_data: None,
}

Proposed Solution

Not sure

Ensure no duplicate transactions within the same bundle inside the validator

Problem

Proposed Solution

Drop Bundles that write-lock programs owned by the AddressLookupProgram

Problem

Proposed Solution

Misleading RPC error on `simulate_bundle`

Problem

Internal RPC error returned when calling simulate_bundle where accounts are not funded and/or not rent-exempt after simulation.

Proposed Solution

Return a more obvious error indicating the account is no longer rent exempt as the reason for failure.

Support base64 encoded pubkeys in `simulate_bundle`

Currently the API defined as if it supports it, but the code path only tries decoding b58 strings.

MEV tip claimer transaction resigning handled improperly

Problem

In sign_and_send_transactions_with_retries, transactions are resigned and resent to the network after some amount of time if they haven't landed. This is to avoid blockhash expirations.

This is a problem if the original transaction lands after it's resigned, especially when resending transactions that create or destroy accounts. For instance, a transaction that creates an account with signature S is sent until it expires. Transaction T is then resigned with S' and the code loses track of S. If S lands after this, S' will keep getting retried to send but fail preflight because the account it's trying to create already exists.

Proposed Solution

Keep track of signatures sent for a particular transaction and if any of them land, remove the transaction

Bad code path hit if validator receives any bundles mentioning tip PDAs while the program isn't deployed.

The validator should check that the program exists in the DB and only then try to create the init tx.

1.17.16 does not build

root@m13:~# git clone https://github.com/jito-foundation/jito-solana.git --recurse-submodules
Cloning into 'jito-solana'...
remote: Enumerating objects: 369101, done.
remote: Counting objects: 100% (48990/48990), done.
remote: Compressing objects: 100% (12758/12758), done.
remote: Total 369101 (delta 36532), reused 47489 (delta 35522), pack-reused 320111
Receiving objects: 100% (369101/369101), 343.69 MiB | 13.36 MiB/s, done.

...

root@m13:~/jito-solana# git checkout tags/$TAG
Note: switching to 'tags/v1.17.16-jito'.

...

root@m13:~/jito-solana# git submodule update --init --recursive
root@m13:~/jito-solana# CI_COMMIT=$(git rev-parse HEAD) scripts/cargo-install-all.sh --validator-only ~/.local/share/solana/install/releases/"$TAG"
OSTYPE IS: linux-gnu
Install location: /root/.local/share/solana/install/releases/v1.17.16-jito (release)
+ cd target/perf-libs
+ [[ -r /root/.cache/solana-perf-v0.19.3.tgz ]]
+ cp /root/.cache/solana-perf-v0.19.3.tgz solana-perf.tgz
+ tar zxvf solana-perf.tgz
./cuda-10.0/
./cuda-10.0/libcl-crypt.so
./cuda-10.0/cuda-version.txt
./cuda-10.0/libcuda-crypt.so
./cuda-10.1/
./cuda-10.1/libcl-crypt.so
./cuda-10.1/cuda-version.txt
./cuda-10.1/libcuda-crypt.so
./cuda-10.2/
./cuda-10.2/libcl-crypt.so
./cuda-10.2/cuda-version.txt
./cuda-10.2/libcuda-crypt.so
./libpoh-simd.so
./libsigning.so
./signing_public.h
./signing.signed.so
./signing.so
./signing_test
./solana-perf-HEAD.txt
+ [[ ! -r /root/.cache/solana-perf-v0.19.3.tgz ]]
+ echo v0.19.3-1
+ /root/jito-solana/cargo build --profile release --bin solana --bin solana-bench-tps --bin solana-faucet --bin solana-gossip --bin solana-install --bin solana-keygen --bin solana-ledger-tool --bin solana-log-analyzer --bin solana-net-shaper --bin solana-validator --bin rbpf-cli --bin solana-genesis
+ exec cargo +1.73.0 build --profile release --bin solana --bin solana-bench-tps --bin solana-faucet --bin solana-gossip --bin solana-install --bin solana-keygen --bin solana-ledger-tool --bin solana-log-analyzer --bin solana-net-shaper --bin solana-validator --bin rbpf-cli --bin solana-genesis
   Compiling proc-macro2 v1.0.69
   Compiling unicode-ident v1.0.2
   Compiling libc v0.2.149
   Compiling version_check v0.9.4
   Compiling autocfg v1.1.0
   Compiling cfg-if v1.0.0
   Compiling syn v1.0.109
   Compiling serde v1.0.189
   Compiling typenum v1.15.0

...

   Compiling jsonrpc-derive v18.0.0
   Compiling ed25519-dalek-bip32 v0.2.0
   Compiling rolling-file v0.2.0
   Compiling ctrlc v3.4.1
error: failed to run custom build command for `etcd-client v0.11.1`

Caused by:
  process didn't exit successfully: `/root/jito-solana/target/release/build/etcd-client-46450057aa650c5c/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-changed=proto
  cargo:rerun-if-changed=proto/auth.proto
  cargo:rerun-if-changed=proto/kv.proto
  cargo:rerun-if-changed=proto/rpc.proto
  cargo:rerun-if-changed=proto/v3election.proto
  cargo:rerun-if-changed=proto/v3lock.proto
  cargo:rerun-if-changed=proto

  --- stderr
  thread 'main' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/prost-build-0.11.9/src/lib.rs:1457:10:
  Could not find `protoc` installation and this build crate cannot proceed without
      this knowledge. If `protoc` is installed and this crate had trouble finding
      it, you can set the `PROTOC` environment variable with the specific path to your
      installed `protoc` binary.If you're on debian, try `apt-get install protobuf-compiler` or download it from https://github.com/protocolbuffers/protobuf/releases

  For more information: https://docs.rs/prost-build/#sourcing-protoc

  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...

Option to add a second trusted relayer

While running a local relayer is exciting, an option to fail back to a trusted relayer ( Jito or another validator's ) woud be useful when something fails on the local relayer so that we dont miss any bundles. Logging it for future consideration even if it may not be the most immediate need.

--relayer-url http://127.0.0.1/11226
--relayer-url-fail http://ny.mainnet.relayer.jito.wtf:8100 \

Missed transfer before epoch ends causes improper snapshots #18

we take a snapshot on the last slot of an epoch and if the next validator didn’t transfer out before the epoch is over then our snapshot balance will be wrong

Full auth workflow should happen if relayer or block engine disconnects

Problem

If there's an error refreshing the block engine or relayer, it will reconnect and look at the previous expiry of tokens before determining if it should do full access token refresh or incremental refresh.

This makes it hard to swap out keys on the backend since it will take several hours/days before a refresh occurs.

Proposed Solution

If we can't auth, the full refresh flow should take place. Furthermore, it should ideally happen in sync with client connects/disconnects and not as a background thread

Consider using solana-sys tuner and iptables to drop TPU and TPU forward packets in the kernel

Problem

We currently have a packet redirection channel that gets hooked into our relayer stage and it drops packets if the relayer is connected. However, this requires using resources in the validator to reach that point.

Proposed Solution

We could add an extra message to solana sys-tuner that tells it when to and when not to drop packets using IP table rules. These would get dropped in the kernel and would be much more efficient

Advertise correct relayer TPU port

When validator is connected to both relayer and block engine, it is advertising block engine (tx recv) tpu port.

Suspicion - we spawn two threads independently in validator, block engine thread probably doing getTpuConfig and advertising that unaware of relayer thread?

Simplify rent_collection when loading accounts from AccountOverrides

Problem

Logic for loading in account during Accounts::load_transaction is confusing around AccountOverrides.
https://discord.com/channels/428295358100013066/560496939779620864/995088256502878230

Proposed Solution

Always try to collect rent from accounts loaded in Accounts::load_transaction from the AccountOverrides

`RpcClient::simulate_bundle` returns error

Problem

let bundle = VersionedBundle {
  transactions: vec![tx.clone()],
  };
 let sim = rpc_client.simulate_bundle(&bundle).await.unwrap();

returns the following error:

ClientError { request: Some(SimulateBundle), kind: RpcError(RpcResponseError { request_id: 1, code: -32602, message: "pre/post_execution_accounts_configs must be equal in length to the number of transactions", data: Empty }) }',

This is because simulate_bundle with no config fills out the default config as:

RpcSimulateBundleConfig {
                simulation_bank: Some(SimulationSlotConfig::Commitment(self.commitment())),
                ..RpcSimulateBundleConfig::default()
            }

and the validator runs the following check:

if !(config.pre_execution_accounts_configs.len()
                == rpc_bundle_request.encoded_transactions.len()
                && config.post_execution_accounts_configs.len()
                    == rpc_bundle_request.encoded_transactions.len())
            {
                return Err(Error::invalid_params(
                    "pre/post_execution_accounts_configs must be equal in length to the number of transactions",
                ));
            }

Proposed Solution

in simulate_bundle, we can have it fill out pre_execution_accounts_configs and post_execution_accounts_configs with vec![None; bundle.transactions.len()] or handle that better in setting the defaults when the simulate_bundle RPC call is made in the validator

If no relayer configuration is provided, udp packets always get dropped

Problem

Proposed Solution

error: failed to get `anchor-lang` as a dependency of package `solana-sdk v1.15.0`

help, can't build!

git clone https://github.com/jito-foundation/jito-solana.git
Cloning into 'jito-solana'...
remote: Enumerating objects: 217494, done.
remote: Counting objects: 100% (7644/7644), done.
remote: Compressing objects: 100% (1393/1393), done.
remote: Total 217494 (delta 6414), reused 7319 (delta 6209), pack-reused 209850
Receiving objects: 100% (217494/217494), 294.48 MiB | 7.05 MiB/s, done.
Resolving deltas: 100% (158941/158941), done.
$cd jito-solana/
$cargo build
    Updating git repository `https://github.com/solana-labs/crossbeam`
    Updating crates.io index
error: failed to get `anchor-lang` as a dependency of package `solana-sdk v1.15.0 (/home/e/r/ts/jito-solana/sdk)`

Caused by:
  failed to load source for dependency `anchor-lang`

Caused by:
  Unable to update /home/e/r/ts/jito-solana/anchor/lang

Caused by:
  failed to read `/home/e/r/ts/jito-solana/anchor/lang/Cargo.toml`

Caused by:
  No such file or directory (os error 2)

Make sure relayer + block engine packets are passed through all stages of the TPU

Problem

Proposed Solution

GRPC error in block_engine_stage-stream_error

After switching from the standard solana-validator to over to jito-solana, we are seeing the following error in the logs exactly once per hour.

ERROR solana_metrics::metrics datapoint: block_engine_stage-stream_error count=5i error="grpc error: status: Unknown, message: \"error reading a body from connection: stream error received: not a result of an error\", details: [], metadata: MetadataMap { headers: {} }"

We are not seeing the block_engine_stage-stats.

Version: 1.14.17
Vote account: D7WodK26tETSqyWBW35vvMGfige9afLEyTsm1Pvvropd
Identity: LitxAVo3RnYXD2sX1TyRJxfnKy48amXgyGiysPZjZwE

Command line:

BLOCK_ENGINE_URL=https://amsterdam.mainnet.block-engine.jito.wtf
RELAYER_URL=http://amsterdam.mainnet.relayer.jito.wtf:8100
SHRED_RECEIVER_ADDR=74.118.140.240:1002

solana-validator \
          --identity LitxAVo3RnYXD2sX1TyRJxfnKy48amXgyGiysPZjZwE.json \
          --vote-account vote-account-keypair-ogden.json \
          --rpc-port 8899 \  
          --entrypoint entrypoint.mainnet-beta.solana.com:8001 \
          --limit-ledger-size 80000000 \
          --log /home/validator/log/validator.log \
          --accounts /NVME1/accounts \
          --ledger /NVME0/ledger \
          --dynamic-port-range 9711-9724 \  
          --private-rpc \              
          --full-rpc-api \                                                          
          --enable-rpc-transaction-history \                                        
          --enable-cpi-and-log-storage \
          --no-check-vote-account \     
          --authorized-voter LitxAVo3RnYXD2sX1TyRJxfnKy48amXgyGiysPZjZwE.json \
          --tip-payment-program-pubkey T1pyyaTNZsKv2WcRAB8oVnk93mLJw2XzjtVYqCsaHqt \
          --tip-distribution-program-pubkey 4R3gSG8BpU4t19KYj8CfnbtRpnT8gtk4dvTHxVRwc2r7 \
          --merkle-root-upload-authority GZctHpWXmsZC1YHACTGGcHhYxjdRqQvTpYkb9LMvxDib \
          --commission-bps 800 \
          --relayer-url ${RELAYER_URL} \
          --block-engine-url ${BLOCK_ENGINE_URL} \
          --shred-receiver-address ${SHRED_RECEIVER_ADDR} \
          --no-snapshot-fetch --no-genesis-fetch \
          --geyser-plugin-config ./mygeyser.json

Add more stringent authentication with relayer and block engine.

Problem

Authentication signs a static message, which makes replay of that message easy.

Proposed Solution

We should use a recently signed blockhash when authenticating with both systems.

Tip Program Management Needs Better Concurrency Guarantees

Problem

Right now we don't have the best concurrency control for anything related to the Tip Program. For instance, a thread in BankingStage may reference a transaction that read locks the tip program config, but BundleStage will attempt to execute a transaction that write-locks this to change the tip receiver.

Proposed Solution

Have better locking guarantees around the tip manager program