Add a check to ensure that each token in the list of flashloan_tokens has sufficient liquidity to enable taking a flashloan. Currently, it is possible to specify flashloan tokens that have no liquidity and/or are not supported on the exchanges which originate the bot's flashloans (e.g. Bancor3). Adding a dynamic check will ensure that users receive an error message (or failure with message) if an invalid flashloan token is specified.

As a temporary measure to help other users who might face similar issues as we did with Brownie and pyyaml==5.4.1, we propose updating our README file with instructions to deal with these problems.

We would like to include the temporary workaround of installing pyyaml==5.4.1 using Conda instead of pip and make a note about potential Brownie-related issues. This would serve as a temporary guide until we find a permanent solution to the problems.

We welcome any suggestions or contributions to improve the clarity and completeness of these instructions.

Issue

Currently, it appears that several Exceptions within our codebase are set to fail silently. This behaviour can hinder debugging efforts and makes it difficult to identify when and where issues are occurring.

Expected Behavior

When an Exception occurs, it should be properly caught, logged, and handled. This logging should include comprehensive details about the exception such as the type of exception, the location (file, method, line number), and any other relevant context information to help with debugging efforts.

Actual Behavior

Presently, some exceptions are caught but fail silently without leaving any trace or logging any details, making it challenging to identify and rectify issues.

Steps to Reproduce

Due to the silent nature of the failures, it's hard to provide specific steps to reproduce. However, a thorough review of the codebase should reveal instances where exceptions are being caught but not properly handled or logged. One such example can be found in the find_arbitrage method for each mode type in the modes/ directory.

Proposed Resolution

The codebase needs to be refactored to ensure that all exceptions are properly handled. This involves:

• Reviewing the entire codebase to identify instances where exceptions are caught but not logged or handled properly.
• Refactoring these sections to include appropriate error handling and logging mechanisms.
• Implementing tests to ensure that exceptions are handled as expected and that all logs are correctly produced.

2023-08-31 13:54:11,234 [fastlane:ERROR] - Error in main loop: list index out of range
Traceback (most recent call last):
File "/root/fastlanebot/main.py", line 546, in
main()
File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/root/fastlanebot/main.py", line 332, in main
run(
File "/root/fastlanebot/main.py", line 522, in run
forks_to_cleanup = delete_tenderly_forks(forks_to_cleanup, mgr)
File "/root/fastlanebot/fastlane_bot/events/utils.py", line 1383, in delete_tenderly_forks
forks_to_keep = [forks_to_cleanup[-1], forks_to_cleanup[-2]]
IndexError: list index out of range

An issue occurs when running multiple bot instances using the same Alchemy API key. The error message indicates:

Error in main loop: local variable 'rate_limiter' referenced before assignment

The cause seems to be that 'rate_limiter' is referenced before being assigned, likely due to shared API key conflicts.

Proposed Resolution:

Define an else condition to set the 'rate_limiter' variable in events/manager.py when it isn't otherwise assigned. This will prevent the variable from being unassigned in the first place, eliminating the error in the case of an API rate limit.

There are two significant issues related to the DEFAULT_MIN_PROFIT_BNT value. Firstly, to prevent potential losses for users in rare instances of high gas prices, we need to increase the DEFAULT_MIN_PROFIT_BNT value to 80. Secondly, in the config/network.py file, a recent bug was introduced that hardcodes the DEFAULT_MIN_PROFIT_BNT value, which was apparently done to enable automated tests to run properly.

Expected behavior

The DEFAULT_MIN_PROFIT_BNT value should be set at 80 to safeguard user profit in the face of fluctuating gas prices.
The DEFAULT_MIN_PROFIT_BNT value should not be hardcoded in config/network.py. Instead, it should be dynamically handled, keeping in view the requirements of automated tests.

Actual behavior

The DEFAULT_MIN_PROFIT_BNT value currently set may not prevent user loss during times of high gas prices.
In the file config/network.py, the DEFAULT_MIN_PROFIT_BNT value is hardcoded, creating a potential issue for the adaptability and flexibility of the bot.

Steps to reproduce

1. Navigate to the file where DEFAULT_MIN_PROFIT_BNT is set, observe the current value.
2. In the config/network.py file, find where the DEFAULT_MIN_PROFIT_BNT value is hardcoded.

Proposed Resolution

Add the DEFAULT_MIN_PROFIT_BNT value to 80 to @click options in main.py.

Refactor the code in config/network.py to avoid hardcoding the DEFAULT_MIN_PROFIT_BNT value. Instead, implement a solution that dynamically fetches this value, thus maintaining the adaptability of the bot while ensuring automated tests can run successfully.

Issue
The variables DEFAULT_MIN_PROFIT_BNT and DEFAULT_MIN_PROFIT both refer to the same value. Having these as separate variables could lead to confusion and potential bugs in the code.

Proposed Solution
We should consolidate these two variables into one, using the name DEFAULT_MIN_PROFIT_BNT for clarity that it is denominated in BNT units, and to maintain consistency across the codebase.

Steps

1. Remove all instances of DEFAULT_MIN_PROFIT.
2. Replace these instances with DEFAULT_MIN_PROFIT_BNT.
3. Ensure that all tests still pass after this change.

Additional context
This change will improve code readability and maintenance.

I had the below failure when running the tests on Ubuntu (they run fine on Mac and Github)

fastlane_bot/tests/nbtest/test_902_ValidatorSlow.py ....F. [100%]

===================================== FAILURES ======================================
___________________________ test_test_validator_bancor_v3 ___________________________

def test_test_validator_bancor_v3():
# ------------------------------------------------------------

    # +
    arb_mode="bancor_v3"

    arb_finder = bot._get_arb_finder(arb_mode)
    finder = arb_finder(
                flashloan_tokens=flashloan_tokens,
                CCm=CCm,
                mode="bothin",
                result=bot.AO_CANDIDATES,
                ConfigObj=bot.ConfigObj,
            )
    r = finder.find_arbitrage()

    arb_opp = r[0]

    validated = bot.validate_optimizer_trades(arb_opp=arb_opp, arb_mode=arb_mode, arb_finder=finder)

  assert arb_opp != validated

E AssertionError: assert (6.985024713911116, USDC-eB48 ... USDT-1ec7\n0xc4771395e138...304634371-1', tknin='USDT-1ec7', amtin=50.154519975185394, tknout='USDC-eB48', amtout=-50.14950501918793, error=None))) != (6.985024713911116, USDC-eB48 ... USDT-1ec7\n0xc4771395e138...304634371-1', tknin='USDT-1ec7', amtin=50.154519975185394, tknout='USDC-eB48', amtout=-50.14950501918793, error=None)))

fastlane_bot/tests/nbtest/test_902_ValidatorSlow.py:253: AssertionError

Body:

This issue proposes the refactoring of bot.py to reduce its current length and complexity. This optimization would enhance maintainability, readability, and scalability of the codebase.

Proposed Steps:

1. Break down large functions into smaller, more manageable units.
2. Eliminate redundancy and repetitiveness where possible.
3. Implement modular programming to enhance code reusability.
4. Apply design patterns where appropriate to simplify complex logic.
5. Perform code reviews to ensure the quality and efficiency of the refactoring process.

The flashloan_tokens list, which is intended to be set via @click options in main.py, is currently being overwritten when bot.py is run. This issue disrupts the intended functionality and disregards the input provided by the user via the @click options.

Expected behavior

After setting the flashloan_tokens list in main.py using the @click options (with a string of token symbols separated by commas), the bot should respect this configuration and utilize these settings when it is run.

Actual behavior

Despite the correct setup of flashloan_tokens in main.py, the bot.py script overwrites this list. As a result, the user-defined input provided through the @click options isn't respected when running the bot.

Steps to reproduce

1. In main.py, set the flashloan_tokens list using @click options.
2. Run the bot.
3. Observe that the bot run process overwrites the flashloan_tokens list and does not use the previously set values. This can be done by printing the flash loan tokens from within the bot.py functionality.

Proposed Resolution

A fix should be implemented in bot.py to prevent it from overwriting the flashloan_tokens list. Instead, bot.py should retrieve the list set by the user in main.py via the @click options. This approach would respect user's input and enhance the functionality of the bot script.

Tracking issue for:

• https://github.com/bancorprotocol/fastlane-bot/security/code-scanning/2

Description

In order to analyze historical events on the Ethereum network and recreate specific scenarios, we propose adding a "Replay Mode" to our application. This mode would allow the application to accept an Ethereum block number as an argument and use Tenderly Forks to simulate the state of the Ethereum blockchain at that specific block.

Features

• Input Ethereum Block Number: The user should be able to specify a block number to recreate the scenario at that point in time.
• Use Tenderly Forks: Leverage Tenderly Forks to accurately recreate the specific state of the Ethereum blockchain.
• Gather Current State of Pools: Directly call specific pool contracts on various exchanges to gather the current state.
• Monitor Contract Events: Starting from the provided block number, iterate through blocks monitoring contract events from block A to B.
• Update Pool States: As contract events are detected, update the state of the pools accordingly.
• Bot Search: Post state updates, allow the bot to search and perform its intended function based on the recreated historical scenario.

Expected Behavior

When in Replay Mode and given a block number, the application should:

• Initialize the Ethereum state using Tenderly Forks.
• Gather the current state of pools from various exchanges.
• Monitor and process contract events from the specified block until the current block.
• Update the state of the pools according to detected events.
• Allow the bot to perform its operations based on the historical scenario.

Use Case

This feature will be especially useful for analyzing past events, understanding how the bot would have reacted in different historical scenarios, and potentially improving its future behavior.

Possible Implementation

1. Integrate with the Tenderly SDK to utilize the Forks feature.
2. Implement functionality to directly call pool contracts and gather the current state.
3. Monitor Ethereum contract events from a given block number and update the pool states accordingly.
4. Extend the bot's capabilities to operate in this simulated historical environment.

Setup

python setup.py install

is this actually necessary? can't you just install then requirements, then go to the directory and run python main.py from there? one reason why I am asking: if you have to there anyway, why install it if not necessary; and if you can run it from anywhere in the system, then main.py is probably not a great name; also fwiw maybe people would like to run multiple versions of the bot, which again may be an issue when install has been run

Legacy

legacy install section should go to the back

Options

• put the example to the top, then the explanations
• order options by importance (eg arb_mode, exchanges, flashloan_tokens, polling_interval and whatever is left)
• provide full range of possible choices for each of the options, or recommended values for numericals
• except for flashloan tokens, provide the top 10 tokens in this document, and link to a list of all tokens

ChangeLog

the change log section here seems to be redundant; either maintain it or link to the change log

Body:

This issue focuses on the need for a GitHub Action that automates the restart of production jobs whenever a new version update is available. This will ensure uninterrupted and consistent operation of the application.

Proposed Steps:

1. Define a GitHub Action that monitors for new version updates.
2. Upon detection of an update, automate the process of restarting production jobs.
3. Incorporate fail-safe measures to handle unsuccessful restart attempts.
4. Test the GitHub Action in a controlled environment before deploying.
5. Update relevant documentation to reflect this new automation process.

Removed logging phrases during cleanup broke some tests. Need to add logging back accordingly.

While setting up our project, we encountered several significant issues related to dependency installations. These issues majorly revolved around the pyyaml==5.4.1 package and the Brownie package.

Primary issue faced was during the installation of the pyyaml==5.4.1 package via pip. As pyyaml is a requirement of Brownie, this hindered our setup process significantly. The temporary workaround involved installing it via Conda instead of pip, but this solution isn't ideal for the long run.

Another significant problem was the time taken for setup. Our team members reported setup times of 2+ hours, which is highly inefficient. The common factor in these lengthy setups was again Brownie, or in some cases, dependencies of Brownie. This issue was especially prevalent for machines that initially only had python 11 installed and required to install a new python version.

To resolve these issues, we are considering the following potential solutions:

1. Implementing our own version of multicall and discontinuing the use of Brownie.
2. Shifting our publication to Conda from PyPi.

Before proceeding with these, we would like to have a thorough discussion and explore all possible solutions.

I saw on the PyDigger site that there is a broken link to github.

In the setup.py file,
url="https://github.com/bancorprotocol/fastlane-bot",

the url is broken,
i am opening a pr to fix it.

During the transition of our codebase from version 1.0 to version 2.0, we lost significant portions of the repository's history lineage. This has resulted in difficulties understanding the evolution of the codebase and retrieving the history of specific changes, which is crucial for debugging, reviewing code changes, and maintaining good development practices.

Expected behavior

The repo's history should contain important changes made throughout the transition from v1.0 to v2.0. Even though a substantial update was made, the previous versions' changes and commits, to some extent, should remain intact and traceable.

Defaults to zero.
Pass in key-value pairs to specify thresholds for each token pair.

Currently, our logging system overwrites previous bot instance logs with those of newer instances. Additionally, we lack a specialized logging mechanism for successful transactions. This lack of permanent logging and special success logs can impede our ability to monitor the performance of our bot instances effectively and debug issues retrospectively.

Expected behavior

Logs from different bot instances should not overwrite each other but coexist for future reference. Furthermore, successful transactions should have their own special logs to allow for a more effortless analysis of the bot's successful operations.

Actual behavior

The system currently overwrites the logs of older bot instances with those of newer ones. Moreover, we lack a dedicated logging mechanism for successful transactions, making it challenging to differentiate between successful and failed transaction logs.

Steps to reproduce

1. Run a bot instance and observe the generated logs.
2. Run a new bot instance and check the logs again.
3. Notice that the logs of the older instance have been overwritten by the newer one.
4. Additionally, observe that there are no specialized logs for successful transactions.

Proposed Resolution

We should introduce a more robust logging mechanism that can persist logs from different bot instances without overwriting. A unique identifier (like timestamp or instance ID) for each bot instance can be prepended to the logs to differentiate between different instances.

Moreover, we should also add a separate logging mechanism that logs successful transactions distinctly. This way, we can filter successful transactions quickly and analyze the success rate of our bot without much hassle.

Description

I propose the addition of a target_tokens setting that will allow users to narrow down the search space to only focus on some specific tokens. This feature would be immensely helpful in various scenarios where the focus needs to be on particular tokens.

Use Case

The target_tokens setting could be used in filtering, search, or any other operation that requires a specific focus on certain tokens. It would enhance the efficiency and speed of the operation by narrowing down the search space.

Proposed Implementation

The target_tokens setting should be implemented as a comma separated string for @click option integration. When specified, the system would only consider pools which contain target tokens and flashloan tokens.

Will be moving some existing stand-alone issues to become sub-issues here. For example, Add filter for dust threshold #72, Add top_n_by_pair option to filter popular pairs by marginal rate. #74, among others...

the CLI of the bot should have a way to read/print the current version number

python main.py --version

==> 2.1

Dynamic target tokens redefined by latest event pools.

On startup, the bot should print out all relevant parameters that have been provided in the options (after checking), in particular

• which mode it is running in
• which flashloan tokens are included
• which exchanges are included
• ...

Also the bot should signal when it is ready to actually do stuff, ie when the startup sequence has finished.

Currently arb opportunities are selected from found opportunities after the optimizer finds them. The problem is that this doesn't weed out false positives that are frequently found by the Optimizer. This change will further process opportunities through the math validation stage, at which point we can much more accurately select an opportunity to continue with.

This change is a general improvement & a requirement for arb routes that don't include Bancor exchanges.

This involves a refactor of the bot.py _handle_trade_instructions function, moving a large segment of its logic upstream.

Description:

Currently, it seems that this mode can sometimes result in losses for certain transactions. This is happening because there is currently no data validation for b3_two_hop transactions by default.

To resolve this issue, need to incorporate data validation for b3_two_hop by default. This can be achieved by adding this mode to the condition check at line 480 in the bot.py file.

Body:

This issue aims to address the removal of top-level files that are no longer in use. The initiative will improve the clarity of the project structure and reduce unnecessary clutter in the repository.

Proposed Steps:

1. Identify and list all unused top-level files.
2. Verify the unused status of each file to prevent inadvertent removal of necessary files.
3. Remove the identified files from the repository.
4. Update any references or dependencies that might still point to the removed files.
5. Commit changes with a clear message indicating the cleanup activity.

Coordinate with the dust threshold option. (see related issue)

Those messages from running test_902_ValidatorSlow probably relate to bad settings in the code

2023-07-26 10:36:44,941 [fastlane:WARNING] - base_exchange must be bancor_v3 for single_triangle_bancor3, setting it to bancor_v3

2023-07-26 10:36:44,941 [fastlane:INFO] - flashloan_tokens for arb_mode=single_triangle_bancor3 will be overwritten.

Body:

This issue proposes the integration of the Bancor V3 Vortex Trigger feature.

Proposed Steps:

1. Activate vortex trigger in smart contract interactions.
2. Add unit tests to validate new feature functionality.

should add cmd pip install fastlane-bot below the first line

This is the run_tests.py that runs the tests

That's how it looks like when it runs on Github (suggesting JupyText is not available)

In other words -- unless someone does run_tests on the local machine first, NBTests will not be converted into .py tests AND THEREFORE THOSE TESTS WILL NOT RUN ON THE SERVER, EVEN THOUGH IT LOOKS LIKE THEY DO.

This is obviously dangerous and should be rectified.

Describe the bug
The --flashloan_tokens flag is currently set as a string, leading to bot failure when this argument is set via the flag.

To Reproduce
Steps to reproduce the behavior:

1. Set --flashloan_tokens flag as a string value.
2. The bot fails.

Expected behavior
--flashloan_tokens flag should accept a list of tokens as an argument without causing the bot to fail.

Proposed Solution
Change the flag type of --flashloan_tokens from a string to a list. Move the default list of tokens from bot.py to main.py for easier readability.

Hi bro, thanks for your great work.
I run you code and meat some problems. All transcations submitted to Flashbots RPC stuck in mempool and failed finally.

Logs below:

2023-04-10 13:43:34,359 - fastlane_bot - INFO - Submitted to Flashbots RPC, response: {'id': 0, 'result': '0x4669397e6819a22100f966f149b838ff9692ac4c1e2b98d4b7a75f75ffcf6899', 'jsonrpc': '2.0'}
2023-04-10 13:45:34,545 - fastlane_bot - INFO - Transaction is stuck in mempool, exception: Transaction HexBytes('0x4669397e6819a22100f966f149b838ff9692ac4c1e2b98d4b7a75f75ffcf6899') is not in the chain after 120 seconds

And Alchemy Mempool Watcher look like below:

Do you have any suggestions?

Description

We currently have GitHub Action automation in place that bumps the patch version for every merged PR. However, for better release management and semantic versioning, we need to update the automation to dynamically bump major, minor, and patch versions based on branch naming or PR tagging conventions.

Proposal

• Major Version Bump: Implement automation to detect a specific branch name pattern (e.g., feature/*) or PR tag (e.g., major) to bump the major version.
• Minor Version Bump: Implement automation to detect a specific branch name pattern (e.g., enhancement/*) or PR tag (e.g., minor) to bump the minor version.
• Patch Version Bump: Continue the current behavior, or implement additional conventions for patch version bumps.

Expected Behavior

• If a PR is merged with a major tag or from a branch matching the major pattern, the major version should be bumped.
• If a PR is merged with a minor tag or from a branch matching the minor pattern, the minor version should be bumped.
• If a PR is merged without any specific tags or from a branch not matching the major/minor patterns, the patch version should be bumped.

Current Behavior

Currently, the automation only bumps the patch version for every merged PR, regardless of the branch naming or PR tagging.

Possible Implementation

We can leverage the existing GitHub Action workflows and extend with additional logic to check for specific branch naming or PR tagging conventions.

Github action is failing due to inability to resolve host.

DAI is no longer supported on Bancor3 exchange, and therefore cannot be flash loaned - however, the current default flashloan_tokens include DAI, which results in a logging error message [fastlane:ERROR] - [TODO CLEAN UP]list index out of range. Removing DAI from the default flashloan_tokens fixes this.