gnosis / etherbalance

etherbalance

An ethereum ether and ERC20 token balance monitoring application.

See the example config file, and command line options (cargo run -- --help):

FLAGS:
    -h, --help              Prints help information
        --print-balances    Print balances to stdout on update
    -V, --version           Prints version information

OPTIONS:
        --bind <bind>                          Serve the prometheus metrics at this address [default: 0.0.0.0:8080]
        --config <config>                      Path to the config file
        --node <node>                          Url of the ethereum node to communicate with
        --update-interval <update-interval>    Update the balances in this interval in seconds [default: 100]

The balance information is exposed as a prometheus metric at /metrics. With the example config file:

# HELP etherbalance_balance The ether or IERC20 balance of an ethereum address.
# TYPE etherbalance_balance gauge
etherbalance_balance{address="0x3f5ce5fbfe3e9af3971dd833d26ba9b5c936f0be",address_name="company-wallet",token_name="ether",tag=""} 208965276689158900000000
etherbalance_balance{address="0x3f5ce5fbfe3e9af3971dd833d26ba9b5c936f0be",address_name="company-wallet",token_name="usdc",tag=""} 16234719511522
etherbalance_balance{address="0x3f5ce5fbfe3e9af3971dd833d26ba9b5c936f0be",address_name="company-wallet",token_name="usdt",tag=""} 110017919015055
etherbalance_balance{address="0xbe0eb53f46cd790cd13851d5eff43d12404d33e8",address_name="personal-wallet",token_name="ether",tag="tag"} 2318528098086858200000000
etherbalance_balance{address="0xbe0eb53f46cd790cd13851d5eff43d12404d33e8",address_name="personal-wallet",token_name="usdc",tag="tag"} 49434572690562
etherbalance_balance{address="0xbe0eb53f46cd790cd13851d5eff43d12404d33e8",address_name="personal-wallet",token_name="usdt",tag="tag"} 717944090350919
# HELP etherbalance_last_update Unix time of last update of balances.
# TYPE etherbalance_last_update gauge
etherbalance_last_update 1586257843.7633243

And additionally on stdout with --print-balances:

address personal-wallet ether balance is 2318528098086858312934419
address personal-wallet usdt balance is 717944090350919
address personal-wallet usdc balance is 49434572690562
address company-wallet ether balance is 208965276689158926934455
address company-wallet usdt balance is 110017919015055
address company-wallet usdc balance is 16234719511522

This information is updated in the background with the specified --update-interval. It is not updated on metric request as is custom for Prometheus metrics because we want to avoid overloading the ethereum node.

Development

contracts/IERC20.sol is OpenZeppelin's version of IERC20. It is used to generate contracts/IERC20.json which in turn is used by ethcontract-rs to generate the rust smart contract bindings. This file is included in the repository instead of being generated when building to make building as easy as possible. The smart contract is not expected to change so the file will not need updating.

etherbalance's Issues

Account specific balance alert threshold

Proposal

Allow to specify low balance threshold for each monitored account.

Motivation

We have reached a point where we are attempting to monitor several accounts with this service. Each of these accounts is submitting different types of transactions for several different purposes (i.e. with varying expected transaction fees at varying frequencies).

Example/Use Case

For example solvers submit solutions and spend lots of gas doing so, while the synthetix liquidity provision bot sparsely places orders on an exchange. This implies that that the classification of a "low balance" in one account might be drastically different from that of another. In order to safely ensure a solver is running with sufficient balance over a long weekend we would consider 4 ETH to be a low balance, while a safety net for synthetix bot is 1 ETH.

Use tini?

We should probably wrap this inside tini. We had an issue with our dex-services binary not handling signals well.

cf. https://github.com/gnosis/dex-services/blob/16ef8cf025c41b80318544df9a45890cb764deff/docker/rust/Dockerfile#L41-L44_

Still sending Error alerts

Somehow, this code block (even though it is just a print statement) is still appears as an error in our alerts channel.

I have also checked with devops to see if they have some of their own triggers for such alerts (like any string containing "failed" or something).

etherbalance/src/main.rs

Lines 151 to 161 in c13d061

	Err(err) => {
	success_metric
	.with_label_values(&["failure", &format!("{:#x}", params.address)])
	.inc();
	println!(
	"failed to get balance for address {} token {}: {}",
	&format!("{:#x}", params.address),
	params.token_name,
	err
	);
	}

Batch requests

We are making a lot of requests in a row. It would be more efficient to batch them together. There is a BatchTransport in web3.

Don't alert on Error. Instead record metric.

As proposed in slack, I think we should merely increment a metric for success/failure counts here instead of alerting on every RPC connection timeout like this one

ERROR (pod: dev-dfusion-etherbalance-xdai-64db97c847-rt4vj):
failed to get balance for address 0xf794…5a70 token ether: Transport error: Error(Connect, Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })

Looking at the following code block, it seems we are only actually printing the statement about failing to get balance so I suspect eliminating this error in our alert channel will also involve some communication with DevOps as well.

etherbalance/src/main.rs

Lines 132 to 145 in b6bf3e2

	match params.balance {
	Ok(balance) => balance_metric
	.with_label_values(&[
	params.address_name,
	params.token_name,
	&format!("{:#x}", params.address),
	params.tag,
	])
	.set(u256_to_f64(balance)),
	Err(err) => println!(
	"failed to get balance for address {} token {}: {}",
	params.address, params.token_name, err
	),
	}