coinbase / mesh-bitcoin

Bitcoin Mesh API Implementation

Mesh Bitcoin

MESH-BITCOIN IS CONSIDERED ALPHA SOFTWARE. USE AT YOUR OWN RISK.

This project is available open source under the terms of the [Apache 2.0 License](https://opensource.org/licenses/Apache-2.0).

Overview

The mesh-bitcoin repository provides a reference implementation of the Mesh API for Bitcoin in Golang. This repository was created for developers of Bitcoin-like (a.k.a., UTXO) blockchains, who may find it easier to fork this reference implementation than write one from scratch.

Mesh is an open-source specification and set of tools that makes integrating with blockchains simpler, faster, and more reliable. The Mesh API is specified in the OpenAPI 3.0 format.

Requests and responses can be crafted with auto-generated code using Swagger Codegen or OpenAPI Generator, are human-readable (easy to debug and understand), and can be used in servers and browsers.

Features

• Mesh API implementation (both Data API and Construction API)
• UTXO cache for all accounts (accessible using the Mesh /account/balance API)
• Stateless, offline, curve-based transaction construction from any SegWit-Bech32 Address
• Automatically prune bitcoind while indexing blocks
• Reduce sync time with concurrent block indexing
• Use Zstandard compression to reduce the size of data stored on disk without needing to write a manual byte-level encoding

System Requirements

The mesh-bitcoin implementation has been tested on an AWS c5.2xlarge instance. This instance type has 8 vCPU and 16 GB of RAM.

Getting Started

1. Adjust your network settings to the recommended connections.
2. Install and run Docker as directed in the Deployment section below.
3. Run the Testnet:Online command.

Network Settings

To increase the load that mesh-bitcoin can handle, we recommend tunning your OS settings to allow for more connections. On a linux-based OS, you can run these commands (source):

sysctl -w net.ipv4.tcp_tw_reuse=1
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
sysctl -w net.ipv4.tcp_max_syn_backlog=10000
sysctl -w net.core.somaxconn=10000
sysctl -p (when done)

We have not tested mesh-bitcoin with net.ipv4.tcp_tw_recycle and do not recommend enabling it.

You should also modify your open file settings to 100000. This can be done on a linux-based OS with the command: ulimit -n 100000.

Memory-Mapped Files

mesh-bitcoin uses memory-mapped files to persist data in the indexer. As a result, you must run mesh-bitcoin on a 64-bit architecture (the virtual address space easily exceeds 100s of GBs).

If you receive a kernel OOM, you may need to increase the allocated size of swap space on your OS. There is a great tutorial for how to do this on Linux here.

Development

While working on improvements to this repository, we recommend that you use these commands to check your code:

• make deps to install dependencies
• make test to run tests
• make lint to lint the source code
• make salus to check for security concerns
• make build-local to build a Docker image from the local context
• make coverage-local to generate a coverage report

Image Installation

Running these commands will create a Docker image called mesh-bitcoin:latest.

Installing from GitHub

To download the pre-built Docker image from the latest release, run:

curl -sSfL https://raw.githubusercontent.com/coinbase/mesh-bitcoin/master/install.sh | sh -s

Do not try to install mesh-bitcoin using GitHub Packages!

Installing from Source

After cloning this repository, run:

make build-local

Run Docker

Running these commands will start a Docker container in detached mode with a data directory at <working directory>/bitcoin-data and the Mesh API accessible at port 8080.

Required Arguments

MODE - Determines whether Mesh can make outbound connections.

• Type: String
• Options: ONLINE, OFFLINE
• Default: None

NETWORK - The Ethereum network to launch or communicate with.

• Type: String
• Options: MAINNET, ROPSTEN, RINKEBY, GOERLI or TESTNET
• Default: ROPSTEN, but only for backwards compatibility if you use TESTNET

PORT - The port to use for Mesh.

• Type: Integer
• Options: 8080, any compatible port number.
• Default: None

Command Examples

You can run these commands from the command line. If you cloned the repository, you can use the make commands shown after the examples.

Mainnet:Online

Uncloned repo:

docker run -d --rm --ulimit "nofile=100000:100000" -v "$(pwd)/bitcoin-data:/data" -e "MODE=ONLINE" -e "NETWORK=MAINNET" -e "PORT=8080" -p 8080:8080 -p 8333:8333 mesh-bitcoin:latest

Cloned repo:

make run-mainnet-online

Mainnet:Offline

Uncloned repo:

docker run -d --rm -e "MODE=OFFLINE" -e "NETWORK=MAINNET" -e "PORT=8081" -p 8081:8081 mesh-bitcoin:latest

Cloned repo:

make run-mainnet-offline

Testnet:Online

Uncloned repo:

docker run -d --rm --ulimit "nofile=100000:100000" -v "$(pwd)/bitcoin-data:/data" -e "MODE=ONLINE" -e "NETWORK=TESTNET" -e "PORT=8080" -p 8080:8080 -p 18333:18333 mesh-bitcoin:latest

Cloned repo:

make run-testnet-online

Testnet:Offline

Uncloned repo:

docker run -d --rm -e "MODE=OFFLINE" -e "NETWORK=TESTNET" -e "PORT=8081" -p 8081:8081 mesh-bitcoin:latest

Cloned repo:

make run-testnet-offline

Architecture

mesh-bitcoin uses the syncer, storage, parser, and server package from mesh-sdk-go instead of a new Bitcoin-specific implementation of packages of similar functionality. Below you can find an overview of how everything fits together:

Concurrent Block Syncing

To speed up indexing, mesh-bitcoin uses concurrent block processing with a "wait free" design (using the channels function instead of the sleep function to signal which threads are unblocked). This allows mesh-bitcoin to fetch multiple inputs from disk while it waits for inputs that appeared in recently processed blocks to save to disk.

Test the Implementation with the mesh-cli Tool

To validate mesh-bitcoin, install mesh-cli and run one of these commands:

• mesh-cli check:data --configuration-file mesh-cli-conf/testnet/config.json - This command validates that the Data API information in the testnet network is correct. It also ensures that the implementation does not miss any balance-changing operations.
• mesh-cli check:construction --configuration-file mesh-cli-conf/testnet/config.json - This command validates the blockchain’s construction, signing, and broadcasting.
• mesh-cli check:data --configuration-file mesh-cli-conf/mainnet/config.json - This command validates that the Data API information in the mainnet network is correct. It also ensures that the implementation does not miss any balance-changing operations.

Read the How to Test your Mesh Implementation documentation for additional details.

Contributing

You may contribute to the mesh-bitcoin project in various ways:

• Asking Questions
• Providing Feedback
• Reporting Issues

Read our Contributing documentation for more information.

You can also find community implementations for a variety of blockchains in the mesh-ecosystem repository.

Documentation

You can find the Mesh API documentation here.

Check out the Getting Started section to start diving into Mesh.

Related Projects

• mesh-sdk-go — The mesh-sdk-go SDK provides a collection of packages used for interaction with the Mesh API specification.
• mesh-specifications — Much of the SDK code is generated from this repository.
• mesh-cli — Use the mesh-cli tool to test your Mesh API implementation. The tool also provides the ability to look up block contents and account balances.

Sample Implementations

You can find community implementations for a variety of blockchains in the mesh-ecosystem repository.

License

This project is available open source under the terms of the Apache 2.0 License.

© 2022 Coinbase

mesh-bitcoin's Issues

Add performance mode to use unlimited RAM

Describe the solution you'd like
Add performance mode to use unlimited RAM (implementation currently optimized to use <= 16 GB of RAM)

Update README file

Describe the bug
The README file needs some process and clarification updates.

To Reproduce
Steps to reproduce the behavior:

• Open the README file.
• Read it.
• Wonder why it's so hard to understand.

Expected behavior

• Open the README file.
• Read it.
• Understand rosetta-bitcoin better.

Additional context

Fee calculation is wrong

So the preprocess request takes the utxo input operations, but not the outputs. Then it estimates the size and returns it as options to be passed to the metadata endpoint. The metadata endpoint then calculates a fee, which is too low, since the estimated size doesn't account for the outputs. Then the payloads takes the inputs and outputs where the first output is the amount to send and the second output is sum(inputs) - amount_to_send - suggested_fee, resulting in the transaction getting rejected.

CRB

https://github.com/devcontainers/images/blob/main/src/universal/.devcontainer/local-features/setup-user/devcontainer-feature.json

Mainnet node takes 64+GB ram to run effectively

Describe the bug
The btc mainnet fully synced node consumes upward of 64GB ram to run (The recommended ram size on docs is 16GB). We have tried running on small memory machines (<20GB or <40GB) by tweaking the badger config and using swap memory instead but giving any lesser amount of ram causes the indexing/block adding to be extremely slow. Newer blocks would constantly lag behind from raw node by about 10min+. Moreover small memory nodes will constantly crash with "Error EOF" issues and are very slow to get data via Rosetta APIs.

Current badger settings
MaxTableSize = PerformanceMaxTableSize (Also tried the default size but no effect)
MaxLogSize = PerformanceLogSize (Also tried the default size but no effect)
CacheSize = 6GB (This seems to have the biggest impact on reducing memory usage, but smaller memory causes slow indexing)

To Reproduce
Run a fully synced mainnet node with swap disabled.

Expected behavior
Expecting to be able to run the node in small machines.

Additional context
We should either update the docs to reflect the real usage value or some more tuned settings to run this effectively at scale.

Some screenshots of memory usage:
(Note: the sudden drops are Rosetta crashes sadly, even after giving such a big vm)

Improve the diagrams in the README file

Is your feature request related to a problem? Please describe.
The architecture and block syncing images are hard to follow.

Describe the solution you'd like
Nicer-looking images would be nice.

Describe alternatives you've considered
n/a

Additional context
n/a

Thoughts on using ZMQ for indexer syncing

Prob a question for @patrick-ogrady

Any thoughts on using the bitcoin zmq pubsub to sync the indexer instead of the current polling based method?
This could greatly reduce both the sync time and load on the btc rpc as we can just use the messages to populate the indexer db.

I see that zmq was disabled in the bitcoind build so you might have already investigated this.
If you have already considered I'd love to know some thoughts on this.

Unreviewed commit on patrick/initial-commit!

Unreviewed commit(s) COMPARE was merged to patrick/initial-commit by patrick-ogrady.

Comment with appropriate number of 👍 to approve.

---

Reviewed requirements satisfied? No
Ancestors reviewed? Yes

Rosetta keeps crashing frequently with bitcoin EOF error

Rosetta keeps crashing around 388k index and 560k tip with different errors like EOF.

Any idea why this is giving EOF?

WARN    bitcoind stderr bitcoin/node.go:42      closing logger  {"error": "EOF"}
github.com/coinbase/rosetta-bitcoin/bitcoin.logPipe
        /app/src/bitcoin/node.go:42
github.com/coinbase/rosetta-bitcoin/bitcoin.StartBitcoind.func2
        /app/src/bitcoin/node.go:88
golang.org/x/sync/errgroup.(*Group).Go.func1
        /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57
2020-11-27T00:06:04.857Z        WARN    pruner  indexer/indexer.go:284  exiting pruner
github.com/coinbase/rosetta-bitcoin/indexer.(*Indexer).Prune
        /app/src/indexer/indexer.go:284
main.startOnlineDependencies.func3
        /app/src/main.go:108
golang.org/x/sync/errgroup.(*Group).Go.func1
        /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57
2020-11-27T00:06:04.857Z        WARN    bitcoind        bitcoin/node.go:98      sending interrupt to bitcoind
github.com/coinbase/rosetta-bitcoin/bitcoin.StartBitcoind.func3
        /app/src/bitcoin/node.go:98
golang.org/x/sync/errgroup.(*Group).Go.func1
        /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57
2020-11-27T00:06:03.487Z        WARN    bitcoind        bitcoin/node.go:42      closing logger  {"error": "EOF"}
github.com/coinbase/rosetta-bitcoin/bitcoin.logPipe
        /app/src/bitcoin/node.go:42
github.com/coinbase/rosetta-bitcoin/bitcoin.StartBitcoind.func1
        /app/src/bitcoin/node.go:84
golang.org/x/sync/errgroup.(*Group).Go.func1
        /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57

More info:
we were also calling /network/status while this was running which might have caused this error because the EOF seems like a common error in APIs. However if thats the case then the API crashing the bitcoin node seems very odd

Support for including mempool coins in /account/coins result

Is your feature request related to a problem? Please describe.
While adding the /account/coins api support we postponed support for including mempool coins. It would be nice to add support for this now.

Additional context
@patrick-ogrady I'll be happy to do a PR for this (if none is in the works currently). But I don't have much clue about how to go about doing this 😄 . If I can get some hints/tips on this I can get started right away.

Support Multi-Sig Sends

Describe the solution you'd like
Support Multi-Sig Sends

[bug] invalid memory address or nil pointer dereference when syncing

Steps to Reproduce:

1. run make run-testnet-online
2. let the node sync until around block 132800-132900
3. see golang panic (below)

Logs from first time producing error - panic after processing block 132914:

2020-11-09T21:55:46.619Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=000000002d44c2249c29c1130d617b13beb5dd5261f1287e7c1ac3e90a51cdf9 height=132908 version=0x00000002 log2_work=57.187135 tx=724624 date='2013-11-10T20:52:01Z' progress=0.037070 cache=55.6MiB(401169txo)
2020-11-09T21:55:46.627Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=00000000279266a5913eaf293c7cfefd5e8cb5e5f10b1f7ccddc362ea59d83a1 height=132909 version=0x00000002 log2_work=57.187135 tx=724625 date='2013-11-10T20:52:09Z' progress=0.037070 cache=55.6MiB(401170txo)
2020-11-09T21:55:46.637Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=000000003dda93b55aefbc85d6c4ee08a6151f05d51b84bb64386763164f37a1 height=132910 version=0x00000002 log2_work=57.187135 tx=724626 date='2013-11-10T20:52:13Z' progress=0.037070 cache=55.6MiB(401171txo)
2020-11-09T21:55:46.645Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=000000002e4da492f8ab8935ba8c73db05c9d79ab6db94fc844ccca13ffdce7a height=132911 version=0x00000002 log2_work=57.187135 tx=724627 date='2013-11-10T20:52:17Z' progress=0.037070 cache=55.6MiB(401172txo)
2020-11-09T21:55:46.651Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=000000003f55e3178660739e3ea48b277fc7176dcde5cd706813508c5dfb008a height=132912 version=0x00000002 log2_work=57.187135 tx=724628 date='2013-11-10T20:52:23Z' progress=0.037070 cache=55.6MiB(401173txo)
2020-11-09T21:55:46.657Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=000000002c720d115a6948ba146eeb4c7bbd698680b63bbf7818f489cea8d616 height=132913 version=0x00000002 log2_work=57.187136 tx=724629 date='2013-11-10T20:52:26Z' progress=0.037070 cache=55.6MiB(401174txo)
2020-11-09T21:55:46.664Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=000000001c8093b31d0a54a3aee3a5fefdd58ac535382b0cb084ca6222b84ff0 height=132914 version=0x00000002 log2_work=57.187136 tx=724630 date='2013-11-10T20:52:33Z' progress=0.037070 cache=55.6MiB(401175txo)
2020/11/09 21:55:46 Syncing 132711-132874
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xa1aa87]

goroutine 98 [running]:
github.com/coinbase/rosetta-sdk-go/storage.(*BlockStorage).AddBlock(0xc00009d740, 0xe92420, 0xc0000ccb80, 0x0, 0x0, 0x0)
	/go/pkg/mod/github.com/coinbase/[email protected]/storage/block_storage.go:545 +0xe7
github.com/coinbase/rosetta-bitcoin/indexer.(*Indexer).BlockAdded(0xc000626100, 0xe92420, 0xc0000ccb80, 0x0, 0x0, 0x0)
	/app/src/indexer/indexer.go:320 +0xae
github.com/coinbase/rosetta-sdk-go/syncer.(*Syncer).processBlock(0xc017cd23c0, 0xe92420, 0xc0000ccb80, 0xc0011a8f00, 0xc000083d01, 0xc00e87e110)
	/go/pkg/mod/github.com/coinbase/[email protected]/syncer/syncer.go:318 +0x154
github.com/coinbase/rosetta-sdk-go/syncer.(*Syncer).processBlocks(0xc017cd23c0, 0xe92420, 0xc0000ccb80, 0xc000bddd20, 0x2070a, 0xc000029e40, 0x0)
	/go/pkg/mod/github.com/coinbase/[email protected]/syncer/syncer.go:483 +0x12a
github.com/coinbase/rosetta-sdk-go/syncer.(*Syncer).syncRange(0xc017cd23c0, 0xe92420, 0xc0000ccb80, 0x2070a, 0x2, 0x0)
	/go/pkg/mod/github.com/coinbase/[email protected]/syncer/syncer.go:628 +0x42e
github.com/coinbase/rosetta-sdk-go/syncer.(*Syncer).Sync(0xc017cd23c0, 0xe92420, 0xc0000ccb80, 0x20667, 0xffffffffffffffff, 0x0, 0x0)
	/go/pkg/mod/github.com/coinbase/[email protected]/syncer/syncer.go:698 +0x3af
github.com/coinbase/rosetta-bitcoin/indexer.(*Indexer).Sync(0xc000626100, 0xe92420, 0xc0000ccb80, 0x0, 0x0)
	/app/src/indexer/indexer.go:270 +0x2e5
main.startOnlineDependencies.func2(0x0, 0x0)
	/app/src/main.go:104 +0x3c
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc000155b30, 0xc00061e300)
	/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57 +0x59
created by golang.org/x/sync/errgroup.(*Group).Go
	/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:54 +0x66

Logs from second time producing same error but at a different block height - 132809

2020-11-09T22:45:10.990Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=000000001249293c57bea9ad24483dab5f9a8537f372a9ab7b5ed985006095e3 height=132803 version=0x00000002 log2_work=57.187119 tx=724510 date='2013-11-10T20:44:26Z' progress=0.037064 cache=55.6MiB(401065txo)
2020-11-09T22:45:11.001Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=000000001b96a6567aa04e7641dcd937957aa8c584f141e71cbb25a71a724179 height=132804 version=0x00000002 log2_work=57.187119 tx=724511 date='2013-11-10T20:44:28Z' progress=0.037064 cache=55.6MiB(401066txo)
2020-11-09T22:45:11.015Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=00000000042681c96a953abf2d43a0bb4fa445e603c0e6005b103377ec351fd3 height=132805 version=0x00000002 log2_work=57.187119 tx=724512 date='2013-11-10T20:44:34Z' progress=0.037064 cache=55.6MiB(401067txo)
2020/11/09 22:45:11 Syncing 132711-132788
2020-11-09T22:45:11.032Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=00000000126dc0b2a376210d72dbc785c600f2c82ef5e23f2d133e79ae094661 height=132806 version=0x00000002 log2_work=57.187119 tx=724514 date='2013-11-10T20:44:41Z' progress=0.037064 cache=55.6MiB(401066txo)
2020-11-09T22:45:11.046Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=000000000b62a79ff975c7f42918eefc9e30412f004ac74e50fbd336b617d91d height=132807 version=0x00000002 log2_work=57.187119 tx=724515 date='2013-11-10T20:44:47Z' progress=0.037064 cache=55.6MiB(401067txo)
2020-11-09T22:45:11.061Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=00000000284b13d618fad2cb5b51356b88b3e45cf9f1d49788810f89d9c5ea84 height=132808 version=0x00000002 log2_work=57.18712 tx=724516 date='2013-11-10T20:44:54Z' progress=0.037064 cache=55.6MiB(401068txo)
2020-11-09T22:45:11.075Z	DEBUG	bitcoind	bitcoin/node.go:56	UpdateTip: new best=00000000112dd164508c156037048123be1c9c6479dce4b001cde45ee69f9175 height=132809 version=0x00000002 log2_work=57.18712 tx=724517 date='2013-11-10T20:45:03Z' progress=0.037064 cache=55.6MiB(401069txo)
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xa1aa87]

goroutine 86 [running]:
github.com/coinbase/rosetta-sdk-go/storage.(*BlockStorage).AddBlock(0xc0248b9770, 0xe92420, 0xc0000ccb80, 0x0, 0x0, 0x0)
	/go/pkg/mod/github.com/coinbase/[email protected]/storage/block_storage.go:545 +0xe7
github.com/coinbase/rosetta-bitcoin/indexer.(*Indexer).BlockAdded(0xc00055c400, 0xe92420, 0xc0000ccb80, 0x0, 0x0, 0x0)
	/app/src/indexer/indexer.go:320 +0xae
github.com/coinbase/rosetta-sdk-go/syncer.(*Syncer).processBlock(0xc0002386c0, 0xe92420, 0xc0000ccb80, 0xc017ca7540, 0xc000682b01, 0xc027810ef0)
	/go/pkg/mod/github.com/coinbase/[email protected]/syncer/syncer.go:318 +0x154
github.com/coinbase/rosetta-sdk-go/syncer.(*Syncer).processBlocks(0xc0002386c0, 0xe92420, 0xc0000ccb80, 0xc000175d20, 0x206b4, 0x2a, 0x0)
	/go/pkg/mod/github.com/coinbase/[email protected]/syncer/syncer.go:483 +0x12a
github.com/coinbase/rosetta-sdk-go/syncer.(*Syncer).syncRange(0xc0002386c0, 0xe92420, 0xc0000ccb80, 0x206b4, 0x2, 0x0)
	/go/pkg/mod/github.com/coinbase/[email protected]/syncer/syncer.go:628 +0x42e
github.com/coinbase/rosetta-sdk-go/syncer.(*Syncer).Sync(0xc0002386c0, 0xe92420, 0xc0000ccb80, 0x20667, 0xffffffffffffffff, 0x0, 0x0)
	/go/pkg/mod/github.com/coinbase/[email protected]/syncer/syncer.go:698 +0x3af
github.com/coinbase/rosetta-bitcoin/indexer.(*Indexer).Sync(0xc00055c400, 0xe92420, 0xc0000ccb80, 0x0, 0x0)
	/app/src/indexer/indexer.go:270 +0x2e5
main.startOnlineDependencies.func2(0x0, 0x0)
	/app/src/main.go:104 +0x3c
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc000155b30, 0xc00000e9a0)
	/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57 +0x59
created by golang.org/x/sync/errgroup.(*Group).Go
	/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:54 +0x66

Fields zero value update for response of /network/options

Describe the bug
A clear and concise description of what the bug is.
From https://www.rosetta-api.org/docs/models/Allow.html, call_methods and balance_exemptions should be required, but in the source code, they are nil instead zero value which is empty array. We should update call_methods and balance_exemptions to zero value to comply rosetta spec

Some coins returned by account/coins are not spendable

There should be a way to query only spendable coins. Unspendable coins that land in a transaction cause the transaction to fail.

Allow Memo to be added to transaction

Is your feature request related to a problem? Please describe.
Memos can be attached to Bitcoin transactions, but at the moment Rosetta Bitcoin doesn't have code to handle OP_RETURN.

Describe the solution you'd like
Allow a memo to be added to a Bitcoin transaction as an additional output operation.
A very simple idea would be to add a Memo field to Operation.Metadata. Then in the construction_service payloads function, do a check if the output is a memo and use txscript.NullDataScript([]byte(metadata.Memo)) instead of txscript.PayToAddrScript(addr).

Would gladly give this a shot at doing a PR. If you think it should go in a different direction, please share, any guidance more than welcome.

Coinbase restore

Motivation behind --enable-glibc-back-compat

Hey Patrick,

Why is --enable-glibc-back-compat set without doing either a static or a depends build? I can see how dependencies needed in core would have issues depending on what versions of glibc they were built with and the core wouldn't have wrapped functions for those at runtime. Usually this flag is set when dependencies are also built so deterministic builds can be built with a lowest common glibc in mind. What is the minimum version you guys must support and why not allow it to be standard bionic install or otherwise just build via depends?

/mempool/transaction implementation

Describe the solution you'd like
Rosetta API /mempool/transaction implementation

Unreviewed commit on master!

Unreviewed commit(s) COMPARE was merged to master by patrick-ogrady.

Comment with appropriate number of 👍 to approve.

---

Reviewed requirements satisfied? No
Ancestors reviewed? Yes

Add CI test using rosetta-cli to run on each PR

Is your feature request related to a problem? Please describe.
Right now we don't run rosetta-cli with any PR. Would be good to run some basic rosetta cli check, if not the data check which can take time (unless we can have a dummy network which can mimic the actual and be fast)

Describe the solution you'd like
Add CI test using rosetta-cli to run on each PR (likely on a regtest network)

Add support for /account/coins

Is your feature request related to a problem? Please describe.
Rosetta spec added a new api /accounts/coins, rosetta-bitcoin should support it.

Describe the solution you'd like
Add a new endpoint /account/coins based on Rosetta spec
https://www.rosetta-api.org/docs/AccountApi.html#accountcoins

Additional context
Let me know if someone is already working on it. I'd be happy to create a PR if I can get some help.

Mode to sync chain first + index later

I've been playing around with Rosetta for LBRY, and I've noticed I have lots of trouble on the initial sync + index when we start getting some larger blocks come in. I did see a recent discussion around improvements, but I haven't tried them yet (just saw them today).

Logs when this happens - almost looks like it fails to process a block in time, and then logs that block on shutdown along with the http request errors:

ss=0.005344 cache=2073.6MiB(5280827txo) IBD
2020-12-07T12:29:34Z UpdateTip: new best=2e51cbf8dfbd8820dd3bbdd55d1097c625414b7b8cb53190890eddfb135e1ab5 height=837104 version=0x20000000 log2_work=73.933872 txb=602 tx=20026401 date='2020-09-26T17:06:41Z' progress=0.005351 cache=2075.2MiB(5285550txo) IBD
2020-12-07T12:29:50Z UpdateTip: new best=6d6bfe8ca9db93626e88cc19e8ace45738730679995a0cf99bdba003b6cf19bb height=837117 version=0x20000000 log2_work=73.933907 txb=177 tx=20028012 date='2020-09-26T17:39:32Z' progress=0.005353 cache=2075.6MiB(5287003txo) IBD
2020-12-07T12:40:53Z tor: Thread interrupt
2020-12-07T12:40:53Z addcon thread exit
2020-12-07T12:40:53Z torcontrol thread exit
2020-12-07T12:40:53Z opencon thread exit
2020-12-07T12:40:53Z Shutdown: In progress...
2020-12-07T12:40:53Z net thread exit
2020-12-07T13:05:39Z UpdateTip: new best=7bfa4f19d91c3ed3dfa304ee3e82e2d7765bd8335d7802bcc24ad0e19ac02945 height=837138 version=0x20000000 log2_work=73.933963 txb=120 tx=20029990 date='2020-09-26T18:35:35Z' progress=0.005355 cache=57.7MiB(0txo) IBD
2020-12-07T13:05:39Z msghand thread exit
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z scheduler thread interrupt
2020-12-07T13:05:39Z Dumped mempool: 0.001684s to copy, 0.000714s to dump
2020-12-07T13:05:40Z Shutdown: done

Would it make sense to have a mode that lets the blockchain sync up fully before processing the blocks?

Indexer is extremely slow

I'm running rosetta-bitcoin on a 8vCPU 64GB vm with SSD storage backing the data. The bitcoin node is already synced till the tip however the indexer is too slow in catching up.

The indexer starts quite fast in the beginning but it slows down around 300k block and then syncs one block every 1-0.5s which will take forever to catchup with tip.

Any idea why this could be happening?
Attaching a screenshot of logs (You can see badger taking up minutes for compaction and also each block time diff is about a second)

defichain: BlockAdded does not get called (block not found)

Describe the bug
Not sure if it is appropriate to ask here, but hope to get some guidance anyway:
I've been trying to replace bitcoind with defid (defichain is bitcoin fork I believe) but indexer / badger-db does not save blocks / txs;

running online mainnet (after sync)
req:

curl -d '{"network_identifier": {"blockchain": "DFI", "network": "Mainnet"}, "block_identifier": {"index": 1}}' localhost:8080/block

resp:

{"code":4,"message":"Block not found","retriable":false,"details":{"context":
"block not found: {\"index\":1}"}}

some logs around request:

2021-02-15T13:36:56.556Z        DEBUG   bitcoind        bitcoin/node.go:56      UpdateTip: new best=be3a3f6c3979bf28d081f8da482a8f02f457d85fbb47d74273252043f2dc4f10 height=646393 version=0x20000000 log2_work=79.507669 tx=1472991 date='2021-02-15T13:36:56Z' progress=1.000000 cache=0.0MiB(257txo)
2021-02-15T13:38:19.846Z        DEBUG   bitcoind        bitcoin/node.go:56      UpdateTip: new best=b4138c1c7934665a702f44ba3275ef18ae11164093dfdebf17a7a514452ff71a height=646394 version=0x20000000 log2_work=79.507671 tx=1473004 date='2021-02-15T13:38:18Z' progress=1.000000 cache=0.0MiB(266txo)
2021-02-15T13:38:23.321Z        DEBUG   memory  utils/utils.go:62       stats   {"heap (MB)": 4433.795745849609, "max heap (MB)": 4479.408668518066, "stack (MB)": 3.59375, "system (MB)": 5066.585372924805, "garbage collections": 84}
2021-02-15T13:39:07.317Z        DEBUG   server  services/logger.go:57   POST    {"code": 500, "uri": "/block", "time": "10.537418ms"}

reading logs I see that BlockAdded does not get called

// BlockAdded is called by the syncer when a block is added.
func (i *Indexer) BlockAdded(ctx context.Context, block *types.Block) error {
	//FIXME: delete
	fmt.Printf("BlockAdded got called; block: %v\n", block)

I wonder if I missed some configuration change or there is some difference in defichain that should be respected?

To Reproduce
Steps to reproduce the behavior:
defichain uses different ports:

	// mainnetRPCPort = 8332
	// testnetRPCPort = 18332
	mainnetRPCPort = 8554
	testnetRPCPort = 18554

bitcoin-mainnet.conf

datadir=/data/bitcoind
bind=0.0.0.0
rpcbind=0.0.0.0
bantime=15
rpcallowip=0.0.0.0/0
rpcthreads=16
rpcworkqueue=1000
disablewallet=1
txindex=0
port=8555
rpcport=8554
rpcuser=rosetta
rpcpassword=rosetta

# allow manual pruning
prune=1

bitcoin.json

{
 "network": {
  "blockchain": "DFI",
  "network": "mainnet"
 },
 "online_url": "http://localhost:8080",
 "data_directory": "",
 "http_timeout": 10,
 "sync_concurrency": 8,
 "transaction_concurrency": 16,
 "tip_delay": 300,
 "disable_memory_limit": false,
 "log_configuration": false,
 "data": {
  "historical_balance_disabled": true
 }
}

bitcoin/types.go

	Blockchain string = "DFI"
	// Blockchain string = "Bitcoin"
...
	MainnetGenesisBlockIdentifier = &types.BlockIdentifier{
		// Hash: "000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f",
		Hash: "279b1a87aedc7b9471d4ad4e5f12967ab6259926cd097ade188dfcf22ebfe72a",
        }
...
	MainnetCurrency = &types.Currency{
		Symbol:   "DFI",
		Decimals: Decimals,
	}

rosetta-cli-conf/mainnet/config.json

{
  "network": {
    "blockchain": "DFI",
    "network": "mainnet"
  },
  "data_directory": "cli-data",
  "http_timeout": 300,
  "max_retries": 5,
  "max_online_connections": 1000,
  "tip_delay": 1800,
  "compression_disabled": true,
  "memory_limit_disabled": true,
  "data": {
    "initial_balance_fetch_disabled": true,
    "end_conditions": {
      "reconciliation_coverage": {
        "coverage": 0.95,
        "from_tip": true
      }
    }
  }
}

make lint fails

I ran make lint on the latest master and saw this:
node.go:46:14: wrapperFunc: use strings.ReplaceAll method in strings.Replace(str, "\n", "", -1) (gocritic) message := strings.Replace(str, "\n", "", -1

Install script for pre-built Docker image doesn't work

To download the pre-built Docker image from the latest release, run:

curl -sSfL https://raw.githubusercontent.com/coinbase/rosetta-bitcoin/master/install.sh | sh -s

It appears that the latest release is missing the .tar.gz

Indexer shuts down node if prune mode is disabled?

Indexer was still running, but full node was shutdown

2020-12-15T00:09:40.945Z	WARN	pruner	indexer/indexer.go:306	unable to prune lbrycrdd	{"prune height": 868152, "error": "invalid response: 500 Internal Server Error {\"result\":null,\"error\":{\"code\":-1,\"message\":\"Cannot prune blocks because node is not in prune mode.\"},\"id\":1}\n: error pruning blockchain"}
github.com/lbryio/rosetta-lbry/indexer.(*Indexer).Prune
	/app/src/indexer/indexer.go:306
main.startOnlineDependencies.func3
	/app/src/main.go:108
golang.org/x/sync/errgroup.(*Group).Go.func1
	/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57
2020-12-15T00:09:57.558Z	DEBUG	lbrycrdd	lbry/node.go:56	UpdateTip: new best=d66047065472d35f8cdf561b9b652c51d654f762a46fb06ec7828b521b1c4cf8 height=878723 version=0x20000000 log2_work=74.026143 txb=151 tx=26229131 date='2020-12-12T17:54:19Z' progress=0.182868 cache=956.5MiB(2445908txo) IBD
2020-12-15T00:10:13.007Z	DEBUG	lbrycrdd	lbry/node.go:56	UpdateTip: new best=c700e808aa053f8b3254e20b859b80137164e4e4d3c81c8f1f34c9e3ba683422 height=878730 version=0x20000000 log2_work=74.026155 txb=167 tx=26230115 date='2020-12-12T18:04:45Z' progress=0.183342 cache=957.5MiB(2449716txo) IBD
2020-12-15T00:10:29.568Z	DEBUG	lbrycrdd	lbry/node.go:56	UpdateTip: new best=7d764d3b11ecb9d62b745b1eb60d8ad57e327f00d07de4a0cba85a25f0e88547 height=878738 version=0x20000000 log2_work=74.026171 txb=11 tx=26231889 date='2020-12-12T18:24:08Z' progress=0.184238 cache=958.8MiB(2453162txo) IBD
2020-12-15T00:10:49.008Z	DEBUG	lbrycrdd	lbry/node.go:56	UpdateTip: new best=99d6e11b5f4ebd6bb3a4c2211b36f82c93ff51979b2c2ba5c61d8c291c15a8bf height=878744 version=0x20000000 log2_work=74.026184 txb=728 tx=26234057 date='2020-12-12T18:46:58Z' progress=0.185305 cache=960.2MiB(2456898txo) IBD
2020-12-15T00:10:52.745Z	DEBUG	memory	utils/utils.go:62	stats	{"heap (MB)": 3010.34708404541, "max heap (MB)": 8235.030403137207, "stack (MB)": 1.375, "system (MB)": 12090.720947265625, "garbage collections": 245}
2020-12-15T00:10:57.140Z	WARN	pruner	indexer/indexer.go:286	exiting pruner
github.com/lbryio/rosetta-lbry/indexer.(*Indexer).Prune
	/app/src/indexer/indexer.go:286
main.startOnlineDependencies.func3
	/app/src/main.go:108
golang.org/x/sync/errgroup.(*Group).Go.func1
	/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57
2020-12-15T00:10:57.141Z	WARN	lbrycrdd	lbry/node.go:98	sending interrupt to lbrycrdd
github.com/lbryio/rosetta-lbry/lbry.Startlbrycrdd.func3
	/app/src/lbry/node.go:98
golang.org/x/sync/errgroup.(*Group).Go.func1
	/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57
2020-12-15T00:10:57.194Z	DEBUG	lbrycrdd	lbry/node.go:56	tor: Thread interrupt
2020-12-15T00:10:57.215Z	DEBUG	lbrycrdd	lbry/node.go:56	addcon thread exit
2020-12-15T00:10:57.215Z	DEBUG	lbrycrdd	lbry/node.go:56	torcontrol thread exit
2020-12-15T00:10:57.215Z	DEBUG	lbrycrdd	lbry/node.go:56	opencon thread exit
2020-12-15T00:10:57.215Z	DEBUG	lbrycrdd	lbry/node.go:56	net thread exit
2020-12-15T00:10:57.220Z	DEBUG	lbrycrdd	lbry/node.go:56	Shutdown: In progress...
2020-12-15T00:10:58.229Z	DEBUG	lbrycrdd	lbry/node.go:56	msghand thread exit
2020-12-15T00:10:58.248Z	DEBUG	lbrycrdd	lbry/node.go:56	scheduler thread interrupt
2020-12-15T00:10:58.257Z	DEBUG	lbrycrdd	lbry/node.go:56	Dumped mempool: 0.002057s to copy, 0.001324s to dump

P2PK outputs are not indexed under the correct address

Is your feature request related to a problem? Please describe.
P2PK outputs (OP_DATA_65 OP_CHECKSIG, OP_DATA_33 OP_CHECKSIG) are indexed under their output hex value, because bitcoin client does not return an address for those under scriptPubkey when decoding such transactions (https://github.com/coinbase/rosetta-bitcoin/blob/master/bitcoin/client.go#L777). This results in balances being incorrect for addresses that receive P2PK outputs.

Describe the solution you'd like
P2PK outputs should be correctly detected and indexed under their corresponding address to avoid balance discrepancies. as it is the case in all wallets.

Describe alternatives you've considered
An alternative to this could be aggregating balances for both the address and its P2PK hex script when querying the balance, but it would be best to fix this at the indexing level.

Allow max concurrency limits configuration on the indexer

Is your feature request related to a problem? Please describe.
allowing max concurrency config will help in running index on smaller cpu machines and also free up some bitcoin rpc bandwidth for other uses.