btcsuite / btcd

An alternative full node bitcoin implementation written in Go (golang)

btcd

btcd is an alternative full node bitcoin implementation written in Go (golang).

This project is currently under active development and is in a Beta state. It is extremely stable and has been in production use since October 2013.

It properly downloads, validates, and serves the block chain using the exact rules (including consensus bugs) for block acceptance as Bitcoin Core. We have taken great care to avoid btcd causing a fork to the block chain. It includes a full block validation testing framework which contains all of the 'official' block acceptance tests (and some additional ones) that is run on every pull request to help ensure it properly follows consensus. Also, it passes all of the JSON test data in the Bitcoin Core code.

It also properly relays newly mined blocks, maintains a transaction pool, and relays individual transactions that have not yet made it into a block. It ensures all individual transactions admitted to the pool follow the rules required by the block chain and also includes more strict checks which filter transactions based on miner requirements ("standard" transactions).

One key difference between btcd and Bitcoin Core is that btcd does NOT include wallet functionality and this was a very intentional design decision. See the blog entry here for more details. This means you can't actually make or receive payments directly with btcd. That functionality is provided by the btcwallet and Paymetheus (Windows-only) projects which are both under active development.

Requirements

Go 1.17 or newer.

Installation

https://github.com/btcsuite/btcd/releases

Linux/BSD/MacOSX/POSIX - Build from Source

• Install Go according to the installation instructions here: http://golang.org/doc/install

• Ensure Go was installed properly and is a supported version:

$ go version
$ go env GOROOT GOPATH

NOTE: The GOROOT and GOPATH above must not be the same path. It is recommended that GOPATH is set to a directory in your home directory such as ~/goprojects to avoid write permission issues. It is also recommended to add $GOPATH/bin to your PATH at this point.

• Run the following commands to obtain btcd, all dependencies, and install it:

$ cd $GOPATH/src/github.com/btcsuite/btcd
$ go install -v . ./cmd/...

• btcd (and utilities) will now be installed in $GOPATH/bin. If you did not already add the bin directory to your system path during Go installation, we recommend you do so now.

Updating

Linux/BSD/MacOSX/POSIX - Build from Source

• Run the following commands to update btcd, all dependencies, and install it:

$ cd $GOPATH/src/github.com/btcsuite/btcd
$ git pull
$ go install -v . ./cmd/...

Getting Started

btcd has several configuration options available to tweak how it runs, but all of the basic operations described in the intro section work with zero configuration.

Linux/BSD/POSIX/Source

$ ./btcd

IRC

• irc.libera.chat
• channel #btcd
• webchat

Issue Tracker

The integrated github issue tracker is used for this project.

Documentation

The documentation is a work-in-progress. It is located in the docs folder.

Release Verification

Please see our documentation on the current build/verification process for all our releases for information on how to verify the integrity of published releases using our reproducible build system.

License

btcd is licensed under the copyfree ISC License.

btcd's Issues

Blockchain download gets stuck

On Win7x64, the blockchain download stopped acquiring new blocks after block 103520 or so. Killing the process and restarting it resumed normal downloading operations.

output feature request.. time in addition to block height

the date of a given block be listed along with the Block height; time gives one a perspective on how far behind one is vs just the raw block #'s themselves

[16:08:57 2013-10-01] [INF] [BTCD] [BMGR] Processed 2 blocks (299 transactions) in the last 48.133185164s - Block height 185877

Optimize SHA256

Benchmarks have shown that the current Go standard library Sha256 interface is roughly 2.5-3x slower than the OpenSSL equivalents. Since a double SHA256 is performed on every block header and every transaction, this is a prime candidate for optimization as even small gains can have a noticeable impact.

Btcd.exe leaves the description field blank when shown in Process Explorer(WinXP 32 SP3).

Support unique logging level per-subsystem

It is often useful to get more detailed debug/trace information about a specific subsystem without doing so for all of the subsystems.

This is also required to be able to implement #15.

btcctl enhancement - set debuglevel dynamically

feature request so davec doesn't forget (or if anybody else jumps on this first nobody will mind):

btcctl debuglevel info
btcctl debuglevel debug
btcctl debuglevel PEER=debug
btcctl debuglevel BMGR=debug

Install problems on Linux

Trying to get a btcd install onto a pristine Ubuntu instance I'm hitting errors:

# github.com/davecgh/go-spew/spew
/usr/lib/go/src/pkg/github.com/davecgh/go-spew/spew/dump.go:211: vt.ConvertibleTo undefined (type reflect.Type has no field or method ConvertibleTo)
/usr/lib/go/src/pkg/github.com/davecgh/go-spew/spew/dump.go:217: vv.Convert undefined (type reflect.Value has no field or method Convert)
github.com/conformal/goleveldb/leveldb
# github.com/conformal/goleveldb/leveldb
/usr/lib/go/src/pkg/github.com/conformal/goleveldb/leveldb/session_util.go:28: method d.file.Num is not an expression, must be called
github.com/conformal/btcjson
# github.com/conformal/btcjson
/usr/lib/go/src/pkg/github.com/conformal/btcjson/jsonapi.go:897: function ends without a return statement

Prior to install I added a few packages:
sudo apt-get install git golang mercurial gcc
I then issued the go get command per the installation instructions:
go get github.com/conformal/btcd
No luck. I'm completely go ignorant but keen to try this out - any suggestions?

IPv6 listen does not work on OpenBSD

As the title suggests, btcd only listens on IPv4 for OpenBSD.

I2P support

There exists a fork of the Satoshi client with native I2P support via the SAM API.

https://bitcointalk.org/index.php?topic=151181.0;all

http://www.i2p2.de/sam

It would be good if btcd could also operate on that network.

Add rate limiting for free transactions accepted to mempool

The mempool needs to have transactions which are less than the minimum relay fee (and are therefore considered free) rate limited to help prevent potential flooding of tiny transactions.

Finish code to handle GroupKey with Tor in address manager

In order to provide full tor support, the GroupKey function needs to be completed for tor per the comments in the code.

Add support for UPnP

In order for things to "just work" for most users, btcd needs to support UPnP for port mapping and obtaining the external IP address that is advertised.

Prevent 'Selfish mining' vulnerability (Eyal & Sirer 2013)

Put simply: "minority can earn revenue in excess of their contribution" as written in this paper: http://arxiv.org/abs/1311.0243

I had a look at the math myself and it does stand. Surprisingly, the return from keeping your mined block secret for a few minutes outweighs the risk of someone beating you to it (with a big enough pool, such as the ones that already exist).

The paper suggests a solution (should take less than 5 minutes):

"We propose a simple, backwards-compatible change to the Bitcoin protocol to
address this problem and raise the threshold. Specically, when a miner learns
of competing branches of the same length, it should propagate all of them, and
choose which one to mine on uniformly at random. In the case of two branches
of length 1, as discussed in Section 4, this would result in half the nodes (in
expectancy) mining on the pool's branch and the other half mining on the other
branch. This yields
Y = 1/2, which in turn yields a threshold of 1/4.
Each miner implementing our change decreases the selsh pool's ability to
increase through control of data propagation. This improvement is independent
of the adoption of the change at other miners, therefore it does not require a
hard fork. This change to the protocol does not introduce new vulnerabilities to
the protocol: Currently, when there are two branches of equal length, the choice
of each miner is arbitrary, e�ectively determined by the network topology and latency. Our change explicitly randomizes this arbitrary choice, and therefore
does not introduce new vulnerabilities." (Eyal & Sirer 2013)

Optimize Transaction Lookups

Profiling has shown that transaction lookups are a prime candidate for optimization as they are currently at just around 28% of the overall block handling time. While it is expected that transaction lookups will take a large portion of the block handling time due to the very nature of what they entail, even minor speedups in this area can have a large impact on the overall performance.

Support for connect only + listening

The current approach of connect where you only connect to the specified peers also disables listening.

This make sense for most use cases, however there is also the case where a user might want to connect only to specific peers as well as listen while controlling the access via their firewall.

One such scenario is a user could setup a tor hidden service while still only connecting to trusted nodes.

Suggested by todd

Using the --tor flag should default listen address to localhost only

Once #33 is implemented, the --tor flag should change the default listener from :8333 (all interfaces) to 127.0.0.1:8333 by default as it will ensure external peers can't connect via the public interface by default. The user will still be able to override this if they also want to provide public access via the --listen flag.

Also, tor does not work over IPv6, so the above default will stop listening on IPv6 by default when --tor is specified as well.

Finish code to handle Tor function in address manager

In order to provide full tor support, the Tor function needs to be completed per the comments in the code.

Optimize NotifyNewTxListener

Profiling has shown that the NotifyNewTxListener code is a prime candidate for optimization as it is currently the third most time consuming operation at over 8% of the time. This is way too high for notifications.

Upon inspection, the issue stems from the fact it is doing a lookup of transactions directly from the database, which is incredibly expensive, instead of simply using them from the block.

Replace --port with --listen

Instead of only accepting a port to listen on, the cli parameters should allow the user to specify listen interfaces.

Examples:

Flags	Comment
--listen=	all interfaces on default port which is changed by --testnet and --regtest (This is the default)
--listen=0.0.0.0	all ipv4 interfaces on default port which is changed by --testnet and --regtest
--listen=::	all ipv6 interfaces on default port which is changed by --testnet and --regtest
--listen=:8333	all interfaces on port 8333
--listen=0.0.0.0:8333	all ipv4 interfaces on port 8333
--listen=[::]:8333	all ipv6 interfaces on port 8333
--listen=127.0.0.1:8333	only ipv4 localhost on port 8333
--listen=[::1]:8333	only ipv6 localhost on port 8333
--listen=:8336	all interfaces on non-standard port 8336
--listen=0.0.0.0:8336	all ipv4 interfaces on non-standard port 8336
--listen=[::]:8336	all ipv6 interfaces on non-standard port 8336
--listen=127.0.0.1:8337 --listen=[::1]:8333	ipv4 localhost on port 8337 and ipv6 localhost on port 8333
--listen=:8333 --listen=:8337	all interfaces on ports 8333 and 8337

btcdHomeDir() doesn't account for plan 9 $home; code is copied into several utils e.g. addblock

maybe this function can be put into btcutil, and take advantage of runtime.GOOS or os/user.Current().HomeDir.

places for data:

• posix os: $HOME/.btcd
• os x: $HOME/Library/App Data/btcd
• windows: %APPDATA%/btcd
• plan 9: $home/btcd?

Add support for --externalip flag

In addition to obtaining the external IP per issue #35 an --externalip flag should be provided to allow the user to override it. The will also enable specifying an onion address as the external IP for proper tor hidden service support.

"installation package is not supported by this processor type" on Windows XP SP3

When trying to install btcd_standalone_0.3.3-alpha_x86.msi

Optimize btcwire Serialization

Latest profiling since the last round of optimizations has shown that 3rd most time consuming process, at 7% of the overall processing time, is being spent doing serialization (mostly for the purposes of hashing).

Performing some benchmarks has shown the binary.Write and binary.Read are significantly slower (due to using reflection and interfaces) than serializing directly into a buffer and writing it. They also create additional internal buffers to do their work, so it results in a lot of extra allocations which means more garbage and GC cycles.

For example, the following benchmark results of serializing a transaction using the aforementioned techniques show it's roughly 10x faster. Considering the sheer number of transactions in the block chain that are serialized and the fact they are serialized more than once per transaction, this adds up.

BenchmarkWriteTx          500000              5437 ns/op
BenchmarkWriteTxNew      5000000               505 ns/op

x509.MarshalECPrivateKey undefined

Using the command
$ go get github.com/conformal/btcd
I get the following error:

dev/go/src/github.com/conformal/btcd/rpcserver.go:393: undefined: x509.MarshalECPrivateKey

Optimize Transaction Hashing

Profiling has shown that transaction hashing is a reasonable candidate for optimization as it is currently takes around 6% of the block handling time. Since hashing is such a large part of block and transaction handling, this is not all that unexpected. However, there are various cases where a transaction is hashed more than once instead of using the cached version to avoid recomputing it.

Create API documentation for RPC commands

Per the title, documentation is needed for the available RPC calls along with example usage.

This includes the legacy RPC commands provided for bitcoind compatibility as well as the new websocket calls. This will be the primary mechanism that callers obtain chain services, so this documentation needs to be clean, written in easy to understand language, and provide working examples of common scenarios.

all import paths are not reachable ;-)

package opensource.conformal.com/go/btcd-internal/addrmgr: unrecognized import path "opensource.conformal.com/go/btcd-internal/addrmgr"

Switch to Common APPDATA directory on Windows

Currently, btcd uses %LOCALAPPDATA% for the application data directory. This means when btcd runs, the chain data, config file, logs, etc all go to a separate directory depending on the user it is running as.

The intent is for btcd to share this information, so saving this information on a per-user basis is not desirable.

This becomes particularly apparent with the recent addition of the Windows service code where the data goes to the service account when running as a service, but to the user's account when running as a user.

Optimize RPC Websockets Notifications

The code is currently doing multiple loops over the transaction inputs and outputs (one per listener). This should be optimized to a single loop.

reverify

an option to re verify the block database; computer crashes and other factors can potentially corrupt the block database in ways that are hard to detect other than re-verifying it. Upon a verify failure obviously it would need to download the corrupt data. Right now btcd trusts everything in the database is accurate and verified.

/? should display help

On windows "btcd /?" launches the application. Most apps use this command line to display the help/command line options page.

Get the external IP to send in the version message

Currently, the IP sent in a version message is the IP of the interface doing the sending. This really should attempt to get the IP from an external source so the advertised IP is the externally accessible IP and not the one that is likely behind a NAT.

A first cut can probably just query a few external sources. Future development could include querying UPnP or NAT-PMP (if available) for the internet gateway.

Add support for minimum transaction fee to memory pool acceptance

The memory pool code has a TODO for disallowing acceptance for transactions that don't have at least a minimum required fee. This ticket is to track its progress.

btcctl addpeer/connect/rmpeer/noconnect feature request

it certainly would be nice to dynamically add a peer or remove a peer or add a connect peer or remove a connect peer without restarting ;-)

getrawtransaction should check mempool

getrawtransaction should check the mempool for transactions as well as the database.

[ERR] BTCD: Unable to start server on :8333: listen tcp6 :8333: .......

Tested it on Windows XP SP3.
The below is the log.
...
C:\Program Files\Btcd Suite\Btcd>btcd.exe
18:01:13 2013-11-27 [WRN] BTCD: open C:\Documents and Settings\user\Application Data\Btcd\btcd.conf:
The system cannot find the file specified.
18:01:13 2013-11-27 [INF] BTCD: Version 0.3.3-alpha
18:01:13 2013-11-27 [INF] BTCD: BMGR: Loading block database from 'C:\Documents and Settings\user\Ap
plication Data\Btcd\data\mainnet\blocks_leveldb'
18:01:13 2013-11-27 [INF] BTCD: BMGR: Block database loaded with block height 0
18:01:13 2013-11-27 [ERR] BTCD: Unable to start server on :8333: listen tcp6 :8333: An address incom
patible with the requested protocol was used.

C:\Program Files\Btcd Suite\Btcd>

Allow btcd to be run as a Windows service

Since btcd is intended to be a long running processes the starts and stop with the computer, it should seamlessly integrate as a service in Windows.

Reconnect to persistent peers should have a backoff

Rather than trying to reconnect to persistent peers every 10 seconds, each failure should cause the retry interval to decay up to some maximum to be determined.

leveldb: unexpected EOF

Cross-posting from here: https://blog.conformal.com/btcd-not-your-moms-bitcoin-daemon/#comment-265

Took almost 5 hours to sync (testnet3) from a local bitcoind instance. The second time I tried to load btcd, it crashed with a "unexpected EOF" error.

➜ ~GOPATH ./bin/btcd -C /Users/shripadk/Bitcoin-btcd/bitcoin.conf
[21:02:27 2013-10-12] [INF] [BTCD] [BMGR] Loading block database from ‘/Users/shripadk/.btcd/data/testnet/blocks_leveldb’
[21:02:27 2013-10-12] [ERR] [BTCD] unexpected EOF

Here is the log output.

➜ ~GOPATH cat ~/.btcd/data/testnet/blocks_leveldb/LOG
=============== Oct 12, 2013 (IST) ===============
21:04:41.482844 db@open opening
21:04:41.483848 journal@recovery F·1
21:04:41.483859 journal@recovery recovering @2791
21:04:41.484765 journal@drop journal-2791 S·104B “length overflows block”

OS: Mac OS X 10.7.2
Go version: go version go1.1.2 darwin/amd64

I was (again) able to reproduce this problem today.
Specifically, it happens when the disk space is low with a running (synced) btcd process. I noticed this when OSX reported that my SSD had "Zero bytes free". I shutdown the process before freeing up some space and restarted it. I get the same error as before: "length overflows block". I'm not really familiar with leveldb but from the logs I think the journal is going into an inconsistent state under this condition.

Optimize most recently used inventory handling

Profiling has shown that manipulating the mru inventory map is currently taking 5% of the block handling time. This is way higher than it should be.

This issue documents my intention to optimize the most recently used inventory handling.

Optimize ECDSA signature verification

The Go standard library ECDSA signature verification is significantly slower than OpenSSL.

go-sqlite3 "Too many errors"

Hi. I am trying to get this running with Go: go1.1.2 darwin/amd64

Running:
go get github.com/conformal/btcd

Gives me the output

$ go get github.com/conformal/btcd
github.com/conformal/btcd
github.com/mattn/go-sqlite3(__DATA/__data): unexpected reloc for dynamic symbol close
github.com/mattn/go-sqlite3(__DATA/__data): unexpected reloc for dynamic symbol access
github.com/mattn/go-sqlite3(__DATA/__data): unexpected reloc for dynamic symbol getcwd
github.com/mattn/go-sqlite3(__DATA/__data): unexpected reloc for dynamic symbol stat$INODE64
github.com/mattn/go-sqlite3(__DATA/__data): unexpected reloc for dynamic symbol fstat$INODE64
github.com/mattn/go-sqlite3(__DATA/__data): unexpected reloc for dynamic symbol ftruncate
github.com/mattn/go-sqlite3(__DATA/__data): unexpected reloc for dynamic symbol fcntl
github.com/mattn/go-sqlite3(__DATA/__data): unexpected reloc for dynamic symbol read
github.com/mattn/go-sqlite3(__DATA/__data): unexpected reloc for dynamic symbol pread
github.com/mattn/go-sqlite3(__DATA/__data): unexpected reloc for dynamic symbol write
github.com/mattn/go-sqlite3(__DATA/__data): unexpected reloc for dynamic symbol pwrite
github.com/mattn/go-sqlite3(__DATA/__data): unexpected reloc for dynamic symbol fchmod
github.com/mattn/go-sqlite3(__DATA/__data): unexpected reloc for dynamic symbol unlink
github.com/mattn/go-sqlite3(__DATA/__data): unexpected reloc for dynamic symbol mkdir
github.com/mattn/go-sqlite3(__DATA/__data): unexpected reloc for dynamic symbol rmdir
github.com/mattn/go-sqlite3(__DATA/__data): unexpected reloc for dynamic symbol mmap
github.com/mattn/go-sqlite3(__DATA/__data): unexpected reloc for dynamic symbol munmap
github.com/mattn/go-sqlite3(__DATA/__data): unhandled relocation for close (type 28 rtype 120)
github.com/mattn/go-sqlite3(__DATA/__data): unhandled relocation for access (type 28 rtype 120)
github.com/mattn/go-sqlite3(__DATA/__data): unhandled relocation for getcwd (type 28 rtype 120)
github.com/mattn/go-sqlite3(__DATA/__data): unhandled relocation for stat$INODE64 (type 28 rtype 120)
too many errors

Any thoughts?

Tor support should not require the use of a proxy for non-Tor connections

It can be desirable to operate a node as a bridge between regular nodes and hidden service nodes without routing the regular connections through Tor.

The --tor option should take an argument to specify the IP address/port of the Tor proxy to be used for hidden service nodes.

RPC Server Credential Improvements

The RPC server should be disabled if no username/password is specified so it's not open by default. Also, if a username is specified without a password, it should prompt for one.

inconsistent formatting of IPv6 address:ports

[15:51:44 2013-10-01] [INF] [BTCD] [BMGR] Syncing to block height 261154 from peer 2001:4801:7819:74:b745:b9d5:ff10:aaec:8333

vs

[15:51:53 2013-10-01] [ERR] [BTCD] [SRVR] Failed to connect to [2001:0:4137:9e76:2c94:31b4:479b:3d06]:8333 via proxy [::1]:9050: TTL expired

Catching up with the blockchain is slow

I have been syncing on and off for over 48 hours and have not caught up with the entire blockchain yet. Both CPU usage (<5%) and Disk usage (1MB/s) appear to be low.

I am running an instance of BitcoinQT on the same box and used --addpeer to try to kick it in the butt. Right now, I appear to be averaging less than a block every 5 seconds. Issue #12 mentions that various speed optimizations are planned.

20:13:59 2013-10-23 [INF] BTCD: BMGR: Processed 2 blocks in the last 11.59s (273
 transactions, height 235912, 2013-05-12 21:08:55 -0400 EDT)
20:14:10 2013-10-23 [INF] BTCD: BMGR: Processed 1 block in the last 10.85s (870
transactions, height 235913, 2013-05-12 21:40:11 -0400 EDT)
20:14:28 2013-10-23 [INF] BTCD: BMGR: Processed 2 blocks in the last 18.21s (846
 transactions, height 235915, 2013-05-12 21:53:58 -0400 EDT)
20:14:47 2013-10-23 [INF] BTCD: BMGR: Processed 2 blocks in the last 18.41s (101
6 transactions, height 235917, 2013-05-12 22:12:24 -0400 EDT)
20:14:58 2013-10-23 [INF] BTCD: BMGR: Processed 3 blocks in the last 11.43s (454
 transactions, height 235920, 2013-05-12 22:30:06 -0400 EDT)
20:15:16 2013-10-23 [INF] BTCD: BMGR: Processed 2 blocks in the last 18.22s (104
4 transactions, height 235922, 2013-05-12 22:47:41 -0400 EDT)
20:15:27 2013-10-23 [INF] BTCD: BMGR: Processed 2 blocks in the last 10.92s (601
 transactions, height 235924, 2013-05-12 23:01:45 -0400 EDT)
20:15:45 2013-10-23 [INF] BTCD: BMGR: Processed 5 blocks in the last 17.46s (103
8 transactions, height 235929, 2013-05-12 23:16:12 -0400 EDT)
20:15:56 2013-10-23 [INF] BTCD: BMGR: Processed 2 blocks in the last 11s (793 tr
ansactions, height 235931, 2013-05-12 23:38:01 -0400 EDT)
20:16:14 2013-10-23 [INF] BTCD: BMGR: Processed 1 block in the last 18.08s (587
transactions, height 235932, 2013-05-12 23:49:04 -0400 EDT)
20:16:27 2013-10-23 [INF] BTCD: BMGR: Processed 2 blocks in the last 12.82s (486
 transactions, height 235934, 2013-05-12 23:56:22 -0400 EDT)

btcd enhancement - syslog support

given plans are to fork in the background, unixy people will find syslog support consistent with other daemons and quite desirable

Finish code for standard input checks in mempool

Rather than only accepting transactions to the memory pool which conform to the chain rules, add additional checks to restrict inputs per the comments in the code.

Half-node example

Hi,

apologies for this not really being an issue, but it might be an interesting exercise to replicate this example using your library packages:

https://raw.github.com/jgarzik/pynode/mini-node/node.py

I'd like to mostly just listen to the bitcoin network with a mind to writing a mining pool in go (obviously a new block solution would have to be submitted back...). There's a lot of code in btcd, and was wondering if you could show an example of a "barebones" approach like jgarzik has?

Also, is it fully the case that btcd does not currently support multiple peers? Do you advise a personal instance of bitcoind to connect to for the moment? Should the peer management be a separate library package?

Thanks for the great code! Have learnt a lot about Bitcoin reading through it!

Cheers,
Donovan.

Replace --rpcport with --rpclisten

Instead of only accepting an RPC port to listen on, the cli parameters should allow the user to specify listen interfaces.

This should behave the same as issue #33.

However, before this is implemented, support for TLS is required.