samourai-wallet / samourai-dojo

End-to-end bitcoin wallet backing server for powering Samourai Wallet and other light wallets

Samourai Dojo

This repository is now maintained at https://code.samourai.io/dojo/samourai-dojo

Samourai Dojo is the backing server for Samourai Wallet. Provides HD account & loose addresses (BIP47) balances & transactions lists. Provides unspent output lists to the wallet. PushTX endpoint broadcasts transactions through the backing bitcoind node.

View API documentation

Installation

MyDojo (installation with Docker and Docker Compose)

This setup is recommended to Samourai users who feel comfortable with a few command lines.

It provides in a single command the setup of a full Samourai backend composed of:

• a bitcoin full node only accessible as an ephemeral Tor hidden service,
• the backend database,
• the backend modules with an API accessible as a static Tor hidden service,
• a maintenance tool accessible through a Tor web browser,
• a block explorer (BTC RPC Explorer) accessible through a Tor web browser,
• an optional indexer of Bitcoin addresses (addrindexrs) providing fast and private rescans of HD accounts and loose addresses.

See the documentation for detailed setup instructions.

Manual installation (developers only)

A full manual setup isn't recommended if you don't intend to install a local development environment.

Theory of Operation

Tracking wallet balances via xpub requires conforming to BIP44, BIP49 or BIP84 address derivation scheme. Public keys received by Dojo correspond to single accounts and derive all addresses in the account and change chains. These addresses are at M/0/x and M/1/y, respectively.

Dojo relies on the backing bitcoind node to maintain privacy.

Architecture

Dojo is composed of 3 modules:

• API (/account): web server providing a REST API and web sockets used by Samourai Wallet and Sentinel.
• PushTx (/pushtx): web server providing a REST API used to push transactions on the Bitcoin P2P network.
• Tracker (/tracker): process listening to the bitcoind node and indexing transactions of interest.

API and PushTx modules are able to operate behind a web server (e.g. nginx) or as frontend http servers (not recommended). Both support HTTP or HTTPS (if SSL has been properly configured in /keys/index.js). These modules can also operate as a Tor hidden service (recommended).

Authentication is enforced by an API key and Json Web Tokens.

Implementation Notes

Tracker

• ZMQ notifications send raw transactions and block hashes. Keep track of txids with timestamps, clearing out old txids after a timeout
• On realtime transaction:
  • Query database with all output addresses to see if an account has received a transaction. Notify client via WebSocket.
  • Query database with all input txids to see if an account has sent coins. Make proper database entries and notify via WebSocket.
• On a block notification, query database for txids included and update confirmed height
• On a blockchain reorg (orphan block), previous block hash will not match last known block hash in the app. Need to mark transactions as unconfirmed and rescan blocks from new chain tip to last known hash. Note that many of the transactions from the orphaned block may be included in the new chain.
• When an input spending a known output is confirmed in a block, delete any other inputs referencing that output, since this would be a double-spend.

Import of HD Accounts and data sources

• First import of an unknown HD account relies on a data source (local bitcoind, local indexer or OXT). After that, the tracker will keep everything current.

• Using the local bitcoind (default option) or the local indexer makes you 100% independent of Samourai Wallet's infrastructure and is recommended for better privacy.

• Activation of bitcoind as the data source:

  • Edit /keys/index.js and set "indexer.active" to "local_bitcoind". OXT API will be ignored.

• Activation of the local indexer as the data source:

  • Edit /keys/index.js and set "indexer.active" to "local_indexer". OXT API will be ignored.

• Activation of OXT as the data source (through socks5):

  • Edit /keys/index.js and set "indexer.active" to "third_party_explorer".

• Main drawbacks of using your local bitcoind for these imports:

  • This option is considered as experimental.
  • It doesn't return the full transactional history associated to an HD account or to an address but only transactions having an unspent output controlled by the HD account or the address.
  • It's slightly slower than using the option relying on the OXT API.
  • It may fail to correctly import an existing wallet if this wallet had a large activity.
  • If you use bitcoind and if the import seems to return an invalid balance, you can use the "XPUB rescan" function provided by the maintenance tool. This function allows you to force the minimum number of addresses to be derived and the start index for the derivation.

• Main drawbacks of using your local indexer for these imports:

  • It requires 120GB of additional disk space during its initialization.

As a rule of thumb, we recommend to use the local indexer as the source of imports and to setup your Dojo with a new clean wallet. It increases your privacy and it removes all potential issues with the import of a large wallet.

samourai-dojo's Issues

Shutdown of Dojo and bitcoind

When Dojo is stopped with "dojo.sh stop", we have a delay of 15s between the shutdown of bitcoind and the shutdown of all docker containers.

The sweet spot for the duration of the delay seems to depend on multiple factors (hardware, operations processed by bitcoind before the shutdown, etc).

If the delay is too short, bitcoind will rewind the chain on next startup (takes a lot of time). On the other hand, a long static delay doesn't seem like a good tradeoff in term of UX.

It might be a good idea to replace the static delay by a loop testing if bitcoind is still up.

Dojo with existing full node

Describe the issue
A clear and concise description of what the issue is.

Maybe there's already Docs on this and I'm blind, but... I already have a full node on my laptop. How can I install of Dojo and use my existing full node/blockchain rather than a new Bitcoin install and IBD inside docker?? I'm not too familiar with Docker, except that I used it to set up BTCPayServer.

Screenshots
If applicable, add screenshots to help explain your problem.
Do not submit screenshots containing PII (logins, passwords, keys, IP addresses, etc)

Desktop (please complete the following information):

• OS: [e.g. Ubuntu vX.X] -- currently on Windows 10 with Ubuntu subsytem available, and hoping to use this laptop. Alternatively I have another Ubuntu laptop, but, again, that laptop already has a full node on it that I'd prefer to use instead of resync'ing again
• Dojo Version [e.g. 22]
• Dojo Advanced Setups used (external bitcoind, bitcoind exposed to external apps, etc) -- this is what I'm looking for--Dojo with external bitcoind

Additional context
Add any other context about the problem here.
I'm on Windows 10 (with Docker), but have the Ubuntu subsystem available. Not sure how much that will complicate things.

Suggestion: alternatives for initial block download

Initial block download is still a bottleneck for any Bitcoin project requiring a full synch.

Although this is a somewhat controversial topic (particularly a torrent for a recent synched blockchain), there is a tradeoff. Users could be up and running faster if the process is optimized.

A suggestion is to include on the Dojo script something similar to what the RaspiBlitz project has done, giving users other alternatives:

The script to perform this action is available here

Redirect to Maintenance Tool url

It seems that many users forget to append "/admin" after the onion address when trying to access the Maintenance Tool.

As a small UX improvement, it might be a good idea to redirect "/" to "/admin".

Testnet config

Hello,

is it possible to have two samourai-dojo running on the same machine, but different bitcoin chain?

• one for Bitcoin mainnet
• another one for Bitcoin testnet

Thank you!

Appropriate steps for getting XPUB

I'm following the steps below but not being able to get XPUB information resulting in an error:
Cannot GET /xpub/xpub123456[...]

Steps taken:

1. Get authentication token from a POST request to:
   url: http://[my_onion_address]/v2/auth/login
   post_fields: {"apikey": [my_API_key]}
   Result: Success with Json:
   {'authorizations': {'access_token': 'my_token', 'refresh_token': 'my_token'}}

2. Use the access_token to send a POST request to add HD address:
   url: http://[my_onion_address]/v2/xpub
   post_fields: {"xpub": [my_xpub], "type": "new", "at": [my_at]}
   Result: Success with Json:
   {"status": "ok"}

3. Send a GET request to receive details of HD address:
   url: http://[my_onion_address]/v2/xpub/[my_xpub]/info?at=[my_at]
   Result: Fails - receive message back:
   Cannot GET /xpub/xpub123456[...]/info

Note:
By changing the GET url to:
http://[my_onion_address]/v2/support/xpub/[my_xpub]/info?at=[my_at]

An error json is returned:

{
  "status": "error",
  "error": "Your current access rights do not allow this operation"
}

I would believe I'm probably using the wrong url path for the GET request.

Dojo stops communicating with bitcoind after some time

Sometimes after a day, sometimes after a week of usage, Samourai Wallet can't fetch UTXOs. I check on Dojo and the tracker logs say "Bitcoind rpc unreachable. Retrying in 20 seconds." In fact Bitcoind is running and the rpc is working because other apps (LND, cgminer) are using it no problem. Restarting BItcoind solves the issue. Dojo thinks bitcoind is not ready to use when in fact it is. Side note: There should be a clear indication inside SW which shows if there is connection to Dojo or not. edit: I use Dojo with external bitcoind, running on windows 10.

Support for regtest

It seems dojo does not support regtest network right now. It would be great if dojo can support regtest for better development.

Maintenance Tool - Display error messages

Suggestion from a user: display error messages related to the rescan of an address/xpub in the UI of the Maintenance Tool (instead of the generic "Rescan complete").

On Pruned bitcoind support

I had my dojo connected to a pruned bitcoind on a raspberry pi the other day and it seems to be working just fine. The node container successfully imported all the blocks' headers into mysql db container and I could pair my mobile Samourai-Wallet and Whirlpool-gui with it. Everything works as if my bitcoind was an archiving node.

My question is: If users don't want to import their existing seeds/keys; is it correct to say Dojo supports pruned bitcoind out of the box?

I believe supporting pruned bitcoind will bring more users to the Samourai ecosystem.

Tor Logs

Wondering if these are problems by any chance...

./dojo.sh logs tor output:

tor | Jun 09 22:48:54.836 [warn] CookieAuthFileGroupReadable is set, but will have no effect: you must specify an explicit CookieAuthFile to have it group-readable.

Adding debug=tor to bitcoin.conf, so ./dojo.sh logs bitcoind output:

2019-06-09T23:21:58Z tor: Error connecting to Tor control socket
2019-06-09T23:21:58Z tor: Not connected to Tor control port 127.0.0.1:9051, trying to reconnect

bitcoind exited with code 126

after upgrading to version 1.2.0

nginx is up-to-date
Attaching to bitcoind, nginx, nodejs, db, tor
bitcoind exited with code 126
nginx       | Operation timed out
bitcoind    | wait-for-it.sh: tor:9050 is available after 0 seconds
bitcoind    | ## Start bitcoind #############################
bitcoind    | /restart.sh: line 42: /usr/local/bin/bitcoind: cannot execute binary file: Exec format error
bitcoind exited with code 126

Improve documentation of the API

Documentation of the API should make clear that some endpoints (endpoints using HTTP POST requests) are expecting parameters passed in the body of the request and in URL encoded format.

Suggestion: Use latest bitcoinjs-lib

Hello, I am the current maintainer of BitcoinJS.

Looking through your source, a quick find and replace of bitcoin.HDNode with bitcoin.bip32 should work with the latest version of BitcoinJS Lib.

Also, address generation is much easier with our payments API.

v6 might remove the old address methods since they are redundant at this point.

We also have Psbt support now, so you might be able to encorporate it into some sort of offline signing feature etc. (Though that would probably be client-side and not dojo)

Please let me know if your devs have any questions, you can find me on keybase in the public team bitcoinjslib

Can't pair testnet wallet with testnet dojo

Describe the issue
I'm not able to pair my testnet wallet (version 0.99.87, commit 16926a8) with my testnet dojo (v1.2.0 for testnet), any suggestions on how to troubleshoot? after creating a new wallet and pairing via QR code I check if the xpub or upub in the dojo admin xpub info tab and it says it's not tracked.. I confirmed by sending testnet coin and it's not received in the wallet

Is there something I need to do with my NODE_API_KEY when I build my wallet? or is it loaded in the QR code pairing process?

Desktop (please complete the following information):

• OS: Ubuntu 18.04.3 LTS
• Dojo Version 1.2.0 for testnet
• Dojo Advanced Setups: no changes to config except to change from mainnet to testnet

Additional context
I side-loaded a fresh wallet build from latest develop branch, selected testnet when starting. I then paired to dojo admin pairing QR code.

install tor from archive of source files

A most welcome addition, however, there's one problem. Tor is still being built by cloing the git repo at torproject.com. If access to torproject.com is blocked, this doesn't work. Alternative would be to download the source from a website clone. For example: https://www.theonionrouter.com/tor-0.3.5.8.tar.gz

If the script were changed to pull the source tar instead of cloning from git a potential blocked user could edit the dockerfile and enter a clone he has access to.

Originally posted by @burcakbaskan in #74 (comment)

Bootstapping to Tor

Tor unable to get past 5% Bootstrapping

When running dojo setup, I'm unable to connect to Tor, SSL state failed. however, I'm able to install and run Tor service outside the docker.

I'm not sure what the issue is, as I'm clearly not being blocked by ISP and have multiple Tor services running currently. For some reason this will not allow me to connect.

I've also ensured timezone is accurate. Tried a restarting dojo, and restarting the Tor docker alone. Still unresolved.

Is it possible to add Bridges, as that seems to be the fix on every forum I've read. Issue being, I'm unable to install obfs4 inside the docker in order to add the bridges arg in the torrc.

I saw the previous enhancement and hack, is this a plausible solution for this case?

Tor service is unable to connect to directory server.

When I run the installation (with my own Bitcoin full node) I see the tor service hung up at 5% with the following error messages:

tor | Jul 13 01:24:21.000 [warn] Problem bootstrapping. Stuck at 5%: Connecting to directory server. (No route to host; NOROUTE; count 8; recommendation warn; host FE296180018833AF03A8EACD5894A614623D3F76 at 149.56.45.200:9001)
tor | Jul 13 01:24:21.000 [warn] 7 connections have failed:
tor | Jul 13 01:24:21.000 [warn] 7 connections died in state connect()ing with SSL state (No SSL object)
tor | Jul 13 01:24:21.000 [warn] Problem bootstrapping. Stuck at 5%: Connecting to directory server. (No route to host; NOROUTE; count 9; recommendation warn; host 7EA6EAD6FD83083C538F44038BBFA077587DD755 at 194.109.206.212:443)

It retries every minute or so but never succeeds to connect. Someone has any idea what that could be?
Regards
Tobi

Installation stuck at mysqld installation

During the first installation, the computer crashed and had to be reboot (this happened when bitcoin-d was already verifying blocks).
Then had to restart the installation but now seems to be stuck on the line below.

Add a random delay to broadcast scheduled transactions

Assuming I'm reading the code correctly, scheduled transactions are broadcast immediately after a new block is mined.

For example, imagine we had a transaction scheduled for block 3:

Block height	Time	Transaction broadcast/time
1	+0 minutes	False
2	+10 minutes	False
3	+20 minutes	True, at +20 minutes
4	+30 minutes	False

An attacker could assume that this transaction might have been scheduled. Compare this to:

Block height	Time	Transaction broadcast/time
1	+0 minutes	False
2	+10 minutes	False
3	+20 minutes	True, at +25 minutes
4	+30 minutes	False

Where an attacker could be sure the transaction was not scheduled as it came between two different blocks (and thus was not triggered by a block being mined).

Adding a random time after the target block (say, 0 seconds to 20 minutes) would prevent an attacker from determining that any transactions broadcast between two blocks were not broadcasted by Dojo's implementation of scheduled transactions.

Upgrade not working or version info wrong

I downloaded the version 1.2.0 from GitHub and followed the instruction here https://github.com/Samourai-Wallet/samourai-dojo/blob/develop/doc/DOCKER_setup.md#upgrade.

When running sudo ./dojo.sh upgrade I will get asked it I want to install 1.1.0 - even after copying the files to the correct location.

Whould be an improvement to not having to download and move manually.

Add option --nolog to dojo.sh install command

Describe the solution you'd like
Add an option (--nolog ?) to the install and upgrade commands of dojo.sh preventing dojo logs to be displayed automatically.

heapdump outdated

Please upgrade heapdump, currently used version doesn't build with recent (11 and 12) nodejs versions bnoordhuis/node-heapdump#121

Fresh install of v1.1 fails when bitcoind is deactivated

Tested with v1.1 currently in 'develop' branch.

Steps

• Download archive for branch 'develop' and uncompress on disk,
• Deactivate the installation of bitcoind in /docker/my-dojo/conf/docker-bitcoind.conf.tpl
• Launch the installation with "dojo.sh install"

Results

Pulling bitcoind (samouraiwallet/dojo-bitcoind:1.1.0)...
ERROR: The image for the service you're trying to recreate has been removed. If you continue, volume data could be lost. Consider backing up
 your data before continuing.

Continue with the new image? [yN]y
Pulling bitcoind (samouraiwallet/dojo-bitcoind:1.1.0)...
ERROR: pull access denied for samouraiwallet/dojo-bitcoind, repository does not exist or may require 'docker login'

Expose bitcoind RPC API to the host machine

It seems that some users are willing to use the bitcoind packaged in docker as a source for external apps running on the docker host (e.g;: LN node, etc).

May be we could expose the port of the RPC API to the localhost, with something like

bitcoind:
  ports:
    - "127.0.0.1:28256:28256"

Make upgrade process more resilient to extra characters

If extra characters (like space or tab) are mistakenly added before the name of a property in a docker-.conf file, the upgrade process overwrites the value of the property with the default value set in the docker-.conf.tpl file.

We should make the upgrade process more tolerant to this kind of error.

Improve the transaction cache

Describe the bug
The transaction cache managed by lib.bitcoind-rpc.Transaction ignores the "fees" parameter.
As a result, if the endpoint GET /tx is first called without the fees parameter, next calls to the endpoint will return the cached version even if the "fees" parameter is set

Expected behavior
The fees paraemter should be part of the key used to identify the entries of the transactions cache.

Mysql 5.7 docker image not supported on arm64 devices

I think this is a long standing issue at docker-library/mysql#318

Consider swapping for an alternative build based on architecture switch or using mariadb or other compatible option.

more info: https://hub.docker.com/r/mysql/mysql-server/

External bticoind not tracking new blocks

Describe the issue
Using an external bitcoind I am getting the following output almost all the time, after the dojo catches up to the head block.

$ ./dojo.sh logs tracker
[20191215 16:09:15.977 071 MiB] Processing inactive Mempool (0 transactions)
[20191215 16:09:25.981 071 MiB] Processing inactive Mempool (0 transactions)
[20191215 16:09:34.448 072 MiB] Processing unconfirmed transactions
[20191215 16:09:34.451 072 MiB]  Finished processing unconfirmed transactions 0.0s, 0 tx, 0ms/tx
[20191215 16:09:35.979 072 MiB] Processing inactive Mempool (0 transactions)
[20191215 16:09:45.982 072 MiB] Processing inactive Mempool (0 transactions)
[20191215 16:09:55.979 072 MiB] Processing inactive Mempool (0 transactions)
[20191215 16:10:05.983 072 MiB] Processing inactive Mempool (0 transactions)
[20191215 16:10:15.984 072 MiB] Processing inactive Mempool (0 transactions)
[20191215 16:10:25.980 072 MiB] Processing inactive Mempool (0 transactions)

My Whirlpool is up-to-date initially but then gets further and further out of date because the dojo doesn't find any new blocks! I would expect that the dojo would track the new blocks coming in, even if using and external bitcoind.

Screenshots
My settings in the .conf file are:

#
# INSTALL AND RUN BITCOIND INSIDE DOCKER
#

# Install and run bitcoind inside Docker
# Set this option to 'off' for using a bitcoind hosted outside of Docker (not recommended)
# Value: on | off
BITCOIND_INSTALL=off

# IP address of bitcoind used by Dojo
# Set value to 172.28.1.5 if BITCOIND_INSTALL is set to 'on'
# Type: string
BITCOIND_IP=172.17.0.1

# Port of the RPC API
# Set value to 28256 if BITCOIND_INSTALL is set to 'on'
# Type: integer
BITCOIND_RPC_PORT=8332

# Port exposing ZMQ notifications for raw transactions
# Set value to 9501 if BITCOIND_INSTALL is set to 'on'
# Type: integer
BITCOIND_ZMQ_RAWTXS=18502

# Port exposing ZMQ notifications for block hashes
# Set value to 9502 if BITCOIND_INSTALL is set to 'on'
# Type: integer
BITCOIND_ZMQ_BLK_HASH=18501

These are the logs I get when restarting the dojo.

$ ./dojo.sh restart                                                                                                                                                              
Stopping nginx  ... done                                                                                                                                                                                                                      
Stopping nodejs ... done                                                                                                                                                                                                                      
Stopping tor    ... done                                                                                                                                                                                                                      
Stopping db     ... done                                                                                                                                                                                                                      
Removing nginx  ... done                                                                                                                                                                                                                      
Removing nodejs ... done                                                                                                                                                                                                                      
Removing tor    ... done                                                                                                                                                                                                                      
Removing db     ... done                                                                                                                                                                                                                      
Removing network my-dojo_dojonet                                                                                                                                                                                                              
Removing network my-dojo_dmznet                                                                                                                                                                                                               
Creating network "my-dojo_dojonet" with driver "bridge"                                                                                                                                                                                       
Creating network "my-dojo_dmznet" with driver "bridge"                                                                                                                                                                                        
Creating tor    ... done                                                                                                                                                                                                                      
Creating db  ... done                                                                                                                                                                                                                         
Creating nodejs ... done                                                                                                                                                                                                                      
Creating nginx  ... done                                                                                                                                                                                                                      
$ ./dojo.sh logs tracker                                                                                                                                                         
[20191215 16:22:06.041 072 MiB] Processing inactive Mempool (0 transactions)                                                                                                                                                                  
[20191215 16:22:16.041 072 MiB] Processing inactive Mempool (0 transactions)                                                                                                                                                                  
[20191215 16:22:26.044 072 MiB] Processing inactive Mempool (0 transactions)                                                                                                                                                                  
[20191215 16:22:36.042 072 MiB] Processing inactive Mempool (0 transactions)                                                                                                                                                                  
[20191215 16:22:46.045 072 MiB] Processing inactive Mempool (0 transactions)                                                                                                                                                                  
[20191215 16:22:56.045 072 MiB] Processing inactive Mempool (0 transactions)                                                                                                                                                                  
[20191215 16:23:06.046 072 MiB] Processing inactive Mempool (0 transactions)                                                                                                                                                                  
[20191215 16:23:16.046 072 MiB] Processing inactive Mempool (0 transactions)                                                                                                                                                                  
[20191215 16:23:26.047 072 MiB] Processing inactive Mempool (0 transactions)                                                                                                                                                                  
[20191215 16:23:36.047 072 MiB] Processing inactive Mempool (0 transactions)                                                                                                                                                                  
[20191215 16:24:05.719 045 MiB] Process ID: 140                                                                                                                                                                                               
[20191215 16:24:05.723 045 MiB] Preparing the tracker                                                                                                                                                                                         
[20191215 16:24:05.826 044 MiB] Created a database pool of 10 connections                                                                                                                                                                     
[20191215 16:24:05.856 044 MiB] HTTP server listening on port 8082                                                                                                                                                                            
[20191215 16:24:07.388 044 MiB] Tracker Startup (normal mode)                                                                                                                                                                                 
[20191215 16:24:07.397 044 MiB] Sync 24 blocks                                                                                                                                                                                                
[20191215 16:24:07.459 047 MiB] Beginning to process new block.                                                                                                                                                                               
[20191215 16:24:08.664 086 MiB]  Added block 608214 (id=608232)                                                                                                                                                                               
[20191215 16:24:08.740 088 MiB]  Finished block 608214, 1.2s, 1028 tx, 1ms/tx                                                                                                                                                                 
[20191215 16:24:08.855 090 MiB] Beginning to process new block.                                                                                                                                                                               
[20191215 16:24:10.017 101 MiB]  Added block 608215 (id=608234)                                                                                                                                                                               
[20191215 16:24:10.091 101 MiB]  Finished block 608215, 1.2s, 1214 tx, 1ms/tx                                                                                                                                                                 
[20191215 16:24:10.228 105 MiB] Beginning to process new block.                                                                                                                                                                               
[20191215 16:24:12.821 121 MiB]  Added block 608216 (id=608235)                                                                                                                                                                               
[20191215 16:24:12.944 122 MiB]  Finished block 608216, 2.6s, 2704 tx, 1ms/tx                                                                                                                                                                 
[20191215 16:24:13.056 125 MiB] Beginning to process new block.                                                                                                                                                                               
[20191215 16:24:14.966 133 MiB]  Added block 608217 (id=608236)                                                                                                                                                                               
[20191215 16:24:15.061 134 MiB]  Finished block 608217, 2.0s, 1845 tx, 1ms/tx                                                                                                                                                                 
[20191215 16:24:15.195 138 MiB] Beginning to process new block.                                                                                                                                                                               
[20191215 16:24:17.303 128 MiB]  Added block 608218 (id=608237)                                                                                                                                                                               
[20191215 16:24:17.410 128 MiB]  Finished block 608218, 2.2s, 2175 tx, 1ms/tx                                                                                                                                                                 
[20191215 16:24:17.595 132 MiB] Beginning to process new block.                                                                                                                                                                               
[20191215 16:24:19.298 130 MiB]  Added block 608219 (id=608238)                                                                                                                                                                               
[20191215 16:24:19.403 130 MiB]  Finished block 608219, 1.7s, 1721 tx, 1ms/tx                                                                                                                                                                 
[20191215 16:24:19.554 135 MiB] Beginning to process new block.                               
.
.
.
[20191215 16:24:47.325 151 MiB]  Finished block 608230, 2.6s, 2805 tx, 1ms/tx
[20191215 16:24:47.405 153 MiB] Beginning to process new block.
[20191215 16:24:48.138 131 MiB]  Added block 608231 (id=608250)
[20191215 16:24:48.222 131 MiB]  Finished block 608231, 0.8s, 651 tx, 1ms/tx
[20191215 16:24:48.478 140 MiB] Beginning to process new block.
[20191215 16:24:49.705 126 MiB]  Added block 608232 (id=608251)
[20191215 16:24:49.741 126 MiB]  Finished block 608232, 1.2s, 111 tx, 11ms/tx
[20191215 16:24:49.952 133 MiB] Beginning to process new block.
[20191215 16:24:51.480 140 MiB]  Added block 608233 (id=608252)
[20191215 16:24:51.531 140 MiB]  Finished block 608233, 1.5s, 1115 tx, 1ms/tx
[20191215 16:24:51.723 147 MiB] Beginning to process new block.
[20191215 16:24:53.843 150 MiB]  Added block 608234 (id=608253)
[20191215 16:24:53.907 150 MiB]  Finished block 608234, 2.1s, 1821 tx, 1ms/tx
[20191215 16:24:54.093 157 MiB] Beginning to process new block.
[20191215 16:24:56.173 152 MiB]  Added block 608235 (id=608254)
[20191215 16:24:56.319 140 MiB]  Finished block 608235, 2.2s, 1667 tx, 1ms/tx
[20191215 16:24:56.493 146 MiB] Beginning to process new block.
[20191215 16:24:58.670 154 MiB]  Added block 608236 (id=608255)
[20191215 16:24:58.743 154 MiB]  Finished block 608236, 2.2s, 2115 tx, 1ms/tx
[20191215 16:24:58.908 160 MiB] Beginning to process new block.
[20191215 16:25:01.869 156 MiB]  Added block 608237 (id=608256)
[20191215 16:25:02.047 136 MiB]  Finished block 608237, 3.1s, 3226 tx, 1ms/tx
[20191215 16:25:02.050 136 MiB] Listening for blocks
[20191215 16:25:02.053 136 MiB] Processing unconfirmed transactions
[20191215 16:25:02.055 136 MiB]  Finished processing unconfirmed transactions 0.0s, 0 tx, 0ms/tx
[20191215 16:25:02.058 136 MiB] Listening for pushTx
[20191215 16:25:02.060 136 MiB] Listening for pushTx orchestrator
[20191215 16:25:02.061 136 MiB] Listening for mempool transactions
[20191215 16:25:02.068 136 MiB] Processing active Mempool (0 transactions)
[20191215 16:25:12.070 136 MiB] Processing active Mempool (0 transactions)
[20191215 16:25:22.078 136 MiB] Processing active Mempool (0 transactions)
[20191215 16:25:32.081 136 MiB] Processing active Mempool (0 transactions)
[20191215 16:25:42.083 136 MiB] Processing active Mempool (0 transactions)
[20191215 16:25:52.084 136 MiB] Processing active Mempool (0 transactions)
[20191215 16:26:02.085 070 MiB] Processing active Mempool (0 transactions)
[20191215 16:26:12.086 070 MiB] Processing active Mempool (0 transactions)
[20191215 16:26:22.086 070 MiB] Processing active Mempool (0 transactions)
[20191215 16:26:32.088 070 MiB] Processing active Mempool (0 transactions)

Desktop (please complete the following information):

• OS: Ubuntu v18.04
• Dojo Version 1.3.0
• External bitcoind used in a docker-compose setup

Additional context
I'm using my docker-compose setup for the full node here: https://github.com/dmp1ce/lnd-tor-docker-compose

Bitcoind docker image not supported on ARM devices

x86_64-linux build of Bitcoind is pulled here

Replacing it with https://bitcoincore.org/bin/bitcoin-core-0.18.0/bitcoin-0.18.0-arm-linux-gnueabihf.tar.gz (and commenting out the SHA256 check) works fine (though then that'd obviously break x86_64 support). This needs to be determined at build time.

Is it possible to not use Tor and Nginx?

Hello,

I'm wondering if after commenting out nginx service and tor service in the docker-compose.yml, everything will still work fine?

Or are these two services a hard requirement?

Thank you!

Websocket API

Hi,

is there a websocket API to notify the client that a wallet balance changed?

Thanks!

Change condition determining the Tracker mode?

On startup, the tracker determines its catchup mode by checking the height of the tip on the full node.

This condition doesn't seem adequate for the users who run Dojo on top of an external bitcoind which is already synchronized (=> the tracker switches in "normal" mode and download all the blocks since block 1).

Now that Dojo allows the use of an external bitcoind, it might be better to check the height of the tip in the database in order to determine the catchup mode.

Please add documentation about how to use a pre-existing bitcoind installation

I very much appreciate the beginner friendly setup method.
I suspect there are quite a number of intermediate users who already run a full node and would like to add dojo on top of it without fouling up their current bitcoin demon.

The install script may be able to handle this already (I am wary to test it on my lightning node server without knowing what will happen).
If that is the case then please just update the documentation to explain what will happen if you run the setup script on a Linux machine already running bitcoind.

Great Work!

Docker compose script fails to setup database tables/schemas

After a lot of failures (:cry:) I've concluded that the docker-compose script does not actually setup a operational Dojo stack.

1_db.sql.tpl gets copied to 1_db.sql here and then is never used again. This means that the database is at no point initialized and thus the entity of Dojo doesn't work (Oops, guess I'm the first non-dev user! 👀).

I assume this code should also contain another block above that is roughly:

if [ -f /docker-entrypoint-initdb.d/1_db.sql ]; then
  mysql -h"db" -u"root" -p"$MYSQL_ROOT_PASSWORD" "$MYSQL_DATABASE" < /docker-entrypoint-initdb.d/1_db.sql
  echo "Updated database with 1_db.sql"
fi

Running out of memory because of bitcoind

Hello,

here is my bitcoind config file ~/services/samourai-dojo/docker/my-dojo/conf/docker-bitcoind.conf:

# Max number of connections to network peers
# Type: integer
BITCOIND_MAX_CONNECTIONS=16

# Mempool maximum size in MB
# Type: integer
BITCOIND_MAX_MEMPOOL=1024

# Db cache size in MB
# Type: integer
BITCOIND_DB_CACHE=2024

# Number of threads to service RPC calls
# Type: integer
BITCOIND_RPC_THREADS=6

But still, bitcoind is eating up machine's memory as you can see here:

and here:

Eachtime, it won't sync further than 2016/10 (crash before, and then kind of start over). It's been running for two weeks...

My machine has 4Gb of ram.

Thanks for your inputs!

Bitcoind RPC API is still unreachable

I had successfully installed dojo on macOS Mojave using docker.
Blockchain also started downloading.. but somewhere in between , system restarted on its own.

Now I am getting this is tracker logs:
Bitcoind RPC API is still unreachable ... and it keeps waiting for the blocks.
Any issue with tor? I am not sure whats going on. Any help is appreceated

nginx exited with code 1

connect joinmarket with bitcoind docker image from the dojo.

Is there a way, that the joinmarket project (https://github.com/JoinMarket-Org/joinmarket-clientserver) can use the bitcoin fullnode from the dojo?

May be the wallet is missing ?

dojo.sh bitcoin-cli getwalletinfo
error code -32601
error message:
Method not Found

This means the wallet "function" is disabled.
Would the dojo run with a activated fullnode wallet-function?

Or is it not recomended?

Restrict access to the Maintenance Tool

It"s currently possible to sign in with the AdminKey or the ApiKey.

Access rights to the different features are properly enforced but allowing the authentication with the ApiKey may mislead users who mistakenly used the ApiKey instead of the AdminKey.

Proposed solution: reject authentication to the maintenance tool with the ApiKey

BTCPayServer-Docker integration

Is your feature request related to a problem? Please describe.
We have a similar docker-compose builder system for btpayserver at https://github.com/btcpayserver/btcpayserver-docker, which also deploys a full bitcoin node(and optionally with txindex and unpruned) along with a tor node.

Describe the solution you'd like
It would be nice to either or else find a simplified wayc to deploy dojo through the plugin system inbtcpayserver-docker(some info here: https://github.com/btcpayserver/btcpayserver-docker#how-can-i-customize-the-generated-docker-compose-file). I think it should be fairly easy:

• build and publish docker images for dojo( no need for bitcoind, tor and mysql from what I can tell)
• allow specifying all needed options to connect dojo to custom mysql, bitcoind and tor
• allow auto generation of the auth token stuff
• create docker fragment in BTCPay( started messing about with this at Kukks/btcpayserver-docker@031e647)

Describe alternatives you've considered
Have an option here to interoperate with that setup but I think it would just be overhead here

2nd option (Full install using Docker for MacOS) - Is not working on Mac

I followed all steps in option 2 - (Full install using Docker for MacOS).

1. Docer installed
2. tor is installed
3. Dojo copied on the relevant path.

Stuck at step ./dojo.sh install
It is giving me error "Line 7, too many arguments". It seems this .sh script is not supported.
Any idea whats going on. Did sombebody test it yet?

Dojo with exposed bitcoind ports doesn't start (linux)

Reproduced on OS: Linux & Windows
Reproduced on Dojo:: v1.1 & develop

Steps to reproduce the issue

- stop dojo

- edit docker-bitcoind.conf and set
BITCOIND_RPC_EXTERNAL=on
BITCOIND_RPC_EXTERNAL_IP=127.0.0.1

- start dojo

Error

WARNING: The BITCOIND_RPC_EXTERNAL_IP variable is not set. Defaulting to a blank string.
ERROR: The Compose file '/home/ubuntu/API-Server-refactoring-node8/docker/my-dojo/overrides/bitcoind.rpc.expose.yaml' is invalid because:
services.bitcoind.ports contains an invalid type, it should be a number, or an object
services.bitcoind.ports contains an invalid type, it should be a number, or an object
services.bitcoind.ports contains an invalid type, it should be a number, or an object
services.bitcoind.ports contains an invalid type, it should be a number, or an object
services.bitcoind.ports contains an invalid type, it should be a number, or an object

Wallet connected to Dojo takes a minute to sync

Describe the issue
Dojo takes a lot of time to sync. It started happening only recently. The biggest bottleneck seems to be the import request/command. It takes more than 40 seconds. Why does it import the same addresses again and again though? Shouldn't this be a one time process? When I send a transaction it also takes 40+ secs to show up in the wallet.

Screenshots

Desktop (please complete the following information):

• OS: Ubuntu v18.04.2
• Dojo Version v1.0.0
• Dojo Advanced Setups used: external bitcoind

Additional context
The VM runs on 4 cores and has 2.5 GiB RAM. It uses ~1.8GiB

Error: Content Security Policy: The page’s settings blocked the loading of a resource

I'm getting the following error message while trying to login.

Please note that this happens only in one of the 3 different TOR browsers I've used.

But on this specific machine, I cannot login to the onion address and admin page. I've tried reinstalling the browser but the error persists.

When checking the browser console there are several error messages like the below:

Content Security Policy: The page’s settings blocked the loading of a resource at http://[address].onion/admin/index.js (“script-src”).

Errors in logfile

nginx       | Operation timed out
nginx exited with code 1

and it's not possible to get a connection from the onion browser...

tor         | Aug 21 13:55:20.000 [warn] Onion service connection to [scrubbed] failed (connection refused) 

System is: RPi4
All cointainers are up and running.

Blockchains is synced, locally.

Using a blockchain datadir at the host machine mounted at bitcoind docker

Wondering if there's a way to mount a fully synched blockchain folder at the host machine to the Dojo bitcoind docker image.

This would avoid having to download a duplicate blockchain and potentially save time for users that already have a full node running at the host machine.

Using Dojo with Electrum Personal Server

Describe the issue

This is regarding Electrum Personal Server using Dojo as its bitcoin node:
chris-belcher/electrum-personal-server#148

In dojo bitcoind configuration, seems that RPC can be configured not to go through tor.

So, dojo and eps are actually incompatible at this point, only because of walletdisabled setting.
Is that right?

Desktop (please complete the following information):

• OS: Debian 9
• Dojo Version 1.2.0
• Dojo Advanced Setups used none

Bug - Impossible to restore addresses

@pxsocs is having issues on Telegram with importing new addresses and getting their balance, and I think(?) I might know why:

1. In getWalletInfo() we add all new pubkeys to the DB
2. On the next line, we call walletInfo.ensureAddresses()
3. ensureAddresses() checks every address we're attempting to import against those in the database, it then calls addrService.restoreAddresses() on all the addresses that are NOT in the database

Only issue is, all the addresses ARE in the database because we just imported them in step 1.

Commenting out the line referenced in step 1 resolves this issue. Now multiaddr takes awhile to import new addresses:

And returns their balances:

API always responds {"status":"ok"}

Hello,

my Bitcoind is still syncing. Still, I'm trying to hit the API (exposed by Nginx on port 80 right) but it always responds {"status":"ok"}, and whatever is the URL I curl.

ex: curl -X GET https://myserver/autezdzd ---> {"status":"ok"}

Also, Nginx and Nodejs docker images doesn't log anything.

Any idea?

Thanks a lot :)