Running out of memory because of bitcoind about samourai-dojo

[waterdrop01]

Yes, I confirm it's located under /home/docker-data:

docker volume inspect my-dojo_data-bitcoind
[
    {
        "CreatedAt": "2019-08-29T15:45:19+02:00",
        "Driver": "local",
        "Labels": {
            "com.docker.compose.project": "my-dojo",
            "com.docker.compose.version": "1.24.1",
            "com.docker.compose.volume": "data-bitcoind"
        },
        "Mountpoint": "/home/docker-data/volumes/my-dojo_data-bitcoind/_data",
        "Name": "my-dojo_data-bitcoind",
        "Options": null,
        "Scope": "local"
    }
]

[waterdrop01]

Here is the exact error that cause the bitcoind node to restart and lose all sync history:

2019-08-28T16:49:55.542098260Z 2019-08-28T16:49:22Z UpdateTip: new best=0000000000000000028c5c04a09886f27c6761845583f624d85e7ef6fdf70ceb height=43173
1 version=0x20000000 log2_work=85.328144 tx=158851602 date='2016-09-27T07:22:35Z' progress=0.356932 cache=2809.8MiB(21046402txo)
2019-08-28T16:50:59.884723512Z 2019-08-28T16:50:59Z Socks5() connect to 46.223.150.108:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T16:53:24.843596399Z 2019-08-28T16:53:10Z Socks5() connect to 76.95.175.206:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T16:55:51.728354312Z 2019-08-28T16:55:34Z Socks5() connect to 156.67.138.180:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T17:01:16.670747440Z 2019-08-28T17:00:26Z Socks5() connect to 193.57.121.61:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T17:05:20.942841908Z 2019-08-28T17:05:16Z Socks5() connect to 190.177.75.119:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T17:05:26.617404191Z 2019-08-28T17:05:19Z ping timeout: 1200.006555s
2019-08-28T17:06:25.316118287Z 2019-08-28T17:05:46Z socket sending timeout: 1230s
2019-08-28T17:07:07.518936932Z 2019-08-28T17:06:35Z socket sending timeout: 1251s
2019-08-28T17:08:02.335328420Z 2019-08-28T17:07:06Z Socks5() connect to 106.38.30.169:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T17:08:07.414736540Z 2019-08-28T17:07:34Z socket sending timeout: 1296s
2019-08-28T17:10:27.552881029Z 2019-08-28T17:08:27Z socket sending timeout: 1344s
2019-08-28T17:12:33.760530636Z 2019-08-28T17:11:20Z socket sending timeout: 1492s
2019-08-28T17:14:27.553958236Z 2019-08-28T17:13:25Z socket sending timeout: 1616s
2019-08-28T17:15:06.075160667Z 2019-08-28T17:14:56Z socket sending timeout: 1729s
2019-08-28T17:16:09.305557713Z 2019-08-28T17:15:47Z Socks5() connect to 67.201.161.221:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T17:19:43.357647291Z 2019-08-28T17:19:10Z Socks5() connect to 82.28.229.225:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T17:24:11.311611596Z 2019-08-28T17:23:22Z Socks5() connect to 85.190.0.5:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T17:27:45.036440463Z 2019-08-28T17:27:04Z Socks5() connect to 136.56.39.150:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T17:33:47.195941323Z 2019-08-28T17:32:27Z Socks5() connect to 2601:192:4b40:1337::8816:8333 failed: InterruptibleRecv() timeout or other f$ilure
2019-08-28T17:38:29.497390636Z 2019-08-28T17:37:51Z Socks5() connect to 104.199.238.17:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T17:45:17.888959917Z 2019-08-28T17:43:23Z Socks5() connect to 2001:569:7604:2100:79ea:ea5a:5bc1:344f:8333 failed: InterruptibleRecv() time$ut or other failure
2019-08-28T17:53:28.754037469Z 2019-08-28T17:51:26Z Socks5() connect to 95.244.61.79:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T18:01:00.752129245Z 2019-08-28T17:59:06Z Socks5() connect to 192.100.0.3:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T18:08:22.714664498Z 2019-08-28T18:06:51Z Socks5() connect to 31.13.145.172:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T18:16:31.606654698Z 2019-08-28T18:14:43Z Socks5() connect to 86.91.100.49:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T18:24:50.264159531Z 2019-08-28T18:22:51Z Socks5() connect to 82.64.60.174:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T18:32:09.994512742Z 2019-08-28T18:30:37Z Socks5() connect to 95.33.120.198:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T18:39:01.640836264Z 2019-08-28T18:37:10Z Socks5() connect to 199.59.62.210:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T18:49:11.426815490Z 2019-08-28T18:45:46Z Socks5() connect to 88.198.47.248:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T18:59:25.683247550Z 2019-08-28T18:56:22Z Socks5() connect to mbjkotfqcn5gnsrm.onion:8333 failed: InterruptibleRecv() timeout or other fail
ure
2019-08-28T19:13:50.164366017Z 2019-08-28T19:09:44Z Socks5() connect to 3.105.116.226:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T19:26:55.944384655Z 2019-08-28T19:23:59Z Socks5() connect to 35.161.90.61:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T19:39:49.017255235Z 2019-08-28T19:36:39Z Socks5() connect to 62.210.75.85:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T19:50:11.206827665Z 2019-08-28T19:46:44Z Socks5() connect to iufdq4qdq64m6sge.onion:8333 failed: InterruptibleRecv() timeout or other fail
ure
2019-08-28T20:03:32.196150520Z 2019-08-28T19:58:52Z Socks5() connect to 159.65.8.110:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T20:20:14.686643385Z 2019-08-28T20:16:14Z Socks5() connect to 116.203.60.14:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T20:37:20.327439185Z 2019-08-28T20:32:21Z Socks5() connect to 37.136.46.107:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T20:54:15.249980129Z 2019-08-28T20:49:49Z Socks5() connect to 174.138.51.147:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T21:08:33.069100994Z 2019-08-28T21:04:45Z Socks5() connect to 110.78.180.171:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T21:22:18.714992387Z 2019-08-28T21:18:07Z Socks5() connect to 5.79.120.133:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T21:38:59.297823687Z 2019-08-28T21:35:09Z Socks5() connect to 138.197.145.133:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T21:54:58.188768361Z 2019-08-28T21:51:25Z Socks5() connect to 185.217.241.142:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T22:10:45.406225254Z 2019-08-28T22:06:20Z Socks5() connect to 72.46.58.86:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T22:31:44.355787793Z 2019-08-28T22:25:09Z Socks5() connect to 91.121.160.59:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T22:54:40.715778598Z 2019-08-28T22:49:33Z Socks5() connect to 188.65.212.213:8333 failed: InterruptibleRecv() timeout or other failure
2019-08-28T23:09:21.049338796Z /restart.sh: line 38:    28 Killed                  bitcoind "${bitcoind_options[@]}"

[LaurentMT]

Hey @waterdrop01

I'm not completely sure that your problem is caused by memory. Based on the elements sent, I think the issue might be a lack of disk space available for storing Dojo data (bitcoin blockchain, Dojo database, etc).

If you haven't customized your Docker setup, these data should be stored under a predefined directory (e.g: /var/lib/docker/ directory on linux systems). Can you check the following points:

• what is the directory storing Docker data on your machine?
• how much free disk space is available on the partition storing this directory?

[waterdrop01]

Hi @LaurentMT , thanks for your reply.

Here is the output of df -h:

And this is how docker is started:

ExecStart=/usr/bin/dockerd -g /home/docker-data -H fd:// --containerd=/run/containerd/containerd.sock

So everything looks ok to me...

[LaurentMT]

Can you check if docker data are stored in /home/docker-data? I can't find the -g argument in this doc of dockerd (see --data-root).

[LaurentMT]

What is the environment of your Dojo? (OS, version of Dojo)

[waterdrop01]

Here you are :)

cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS"

./dojo.sh version
Dojo v1.1.0

[LaurentMT]

Thanks. Considering that disk space isn't the issue, I think you're right about RAM being the cause. I think you should try to set the value of BITCOIND_DB_CACHE back to its default values (1024). It's going to slow down the IBD but it's better than a failing IBD.

[waterdrop01]

Ok, I will try that. Editing the file ~/services/samourai-dojo/docker/my-dojo/conf/docker-bitcoind.conf and docker restart bitcoind is enough?

[LaurentMT]

I would recommend to restart Dojo with dojo.sh restart

[waterdrop01]

Ok, I did dojo.sh restart:

./dojo.sh restart
Bitcoin server stopping
Preparing shutdown of dojo. Please wait.
./dojo.sh: line 63: [: -eq: unary operator expected
Force shutdown of Bitcoin server.
Stopping nginx    ... done
Stopping nodejs   ... done
Stopping bitcoind ... done
Stopping db       ... done
Stopping tor      ... done
Removing nginx    ... done
Removing nodejs   ... done
Removing bitcoind ... done
Removing db       ... done
Removing tor      ... done
Removing network my-dojo_dojonet
Removing network my-dojo_dmznet
Network web is external, skipping
Creating network "my-dojo_dojonet" with driver "bridge"
Creating network "my-dojo_dmznet" with driver "bridge"
Creating db  ... done
Creating tor ... done
Creating bitcoind ... done
Creating nodejs   ... done
Creating nginx    ... done

It took quite long time for the whole restart. Then I realize nginx seems to be restarting every few minutes:

docker logs -ft --tail 200 nginx
2019-08-29T15:30:20.485375224Z Operation timed out
2019-08-29T15:36:34.369677894Z Operation timed out
2019-08-29T15:42:49.811220452Z Operation timed out
2019-08-29T15:49:05.713627380Z Operation timed out
2019-08-29T15:55:22.128051853Z Operation timed out
2019-08-29T16:01:38.607317827Z Operation timed out
2019-08-29T16:07:54.921085312Z Operation timed out
2019-08-29T16:14:10.519638547Z Operation timed out
2019-08-29T16:20:26.238413624Z Operation timed out
2019-08-29T16:26:42.729640818Z Operation timed out
2019-08-29T16:32:59.461408790Z Operation timed out
2019-08-29T16:39:16.927905074Z Operation timed out
2019-08-29T16:45:34.153778922Z Operation timed out
2019-08-29T16:51:51.826702005Z Operation timed out
2019-08-29T16:58:09.419132880Z Operation timed out
2019-08-29T17:04:26.836327263Z Operation timed out
2019-08-29T17:10:45.631587805Z Operation timed out
2019-08-29T17:17:04.346046395Z Operation timed out
2019-08-29T17:23:23.819848193Z Operation timed out
2019-08-29T17:29:43.628523775Z Operation timed out
2019-08-29T17:36:02.131707911Z Operation timed out
2019-08-29T17:42:21.290065744Z Operation timed out
2019-08-29T17:48:39.697697523Z Operation timed out
2019-08-29T17:54:59.042067932Z Operation timed out
2019-08-29T18:01:18.193839728Z Operation timed out
2019-08-29T18:07:38.639675438Z Operation timed out
2019-08-29T18:13:57.020911871Z Operation timed out
2019-08-29T18:20:16.460691618Z Operation timed out
2019-08-29T18:26:34.695727080Z Operation timed out
2019-08-29T18:32:53.915226469Z Operation timed out

And looking at bitcoind logs, it says "rolling":

2019-08-29T18:41:43.429599905Z 2019-08-29T18:41:43Z Rolling forward 000000000000000002e16be0a7b22ee5bf0935a94e23bbdccab8d76050e0fad8 (425958)
2019-08-29T18:41:47.177635509Z 2019-08-29T18:41:47Z Rolling forward 000000000000000001565369328b79170649adebd565d17c20354d658c60bbb6 (425959)
2019-08-29T18:41:47.445561946Z 2019-08-29T18:41:47Z Rolling forward 00000000000000000207e7ca7b3e408f641d5169076a3557ee4974e8e73c79eb (425960)
2019-08-29T18:41:48.345759957Z 2019-08-29T18:41:48Z Rolling forward 000000000000000003bd91764f84398ef04fd542a6d32a801ed0c3cbe4be3ff4 (425961)
2019-08-29T18:41:48.547864668Z 2019-08-29T18:41:48Z Rolling forward 0000000000000000030ddfbd127c56d593a4381051d9f1d18e5a71aad1583924 (425962)
2019-08-29T18:41:48.985723895Z 2019-08-29T18:41:48Z Rolling forward 00000000000000000199c0693ef43511bee1c445717248fba25115054ce52444 (425963)
2019-08-29T18:41:49.683192856Z 2019-08-29T18:41:49Z Rolling forward 00000000000000000444335b445483a0eb862d7542872f7bc9d01c9e81ca1b3b (425964)
2019-08-29T18:41:51.263727386Z 2019-08-29T18:41:51Z Rolling forward 000000000000000004c99ddc28747cbd3e4dadb9186dd57451f7c12c2b396ed1 (425965)

Let's wait & see tomorrow...
Thanks again for your help

[LaurentMT]

Closed for inactivity