Database not working/responding and no errors about docker-neo4j

[spacecowboy]

@andyrichardson

By reading the logs I have a strong guess that you are maxing out the
RAM on your droplet. It seems like your instance had to start
swapping, which will render the database unusable:

2017-02-11 15:59:25.918+0000 INFO  [o.n.k.i.DiagnosticsManager] System memory information:
2017-02-11 15:59:25.925+0000 INFO  [o.n.k.i.DiagnosticsManager] Total Physical memory: 992.82 MB
2017-02-11 15:59:25.925+0000 INFO  [o.n.k.i.DiagnosticsManager] Free Physical memory: 186.12 MB
2017-02-11 15:59:25.926+0000 INFO  [o.n.k.i.DiagnosticsManager] Committed virtual memory: 2.04 GB
2017-02-11 15:59:25.926+0000 INFO  [o.n.k.i.DiagnosticsManager] Total swap space: 6.84 GB
2017-02-11 15:59:25.926+0000 INFO  [o.n.k.i.DiagnosticsManager] Free swap space: 6.47 GB
2017-02-11 15:59:25.926+0000 INFO  [o.n.k.i.DiagnosticsManager] JVM memory information:
2017-02-11 15:59:25.926+0000 INFO  [o.n.k.i.DiagnosticsManager] Free  memory: 82.07 MB
2017-02-11 15:59:25.927+0000 INFO  [o.n.k.i.DiagnosticsManager] Total memory: 128.00 MB
2017-02-11 15:59:25.927+0000 INFO  [o.n.k.i.DiagnosticsManager] Max   memory: 128.00 MB

I've done some experiments myself with running Neo4j on low memory VPS
instances and I have concluded that it's not viable to run it on a
machine with just 1GB of RAM (which my own Linode instance has). Assuming
that you also want to run anything other than Neo4j on that machine.

The reason is that besides the Java heap and the Neo4j page cache, you
get certain other memory overheads. Lucence will allocate some of its
own off-heap memory (likely at least 32MB). Java itself seems to
require more memory that the configured heap size.

Investigating the minimum required memory

To view actual memory usage you can use the docker stats
command. Consider for example the following invocation:

docker run --rm --name=neo -e NEO4J_AUTH=none -e NEO4J_dbms_memory_pagecache_size=8M -e NEO4J_dbms_memory_heap_maxSize=100M -p 7474:7474 -p 7687:7687 neo4j:3.1.1-enterprise

Initially you'd suspect this to require about 110MB of RAM right? Well, you'd be wrong:

CONTAINER           CPU %               MEM USAGE / LIMIT       MEM %               NET I/O             BLOCK I/O           PIDS
a8a86807064b        0.35%               370.1 MiB / 23.44 GiB   1.54%               2.62 kB / 690 B     119 MB / 3.01 MB    43

In fact it requires almost 4 times as much as expected. Now let's see what happens if we increase the heap size by 100MB:

docker run --rm --name=neo -e NEO4J_AUTH=none -e NEO4J_dbms_memory_pagecache_size=8M -e NEO4J_dbms_memory_heap_maxSize=200M -p 7474:7474 -p 7687:7687 neo4j:3.1.1-enterprise

Result:

CONTAINER           CPU %               MEM USAGE / LIMIT       MEM %               NET I/O             BLOCK I/O           PIDS
d92e8ecde4a8        0.43%               497.8 MiB / 23.44 GiB   2.07%               7.12 kB / 59.3 kB   119 MB / 3.01 MB    49

Memory usage went up by nearly exactly 100MB. So now, what if we then increase the Neo4j pagecache size to 100MB?

docker run --rm --name=neo -e NEO4J_AUTH=none -e NEO4J_dbms_memory_pagecache_size=100M -e NEO4J_dbms_memory_heap_maxSize=200M -p 7474:7474 -p 7687:7687 neo4j:3.1.1-enterprise

Result:

CONTAINER           CPU %               MEM USAGE / LIMIT       MEM %               NET I/O             BLOCK I/O           PIDS
2474e209f64e        0.38%               472.3 MiB / 23.44 GiB   1.97%               2.62 kB / 690 B     119 MB / 3.01 MB    43

No obvious change on startup, so we can assume that this might
potentially allocate 100MB at a later time then. To test this I
applied some load to the db by repeatedly creating the Movie graph
which caused memory to increase ever so slightly:

CONTAINER           CPU %               MEM USAGE / LIMIT       MEM %               NET I/O             BLOCK I/O           PIDS
77f067c9695c        0.36%               514.2 MiB / 23.44 GiB   2.14%               644 kB / 3.56 MB    120 MB / 4.19 MB    51

Verdict

Looking at the numbers above, this is a rough idea of Neo4j's memory requirements (with some padding to allow for index memory requirements too):

Minimum = 300MB + JavaHeapSize + PageCacheSize

[andyrichardson]

@spacecowboy Thanks for taking the time to experiment with this.

Upgrading to 2GB of RAM looks to have solved the issue. It seemed as though the upgrade hasn't made a difference but in actual fact it is working fine following the upgrade; the network I'm on at the moment (Eduroam - university network) blocks port 7474 so I wasn't able to communicate with the server.

I'm going to close the issue although it would have saved me a lot of time if Neo4j was able to produce errors regarding insufficient RAM. This might be something to think about for future iterations.

---

Upgrading to 2GB RAM
I've upgraded my VPS to 2GB of RAM and an extra core. Running Neo4j alone, this should be more than sufficient - right? Well here's the thing; after doing so and giving Neo4j some time, the browser interface still isn't accessible despite stating otherwise in the console output.

CONTAINER           CPU %               MEM USAGE / LIMIT       MEM %               NET I/O             BLOCK I/O             PIDS
18b0c164b666        0.56%               742.2 MiB / 1.954 GiB   37.10%              788 B / 788 B       71.28 MB / 159.7 kB   31

2017-02-13 13:25:23.236+0000 INFO  [o.n.k.i.DiagnosticsManager] System memory information:
2017-02-13 13:25:23.253+0000 INFO  [o.n.k.i.DiagnosticsManager] Total Physical memory: 1.95 GB
2017-02-13 13:25:23.253+0000 INFO  [o.n.k.i.DiagnosticsManager] Free Physical memory: 715.57 MB
2017-02-13 13:25:23.254+0000 INFO  [o.n.k.i.DiagnosticsManager] Committed virtual memory: 1.93 GB
2017-02-13 13:25:23.254+0000 INFO  [o.n.k.i.DiagnosticsManager] Total swap space: 6.84 GB
2017-02-13 13:25:23.254+0000 INFO  [o.n.k.i.DiagnosticsManager] Free swap space: 6.84 GB
2017-02-13 13:25:23.254+0000 INFO  [o.n.k.i.DiagnosticsManager] JVM memory information:
2017-02-13 13:25:23.255+0000 INFO  [o.n.k.i.DiagnosticsManager] Free  memory: 482.00 MB
2017-02-13 13:25:23.255+0000 INFO  [o.n.k.i.DiagnosticsManager] Total memory: 512.00 MB
2017-02-13 13:25:23.255+0000 INFO  [o.n.k.i.DiagnosticsManager] Max   memory: 512.00 MB

As you can see from the logs above (full debug log here), I've ran Neo4j without changing heap or cache values and all seems fine with regard to memory.

Before upgrade - Inconsistencies
Strangely enough, before upgrading to 2GB RAM, the 1GB RAM database did begin to be accessible after some time but would then become unresponsive later in the day. There was no surge in demand which could have caused it to shut down as I am the only person with access at this time.

[spacecowboy]

@andyrichardson you can bind the 7474 port to whatever you want with docker. That way you can bypass any specific port restrictions you might have.

To send it to port 12524 for example:

docker run --rm --name=neo -p 7474:12524 -p 7687:7687 neo4j:3.1.1-enterprise

Note that the browser assumes the default bolt port to be used (7687). This can be rebound as well but you'll need to edit/disable bolt in the browser settings before you can execute queries.

[andyrichardson]

Great advice, that didn't cross my mind! Thanks again.