All nodes in the cluster are crashed when multiple writing stress testing and 16 million total points written per second about influxdb-cluster

[chengshiwen]

@andyli029 This error seems to be caused by a lot of stress writing sources. Because one stress write source will make the cpu and memory of all data nodes become fully loaded, and multiple stress write sources will only cause all data nodes to go down.

[chengshiwen]

Additional information: 16 million total points written per second under 4 servers with 32C32GB

cat 2stress.sh
/usr/bin/influx-stress insert -r 200s --strict --pps 4000000 --host http://10.90.3.93:8086 >> /root/out_stress.log &
/usr/bin/influx-stress insert -r 200s --strict --pps 4000000 --host http://10.90.3.94:8086 >> /root/out_stress.log &
/usr/bin/influx-stress insert -r 200s --strict --pps 4000000 --host http://10.90.3.95:8086 >> /root/out_stress.log &
/usr/bin/influx-stress insert -r 200s --strict --pps 4000000 --host http://10.90.3.96:8086 >> /root/out_stress.log &

The performance requirements for this write are very challenging.

[andyli029]

@chengshiwen
Maybe I find the core reason: OOM

Mar 14 21:18:03 kernel: Out of memory: Kill process 17944 (influxd) score 847 or sacrifice child
Mar 14 21:18:03 kernel: Killed process 17944 (influxd), UID 0, total-vm:33462440kB, anon-rss:28731572kB, file-rss:0kB, shmem-rss:0kB

the Cluster with replica 2 use more mem than the Cluster with replica 1.

[chengshiwen]

This kind of crash problem is OOMKilled, which is similar to the #19 issue.