dereksdata / hadoop-kickstart

Kickstart snippet for Hadoop RHEL/Centos installation

hadoop-kickstart

Optimised kernel and network settings configuration script for Hadoop installed on RHEL/Centos.

General purpose script for Linux hosts that sets the optimal kernel settings for a Hadoop node maintained to the current Hadoop release. Can be used as part of a kickstart build. Non-destructive and can be safely run multiple times against the same host without fear of misconfiguration.

This is a collaboratively developed script that do the above based upon the current (as of August 2016) Hadoop norms. It is optimised for 10Gbe networking but should still function fine for 1Gbe.

(from http://www.dereksdata.com/single-post/2016/08/18/Safe-Hadoop-Kernel-Configuration-Script)

hadoop-kickstart's Issues

Revalidate Impala disk write cache settings for Impala 2.6

For large impala commits

SetProperty "vm.dirty_background_ratio" "5" "/etc/sysctl.conf"
SetProperty "vm.dirty_ratio" "10" "/etc/sysctl.conf"

Correct kernel.shmma -> kernel.shmmax typo

Verify Tagar's Spark optimisations

to address Spark netty spike in network activity :

net.ipv4.ip_local_port_range = 2000 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_syn_backlog = 4096
net.core.somaxconn = 2048
net.ipv4.tcp_synack_retries = 3
net.ipv4.tcp_fin_timeout = 20
net.ipv4.tcp_max_syn_backlog = 8096
net.core.netdev_max_backlog = 16000
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728
net.core.rmem_default = 262144
net.core.wmem_default = 262144