I used a five node non-autoscaling in a single GCP availability sone. I used small n-1 (3.75GB RAM) Centos 7 images.
sudo vi /etc/sysctl.d/100-elastic.conf
vm.max_map_count=262144
sudo vi /etc/security/limits.conf
# elasticsearch
* soft nofile 65535
* hard nofile 65535
I have five nodes. I am requiring two master eligible nodes.
The first node (external-elastic-one
) has this config after following the security blog:
node.name: external-elastic-one
#node.attr.data.hot: true
#node.attr.data.warm: true
#node.attr.data.cold: true
node.master: true
node.data: true
node.ingest: true
node.ml: false
xpack.ml.enabled: false
discovery.zen.minimum_master_nodes: 2
network.host: 0.0.0.0
discovery.seed_hosts: ["roscigno-ilm-sr76", "roscigno-ilm-q8dx"]
cluster.initial_master_nodes: ["external-elastic-one", "external-elastic-two"]
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
and the second master eligible:
node.name: external-elastic-two
#node.attr.data.hot: true
#node.attr.data.warm: true
#node.attr.data.cold: true
node.master: true
node.data: true
node.ingest: true
node.ml: false
xpack.ml.enabled: false
discovery.zen.minimum_master_nodes: 2
network.host: 0.0.0.0
discovery.seed_hosts: ["roscigno-ilm-sr76", "roscigno-ilm-q8dx"]
cluster.initial_master_nodes: ["external-elastic-one", "external-elastic-two"]
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
node 5 is not master eligible or an ingest node. It is a warm node (node.attr.data.warm: true
)
node.name: external-elastic-five
node.attr.data.warm: true
node.master: false
node.data: true
node.ingest: false
node.ml: false
xpack.ml.enabled: false
discovery.zen.minimum_master_nodes: 2
network.host: 0.0.0.0
discovery.seed_hosts: ["roscigno-ilm-sr76", "roscigno-ilm-q8dx"]
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
Note that I am running Kibana on localhost
and not the hostname as I am going to use an SSH tunnel and map port 8601
on my laptop to port 5601
on the GCE VM
#server.host: "roscigno-ilm-sr76"
server.host: "localhost"
elasticsearch.hosts: ["http://roscigno-ilm-sr76:9200", "http://roscigno-ilm-q8dx:9200", "http://roscigno-ilm-6tml:9200"]
elasticsearch.username: "kibana"
elasticsearch.password: "kibanapassword"
# Disable reporting because of a Chromium issue on my VM
xpack.reporting.enabled: false
This uses the Google gcloud
command to ssh to a VM in my GCE projject and port forward port 8601
on my laptop to port 5601
on the remote VM. See the Kibana config above, I have Kibana listening on the loopback (localhost) address.
gcloud compute ssh --project elastic-customer-success \
--zone us-central1-a roscigno-ilm-sr76 \
-- -L 8601:localhost:5601
If you need to scp files from a GCE VM to your laptop:
gcloud compute scp \
dan_roscigno@roscigno-ilm-sr76:/home/dan_roscigno/install/elasticsearch-7.5.2/config/elastic-certificates.p12 \
. \
--project elastic-customer-success --zone us-central1-a
Dowload latest.
Reuse the JDK supplied with Elasticsearch
export JAVA_HOME=$HOME/install/elasticsearch-7.5.2/jdk
Sample logs used in Logstash getting started are at https://download.elastic.co/demos/logstash/gettingstarted/logstash-tutorial.log.gz