saadnabs / deploygelongcp

A python library that uses gcloud and other utils to deploy Grafana Enterprise Logs on GCP

Table of Contents

• Table of Contents
• Deploying Grafana Enterprise Logs on GCP
  • Getting started
    • Dependencies
    • Setting up a deployment environment
    • Running the script
    • Known Issues
    • Outcome of the scipt should be
    • Finalising the setup
      • Create Instance
      • Create reader access policy and token
      • Install PromTail
  • Potential improvements

Deploying Grafana Enterprise Logs on GCP

This script is provided to run through the most common commands required to create a K8s cluster on GCP, create all the necessary service accounts and permissions, and deploy GEL via a helm chart.

Getting started

Dependencies

• This script is written in python and expects python version 3+

• The following CLI tools are required for this script:

  • gcloud
  • kubectl
  • gsutil
  • helm

• Grafana Enterprise and Logs licenses are required

  • Grafana Enterprise Logs (save as license-gel.jwt)

  • Grafana Enterprise (save as license-ge.jwt)

  • Copy those licenses to your local ./deployGEL/data/licenses folder

Setting up a deployment environment

• Create a default VM instance (Linux) in Google Cloud, allow http/https traffic, and connect via SSH

  gcloud compute instances create $NAME --project=solutions-engineering-248511 --zone=$gcpRegion --machine-type=e2-standard-4 --tags=http-server,https-server 

• Install python

  sudo apt install python3

• Confirm gcloud / gsutil are already installed
• Initialise gcloud to your project by following the cli instructions

  gcloud init

• Install kubectl, instructions

  curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
  sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
  kubectl version --client

• Install helm, instructions

  curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
  chmod 700 get_helm.sh
  ./get_helm.sh
  helm version

• Upload licenses to GCE instance - instructions
  • And move them into the correct folder (deployGELonGCP/data/licenses)
  • Make sure to have a license with "Cluster Name" set to $PREFIX-cluster
• Install git

sudo apt-get install git
git --version

• Download the deployGEL github repo

git clone https://github.com/saadnabs/deployGELonGCP.git

Running the script

• The script takes two optional flags

  • -v: version of your deploy, will be used as a unique identifier when deploying all the components
  • -d: when passed, invokes a delete of the deployment with the version identifier
  • -p: prefix used for all the component names deployed (can be your username, keep it short though others length limits will be hit)

• Bear in mind that the script will be interactive in a few different parts, so please keep an eye on it for the following inputs:

  • Confirmation of having the dependencies
  • Policy conditions (x5)

• It can take about 7 mins (+ ~5min for K8s cluster creation) to complete

An example run of the script, from within the deployGELonGCP folder:

python3.9 deployGEL.py -v 2 -p nabeel

Known Issues

Nov 3, 2021: None identified at the moment.

Outcome of the scipt should be

• A GCP service account created
• GCP IAM permissions added
• A K8s auto cluster created
• A K8s namespace and service account annotated with the GCP service account
• A helm deployment of GEL
• An ingress controller for the GEL gateway
• A deployment of GE

Finalising the setup

Follow these steps after everything is deployed to start sending logs and seeing them in your new GEL instances

• Log in to your Grafana Enterprise instance at http://$external-ip:3000/login as provided by the script

• Go to Stats & licensing page

• And upload the Grafana Enterprise license

• Set up the Grafana Enterprise Logs plugin following these instructions, getting the API Settings by doing the following

  • In K9s, type :service, find the nabeel-gel-enterprise-logs-gateway service’s Cluster IP and copy that or use the service name.
  • Get the token from the tokengen job as done previously

• Create the Logs instance, data source and PromTail access policies (customised these instructions)

Create Instance

• Go to “Grafana Enterprise Logs” → Instances
• Click “Create Instance” with the following settings:
  • Display name: $gelInstanceName
  • Cluster: $kubeClusterName

Create reader access policy and token

• Go to the “Access Policies” tab and click “Create access policy” with the following settings:
  • Display name: demo reader
  • Scope: logs:read
  • Instances: $gelInstanceName
• Click the “Add token” for the new policy created and use the following settings:
  • Display name: reader token
  • Expiration date:
• In the “Token successfully added!” dialog, click the “Create a datasource” button
• Once created, click on the “Data source settings” link, click “Save & Test” at the bottom to confirm the data source works correctly.
• Go back to the “Grafana Enterprise Logs” → “Access policies”
• Click “Create access policy” with the following settings:
  • Display name: demo writer
  • Scope: logs:write
  • Instances: $gelInstanceName
• Click the “Add token” for the new policy created and use the following settings:
  • Display name: writer token
  • Expiration date:
• In the “Token successfully added!” dialog, click the “Copy to clipboard” button to save the Token for use with Promtail
  • $gelWriterToken, e.g. ZGVtbydsdgl0ZXItd3JpdGVyLXRva2Vu323sds5vOjMyQEAsMzE0NXQ1OFE4QDp9MA==

Install PromTail

• Install promtail local using a binary you can download from here
  • Using the promtail-config-template.yaml, run promtail after modifying all the variables prefixed with $
    • $PROMTAIL-DATA-LOCATION, e.g a local directory that promtail can write to
    • $GEL-GATEWAY-URL, e.g the IP to your GEL gateway ingress (or use port-forwarding and localhost as a simple/local solution)
    • $GEL-INSTANCE-INTERNAL-NAME, e.g the internal name of the GEL instance
    • $GEL-INSTANCE-ACCESSPOLICY-TOKEN, e.g token taken from the access policy for writing
    • $HOSTNAME, e.g mymachine or node1, the host name of the machine sending the logs

  ./promtail-darwin-amd64 -config.file=~/$deployGEL/promtail-config.yaml
  

• Click “Explore” on the left menu bar, select the data source at the top to be the GEL data source we added previously and click on the “Log browser” so you see the logs that are coming through, select one of the labels and values, then click “Show logs”

Potential improvements

• Use the latest GEL plugin from the catalogue (currently using a dev instance due to a bug)
• Use Google Cloud deployment manager templates instead of gcloud calls
• Set more restrictive permissions on the service account