exasol / terraform-aws-exasol

Terraform module for deploying Exasol cluster on AWS.

Exasol AWS Terraform Module

A Terraform module to create an Exasol cluster on Amazon AWS.

Prerequisites

• terraform version >= 1.5.1
• aws-cli profile with administrative access
• python3

Usage example

provider "aws" {
  profile     = var.profile
  region      = "eu-central-1"
}

module "exasol" {
  source                          = "exasol/exasol/aws"
  version                         = "<VERSION>"

  cluster_name                    = "exasol-cluster"
  database_name                   = "exadb"
  ami_image_name                  = "R6.2.3-PAYG"
  sys_user_password               = "eXaSol1337DB"
  admin_user_password             = "eXaSol1337OP"
  management_server_instance_type = "m5.xlarge"
  datanode_instance_type          = "m5.2xlarge"
  datanode_count                  = "3"
  standbynode_count               = "0"
  public_ip                       = true

  # These values can be obtained from other modules.
  key_pair_name                   = "exasol-key-pair"
  subnet_id                       = "subnet-ed85b690"
  security_group_id               = "sg-07599522f13906845"

  # Variables used in tags.
  project                         = "exasol-terraformed"
  project_name                    = "exasol-terraformed"
  owner                           = "[email protected]"
  environment                     = "dev"
}

If you don't already have a vpc and security group you can use:

resource "aws_vpc" "exasol_db_vpc" {
  cidr_block = "10.0.0.0/16"
  tags = {
    "exa:owner": "[email protected]",
    "exa:deputy": "[email protected]"
    "exa:project": "exasol-terraformed"
    "exa:project.name": "exasol-terraformed"
    "exa:stage": "development"
    "Name": "VPC for exasol cluster"
  }
}

resource "aws_security_group" "exasol_db_security_group" {
  name = "exasol_cluster_security_group"
  description = "Security group for exasol cluster"
  vpc_id = "${aws_vpc.exasol_db_vpc.id}"

  ingress {
    description = "SSH from VPC"
    from_port = 22
    to_port = 22
    protocol = "tcp"
    cidr_blocks = [
      "0.0.0.0/0"]
  }

  ingress {
    description = "HTTPS from VPC"
    from_port = 443
    to_port = 443
    protocol = "tcp"
    cidr_blocks = [
      "0.0.0.0/0"]
  }

  ingress {
    description = "SQL from VPC"
    from_port = 8563
    to_port = 8563
    protocol = "tcp"
    cidr_blocks = [
      "0.0.0.0/0"]
  }

  ingress {
    description = "BucketFS"
    from_port = 2580
    protocol = "tcp"
    to_port = 2580
    cidr_blocks = [
      "0.0.0.0/0"]
  }

  ingress {
    from_port = 0
    protocol = "-1"
    to_port = 0
    self = true
  }

  egress {
    from_port = 0
    to_port = 0
    protocol = "-1"
    cidr_blocks = [
      "0.0.0.0/0"]
  }

  tags = {
   "exa:owner": "[email protected]",
   "exa:deputy": "[email protected]"
   "exa:project": "exasol-terraformed"
   "exa:project.name": "exasol-terraformed"
   "exa:stage": "development"
   "Name": "VPC for exasol cluster"
  }
}

Then you can configure the exasol module like:

  subnet_id = aws_subnet.exasol_db_vpc.id
  security_group_id = aws_security_group.exasol_db_security_group.id

Inputs

The following configuration variables are available.

Variable name	Default	Description
cluster_name	exasol-cluster	A name for the to be deployed cluster.
database_name	exadb	A name of the Exasol database.
ami_image_id	null	An Exasol release AMI image id, e.g, ami-05fad9f0c2609cef0.
ami_image_name		An Exasol release AMI image name, e.g, R6.1.5-PAYG or R6.2.1-BYOL.
sys_user_password		An Exasol database sys user password.
admin_user_password		An EXAOperation admin user password.
management_server_instance_type	m5.large	An EC2 instance type for management server.
datanode_instance_type	m5.xlarge	An EC2 instance type for Exasol datanodes.
datanode_count	3	The number of Exasol datanodes.
standbynode_count	0	The number of Exasol standby nodes.
license	null	An optional path for the Bring Your Own (BYOL) image license file, e.g, ./mor_byol_license.xml.
public_ip	true	A boolean variable whether to set public IPv4 address to nodes. Skips waiting if set to false.
key_pair_name		An EC2 key pair name to attach to nodes.
subnet_id		A subnet id to deploy the Exasol cluster.
security_group_id		A security group id to attach to nodes. Please ensure that it has correct inbound rules.
project	""	A name for the project used in resource tagging as exa:project and Project.
project_name	""	A name for the project used in resource tagging as exa:project.name
owner	""	An email address of the owner used in resource tagging.
environment	""	An environment name to deploy the cluster used in resource tagging.
waited_on	null	An optional variable that can include other resource id-s to wait before deploying the cluster.

Remarks

• If the ami_image_id is provided it will used. Otherwise, an AMI image id will be used corresponding to the provided ami_image_name value.
• The security_group_id should have at least these ports open for basic operations.
  • 22 for SSH
  • 443 for EXAOperation
  • 8563 for Exasol database
  • 8835 for Cloud UI
• The project input value is also used to create a exa:project tag.
• Similarly, the owner input value is used to create a exa:owner tag.

Outputs

Output name	Description
management_server_ip	The Exasol management server public ip address.
first_datanode_ip	The first Exasol datanode public ip address.

Additional Information

• Changelog
• Developer Guide
• Contribution guidelines

terraform-aws-exasol's Issues

Update to NodeJs 14

terraform apply displays the error "The runtime parameter of nodejs12.x is no longer supported."

The current ticket therefore requests to update to NodeJs 14.
Please note: NodeJS 18 already requires AWS SDK 2.0 containing breaking changes.

Currently main.tf references cloud formation template cloudformation_template_v0.0.3.yml with in turn uses NodeJs 12.
So the content of this resource needs to be updated.

AC

1. old versions are kept in Git repo terraform-aws-exasol and in S3 bucket for backwards compatibility
2. a new version 0.0.4 is added
3. new version 0.0.4 uses NodeJS 14

Update cloud formation templates

Migrate to Github Actions from Travis CI

Add job name to .github/workflows/ci-build.yml

github-keeper can only update the branch protection in this repo if a job name is defined in .github/workflows/ci-build.yml:

Validation Error for .github/workflows/ci-build.yml: matrix github-action jobs with object parameters and no job name are not supported. Please add a name field to the job that combines the matrix parameters into a more readable name. For example "Build with Go ${{matrix.go}} and Exasol ${{ matrix.db }}"

Remove bucketfs changes and bucket creation

These actions should be performed afterwards the cluster is up and running.

Do only one thing here: start and wait until the cluster is ready.

Tags for ec2 instances have wrong value

Expected:

• exa:owner = [email protected]
• exa:project = SageMakerGuide

But:

Cloudformation tags:

Management node tags:

Data node tags:

Update CloudFormation template to use NodeJS 20 Lambdas

The CloudFormation template for deploying the Exasol cluster uses four Lambdas using NodeJS 16 to implement custom behavior. AWS supports NodeJS 16 only until July 2024.

So we need to migrate the Lambda source code to NodeJS 20 which requires migration to the new JS module system.

Add open ports as a variable

Add the comma separated ports as a variable so that users can provide them to the terraform scripts.

Create fails if port 443 is not exposed

If the security group does not expose port 443, the creation fails, since exasol_xmlrpc.py can't connect to the database.

For security reasons I only want to export the SSH port. All other access to the database is done via SSH tunnels.

Dependencies

• exasol/cloudformation-aws-exasol#2

Deployment fails because of outdated NodeJS version

Deployment fails because of outdated NodeJS version:

Resource handler returned message: \"The runtime parameter of nodejs14.x is no longer supported for creating or updating AWS Lambda functions. We recommend you use the new runtime (nodejs20.x) while creating or updating functions.

Add 7.x version AMI with updated cloudformation templates

Deployment fails when setting public ip to false

Currently, when user set the public_ip variable to false the deployment fails since the output value tries to obtain the publicIP attribute from the management node.

Support single node database

Also support creating single node exasol databases using the template form: https://docs.exasol.com/cloud_platforms/aws/installation_cf_template.htm#CloudFormation_Templates_(CFT)

Support custom tags for ec2 instances

Support custom tags for ec2 instances.

Depends on: https://github.com/exasol/Cloud-Templates/issues/12

Creation fails if password contains special characters

For certain passwords the resource creation fails:

│ Error: local-exec provisioner error
│ 
│   with module.exasol.null_resource.exasol_cluster_wait[0],
│   on ../../terraform-aws-exasol/main.tf line 72, in resource "null_resource" "exasol_cluster_wait":
│   72:   provisioner "local-exec" {
│ 
│ Error running command '    python3 ../../terraform-aws-exasol/scripts/exasol_xmlrpc.py \
│       --license-server-address \
│       18.196.183.63 \
│       --username admin \
│       --password Z+Dia7D>w]NDS)hT
│ ': exit status 2. Output: /bin/sh: 5: Syntax error: ")" unexpected

No worries / hope, even so the password is a real example it's no longer valid ;-)

Document how to create cloudformation templates

Update to the latest cloudformation template

Currently we use older version of cloudformation template which only supports Lambda functions with nodejs8.x versions.

Update it to the latest cloudformation template so that it supports Lambda functions with nodejs12.x versions.

Add integration tests

Add integration tests

Run integration tests in CI

Run the integration tests during CI

Outputs broken

The management_server_ip ouput returns data.aws_instance.exasol_first_datanode.public_ip instead of it's value.
Same for the other outputs.

Terraform v0.12.26