fdmsantos / terraform-aws-kinesis-firehose

Dynamic Terraform module, which creates a Kinesis Firehose Stream and others resources like Cloudwatch, IAM Roles and Security Groups that integrate with Kinesis Firehose. Supports all destinations and all Kinesis Firehose Features.

📈 Stats 📈

---

👁️ Visitors 👁️

---

Author of the following AWS Terraform modules :

• AWS Kinesis Firehose
• AWS Opensearch Serverless

---

Author of the following Azure Terraform modules :

• Azure Synapse
• Azure Logic App
• Azure API Connections

terraform-aws-kinesis-firehose's Issues

[error] AWS provider version >= 5.x.x: argument named "buffer_size" is not expected here (or "buffer_interval")

There seems to be a silly issue with buffer_size and buffer_interval being renamed to buffering_size and buffering_interval with AWS provider version >= 5

• https://registry.terraform.io/providers/hashicorp/aws/5.16.1/docs/resources/kinesis_firehose_delivery_stream

At least, might need to add constraint to prevent this:

Updating an existing Application role doens't work

Using the following setup didn't work.

Configure existing Application Role to an application that runs in EC2 Instance with a policy with provided actions

module "firehose" {
  source                              = "fdmsantos/kinesis-firehose/aws"
  version                             = "x.x.x"
  name                                = "firehose-delivery-stream"
  destination                         = "s3" # or destination = "extended_s3"
  configure_existing_application_role = true
  application_role_name               = "application-role"
  create_application_role_policy      = true
  application_role_policy_actions     = [
    "firehose:PutRecord",
    "firehose:PutRecordBatch",
    "firehose:CreateDeliveryStream",
    "firehose:UpdateDestination"
  ]
}

I got an error name = "${local.application_role_name}-policy" is null.

[Enhancement] : `buffering_interval` variable validation doesn't allow to specify values less than 60 seconds.

Description

• Hi there :) Modifying the buffering_interval variable validation would be great to allow specifying values less than 60 seconds.

• From documentation :

The elasticsearch_configuration object supports the following:

• buffering_interval - (Optional) Buffer incoming data for the specified period of time, in seconds between 0 to 900, before delivering it to the destination. The default value is 300s.

[Enhancement] : Provide support for the `opensearch_configuration` object.

Description

• Hi there :) It would be great to provide support for the opensearch_configuration object inside aws_kinesis_firehose_delivery_stream resource.

• One more thing is that recently terraform-provider-aws got an update to allow specifying document_id_options inside the opensearch_configuration object. This option requires at least the 5.32.0 aws provider version. You can also check the related issue.

• DocumentID type - (introduced in May, 2023) - this is the method for setting up document ID. The supported methods are Kinesis Data Firehose-generated document ID and OpenSearch Service-generated document ID. Kinesis Data Firehose-generated document ID is the default option when the document ID value is not set. OpenSearch Service-generated document ID is the recommended option because it supports write-heavy operations, including log analytics and observability, consuming fewer CPU resources at the OpenSearch Service domain and thus, resulting in improved performance.

Prevent perpetual differences during the terraform plan/apply

Hello @fdmsantos,

I work with your module to deploy a firehose stream with configured lambda transformation.
There are a set of parameters that cannot be(at least for now). From the AWS documentation:
Parameters with default values, including NumberOfRetries(default: 3), RoleArn(default: firehose role ARN), BufferSizeInMBs(default: 1), and BufferIntervalInSeconds(default: 60), are not stored in terraform state. To prevent perpetual differences, it is therefore recommended to only include parameters with non-default values.

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kinesis_firehose_delivery_stream

In the current version v3.3.0 the module always trying to set those parameters so in the plan it will be like this:

     ~ extended_s3_configuration {
           # (9 unchanged attributes hidden)

         ~ processing_configuration {
               # (1 unchanged attribute hidden)

             ~ processors {
                   # (1 unchanged attribute hidden)

                 + parameters {
                     + parameter_name  = "BufferIntervalInSeconds"
                     + parameter_value = "60"
                   }
                 + parameters {
                     + parameter_name  = "NumberOfRetries"
                     + parameter_value = "3"
                   }
                 + parameters {
                     + parameter_name  = "RoleArn"
                     + parameter_value = "arn:aws:iam::1111111111:role/env1-dev-test1-firehose-vpcflowlog-stream"
                   }

Setting custom values(suggested here) won't fix everything if the engineer wants to use the default RoleArn.

Please add logic to make applying each parameter optional.
An easy fix that works for me:
variable.tf

variable "transform_lambda_default_parameters" {
 description = "Set parameters with default values for the data transformation lambda: NumberOfRetries, RoleArn, BufferSizeInMBs, BufferIntervalInSeconds"
 type        = bool
 default     = false
}

locals.tf

  lambda_processor = var.enable_lambda_transform ? {
    type = "Lambda"
    parameters = var.transform_lambda_default_parameters ? [
      {
        name  = "LambdaArn"
        value = var.transform_lambda_arn
      },
      ] : [
      {
        name  = "LambdaArn"
        value = var.transform_lambda_arn
      },
      {
        name  = "BufferSizeInMBs"
        value = var.transform_lambda_buffer_size
      },
      {
        name  = "BufferIntervalInSeconds"
        value = var.transform_lambda_buffer_interval
      },
      {
        name  = "NumberOfRetries"
        value = var.transform_lambda_number_retries
      },
      {
        name  = "RoleArn"
        value = var.transform_lambda_role_arn != null ? var.transform_lambda_role_arn : local.firehose_role_arn
      },
    ]
  } : null

Example of the module execution:

module "firehose" {
  count                               = var.flow_log_destination_type == "kinesis-data-firehose" ? 1 : 0
  source                              = "git::https://gitlab.test/test/terraform-modules/terraform-aws-kinesis-firehose.git//?ref=initial"
  name                                = "${var.name}-firehose-vpcflowlog-stream"
  input_source                        = "direct-put"
  destination                         = "s3" # or destination = "extended_s3"
  s3_bucket_arn                       = var.flow_log_s3_bucket_arn
  s3_prefix                           = local.s3_prefix
  s3_error_output_prefix              = local.s3_error_output_prefix
  s3_compression_format               = "GZIP"
  enable_lambda_transform             = true
  transform_lambda_arn                = module.lambda_data_transformation.lambda_function_arns["vpc_flowlogs_transformamtion_lambda"]
  transform_lambda_default_parameters = true
  tags                                = merge(var.tags, { purpose = "Stream transformed VPC Flow logs to the access S3" }, { LogDeliveryEnabled = "true" })
}

Probably it makes sense to add more complex logic to set parameters. My approach limits configuration:

1. Use all custom settings
2. Use defaults only

Add support for MSK Source Configuration

Add support for MSK CLuster Source.

Docs

Allow enable_s3_backup = false for opensearch

The module enforce s3 backup for opensearch. When using a scenario such logs destination (fluentbit -> firehose -> opensearch) the usage of s3 backup is not needed.

Is it possible to allow it as optional instead of required?

Provider specified to strict aws provider major version

Seems like the provider is strict to ~> 5.0 which is not best practice for internal modules. Since we might use other modules that will require higher versions in the future.

Is it possible to change it to >= 5.0?