terraform-aws-modules / terraform-aws-datadog-forwarders

Terraform module to create resources on AWS to forward logs/metrics to Datadog 🇺🇦

AWS Datadog Forwarders Terraform module

Terraform module which creates resources to forward logs and metrics to Datadog on AWS.

The following resources are supported:

• Log Forwarder
• RDS Enhanced Monitoring Forwarder
• VPC Flow Log Forwarder
• Agent PrivateLink VPC Endpoint
• API PrivateLink VPC Endpoint
• Log Forwarder PrivateLink VPC Endpoint
• Metrics PrivateLink VPC Endpoint
• Process PrivateLink VPC Endpoint
• Traces PrivateLink VPC Endpoint

Please refer to the official Datadog datadog-serverless-functions for further information on the forwarder lambda functions, configuraion via environment variables, and integration with PrivateLink endpoints.

Security

There are several factors to keep in mind when working with and/or using this module that largely revolve around the decisions made based on security implications.

1. Based on the functionality provided by the Datadog team at datadog-serverless-functions, the recommended approach for providing your Datadog API key is through AWS Secrets Manager. The creation of this secret is not facilitated by this module and should be created manually (or through some other means where the secret is not passed as plain text into Terraform as input).
   • Note: even though this is the recommended approach, the pattern used in the past of providing the DD_API_KEY as an environment variable is still supported by the module. Users are able to provde any and all environment variables to the forwarders through inputs (i.e. - log_forwarder_environment_variables) to configure the forwarders as desired. See settings.py for more details on what environment variables are supported to configure the forwarders.
2. The use of a KMS key to encrypt/decrypt API and APP keys is required by the rds_enhanced_monitoring_forwarder and vpc_flow_log_forwarder modules/functions per the uptream source at datadog-serverless-functions. The creation of a KMS key has been left out of this module so that users are able to better manage their KMS CMK key (and therefore the policies and usage of said key) as they see fit without over-complicating this module.
3. The roles and their permissions created by this module have several built in conditional checks in order to provide permission sets that allow the desired functionality while following the recommended approach of least privelege access. Nearly all attributes for the IAM roles and their permissions are accessible via inputs - even allowing users to provide their own IAM roles and/or policies to meet their organizational requirements.

Vendored Artifacts

Due to Terraform not dealing with dynamically created files, especially in ephemeral environments like CI/CD pipelines, the decision was made to vendor the Lambda function artifacts within the VPC Flow Log and RDS Enhanced Monitoring modules to avoid issues. When the upstream artifacts are hosted as zipped archives available on GitHub, similar to the Log Forwarder, then this vendoring hack will be removed in favor of pulling directly from the project. Ref Issue #374

Usage

See examples directory for working examples to reference:

# Note: you will need to create this secret manually prior to running
# This avoids having to pass the key to Terraform in plaintext
data "aws_secretsmanager_secret" "datadog_api_key" {
  name = "datadog/api_key"
}

module "datadog_forwarders" {
  source  = "terraform-aws-modules/datadog-forwarders/aws"

  kms_alias             = "alias/datadog" # KMS key will need to be created outside of module
  dd_api_key_secret_arn = data.aws_secretsmanager_secret.datadog_api_key.arn

  tags = {
    Terraform   = "true"
    Environment = "dev"
  }
}

Examples

Examples codified under the examples are intended to give users references for how to use the module(s) as well as testing/validating changes to the source code of the module(s). If contributing to the project, please be sure to make any appropriate updates to the relevant examples to allow maintainers to test your changes and to keep the examples up to date for users. Thank you!

• Complete
• Simple

Requirements

Name	Version
terraform	>= 1.3
aws	>= 5.0

Providers

Name	Version
aws	>= 5.0

Modules

Name	Source	Version
log_forwarder	./modules/log_forwarder	n/a
rds_enhanced_monitoring_forwarder	./modules/rds_enhanced_monitoring_forwarder	n/a
vpc_flow_log_forwarder	./modules/vpc_flow_log_forwarder	n/a

Resources

Name	Type
aws_vpc_endpoint.agent	resource
aws_vpc_endpoint.api	resource
aws_vpc_endpoint.log_forwarder	resource
aws_vpc_endpoint.metrics	resource
aws_vpc_endpoint.processes	resource
aws_vpc_endpoint.traces	resource

Inputs

Name	Description	Type	Default	Required
agent_vpce_policy	Policy to attach to the agent endpoint that controls access to the service. Defaults to full access	any	null	no
agent_vpce_security_group_ids	IDs of security groups to attach to agent endpoint	list(string)	[]	no
agent_vpce_subnet_ids	IDs of subnets to associate with agent endpoint	list(string)	[]	no
agent_vpce_tags	A map of tags to apply to the Datadog agent endpoint	map(string)	{}	no
api_vpce_policy	Policy to attach to the API endpoint that controls access to the service. Defaults to full access	any	null	no
api_vpce_security_group_ids	IDs of security groups to attach to API endpoint	list(string)	[]	no
api_vpce_subnet_ids	IDs of subnets to associate with API endpoint	list(string)	[]	no
api_vpce_tags	A map of tags to apply to the API endpoint	map(string)	{}	no
bucket_attach_deny_insecure_transport_policy	Controls if S3 bucket should have deny non-SSL transport policy attacheds	bool	true	no
bucket_encryption_settings	S3 bucket server side encryption settings	map(string)	{ "sse_algorithm": "AES256" }	no
bucket_name	Lambda artifact S3 bucket name	string	""	no
create_agent_vpce	Controls whether an agent endpoint should be created	bool	false	no
create_api_vpce	Controls whether a API endpoint should be created	bool	false	no
create_bucket	Controls whether an S3 artifact bucket should be created. this is used for the zip archive as well as caching tags	bool	true	no
create_log_forwarder	Controls whether log forwarder resources should be created	bool	true	no
create_log_forwarder_role	Controls whether an IAM role is created for the log forwarder	bool	true	no
create_log_forwarder_role_policy	Controls whether an IAM role policy is created for the log forwarder	bool	true	no
create_log_forwarder_vpce	Controls whether a log forwarder endpoint should be created	bool	false	no
create_metrics_vpce	Controls whether a metrics VPC endpoint should be created	bool	false	no
create_processes_vpce	Controls whether a processes endpoint should be created	bool	false	no
create_rds_em_forwarder	Controls whether RDS enhanced monitoring forwarder resources should be created	bool	true	no
create_rds_em_forwarder_role	Controls whether an IAM role is created for the RDS enhanced monitoring forwarder	bool	true	no
create_rds_em_forwarder_role_policy	Controls whether an IAM role policy is created for the RDS enhanced monitoring forwarder	bool	true	no
create_traces_vpce	Controls whether a traces endpoint should be created	bool	false	no
create_vpc_fl_forwarder	Controls whether VPC flow log forwarder resources should be created	bool	true	no
create_vpc_fl_forwarder_role	Controls whether an IAM role is created for the VPC flow log forwarder	bool	true	no
create_vpc_fl_forwarder_role_policy	Controls whether an IAM role policy is created for the VPC flow log forwarder	bool	true	no
dd_api_key	The Datadog API key, which can be found from the APIs page (/account/settings#api). It will be stored in AWS Secrets Manager securely. If DdApiKeySecretArn is also set, this value will not be used. This value must still be set, however	string	""	no
dd_api_key_secret_arn	The ARN of the Secrets Manager secret storing the Datadog API key, if you already have it stored in Secrets Manager. You still need to set a dummy value for dd_api_key to satisfy the requirement, though that value won't be used	string	""	no
dd_app_key	The Datadog application key associated with the user account that created it, which can be found from the APIs page	string	""	no
dd_site	Define your Datadog Site to send data to. For the Datadog EU site, set to datadoghq.eu	string	"datadoghq.com"	no
kms_alias	Alias of KMS key used to encrypt the Datadog API keys - must start with alias/	string	n/a	yes
log_forwarder_architectures	Instruction set architecture for your Lambda function. Valid values are ["x86_64"] and ["arm64"]. Default is ["x86_64"]	list(string)	[ "x86_64" ]	no
log_forwarder_bucket_prefix	S3 object key prefix to prepend to zip archive name	string	""	no
log_forwarder_bucket_tags	A map of tags to apply to the log forwarder bucket	map(any)	{}	no
log_forwarder_environment_variables	A map of environment variables for the log forwarder lambda function	map(string)	{}	no
log_forwarder_kms_key_arn	KMS key that is used to encrypt environment variables. If this configuration is not provided when environment variables are in use, AWS Lambda uses a default service key	string	null	no
log_forwarder_lambda_tags	A map of tags to apply to the log forwarder lambda function	map(string)	{}	no
log_forwarder_layers	List of Lambda Layer Version ARNs (maximum of 5) to attach to the log forwarder lambda	list(string)	[]	no
log_forwarder_log_kms_key_id	The AWS KMS Key ARN to use for CloudWatch log group encryption	string	null	no
log_forwarder_log_retention_days	Log forwarder CloudWatch log group retention in days	number	7	no
log_forwarder_memory_size	Memory size for the log forwarder lambda function	number	1024	no
log_forwarder_name	Log forwarder lambda name	string	"datadog-log-forwarder"	no
log_forwarder_policy_arn	IAM policy arn for log forwarder lambda function to utilize	string	null	no
log_forwarder_policy_name	Log forwarder policy name	string	""	no
log_forwarder_policy_path	Log forwarder policy path	string	null	no
log_forwarder_publish	Whether to publish creation/change as a new Lambda Function Version	bool	false	no
log_forwarder_reserved_concurrent_executions	The amount of reserved concurrent executions for the log forwarder lambda function	number	100	no
log_forwarder_role_arn	IAM role arn for log forwarder lambda function to utilize	string	null	no
log_forwarder_role_max_session_duration	The maximum session duration (in seconds) that you want to set for the specified role. If you do not specify a value for this setting, the default maximum of one hour is applied. This setting can have a value from 1 hour to 12 hours	number	null	no
log_forwarder_role_name	Log forwarder role name	string	""	no
log_forwarder_role_path	Log forwarder role path	string	null	no
log_forwarder_role_permissions_boundary	The ARN of the policy that is used to set the permissions boundary for the log forwarder role	string	null	no
log_forwarder_role_tags	A map of tags to apply to the log forwarder role	map(string)	{}	no
log_forwarder_runtime	Lambda function runtime	string	"python3.11"	no
log_forwarder_s3_log_bucket_arns	S3 log buckets for forwarder to read and forward logs to Datadog	list(string)	[]	no
log_forwarder_s3_zip_kms_key_id	The AWS KMS Key ARN to use for object encryption	string	null	no
log_forwarder_s3_zip_metadata	A map of keys/values to provision metadata (will be automatically prefixed by x-amz-meta-	map(string)	{}	no
log_forwarder_s3_zip_server_side_encryption	Server-side encryption of the zip object in S3. Valid values are AES256 and aws:kms	string	null	no
log_forwarder_s3_zip_storage_class	Specifies the desired Storage Class for the zip object. Can be either STANDARD, REDUCED_REDUNDANCY, ONEZONE_IA, INTELLIGENT_TIERING, or STANDARD_IA	string	null	no
log_forwarder_s3_zip_tags	A map of tags to apply to the zip archive in S3	map(string)	{}	no
log_forwarder_s3_zip_tags_only	Set to true to not merge var.tags with log_forwarder_s3_zip_tags. Useful to avoid breaching S3 Object 10 tag limit	bool	false	no
log_forwarder_security_group_ids	List of security group ids when forwarder lambda function should run in the VPC	list(string)	null	no
log_forwarder_subnet_ids	List of subnet ids when forwarder lambda function should run in the VPC. Usually private or intra subnets	list(string)	null	no
log_forwarder_tags	A map of tags to apply to the log forwarder resources	map(string)	{}	no
log_forwarder_timeout	The amount of time the log forwarder lambda has to execute in seconds	number	120	no
log_forwarder_use_policy_name_prefix	Whether to use unique name beginning with the specified policy_name for the log forwarder policy	bool	false	no
log_forwarder_use_role_name_prefix	Whether to use unique name beginning with the specified role_name for the log forwarder role	bool	false	no
log_forwarder_version	Forwarder version - see https://github.com/DataDog/datadog-serverless-functions/releases	string	"3.103.0"	no
log_forwarder_vpce_policy	Policy to attach to the log forwarder endpoint that controls access to the service. Defaults to full access	any	null	no
log_forwarder_vpce_security_group_ids	IDs of security groups to attach to log forwarder endpoint	list(string)	[]	no
log_forwarder_vpce_subnet_ids	IDs of subnets to associate with log forwarder endpoint	list(string)	[]	no
log_forwarder_vpce_tags	A map of tags to apply to the log forwarder endpoint	map(string)	{}	no
metrics_vpce_policy	Policy to attach to the metrics endpoint that controls access to the service. Defaults to full access	any	null	no
metrics_vpce_security_group_ids	IDs of security groups to attach to metrics endpoint	list(string)	[]	no
metrics_vpce_subnet_ids	IDs of subnets to associate with metrics endpoint	list(string)	[]	no
metrics_vpce_tags	A map of tags to apply to the metrics endpoint	map(string)	{}	no
processes_vpce_policy	Policy to attach to the processes endpoint that controls access to the service. Defaults to full access	any	null	no
processes_vpce_security_group_ids	IDs of security groups to attach to processes endpoint	list(string)	[]	no
processes_vpce_subnet_ids	IDs of subnets to associate with processes endpoint	list(string)	[]	no
processes_vpce_tags	A map of tags to apply to the processes endpoint	map(string)	{}	no
rds_em_forwarder_architectures	Instruction set architecture for your Lambda function. Valid values are ["x86_64"] and ["arm64"]. Default is ["x86_64"]	list(string)	[ "x86_64" ]	no
rds_em_forwarder_environment_variables	A map of environment variables for the RDS enhanced monitoring forwarder lambda function	map(string)	{}	no
rds_em_forwarder_kms_key_arn	KMS key that is used to encrypt environment variables. If this configuration is not provided when environment variables are in use, AWS Lambda uses a default service key	string	null	no
rds_em_forwarder_lambda_tags	A map of tags to apply to the RDS enhanced monitoring forwarder lambda function	map(string)	{}	no
rds_em_forwarder_layers	List of Lambda Layer Version ARNs (maximum of 5) to attach to the RDS enhanced monitoring forwarder lambda	list(string)	[]	no
rds_em_forwarder_log_kms_key_id	The AWS KMS Key ARN to use for CloudWatch log group encryption	string	null	no
rds_em_forwarder_log_retention_days	RDS enhanced monitoring forwarder CloudWatch log group retention in days	number	7	no
rds_em_forwarder_memory_size	Memory size for the RDS enhanced monitoring forwarder lambda function	number	256	no
rds_em_forwarder_name	RDS enhanced monitoring forwarder lambda name	string	"datadog-rds-enhanced-monitoring-forwarder"	no
rds_em_forwarder_policy_arn	IAM policy arn for RDS enhanced monitoring forwarder lambda function to utilize	string	null	no
rds_em_forwarder_policy_name	RDS enhanced monitoring forwarder policy name	string	""	no
rds_em_forwarder_policy_path	RDS enhanced monitoring forwarder policy path	string	null	no
rds_em_forwarder_publish	Whether to publish creation/change as a new fambda function Version	bool	false	no
rds_em_forwarder_reserved_concurrent_executions	The amount of reserved concurrent executions for the RDS enhanced monitoring forwarder lambda function	number	10	no
rds_em_forwarder_role_arn	IAM role arn for RDS enhanced monitoring forwarder lambda function to utilize	string	null	no
rds_em_forwarder_role_max_session_duration	The maximum session duration (in seconds) that you want to set for the specified role. If you do not specify a value for this setting, the default maximum of one hour is applied. This setting can have a value from 1 hour to 12 hours	number	null	no
rds_em_forwarder_role_name	RDS enhanced monitoring forwarder role name	string	""	no
rds_em_forwarder_role_path	RDS enhanced monitoring forwarder role path	string	null	no
rds_em_forwarder_role_permissions_boundary	The ARN of the policy that is used to set the permissions boundary for the RDS enhanced monitoring forwarder role	string	null	no
rds_em_forwarder_role_tags	A map of tags to apply to the RDS enhanced monitoring forwarder role	map(string)	{}	no
rds_em_forwarder_runtime	Lambda function runtime	string	"python3.8"	no
rds_em_forwarder_security_group_ids	List of security group ids when forwarder lambda function should run in the VPC	list(string)	null	no
rds_em_forwarder_subnet_ids	List of subnet ids when forwarder lambda function should run in the VPC. Usually private or intra subnets	list(string)	null	no
rds_em_forwarder_tags	A map of tags to apply to the RDS enhanced monitoring forwarder resources	map(string)	{}	no
rds_em_forwarder_timeout	The amount of time the RDS enhanced monitoring forwarder lambda has to execute in seconds	number	10	no
rds_em_forwarder_use_policy_name_prefix	Whether to use unique name beginning with the specified rds_em_forwarder_policy_name for the RDS enhanced monitoring forwarder role	bool	false	no
rds_em_forwarder_use_role_name_prefix	Whether to use unique name beginning with the specified rds_em_forwarder_role_name for the RDS enhanced monitoring forwarder role	bool	false	no
rds_em_forwarder_version	RDS enhanced monitoring lambda version - see https://github.com/DataDog/datadog-serverless-functions/releases	string	"3.103.0"	no
tags	A map of tags to use on all resources	map(string)	{}	no
traces_vpce_policy	Policy to attach to the traces endpoint that controls access to the service. Defaults to full access	any	null	no
traces_vpce_security_group_ids	IDs of security groups to attach to traces endpoint	list(string)	[]	no
traces_vpce_subnet_ids	IDs of subnets to associate with traces endpoint	list(string)	[]	no
traces_vpce_tags	A map of tags to apply to the traces endpoint	map(string)	{}	no
vpc_fl_forwarder_architectures	Instruction set architecture for your Lambda function. Valid values are ["x86_64"] and ["arm64"]. Default is ["x86_64"]	list(string)	[ "x86_64" ]	no
vpc_fl_forwarder_environment_variables	A map of environment variables for the VPC flow log forwarder lambda function	map(string)	{}	no
vpc_fl_forwarder_kms_key_arn	KMS key that is used to encrypt environment variables. If this configuration is not provided when environment variables are in use, AWS Lambda uses a default service key	string	null	no
vpc_fl_forwarder_lambda_tags	A map of tags to apply to the VPC flow log forwarder lambda function	map(string)	{}	no
vpc_fl_forwarder_layers	List of Lambda Layer Version ARNs (maximum of 5) to attach to the VPC flow log forwarder lambda	list(string)	[]	no
vpc_fl_forwarder_log_kms_key_id	The AWS KMS Key ARN to use for CloudWatch log group encryption	string	null	no
vpc_fl_forwarder_log_retention_days	VPC flow log forwarder CloudWatch log group retention in days	number	7	no
vpc_fl_forwarder_memory_size	Memory size for the VPC flow log forwarder lambda function	number	256	no
vpc_fl_forwarder_name	VPC flow log forwarder lambda name	string	"datadog-vpc-flow-log-forwarder"	no
vpc_fl_forwarder_policy_arn	IAM policy arn for VPC flow log forwarder lambda function to utilize	string	null	no
vpc_fl_forwarder_policy_name	VPC flow log forwarder policy name	string	""	no
vpc_fl_forwarder_policy_path	VPC flow log forwarder policy path	string	null	no
vpc_fl_forwarder_publish	Whether to publish creation/change as a new fambda function Version	bool	false	no
vpc_fl_forwarder_read_cloudwatch_logs	Whether the VPC flow log forwarder will read CloudWatch log groups for VPC flow logs	bool	false	no
vpc_fl_forwarder_reserved_concurrent_executions	The amount of reserved concurrent executions for the VPC flow log forwarder lambda function	number	10	no
vpc_fl_forwarder_role_arn	IAM role arn for VPC flow log forwarder lambda function to utilize	string	null	no
vpc_fl_forwarder_role_max_session_duration	The maximum session duration (in seconds) that you want to set for the specified role. If you do not specify a value for this setting, the default maximum of one hour is applied. This setting can have a value from 1 hour to 12 hours	number	null	no
vpc_fl_forwarder_role_name	VPC flow log forwarder role name	string	""	no
vpc_fl_forwarder_role_path	VPC flow log forwarder role path	string	null	no
vpc_fl_forwarder_role_permissions_boundary	The ARN of the policy that is used to set the permissions boundary for the VPC flow log forwarder role	string	null	no
vpc_fl_forwarder_role_tags	A map of tags to apply to the VPC flow log forwarder role	map(string)	{}	no
vpc_fl_forwarder_runtime	Lambda function runtime	string	"python3.8"	no
vpc_fl_forwarder_s3_log_bucket_arns	S3 log buckets for VPC flow log forwarder to read and forward to Datadog	list(string)	[]	no
vpc_fl_forwarder_security_group_ids	List of security group ids when forwarder lambda function should run in the VPC	list(string)	null	no
vpc_fl_forwarder_subnet_ids	List of subnet ids when forwarder lambda function should run in the VPC. Usually private or intra subnets	list(string)	null	no
vpc_fl_forwarder_tags	A map of tags to apply to the VPC flow log forwarder resources	map(string)	{}	no
vpc_fl_forwarder_timeout	The amount of time the VPC flow log forwarder lambda has to execute in seconds	number	10	no
vpc_fl_forwarder_use_policy_name_prefix	Whether to use unique name beginning with the specified vpc_fl_forwarder_policy_name for the VPC flow log forwarder role	bool	false	no
vpc_fl_forwarder_use_role_name_prefix	Whether to use unique name beginning with the specified vpc_fl_forwarder_role_name for the VPC flow log forwarder role	bool	false	no
vpc_fl_forwarder_version	VPC flow log lambda version - see https://github.com/DataDog/datadog-serverless-functions/releases	string	"3.103.0"	no
vpc_id	ID of VPC to provision endpoints within	string	null	no

Outputs

Name	Description
agent_endpoint_arn	ARN of the agent VPC endpoint
agent_endpoint_dns_entry	DNS entries of the agent VPC endpoint
agent_endpoint_id	ID of the agent VPC endpoint
agent_endpoint_network_interface_ids	One or more network interfaces for the agent VPC endpoint
agent_endpoint_owner_id	The ID of the AWS account that owns the agent VPC endpoint
agent_endpoint_state	The state of the agent VPC endpoint
api_endpoint_arn	ARN of the API VPC endpoint
api_endpoint_dns_entry	DNS entries of the API VPC endpoint
api_endpoint_id	ID of the API VPC endpoint
api_endpoint_network_interface_ids	One or more network interfaces for API api VPC endpoint
api_endpoint_owner_id	The ID of the AWS account that owns the API VPC endpoint
api_endpoint_state	The state of the API VPC endpoint
log_forwarder_cloudwatch_log_group_arn	The ARN of the log forwarder lambda function CloudWatch log group
log_forwarder_endpoint_arn	ARN of the log forwarder VPC endpoint
log_forwarder_endpoint_dns_entry	DNS entries of the log forwarder VPC endpoint
log_forwarder_endpoint_id	ID of the log forwarder VPC endpoint
log_forwarder_endpoint_network_interface_ids	One or more network interfaces for the log forwarder VPC endpoint
log_forwarder_endpoint_owner_id	The ID of the AWS account that owns the log forwarder VPC endpoint
log_forwarder_endpoint_state	The state of the log forwarder VPC endpoint
log_forwarder_lambda_arn	The ARN of the log forwarder lambda function
log_forwarder_lambda_kms_key_arn	(Optional) The ARN for the KMS encryption key for the log forwarder lambda function
log_forwarder_lambda_qualified_arn	The ARN of the log forwarder lambda function (if versioning is enabled via publish = true)
log_forwarder_lambda_source_code_hash	Base64-encoded representation of raw SHA-256 sum of the log forwarder zip file, provided either via filename or s3_* parameters
log_forwarder_lambda_version	Latest published version of the log forwarder lambda function
log_forwarder_role_arn	The log forwarder lambda role arn
log_forwarder_role_id	The log forwarder lambda role id
log_forwarder_role_name	The log forwarder lambda role name
log_forwarder_role_policy_arn	The ARN of the log forwarder lambda role policy
log_forwarder_role_policy_id	The ID of the log forwarder lambda role policy
log_forwarder_role_policy_name	The name of the log forwarder lambda role policy
log_forwarder_role_unique_id	The stable and unique string identifying the log forwarder lambda role
log_forwarder_s3_bucket_arn	The ARN of the log forwarder bucket. Will be of format arn:aws:s3:::bucketname
log_forwarder_s3_bucket_domain_name	The log forwarder bucket domain name. Will be of format bucketname.s3.amazonaws.com
log_forwarder_s3_bucket_id	The name of the log forwarder bucket
log_forwarder_s3_bucket_regional_domain_name	The log forwarder bucket region-specific domain name. The bucket domain name including the region name
log_forwarder_s3_object_etag	The ETag generated for the log forwarder lambda zip object (an MD5 sum of the object content)
log_forwarder_s3_object_id	The key of the log forwarder lambda zip archive
log_forwarder_s3_object_version	A unique version ID value for the log forwarder lambda zip object, if bucket versioning is enabled
metrics_endpoint_arn	ARN of the metrics VPC endpoint
metrics_endpoint_dns_entry	DNS entries of the metrics VPC endpoint
metrics_endpoint_id	ID of the metrics VPC endpoint
metrics_endpoint_network_interface_ids	One or more network interfaces for the metrics VPC endpoint
metrics_endpoint_owner_id	The ID of the AWS account that owns the metrics VPC endpoint
metrics_endpoint_state	The state of the metrics VPC endpoint
processes_endpoint_arn	ARN of the processes VPC endpoint
processes_endpoint_dns_entry	DNS entries of the processes VPC endpoint
processes_endpoint_id	ID of the processes VPC endpoint
processes_endpoint_network_interface_ids	One or more network interfaces for the processes VPC endpoint
processes_endpoint_owner_id	The ID of the AWS account that owns the processes VPC endpoint
processes_endpoint_state	The state of the processes VPC endpoint
rds_em_forwarder_cloudwatch_log_group_arn	The ARN of the RDS enhanced monitoring forwarder lambda function CloudWatch log group
rds_em_forwarder_lambda_arn	The ARN of the RDS enhanced monitoring forwarder lambda function
rds_em_forwarder_lambda_kms_key_arn	(Optional) The ARN for the KMS encryption key for the RDS enhanced monitoring forwarder lambda function
rds_em_forwarder_lambda_qualified_arn	The ARN of the RDS enhanced monitoring forwarder lambda function (if versioning is enabled via publish = true)
rds_em_forwarder_lambda_source_code_hash	Base64-encoded representation of raw SHA-256 sum of the RDS enhanced monitoring lambda forwarder zip file, provided either via filename or s3_* parameters
rds_em_forwarder_lambda_version	Latest published version of the RDS enhanced monitoring forwarder lambda function
rds_em_forwarder_role_arn	The RDS enhanced monitoring forwarder lambda role arn
rds_em_forwarder_role_id	The RDS enhanced monitoring forwarder lambda role id
rds_em_forwarder_role_name	The RDS enhanced monitoring forwarder lambda role name
rds_em_forwarder_role_policy_arn	The ARN of the RDS enhanced monitoring forwarder lambda role policy
rds_em_forwarder_role_policy_id	The ID of the RDS enhanced monitoring forwarder lambda role policy
rds_em_forwarder_role_policy_name	The name of the RDS enhanced monitoring forwarder lambda role policy
rds_em_forwarder_role_unique_id	The stable and unique string identifying the RDS enhanced monitoring forwarder lambda role.
traces_endpoint_arn	ARN of the traces VPC endpoint
traces_endpoint_dns_entry	DNS entries of the traces VPC endpoint
traces_endpoint_id	ID of the traces VPC endpoint
traces_endpoint_network_interface_ids	One or more network interfaces for the traces VPC endpoint
traces_endpoint_owner_id	The ID of the AWS account that owns the traces VPC endpoint
traces_endpoint_state	The state of the traces VPC endpoint
vpc_fl_forwarder_cloudwatch_log_group_arn	The ARN of the VPC flow log forwarder lambda function CloudWatch log group
vpc_fl_forwarder_lambda_arn	The ARN of the VPC flow log forwarder lambda function
vpc_fl_forwarder_lambda_kms_key_arn	(Optional) The ARN for the KMS encryption key for the VPC flow log forwarder lambda function
vpc_fl_forwarder_lambda_qualified_arn	The ARN of the VPC flow log forwarder lambda function (if versioning is enabled via publish = true)
vpc_fl_forwarder_lambda_source_code_hash	Base64-encoded representation of raw SHA-256 sum of the VPC flow log forwarder lambda zip file, provided either via filename or s3_* parameters
vpc_fl_forwarder_lambda_version	Latest published version of the VPC flow log forwarder lambda function
vpc_fl_forwarder_role_arn	The VPC flow log forwarder lambda role arn
vpc_fl_forwarder_role_id	The VPC flow log forwarder lambda role id
vpc_fl_forwarder_role_name	The VPC flow log forwarder lambda role name
vpc_fl_forwarder_role_policy_arn	The ARN of the VPC flow log forwarder lambda role policy
vpc_fl_forwarder_role_policy_id	The ID of the VPC flow log forwarder lambda role policy
vpc_fl_forwarder_role_policy_name	The name of the VPC flow log forwarder lambda role policy
vpc_fl_forwarder_role_unique_id	The stable and unique string identifying the VPC flow log forwarder lambda role.

License

Apache-2.0 Licensed. See LICENSE.

terraform-aws-datadog-forwarders's Issues

Support for AWS provider 4.x

Is your feature request related to a problem? Please describe.
Can you please add support for AWS Provider 4.0+

Describe the solution you'd like
Support for AWS Provider 4.0+

Additional context
S3 Bucket only module used in this module already released support for AWS Provider 4.x - https://github.com/terraform-aws-modules/terraform-aws-s3-bucket/releases/tag/v3.0.0

Number of tags for s3 bucket object

Description

The number of tags that can be attached to an s3 bucket object is 10 AWS Documentation
For other AWS resources, this limit is 50.

So, in the case when we use more than 10 tags for the whole module we have an error

• [ x ] ✋ I have searched the open/closed issues and my issue is not listed.

⚠️ Note

It doesn't depend on terraform. It's AWS limit.

Versions

• Module version [Required]:
  "terraform-aws-modules/datadog-forwarders/aws//modules/log_forwarder"
  version = "4.1.1"

• Terraform version:
  Terraform v0.14.11

• Provider version(s):
  ├── provider[registry.terraform.io/hashicorp/aws] ~> 4.0
  ├── provider[registry.terraform.io/opsgenie/opsgenie] ~> 0.6
  ├── provider[terraform.io/builtin/terraform]
  ├── module.tags
  │   └── provider[registry.terraform.io/hashicorp/aws] >= 3.38.0
  ├── module.datadog_log_forwarder
  │   ├── provider[registry.terraform.io/hashicorp/aws] >= 4.0.0
  │   ├── provider[registry.terraform.io/hashicorp/null] >= 3.0.0
  │   └── module.this_s3_bucket
  │   └── provider[registry.terraform.io/hashicorp/aws] >= 3.75.0
  ├── module.labels
  └── module.opsgenie_service
  ├── provider[registry.terraform.io/opsgenie/opsgenie] ~> 0.6
  └── provider[registry.terraform.io/hashicorp/null]

Providers required by state:
provider[registry.terraform.io/hashicorp/aws]
provider[registry.terraform.io/hashicorp/null]
provider[terraform.io/builtin/terraform]

Reproduction Code [Required]

Steps to reproduce the behavior:

You need also to add more than 10 tags to the variable tags ;

variable "tags" {
  description = "A map of tags to use on all resources"
  type        = map(string)
  default     = {
  "Name.1"="a"
  "Name.2"="a"
  "Name.3"="a"
  "Name.4"="a"
  "Name.5"="a"
  "Name.6"="a"
  "Name.7"="a"
  "Name.8"="a"
  "Name.9"="a"
  "Name.10"="a"
  "Name.11"="a"
}
}

module "datadog_log_forwarder" {
  source  = "terraform-aws-modules/datadog-forwarders/aws//modules/log_forwarder"
  version = "4.1.1"

  dd_api_key = var.dd_api_key
  dd_site = "datadoghq.eu"
  forwarder_version = "3.41.0"

  bucket_name  = "s-dd-forwarder-${var.env}"
  kms_key_arn = var.cmk_arn

  role_tags = var.role_tags
  tags  = var.tags

  environment_variables = { DD_TAGS = "env:cloud-${var.env}" }
}

Expected behavior

The bucket zip object created without any tags

Actual behavior

Now the bucket zip object tags as other AWS resources, but terraform isn't applied if the tags count more than 10

Terminal Output Screenshot(s)

Error: error updating tags: error setting resource tags (s-dd-forwarder-dev/aws-dd-forwarder-3.41.0.zip): BadRequest: Object tags cannot be greater than 10
	status code: 400, request id: TE64412FP9X2M6RM, host id: QshfEqkYz3YB1fcBPsmuqZ1jfFP3a3JegL4W5b+YPqD7Vck3FwZ8nvIewvdg4hrb+HTUdZS3q1I1bjEiZ/ZQBA==
  on .terraform/modules/datadog_log_forwarder/modules/log_forwarder/main.tf line 131, in resource "aws_s3_bucket_object" "this":
 131: resource "aws_s3_bucket_object" "this" {
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: command terminated with exit code 1

Additional context

Bump s3_module version to 3.1.0 or newer

Description

I updated AWS provider version for my Terraform project from 3.75.2 to 4.9.0 and warning started appearing.

│ Warning: Argument is deprecated │ │ with module.my_module.module.datadog_log_forwarder.module.datadog_log_forwarder.module.this_s3_bucket.aws_s3_bucket.this, │ on .terraform/modules/my_module.datadog_log_forwarder.datadog_log_forwarder.this_s3_bucket/main.tf line 14, in resource "aws_s3_bucket" "this": │ 14: resource "aws_s3_bucket" "this" { │ │ Use the top-level parameter object_lock_enabled and the aws_s3_bucket_object_lock_configuration resource instead

There are newer versions of terraform-aws-s3-bucket module, starting with 3.1.0, that fixed said warning.

• [ X] ✋ I have searched the open/closed issues and my issue is not listed.

Versions

• Module version [Required]: 4.1.1

• Terraform version: 1.2.4

• Provider version(s): 4.9.0

Reproduction Code [Required]

Steps to reproduce the behavior:

Are you using workspaces? No.
Have you cleared the local cache (see Notice section above)? Yes.
List steps in order that led up to the issue you encountered:
terraform init terraform plan terraform apply -auto-approve

Expected behavior

Terraform plan and apply executed without the warning.

Actual behavior

Terraform plan and apply executed with the described warning.

Support AWS Provider v5 - aws_s3_object acl default value changed to null instead of private

Is your request related to a new offering from AWS?

Is this functionality available in the AWS provider for Terraform? See CHANGELOG.md, too.

• Yes ✅: Terraform AWS Provider 5.0.0

Is your request related to a problem? Please describe.

Description

I encountered an error with Terraform AWS provider v4 -> v5 upgrade.

Change that Terraform want to make:

module.datadog_log_forwarder.aws_s3_object.this[0] will be updated in-place
  ~ resource "aws_s3_object" "this" {
      - acl                    = "private" -> null
        id                     = "aws-dd-forwarder-3.71.0.zip"
        tags                   = {
            "DD_FORWARDER_VERSION" = "3.71.0"
        }
        # (14 unchanged attributes hidden)
    }

Error for previous plan:

╷
│ Error: putting S3 object ACL: MissingSecurityHeader: Your request was missing a required header
│ status code: 400
│
│ with module.datadog_log_forwarder.aws_s3_object.this[0],
│ on .terraform/modules/datadog_log_forwarder.datadog_log_forwarder/modules/log_forwarder/main.tf line 131, in resource "aws_s3_object" "this":
│ 131: resource "aws_s3_object" "this" {
│
╵

Describe the solution you'd like.

Set acl = "private" for aws_s3_object.this[0]

Additional context

Terraform AWS Provider Version 5 Upgrade Guide

Number of tags for s3 bucket object

terraform-aws-datadog-forwarders/modules/log_forwarder/main.tf

Line 147 in 752367f

tags = merge(var.tags, var.s3_zip_tags, local.version_tag)

The number of tags that can be attached to an s3 bucket object is 10 (for other resources it is 50) Documentation
So, we have an error:
Error: error updating tags: error setting resource tags (s-dd-forwarder-dev/aws-dd-forwarder-3.41.0.zip): BadRequest: Object tags cannot be greater than 10 status code: 400, request id: TE64412FP9X2M6RM, host id: QshfEqkYz3YB1fcBPsmuqZ1jfFP3a3JegL4W5b+YPqD7Vck3FwZ8nvIewvdg4hrb+HTUdZS3q1I1bjEiZ/ZQBA== on .terraform/modules/datadog_log_forwarder/modules/log_forwarder/main.tf line 131, in resource "aws_s3_bucket_object" "this": 131: resource "aws_s3_bucket_object" "this" {

We need all these tags in other objects and we can't reduce them.

Could you, please, reduce or delete the number of tags in the module that can be attached to this object?

Terraform registry version is still 6.0.1

Description

Hello!
I would like to update this module to version 6.1.0, but the latest version available on the Terraform registry is still 6.0.1.
When I update my tf file and run terraform init, I get the following error:

There is no available version of module "registry.terraform.io/terraform-aws-modules/datadog-forwarders/aws//modules/log_forwarder" (datadog_log_forwarder.tf:5) which matches the given
version constraint. The newest available version is 6.0.1.

Could you update the version of Terraform registry?

• ✋ I have searched the open/closed issues and my issue is not listed.

⚠️ Note

Before you submit an issue, please perform the following first:

1. Remove the local .terraform directory (! ONLY if state is stored remotely, which hopefully you are following that best practice!): rm -rf .terraform/
2. Re-initialize the project root to pull down modules: terraform init
3. Re-attempt your terraform plan or apply and check if the issue still persists

Versions

• Module version [Required]: 6.1.0
• Terraform version: Terraform v1.9.0 on darwin_arm64
• Provider version(s):
  • • provider registry.terraform.io/hashicorp/aws v5.64.0
  • • provider registry.terraform.io/hashicorp/null v3.2.2
  • • provider registry.terraform.io/hashicorp/tls v4.0.5

Reproduction Code [Required]

module "datadog_forwarders_log_forwarder" {
  source  = "terraform-aws-modules/datadog-forwarders/aws//modules/log_forwarder"
  version = "6.1.0"

  runtime                   = "python3.11"
  forwarder_version         = "3.122.0"
  dd_api_key_secret_arn     = data.aws_secretsmanager_secret.dd_api_key.arn
  log_forwarder_bucket_tags = {
    Sample = "tag"
  }
}

Steps to reproduce the behavior:

• Execute terraform init.

Expected behavior

Finish without an error, like the example below (this example is for terraform-aws-modules/datadog-forwarders/aws 6.0.1):

❯ terraform init
Initializing the backend...
Initializing modules...
Downloading registry.terraform.io/terraform-aws-modules/datadog-forwarders/aws 6.0.1 for datadog_forwarders_log_forwarder...
- datadog_forwarders_log_forwarder in .terraform/modules/datadog_forwarders_log_forwarder/modules/log_forwarder
Downloading registry.terraform.io/terraform-aws-modules/s3-bucket/aws 3.6.1 for datadog_forwarders_log_forwarder.this_s3_bucket...
- datadog_forwarders_log_forwarder.this_s3_bucket in .terraform/modules/datadog_forwarders_log_forwarder.this_s3_bucket
Initializing provider plugins...
- Reusing previous version of hashicorp/null from the dependency lock file
- Reusing previous version of hashicorp/aws from the dependency lock file
- Reusing previous version of hashicorp/tls from the dependency lock file
- Using previously-installed hashicorp/null v3.2.2
- Using previously-installed hashicorp/aws v5.64.0
- Using previously-installed hashicorp/tls v4.0.5

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Actual behavior

❯ terraform init
Initializing the backend...
Initializing modules...
╷
│ Error: Unresolvable module version constraint
│ 
│   on datadog_log_forwarder.tf line 5:
│    5: module "datadog_forwarders_log_forwarder" {
│ 
│ There is no available version of module "registry.terraform.io/terraform-aws-modules/datadog-forwarders/aws//modules/log_forwarder" (datadog_log_forwarder.tf:5) which matches the given
│ version constraint. The newest available version is 6.0.1.

Additional context

I'm using terraform-aws-datadog-forwarders module from registry.terraform.io . If there is a walk around, let me know.

Compatability with Terraform 0.15+

Describe the bug
The list function is no longer present in Terraform 0.15.

To Reproduce
Just try to use this module with Terraform 0.15

Expected behavior
Working with the same behavior as in Terraform 0.14

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: macOS 11.2.3
• Version 0.15.0

Additional context
Add any other context about the problem here.

╷
│ Error: Error in function call
│ 
│   on .terraform/modules/datadog_forwarder.this_s3_bucket/outputs.tf line 3, in output "this_s3_bucket_id":
│    3:   value       = element(concat(aws_s3_bucket_policy.this.*.id, aws_s3_bucket.this.*.id, list("")), 0)
│ 
│ Call to function "list" failed: the "list" function was deprecated in
│ Terraform v0.12 and is no longer available; use tolist([ ... ]) syntax to
│ write a literal list.
╵
╷
│ Error: Error in function call
│ 
│   on .terraform/modules/datadog_forwarder.this_s3_bucket/outputs.tf line 8, in output "this_s3_bucket_arn":
│    8:   value       = element(concat(aws_s3_bucket.this.*.arn, list("")), 0)
│ 
│ Call to function "list" failed: the "list" function was deprecated in
│ Terraform v0.12 and is no longer available; use tolist([ ... ]) syntax to
│ write a literal list.
╵
╷
│ Error: Error in function call
│ 
│   on .terraform/modules/datadog_forwarder.this_s3_bucket/outputs.tf line 13, in output "this_s3_bucket_bucket_domain_name":
│   13:   value       = element(concat(aws_s3_bucket.this.*.bucket_domain_name, list("")), 0)
│ 
│ Call to function "list" failed: the "list" function was deprecated in
│ Terraform v0.12 and is no longer available; use tolist([ ... ]) syntax to
│ write a literal list.
╵
╷
│ Error: Error in function call
│ 
│   on .terraform/modules/datadog_forwarder.this_s3_bucket/outputs.tf line 18, in output "this_s3_bucket_bucket_regional_domain_name":
│   18:   value       = element(concat(aws_s3_bucket.this.*.bucket_regional_domain_name, list("")), 0)
│ 
│ Call to function "list" failed: the "list" function was deprecated in
│ Terraform v0.12 and is no longer available; use tolist([ ... ]) syntax to
│ write a literal list.
╵
╷
│ Error: Error in function call
│ 
│   on .terraform/modules/datadog_forwarder.this_s3_bucket/outputs.tf line 23, in output "this_s3_bucket_hosted_zone_id":
│   23:   value       = element(concat(aws_s3_bucket.this.*.hosted_zone_id, list("")), 0)
│ 
│ Call to function "list" failed: the "list" function was deprecated in
│ Terraform v0.12 and is no longer available; use tolist([ ... ]) syntax to
│ write a literal list.
╵
╷
│ Error: Error in function call
│ 
│   on .terraform/modules/datadog_forwarder.this_s3_bucket/outputs.tf line 28, in output "this_s3_bucket_region":
│   28:   value       = element(concat(aws_s3_bucket.this.*.region, list("")), 0)
│ 
│ Call to function "list" failed: the "list" function was deprecated in
│ Terraform v0.12 and is no longer available; use tolist([ ... ]) syntax to
│ write a literal list.
╵
╷
│ Error: Error in function call
│ 
│   on .terraform/modules/datadog_forwarder.this_s3_bucket/outputs.tf line 33, in output "this_s3_bucket_website_endpoint":
│   33:   value       = element(concat(aws_s3_bucket.this.*.website_endpoint, list("")), 0)
│ 
│ Call to function "list" failed: the "list" function was deprecated in
│ Terraform v0.12 and is no longer available; use tolist([ ... ]) syntax to
│ write a literal list.
╵
╷
│ Error: Error in function call
│ 
│   on .terraform/modules/datadog_forwarder.this_s3_bucket/outputs.tf line 38, in output "this_s3_bucket_website_domain":
│   38:   value       = element(concat(aws_s3_bucket.this.*.website_domain, list("")), 0)
│ 
│ Call to function "list" failed: the "list" function was deprecated in
│ Terraform v0.12 and is no longer available; use tolist([ ... ]) syntax to
│ write a literal list.
╵

make kms optional

Is your feature request related to a problem? Please describe.
We're using hashicorp vault to store all secrets so the required kms key is not suitable.

Describe the solution you'd like
make kms keys optional

Create Lambda trigger for RDS EM forwarder

Is your request related to a new offering from AWS?

No

Is this functionality available in the AWS provider for Terraform? See CHANGELOG.md, too.

• Yes ✅: please list the AWS provider version which introduced this functionality

Is your request related to a problem? Please describe.

The RDS enhanced metrics forwarder module doesn't create the Lambda trigger for sending metrics from the RDSOSMetrics CloudWatch log group.

Describe the solution you'd like.

Add a the trigger as either a subscription filter on the log group or a trigger on the Lambda function.

Describe alternatives you've considered.

We've had to Terraform the trigger/filter separately, but there's no obvious reason why it can't be included in this module.

Additional context

Basically step 11 on the installation docs onwards:

https://docs.datadoghq.com/integrations/amazon_rds/?tab=enhanced#installation

S3 GetObject permission needs wildcard qualifier

Shouldn't the GetObject permission be <bucket>/* rather than just <bucket>?

terraform-aws-datadog-forwarders/modules/log_forwarder/policy.tmpl

Line 41 in 3920c11

"Resource": ${s3_log_bucket_arns},

KMS permissions should be tighter ?

The change here:

851257b

... seems to grant the lambdas permission to kms:Decrypt on any KMS key, which seems bad from a security perspective?

Thanks!

Allow for custom KMS key encryption on bucket created instead of standard AES256

Is your feature request related to a problem? Please describe.

Using AWS default encryption key is generally discouraged, best practice is to use customer-managed keys, therefore it would be great if the module would allow specifying a customer-managed key. Ref: tfsec.dev/docs/aws/s3/enable-bucket-encryption

Describe the solution you'd like
Ability to provide a CMK for bucket encryption

Describe alternatives you've considered
None

Additional context
From convo on #9

Again: Unable to import module 'lambda_function': cannot import name '_rand' from 'ddtrace.internal'

Description

Upgraded from 4.1.0 to 5.0.1 and the log forwarding function stopped working. Played with Python version - 3.7 and 3.9 fail, 3.8 works fine. Datadog doc clearly says to use Python 3.8 https://docs.datadoghq.com/logs/guide/forwarder/
"Since version 3.49.0 the Lambda function has been updated to require Python 3.8. If upgrading an older forwarder installation to 3.49.0 or above, ensure the AWS Lambda function is configured to use Python 3.8"

Versions

• Module version [Required]:
  5.0.1
• Terraform version:
  Terraform v1.3.0
  on linux_amd64

• provider registry.terraform.io/hashicorp/aws v4.27.0
• provider registry.terraform.io/hashicorp/external v2.2.3
• provider registry.terraform.io/hashicorp/local v2.3.0
• provider registry.terraform.io/hashicorp/null v3.2.1
• provider registry.terraform.io/hashicorp/random v3.4.3

Reproduction Code [Required]

module "datadog_forwarders" {
  source     = "terraform-aws-modules/datadog-forwarders/aws"
  version    = "5.0.1"

  log_forwarder_name                           = var.name
  create_rds_em_forwarder                      = false
  create_vpc_fl_forwarder                      = false
  dd_api_key_secret_arn                        = data.aws_secretsmanager_secret.datadog_api_key.arn
  kms_alias                                    = aws_kms_alias.datadog.name
  log_forwarder_reserved_concurrent_executions = var.log_forwarder_reserved_concurrent_executions

  tags = var.tags
}

Actual behavior

The function fails with:
Runtime.ImportModuleError: Unable to import module 'lambda_function': cannot import name '_rand' from 'ddtrace.internal' (/var/task/ddtrace/internal/__init__.py)

Additional context

Default Python version should be changed to 3.8

Not forwarding S3 logs to Datadog?

Description

This might be halfway between a "bug" (because of the description) and a feature request at the same time...

The s3_log_bucket_arns variable in https://github.com/terraform-aws-modules/terraform-aws-datadog-forwarders/tree/master/modules/log_forwarder has a description that mentions:

S3 log buckets for forwarder to read and forward logs to Datadog

...yet, looking through the code (and also trying it out), is it possible that it does not take care of adding the actual forwarding of logs to Datadog?

I was expecting to see a aws_s3_bucket_notification resource, which I believe would have taken care of this part, but there does not seem to be one? Or might I be missing something?

• ✋ I have searched the open/closed issues and my issue is not listed.

Versions

• Module version [Required]: 6.0.1

• Terraform version: 1.8.3

• Provider version(s):

+ provider registry.terraform.io/datadog/datadog v3.39.0
+ provider registry.terraform.io/hashicorp/aws v5.49.0
+ provider registry.terraform.io/hashicorp/null v3.2.2
+ provider registry.terraform.io/hashicorp/random v3.6.1

Expected behavior

Lambda function being aware of new logs in S3 bucket and forwarding those to Datadog. Lambda logs/metrics showing activity. New logs appearing in Datadog.

Actual behavior

S3 logs are not getting forwarded by the lambda function. Lambda function logs are empty. Nothing appearing in Datadog logs.

module doesn't return `lambda_arn`

module doesn't return lambda_arn

Error: Unsupported attribute

  on datadog/rds_enhanced_monitoring.tf line 27, in output "debug":
  27:   value = module.datadog_rds_enhanced_monitoring_forwarder.lambda_arn

This value does not have any attributes.

Error: cannot import name '_rand' with latest module version

Description

Please provide a clear and concise description of the issue you are encountering, and a reproduction of your configuration (see the examples/* directory for references that you can copy+paste and tailor to match your configs if you are unable to copy your exact configuration). The reproduction MUST be executable by running terraform init && terraform apply without any further changes.

If your request is for a new feature, please use the Feature request template.

• ✋ I have searched the open/closed issues and my issue is not listed.

⚠️ Note

Before you submit an issue, please perform the following first:

1. Remove the local .terraform directory (! ONLY if state is stored remotely, which hopefully you are following that best practice!): rm -rf .terraform/
2. Re-initialize the project root to pull down modules: terraform init
3. Re-attempt your terraform plan or apply and check if the issue still persists

Versions

• Module version [Required]:

• Terraform version:
  Terraform v1.4.3-dev
  on darwin_amd64

• provider registry.terraform.io/datadog/datadog v3.23.0
• provider registry.terraform.io/hashicorp/aws v4.61.0
• provider registry.terraform.io/hashicorp/null v3.2.1

• Provider version(s):
  Terraform v1.4.3-dev
  on darwin_amd64

• provider registry.terraform.io/datadog/datadog v3.23.0
• provider registry.terraform.io/hashicorp/aws v4.61.0
• provider registry.terraform.io/hashicorp/null v3.2.1

Reproduction Code [Required]

module "datadog_forwarders" {
  source  = "terraform-aws-modules/datadog-forwarders/aws"

  kms_alias             = "alias/datadog"
  dd_api_key_secret_arn = data.aws_secretsmanager_secret_version.datadog_api_key.arn

  create_log_forwarder    = true
  create_rds_em_forwarder = true
  create_vpc_fl_forwarder = false

  vpc_id = "XXXXX"
}

Steps to reproduce the behavior:

terraform apply

Expected behavior

The log forwarder should be created and should forward logs

Actual behavior

Log forwarder throws error:

[ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function': cannot import name '_rand' from 'ddtrace.internal' (/var/task/ddtrace/internal/__init__.py)
Traceback (most recent call last):

Error with 3.7.0: Unable to import module 'lambda_function': cannot import name '_rand' from 'ddtrace.internal'

Describe the bug

Hey there, so I recently attempted an upgrade of this module to the latest version (3.7.0) from 3.1.1 and the forwarder Lambda started failing and spitting out these logs:

[ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function': cannot import name '_rand' from 'ddtrace.internal' (/var/task/ddtrace/internal/__init__.py) 

I was previously pinned to version 3.1.1 of this module and 3.40.0 of the Datadog forwarder and tried to update to 3.7.0 and 3.44.0, respectively. Though if I pin to 3.7.0 and forwarder version 3.40.0, I see this in the plan and it still breaks:

  ~ resource "aws_lambda_function" "this" {
      ~ architectures                  = [
          + "x86_64",
        ]
        id                             = "preprod-datadog-logs-forwarder"
      ~ last_modified                  = "2022-03-23T17:50:35.000+0000" -> (known after apply)
      ~ runtime                        = "python3.7" -> "python3.9"
        tags                           = {
            "DD_FORWARDER_VERSION" = "3.40.0"
            "Env"                  = "preprod"
        }
        # (20 unchanged attributes hidden)


        # (2 unchanged blocks hidden)
    }

I'm guessing this has to do with the architecture and Python runtime updates introduced in v3.5.0, and I found your GitHub issue with the exact same log over here: DataDog/datadog-lambda-python#88 ...but I'm not clear on how to remediate and I'd love any insight or suggestions you might have as it seems you've dug into it pretty deep.

To Reproduce
This is my old configuration:

module "datadog_log_forwarder" {
  source = "clowdhaus/datadog-forwarders/aws"
  version = "3.1.1"

  dd_api_key            = ""
  dd_api_key_secret_arn = var.api_key_secret_arn
  kms_alias             = var.kms_key_alias

  log_forwarder_version = "3.40.0"
  log_forwarder_name    = "${var.environment_name}-datadog-logs-forwarder"
  log_forwarder_environment_variables = {
    DD_TAGS = "env:${local.dd_env}"
  }
  bucket_name             = "my-bucket-name"
  create_vpc_fl_forwarder = false
}

And this is the new configuration that broke, though the log_forwarder_version can be set to 3.40.0 and the error still occurs:

module "datadog_log_forwarder" {
  source = "clowdhaus/datadog-forwarders/aws"
  version = "3.7.0"

  dd_api_key            = ""
  dd_api_key_secret_arn = var.api_key_secret_arn
  kms_alias             = var.kms_key_alias

  log_forwarder_version = "3.44.0"
  log_forwarder_name    = "${var.environment_name}-datadog-logs-forwarder"
  log_forwarder_environment_variables = {
    DD_TAGS = "env:${local.dd_env}"
  }
  bucket_name             = "my-bucket-name"
  create_vpc_fl_forwarder = false
}

Expected behavior

The forwarder Lambda function continues to run without error

Thanks!

Tighten S3 bucket security

tfsec reports the following issues:

• _forwarder.this_s3_bucket/main.tf
• [aws-s3-block-public-policy][HIGH] - Resource 'module.web_bucket:aws_s3_bucket_public_access_block.this[0]' sets block_public_policy explicitly to false
• [aws-s3-enable-bucket-encryption][HIGH] - Resource 'module.web_bucket:aws_s3_bucket.this[0]' defines an unencrypted S3 bucket (missing server_side_encryption_configuration block).
• [aws-s3-ignore-public-acls][HIGH] - Resource 'module.web_bucket:aws_s3_bucket_public_access_block.this[0]' sets ignore_public_acls explicitly to false
• [aws-s3-no-public-access-with-acl][CRITICAL] - Resource 'module.web_bucket:aws_s3_bucket.this[0]' has an ACL which allows public access.
• [aws-s3-no-public-buckets][HIGH] - Resource 'module.web_bucket:aws_s3_bucket_public_access_block.this[0]' sets restrict_public_buckets explicitly to false