Terraform Version
0.12-dev
Affected Resource(s)
template provider, template_dir data source
Expected behavior
This is a proposal to improve the way user files are managed in Terraform.
One user story
User story: "I would like to deploy static websites to S3, using only Terraform."
Problem description:
- Fact nr.1: the current
aws_s3_bucket_object
is capable of uploading a well-defined file to S3
- Fact nr.2: Using
count
(or the soon-to-be for-each) it is possible to create a set of aws_s3_bucket_object
s to upload a set of files. This requires a list-type variable that can be iterated.
- Problem nr.1: there is no Terraform-native way to read a list of files from a directory. (As mentioned in hashicorp/terraform/issues/16697 and worked around with https://github.com/saymedia/terraform-s3-dir)
- Problem nr.2: lists are not supported as Terraform input parameters (at least
-var
didn't support them a few versions ago).
So if I have a directory with a bunch of files (not exactly identified by name for Terraform), I have no way of iterating through them.
The one exception is, if they are templates and I need them rendered on the same machine. In that case the template_dir
resource in the template
provider will do it for me.
Generic user story
It seems that there are a few cases where people want to iterate through a directory. An additional one is hashicorp/terraform/issues/6065 where the user wants to use the file provisioner to keep a directory up-to-date but the file provisioner doesn't keep track of changes. It seems that a provider iterating through directories would be useful.
Originally, I thought this should be a custom directory provider and just write it for my use-case. Then I thought why not try to do it in a Terraform-native-way and try to integrate with the current providers.
The closest provider that iterates through directories is the already mentioned template_dir
resource. Unfortunately, if I generally want to read the folders but not write them, I'm out of luck. As it was raised in #34, there is no template_dir
data source, only resource.
Proposed solution nr.1 (already done)
New data source: template_dir
. This can be considered a non-breaking improvement since it didn't exist before. Based on how the template_dir
resource is defined (and trying to keep in line with it), here's how it would look like:
data "template_dir" "solution1" {
source_dir = "website" # Same as in the resource
exclude = "\.tmp$" # Improvement: regular expression applied to the relative path
vars { # Same as in the resource
"name": "mywebsite"
}
render: true # Improvement: details below (consider files as templates)
}
Output variable rendered
(same as in the resource) would be a map
where the keys are the file names and optionally the values are the rendered files (in case of templates) or empty if render = false
. (You can create a list of files using keys(data.template_dir.solution1.rendered)
and iterate through it, for example to upload them to S3.)
So, the idea is to stuff this functionality into the template provider, even though the template provider is a bit more than just a file-system reader.
I already programmed this solution for 0.11 because I needed it for my use-case. Unfortunately, because of a bug in 0.11 (hashicorp/terraform/issues/19258), this will only work correctly in 0.12. I'm in the process of upgrading it and I'll share afterwards.
Pro: in-line with current expectations on how Terraform works; non-breaking change
Con: It's abusing the template
provider to add functionality.
Proposed solution nr.2
New provider: io
or built-in functions.
It seems that generic file reads (and writes) are in demand. In hashicorp/terraform/issues/16697 there is a discussion about a set of builtin functions to manage files in a directory. It seems that those discussions didn't amount to anything, but they might still be good ideas.
So the request: is it possible to get a bit of brainstorming together for an overarching solution? Is it better to use providers or builtin functions? I'm willing to look into any of it that makes sense because I would rather have a terraform-native solution (eventually) to this problem, than a custom provider.
Until a better idea emerges, I'll keep working on solution nr.1.