This originally spawned from a reddit thread asking how to create resources from a CSV file.
This repo contains a slightly edited version of my answer, along with the code I used just for reference.
There's no simple answer here. I will warn you that the solution is very obtuse and I do not recommend using it.
I took a sample CSV and realized that to get it into terraform, the only two routes are to either convert it to a single JSON object and then read it in with the external data source, or to build a CSV parser out of terraform interpolation syntax. We'll try both.
First, let's look at using the External Data source. This one front-loads the insanity by using JQ to format the data for consumption by the external provider:
data "external" "csv_file" {
program = ["jq", "--slurp", "--raw-input", "--raw-output", "split(\"\n\") | .[1:] | map(select(length > 0) | split(\",\")) | map({\"name\": .[0], \"value\": .[1], \"description\": .[2], \"tag\": .[3]}) | {\"names\": map(.name) | join(\",\"), \"values\": map(.value) | join(\",\"), \"description\": map(.description) | join(\",\"), \"tag\": map(.tag) | join(\",\")}", "${path.module}/sample.csv"]
}
resource "null_resource" "csv_external_data_source_method" {
count = "${length(split(",", data.external.csv_file.result.names))}"
triggers = {
name = "${element(split(",", data.external.csv_file.result.names), count.index)}"
value = "${element(split(",", data.external.csv_file.result.values), count.index)}"
description = "${element(split(",", data.external.csv_file.result.description), count.index)}"
tag = "${element(split(",", data.external.csv_file.result.tag), count.index)}"
}
}
That single resource gives us a terraform plan
output that will count out to the end of the CSV and generate a resource for each CSV entry:
+ null_resource.csv_external_data_source_method[0]
id: <computed>
triggers.%: "4"
triggers.description: "geolocation"
triggers.name: "China-1.0.1.0-24"
triggers.tag: "maxmind"
triggers.value: "1.0.1.0/22"
+ null_resource.csv_external_data_source_method[1]
id: <computed>
triggers.%: "4"
triggers.description: "geolocation"
triggers.name: "China-1.0.2.0-24"
triggers.tag: "maxmind"
triggers.value: "1.0.2.0/23"
You'll obviously need to have jq installed.
It's worth mentioning that this can probably be made less crazy by using a script to parse the data and feed something a little more dynamic to JQ, but JQ and I don't have quite that kind of relationship ;)
This strategy has us performing lots of repetitive interpolation thanks to a few limitations in the null data source. In this example, the data source is pretty short, but the resource where we use the data is incomprehensibly long, with each key in the resource using no fewer than eight interpolation functions:
data "null_data_source" "csv_file" {
inputs = {
file_data = "${chomp(file("${path.module}/sample.csv"))}"
}
}
resource "null_resource" "csv_interpolation_method" {
count = "${length(slice(split("\n", lookup(data.null_data_source.csv_file.outputs, "file_data")), 1, length(split("\n", lookup(data.null_data_source.csv_file.outputs, "file_data")))))}"
triggers = {
name = "${element(split(",", element(slice(split("\n", lookup(data.null_data_source.csv_file.outputs, "file_data")), 1, length(split("\n", lookup(data.null_data_source.csv_file.outputs, "file_data")))), count.index)), 0)}"
value = "${element(split(",", element(slice(split("\n", lookup(data.null_data_source.csv_file.outputs, "file_data")), 1, length(split("\n", lookup(data.null_data_source.csv_file.outputs, "file_data")))), count.index)), 1)}"
description = "${element(split(",", element(slice(split("\n", lookup(data.null_data_source.csv_file.outputs, "file_data")), 1, length(split("\n", lookup(data.null_data_source.csv_file.outputs, "file_data")))), count.index)), 2)}"
tag = "${element(split(",", element(slice(split("\n", lookup(data.null_data_source.csv_file.outputs, "file_data")), 1, length(split("\n", lookup(data.null_data_source.csv_file.outputs, "file_data")))), count.index)), 3)}"
}
}
Nonetheless, we end up with identical terraform plan
output:
+ null_resource.csv_interpolation_method[0]
id: <computed>
triggers.%: "4"
triggers.description: "geolocation"
triggers.name: "China-1.0.1.0-24"
triggers.tag: "maxmind"
triggers.value: "1.0.1.0/22"
+ null_resource.csv_interpolation_method[1]
id: <computed>
triggers.%: "4"
triggers.description: "geolocation"
triggers.name: "China-1.0.2.0-24"
triggers.tag: "maxmind"
triggers.value: "1.0.2.0/23"
Of the two, I actually consider the interpolation technique to be the better choice. It's pure terraform and somehow less obtuse. Somehow. Now, let's walk through some of the interpolation:
split("\n", lookup(data.null_data_source.csv_file.outputs, "file_data"))
- This gives us a list where each element of the list is a line of our CSV file.
slice(split("\n", lookup(data.null_data_source.csv_file.outputs, "file_data")), 1, length(split("\n", lookup(data.null_data_source.csv_file.outputs, "file_data"))))
- We need to remove the CSV header, so we'll
slice()
it starting from line 1 and going to thelength()
of the list itself.
count = "${length(slice(split("\n", lookup(data.null_data_source.csv_file.outputs, "file_data")), 1, length(split("\n", lookup(data.null_data_source.csv_file.outputs, "file_data")))))}"
- The
length()
of the now header-free list is how many times we want tocount
the resource we're creating from the CSV data.
element(slice(split("\n", lookup(data.null_data_source.csv_file.outputs, "file_data")), 1, length(split("\n", lookup(data.null_data_source.csv_file.outputs, "file_data")))), count.index)
- We take our primitive from above and use
element()
to choose the item from the list that matches the currentcount.index
. At this point, it's raw CSV likeChina-1.0.1.0-24,1.0.1.0/22,geolocation,maxmind
split(",", element(slice(split("\n", lookup(data.null_data_source.csv_file.outputs, "file_data")), 1, length(split("\n", lookup(data.null_data_source.csv_file.outputs, "file_data")))), count.index))
- We take the CSV and
split()
it on every comma to get a list where each element is a column of the CSV line
name = "${element(split(",", element(slice(split("\n", lookup(data.null_data_source.csv_file.outputs, "file_data")), 1, length(split("\n", lookup(data.null_data_source.csv_file.outputs, "file_data")))), count.index)), 0)}"
- Finally, we use
element()
to extract the contents of a specific column as the value for each of the keys in our resource.
You can do it, but the lack of any real answer elsewhere on the internet... turns out it's for a reason. There's special code in each of these techniques to ensure that the first line of the CSV isn't processed, so your CSV must have a header just like the example one does if you see fit to use this.
I'm only really writing this up here because it's a topic that I couldn't find really anywhere on the Internet, and since it's possible, I decided I had to figure out just how it would have to work. Perhaps solving this here could lead to a better implementation by someone else, but I'm not really sure about that. Terraform could probably stand to have some proper CSV/JSON support instead. Also, this technically doesn't even follow the spec because it doesn't handle properly-escaped commas which CAN exist in a CSV file.
Cheers.