flosell / iam-policy-json-to-terraform

Small tool to convert an IAM Policy in JSON format into a Terraform aws_iam_policy_document

iam-policy-json-to-terraform

Small tool to convert an IAM Policy in JSON format into a Terraform aws_iam_policy_document

Web Version

Check out a web version of the tool here.

For command line usage and automation, check out the instructions below.

Installation

OSX

$ brew install iam-policy-json-to-terraform

asdf

$ asdf plugin add iam-policy-json-to-terraform https://github.com/carlduevel/asdf-iam-policy-json-to-terraform.git

Other

Download the latest binary from the releases page and put it into your PATH under the name iam-policy-json-to-terraform

Developer

If you're a go developer and have your GOPATH defined and have added your $GOPATH/bin directory to your path, you can simply run this command.

$ go install github.com/flosell/iam-policy-json-to-terraform@latest

Usage

From raw JSON

$ echo '{"Statement":[{"Effect":"Allow","Action":["ec2:Describe*"],"Resource":"*"}]}' | iam-policy-json-to-terraform
data "aws_iam_policy_document" "policy" {
  statement {
    sid       = ""
    effect    = "Allow"
    resources = ["*"]
    actions   = ["ec2:Describe*"]
  }
}

From a JSON policy file

$ iam-policy-json-to-terraform < some-policy.json

If a video demo is more your thing, checkout this nice 2min introduction by the folks at env0.

Local development

Prerequisites

• Clone the repository to a location of your choosing:

  $ git clone [email protected]:flosell/iam-policy-json-to-terraform.git

• Install dependencies and tools:

  $ cd iam-policy-json-to-terraform
  $ make tools

Development

Entry point

make is your primary point of entry for any development activity. Call it without arguments to learn more:

$ make
build                          Test and build the whole application
clean                          Remove build artifacts
fmt                            Format code
fmtcheck                       Run linter
seccheck                       Run security checks
test                           Run all tests
test-readme                    Run the commands mentioned in the README for sanity-checking
tools                          Install additional required tooling
tools-main                     Install additional required tooling for the main version
tools-web                      Install additional required tooling for the web version
web-build                      Build the web version
web-deploy                     Deploy the web version to GitHub pages
web-e2e                        Run end to end tests for web version (requires web-build)
web-e2e-live                   Run end to end tests for web version in live mode for development (requires web-build)
web-serve                      Serve the web version on a local development server
web-visual-regression-approve  Accept changes in Web UI visuals
web-visual-regression-test     Test for changes in Web UI visuals

Web Development

To develop the web-frontend, you'll need to first compile the JavaScript version of iam-policy-json-to-terraform. make web-build will do that, generating a web.js file. Include it and it'll expose a convert(policyName,jsonString) function in the global namespace.

Currently, the complete web-frontend is plain HTML, JS and CSS, all within web/index.html. Edit or refine as needed.

End-To-End Tests for the web frontend exist as TestCafe tests in web_test.js and can be run using make web-e2e.

iam-policy-json-to-terraform's Issues

Parsing issues

Given examples here:

 {
     "Version": "2012-10-17",
     "Id": "PutObjPolicy",
     "Statement": [
           {
                "Sid": "DenyIncorrectEncryptionHeader",
                "Effect": "Deny",
                "Principal": "*",
                "Action": "s3:PutObject",
                "Resource": "arn:aws:s3:::<bucket_name>/*",
                "Condition": {
                        "StringNotEquals": {
                               "s3:x-amz-server-side-encryption": "AES256"
                         }
                }
           },
           {
                "Sid": "DenyUnEncryptedObjectUploads",
                "Effect": "Deny",
                "Principal": "*",
                "Action": "s3:PutObject",
                "Resource": "arn:aws:s3:::<bucket_name>/*",
                "Condition": {
                        "Null": {
                               "s3:x-amz-server-side-encryption": true
                        }
               }
           }
     ]
 }

1st, does not recognise Principal:*

unable to convert: json: cannot unmarshal string into Go struct field jsonStatement.Principal of type map[string]converter.stringOrStringArray

2nd, does not convert json booleans to terraform strings

    condition {
      test     = "Null"
      variable = "s3:x-amz-server-side-encryption"
      values   = [""]
    }

Add to package managers like homebrew

Would be nice to install this using a package managers

brew install iam-policy-json-to-terraform
choco install iam-policy-json-to-terraform
scoop install iam-policy-json-to-terraform

Thanks!

Provide more examples of where to start the JSON, multiple policies in one, etc., plus more informative messaging

I keep pasting different sections of IAM policy JSON into the web converter and nothing happens. Which section of policy do I start at? Can I include multiple policies in one? The tool isn't giving me any feedback.

This sounds great but it is not working for me without some more specific information.

For example:

{
            "PolicyName": "DeployLambdaFunctions",
            "PolicyDocument": {
                "Version": "2012-10-17",
                "Statement": [
                    {
                        "Effect": "Allow",
                        "Action": [
                            "lambda:Get*",
                            "lambda:List*",
                            "lambda:CreateFunction",
                            "lambda:DeleteFunction",
                            "lambda:CreateFunction",
                            "lambda:DeleteFunction",
                            "lambda:UpdateFunctionConfiguration",
                            "lambda:UpdateFunctionCode",
                            "lambda:PublishVersion",
                            "lambda:CreateAlias",
                            "lambda:DeleteAlias",
                            "lambda:UpdateAlias",
                            "lambda:AddPermission",
                            "lambda:RemovePermission",
                            "lambda:InvokeFunction"
                        ],
                        "Resource": [
                            {
                                "Fn::Sub": "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${AppId}-*"
                            }
                        ]
                    }
                ]
            }
        }

...results in nothing:

Other pastes w/ actual formatting issues do give some error output, so it's not like the whole web app isn't working for me.

I'm sure I'm doing something wrong, but if it's not too much trouble to hint at what it could be, that would be awesome!

This could just be my ignorance though!!

Enhancement: Linux ARM build

Provider generic ARM release.

I see you have both Darwin and Darwin_arm (M1/2/3). Could we also get a generic ARM build? Myself and teams are migrating to ARM based compute resources and it would be great to continue using your tool.

Thank you

Try out replacing GopherJS with Web Assembly

GopherJS seems to be a few versions behind normal go releases, occasionally causing issues with library upgrades (e.g. gosec).
Go these days has native support for WebAssembly so this might be an easier way to get our web frontend in front of users;

Two main things to figure out:

• Is it possible to swap out GopherJS with Go WASM with reasonable effort?
• Does the switch preserve the current state of performance, artifact size (TinyGo might help there) and so on?

Be able to convert documents that have embedded terraform interpolations that break JSON parsing

(triggered by #13)

When writing policy JSONs inline as heredoc, one often includes terraform interpolations for certain values, e.g.

locals {
  path_elements = ["foo","bar"]
}

resource "aws_iam_policy" "policy" {
  name        = "test_policy"
  path        = "/"
  description = "My test policy"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "s3:Get*"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::foo/${join("/",local.path_elements)}"
    }
  ]
}
EOF
}

In this example, the extra quotes inside the interpolation make this invalid JSON, we can't parse it.

Add archlinux installation guide

I have added this tool on archlinux aur, feel free to add this on installation guide.

https://aur.archlinux.org/packages/iam-policy-json-to-terraform/

Thank you.

Hey I just used this to move a rather large KMS policy document from inline local into a policy_document and it was a breeze.

Just wanted to say kudos you saved me an hour of my life.

Make web frontend prettier

Context

#24 introduced a basic web frontend to use. At the moment, it provides only very bare-bones functionality with minimal styling. I'd expect this to become a bit nicer to look at while not distracting from the overall functionality

Outcomes

• A web frontend with a nice, clean looking UI - I don't have a strong opinion what "nice" and "clean" should be, however if you are thinking of contributing and are unsure, feel to post your ideas here and I'll give feedback before you invest too much time
• Prioritise function and usability over prettiness - this tool is used by people who quickly need to achieve a task (to convert a json policy into a different format). The design shouldn't distract from this.

Constraints

• My own frontend-skills exist but are rusty - you are free to choose the best tools and frameworks for the job, however I'd prefer minimal learning curves and complexity. In doubt, err on the side of simplicity and well known tools for myself and other contributors to maintain this in the long run
• The tool is primarily used by infrastructure people for their work - I assume most will use it on their desktops and laptops. Therefore, responsive design or mobile friendly features are not a priority (however nice to have).

Related Issues

There are several other issues dealing with the web frontend - check the web-frontend label to get an understanding where overlaps might exist.
#30 would introduce basic mobile friendliness topics, mainly from a SEO perspective.

Feature: rewrite .tf files with embedded policy heredocs

(triggered by #13)

Introduce a new feature that would be able to read in a terraform file that contains JSON in heredoc and replace the occurrances with actual policy documents:

resource "aws_iam_policy" "policy" {
  name        = "test_policy"
  path        = "/"
  description = "My test policy"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "ec2:Describe*"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}
EOF
}

to

data "aws_iam_policy_document" "policy" {
  statement {
    sid       = ""
    effect    = "Allow"
    resources = ["*"]
    actions   = ["ec2:Describe*"]
  }
}

resource "aws_iam_policy" "policy" {
  name        = "test_policy"
  path        = "/"
  description = "My test policy"

  policy = data.aws_iam_policy_document.policy.json
}

Auto-convert in web frontend

Context

#24 introduced a basic web frontend to use. At the moment, users need to push the > button to trigger a conversion. This feature would simplify this by automatically re-converting whenever the input document changes (and is valid).

Outcomes

• Given any existing input, when the users change the input into a valid input document, the output document should automatically change
• Given any existing input, when the users change the input document to become invalid, the output document should not change
• Given any existing input, when the users change the input document to become invalid, an error message should appear

Replace github-release tool in release-script

The the release script currently uses github-release to create releases and update binaries to GitHub.

Looks like this project is no longer maintained and uses deprecated APIs. Use a maintained alternative (e.g. hub) instead.

Web frontend on github pages using gopherjs

I came across this project https://github.com/gopherjs/gopherjs and it seems like it would be possible to transpile golang to js and reuse the code base to build a frontend for this tool which can then be backed by github pages.

Imagine going to the website, pasting the json iam policy there, and having it convert to the terraform data source.

Low priority, nice to have, and possibly a fun weekend project for someone (maybe me...maybe 😄 )

Reverse terraform back to json

I know I could simply get the output

temp_tf_dir=/tmp/tf-get-json-of-iam-policy
mkdir -p $temp_tf_dir
rm $temp_tf_dir/*
cd $temp_tf_dir
# paste contents of data source
# add an output
vim $temp_tf_dir/main.tf
terraform apply

But it would be nice if the tool allowed me to do it without having to create all of the above

Proposals to make it easier to use

I often use vscode to update terraform. When I see json, I usually have to copy the json to my clipboard, open my terminal, and begin with the following command.

cat <<EOF | iam-policy-json-to-terraform

Then hit enter, paste from my clipboard, hit enter, type EOF, and hit enter again.

It's a bit painful but I've done it so many times that I've almost stopped forgetting the order of the hyphenated command (was it policy then json or json then policy?).

Here are some suggestions to make this easier

1. vscode extension that could auto detect json and allow converting all or specific jsons to policy documents

2. A repl command where you can run the command without stdin and then paste the json to get the output

   $ iam-policy-json-to-terraform
   > 
   > {...json...}
   data aws_iam_policy_document...
   

3. Easier to remember command name

   Perhaps using alias commands like

   iam-json-policy-to-terraform
   iam-json-to-tf
   ijtt
   

4. Run command in a module and have it convert all json policies to data policy documents

Would be interested in hearing the owner and community's thoughts on these. Thank you.

Bookmarklet to open web frontend with currently selected JSON document

Context

#24 introduced a basic web frontend to use. I'd assume many people will use this to convert a JSON document they found on in the AWS documentation, in tutorials or other websites.
This feature would make this task easier

Outcomes

Introduce a way to feed selected text from any website into the web frontend input (e.g. a bookmarklet that generates a URL that's read by the web frontend)

Not parsing correctly adding this statement = [

Doing the same thing for me

data "aws_iam_policy_document" "policy" {
  statement = [
    {
      sid       = ""
      effect    = "Allow"
      resources = ["*"]

      actions = [
        "autoscaling:DescribeAutoScalingGroups",
        "autoscaling:DescribeAutoScalingInstances",
        "autoscaling:SetDesiredCapacity",
        "autoscaling:TerminateInstanceInAutoScalingGroup",
        "autoscaling:DescribeTags",
      ]
    },
  ]
}

Error message

An argument named "statement" is not expected here. Did you mean to define a
block of type "statement"?

Originally posted by @k8s-class in #9 (comment)

Remove superflous quotes when dealing with inline terraform expressions

(triggered by #13)

When writing policy JSONs inline as heredoc, one often includes terraform expressions for certain values, e.g.

"Resource": ["${aws_kms_key.key.arn}"]

Currently, those are converted into normal string arrays with interpolations, e.g.

resources = ["${aws_kms_key.key.arn}"]

Since terraform 0.12, there's a more elegant way to express this, would be nice for iam-policy-json-to-terraform to render them:

resources = [ aws_kms_key.key.arn ]

Create new version for packaging purposes

Hello, can you please bump a version so we could update it nixpkgs here: https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/tools/misc/iam-policy-json-to-terraform/default.nix#L5.

Thanks,
Ivan.

Basic mobile-friendliness for web-frontend to appease Google

Context

The tool is primarily used by infrastructure people for their work - I assume most will use it on their desktops and laptops. However, Google penalises web-pages that aren't mobile-friendly at all.

Outcomes

Implement basic mobile friendliness (use Googles Tools as a guide) so that the page isn't penalised any more.
Improve the styling just enough so the tools has some basic usability on mobile devices for those who need it (or stumble upon it on their mobile and want to explore).

Related issues

#25 deals with making the web version prettier overall so someone working on that part might already have made some tooling choices or even made this issue redundant.

Link to the CLI from the web frontend

Some people might first end up on the web version of this tool, yet have use cases (e.g. bulk conversion) where the CLI works better - add a link to GitHub that promotes the CLI

Make it easier to copy-paste output in web-ui

Usually, we probably want to copy the output of the conversion back into our IDE. Explore what we can do to make this easier.

One idea would be to use the toolbar and copy-to-clipboard Prism plugins for this

Issues with validation when using Terraform v0.12.20+ and installing with `go get`

The current output looks like:

data "aws_iam_policy_document" "policy" {
  statement = [
    {
      sid       = ""
      effect    = "Allow"
      resources = ["arn:aws:dynamodb:ap-southeast-2:<account-id>:table/<redacted>"]

      actions = [
        "dynamodb:GetItem",
        "dynamodb:PutItem",
        "dynamodb:DeleteItem",
        "dynamodb:Scan",
        "dynamodb:GetRecords",
        "dynamodb:DescribeTable",
        "dynamodb:DescribeStream",
        "dynamodb:GetShardIterator",
        "dynamodb:ListStreams",
        "dynamodb:Query",
      ]
   }
]

But Terraform v0.12.20+ complains about the statement during the validation step (terraform validate). I had to convert each item to the list to multiple references of statement to make it work.

The error:

An argument named "statement" is not expected here. Did you mean to define a block of type "statement"?

Statement parsing issue

Hello,
This valid policy :

  "Version" : "2012-10-17",
  "Statement" : {
    "Effect" : "Deny",
    "Action" : "*",
    "Resource" : "*",
    "Condition" : {
      "NotIpAddress" : {
        "aws:SourceIp" : [
          "xxx.xxx.xxx.xxx/xx",
          "yyy.yyy.yyy.yyy"
        ]
      },
      "StringNotLike" : {
        "aws:invokedBy" : "*.amazonaws.com"
      }
    }
  }
}

make iam-policy-json-to-terraform return :
unable to convert: json: cannot unmarshal object into Go struct field jsonPolicyDocument.Statement of type []converter.jsonStatement

I had to add square brackets to make it working

Thanks for this very useful tool

Support CloudFormation snippets as input?

This was raised in #48 where the @joemilacek tried to input a snippet of CloudFormation JSON which is currently not supported (we only support plain IAM policy documents).
However, the use-case doesn't seem too far-fetched. It's likely a significant piece of work though so I probably won't work on it unless there's significant need for this feature.

As a start, we could introduce a specific error message that allows us to put analytics on how often this use-case happens.

If you have additional comments on the need for this feature, feel free to comment below.

Add syntax highlighting in web frontend

Context

#24 introduced a basic web frontend to use. At the moment, it provides only very bare-bones functionality with minimal styling.
This issue would add syntax highlighting to both input and output to make it easier to edit and spot things.

Outcomes

• Add a well supported syntax highlighter and use it on input and output (needs to support JSON and HCL)
  • HCL
  • JSON
• Prioritise function and usability over prettiness - this tool is used by people who quickly need to achieve a task (to convert a json policy into a different format). The design shouldn't distract from this.

Constraints

• My own frontend-skills exist but are rusty - you are free to choose the best tools and frameworks for the job, however I'd prefer minimal learning curves and complexity. In doubt, err on the side of simplicity and well known tools for myself and other contributors to maintain this in the long run
• The tool is primarily used by infrastructure people for their work - I assume most will use it on their desktops and laptops. Therefore, responsive design or mobile friendly features are not a priority (however nice to have)

Related Issues

There are several other issues dealing with the web frontend - check the web-frontend label to get an understanding where overlaps might exist

More useful error message if input can be parsed but doesn't contain essential information

Brought up in #48: Input a snippet of CloudFormation JSON which was perfectly parseable but has a different format so nothing could be converted to terraform - the result is simply empty which doesn't give the user a lot of information what's wrong.

Instead, we should raise some sort of warning that indicates that this isn't a valid IAM policy.

Improve web-deployment scripting and pipeline

Currently, the deployment script and pipeline aren't trivial to understand - they make some assumption about how what files are important for the website, do some artifact handling but also implicitly use other stuff.

Clarify this and make it more generic

Superfluous $$'s and quotes

Found a couple of cases where there's extra $'s and "'s being added.

Here's a sample input json:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetBucketLocation",
                "s3:GetObjectAcl",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:ListBucketVersions",
                "s3:DeleteObject",
                "s3:DeleteObjectVersion"
            ],
            "Resource": [
                "arn:aws:s3:::${aws_s3_bucket.output.id}/*",
                "arn:aws:s3:::${aws_s3_bucket.output.id}"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:GetObjectAcl",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:ListBucketVersions",
                "s3:DeleteObject",
                "s3:DeleteObjectVersion"
            ],
            "Resource": [
                "arn:aws:s3:::${aws_s3_bucket.upload.id}/*",
                "arn:aws:s3:::${aws_s3_bucket.upload.id}"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
              "kms:Encrypt",
              "kms:Decrypt",
              "kms:ReEncrypt*",
              "kms:GenerateDataKey*",
              "kms:DescribeKey"
            ],
            "Resource": [
              "${aws_kms_key.key.arn}"
            ]
        }
    ]
}

and the resulting aws_iam_policy_document:

data "aws_iam_policy_document" "policy" {
  statement {
    sid    = "VisualEditor0"
    effect = "Allow"

    resources = [
      "arn:aws:s3:::$${aws_s3_bucket.output.id}/*",
      "arn:aws:s3:::$${aws_s3_bucket.output.id}",
    ]

    actions = [
      "s3:PutObject",
      "s3:GetBucketLocation",
      "s3:GetObjectAcl",
      "s3:GetObject",
      "s3:ListBucket",
      "s3:ListBucketVersions",
      "s3:DeleteObject",
      "s3:DeleteObjectVersion",
    ]
  }

  statement {
    sid    = "VisualEditor1"
    effect = "Allow"

    resources = [
      "arn:aws:s3:::$${aws_s3_bucket.upload.id}/*",
      "arn:aws:s3:::$${aws_s3_bucket.upload.id}",
    ]

    actions = [
      "s3:GetBucketLocation",
      "s3:GetObjectAcl",
      "s3:GetObject",
      "s3:ListBucket",
      "s3:ListBucketVersions",
      "s3:DeleteObject",
      "s3:DeleteObjectVersion",
    ]
  }

  statement {
    sid       = ""
    effect    = "Allow"
    resources = ["$${aws_kms_key.key.arn}"]

    actions = [
      "kms:Encrypt",
      "kms:Decrypt",
      "kms:ReEncrypt*",
      "kms:GenerateDataKey*",
      "kms:DescribeKey",
    ]
  }
}

Here's the diff of manual edits needed to fix the output:

[I] ➜ diff tenant-bucket-access.hcl.orig tenant-bucket-access.hcl
7,8c7,8
<       "arn:aws:s3:::$${aws_s3_bucket.output.id}/*",
<       "arn:aws:s3:::$${aws_s3_bucket.output.id}",
---
>       "arn:aws:s3:::${aws_s3_bucket.output.id}/*",
>       "arn:aws:s3:::${aws_s3_bucket.output.id}",
28,29c28,29
<       "arn:aws:s3:::$${aws_s3_bucket.upload.id}/*",
<       "arn:aws:s3:::$${aws_s3_bucket.upload.id}",
---
>       "arn:aws:s3:::${aws_s3_bucket.upload.id}/*",
>       "arn:aws:s3:::${aws_s3_bucket.upload.id}",
46c46
<     resources = ["$${aws_kms_key.key.arn}"]
---
>     resources = [ aws_kms_key.key.arn ]

Converting variable adds additional $ sign sometimes

Consider having this json file:

{
    "Version": "2012-10-17",
    "Statement": [
      {

        "Sid": "AWSCloudTrailCreateLogStream2014110",
        "Effect": "Allow",
        "Action": [
          "logs:CreateLogStream"
        ],
        "Resource": [
          "arn:aws:logs:${data.aws_region.reg_current.name}:${data.aws_caller_identity.acc_current.account_id}:log-group:${aws_cloudwatch_log_group.trail-log-group.name}:*"
        ]

      },
      {
        "Sid": "AWSCloudTrailPutLogEvents20141101",
        "Effect": "Allow",
        "Action": [
          "logs:PutLogEvents"
        ],
        "Resource": [
          "arn:aws:logs:${data.aws_region.reg_current.name}:${data.aws_caller_identity.acc_current.account_id}:log-group:${aws_cloudwatch_log_group.trail-log-group.name}:*"
        ]
      }
    ]
}

Running iam-policy-json-to-terraform returns this:

data "aws_iam_policy_document" "policy" {
  statement {
    sid       = "AWSCloudTrailCreateLogStream2014110"
    effect    = "Allow"
    resources = ["arn:aws:logs:$${data.aws_region.reg_current.name}:${data.aws_caller_identity.acc_current.account_id}:log-group:${aws_cloudwatch_log_group.trail-log-group.name}:*"]
    actions   = ["logs:CreateLogStream"]
  }

  statement {
    sid       = "AWSCloudTrailPutLogEvents20141101"
    effect    = "Allow"
    resources = ["arn:aws:logs:$${data.aws_region.reg_current.name}:${data.aws_caller_identity.acc_current.account_id}:log-group:${aws_cloudwatch_log_group.trail-log-group.name}:*"]
    actions   = ["logs:PutLogEvents"]
  }
}

Notice how in the resources an extra $ is added to the first variable: "arn:aws:logs:$${data.aws_region.reg_current.name}".
The following variables however are correctly converted.

Version: 1.8.0 via Brew
OS: MacOS

Add homebrew publishing to release-script

#3 published this tool on homebrew. To keep it up to date, we should add homebrew publishing to the release script

This is the command currently used to submit pull requests to bump versions to homebrew:

$ brew bump-formula-pr --strict iam-policy-json-to-terraform --url https://github.com/flosell/iam-policy-json-to-terraform/archive/<VERSION>.tar.gz --sha256 <HASH OF TAR FILE>