pulumi / aws-tf-fork-patch

Tool for patching the AWS TF fork

AWS TF Fork Patcher

Applies a set of automated changes to a AWS TF fork working directory.

1. yarn - restore packages
2. yarn apply --cwd [DIR] - apply patches to a directory
3. yarn suggest --cwd [DIR] - generate suggestions for pending replacements and write into the manual replacements.
4. Search for "TODO" in patches/manualReplacements.json for pending docs fixes.
5. Re-apply patches once edited.

Design Notes

• Consider failure cases:
  1. Worst case is replacing something we shouldn't.
  2. Second worst is missing a replacement without warning.
  3. Throwing an error due to an unexpected scenario is a good outcome.
• Instruct the user how to fix the issue in the patcher.
• Aim for patches to be idempotent. Re-applying a patch multiple times should have no adverse effects.

Generating replacements from diffs

1. Generate a diff git diff v4.35.0..upstream-v4.35.0 > upstream.patch
2. Convert diff to replacements yarn start parse-patch upstream.patch --outFile patches/patchReplacements.json

Approximate method:

Parse diff for website/docs files (ignore renames and internal/service)

1. Find diffs where the same line has a removal and addition (rewrite). Create replacements for where "terraform" or "hashicorp" has been removed.
2. Find diffs where a single note has been removed.
3. Find diffs where a block of text has been removed.

aws-tf-fork-patch's Issues

Flag when using an invalid node version

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

nvm was using node 14 and ran into this error when running make tfgen. Resolved by using Node 18, but the error message could be better.

% make tfgen  
[ -x /home/kdixler/.pulumi-dev/bin/pulumi ] || curl -fsSL https://get.pulumi.com/ | sh
pulumi plugin install resource tls 4.1.0
pulumi plugin install resource github 4.10.0
pulumi plugin install resource kubernetes 3.17.0
pulumi plugin install resource random 4.8.2
cd upstream-tools && yarn install --frozen-lockfile
yarn install v1.22.19
[1/4] Resolving packages...
success Already up-to-date.
Done in 0.04s.
cd upstream && git checkout . && git clean -fdx
Updated 524 paths from the index
cd upstream-tools && yarn --silent run apply
tf-patch apply

Apply AWS TF fork patches onto working directory

Options:
  --version                     Show version number                    [boolean]
  --help                        Show help                              [boolean]
  --cwd                         Target directory                  [default: "."]
  --pre-automated-replacements  Output replacement file path. Defaults to
                                pre-replacements.json if it exists      [string]
  --replacements                Replacements source file path
                                                  [default: "replacements.json"]
  --skip-link-stripping         Disable auto-stripping of links from docs
                                                      [boolean] [default: false]
  --skip-gofmt                  Disable auto-formatting after edits
                                                      [boolean] [default: false]
  --domains                     Domain rules path for link stripping
                                                       [default: "domains.json"]

TypeError: replaced.replaceAll is not a function
    at tryReplace (/home/kdixler/go/src/github.com/pulumi/home/analytics/provider-codegen-errors/.pulumi-aws/upstream-tools/node_modules/tf-patch/cmd/index.js:14856:45)
    at Object.<anonymous> (/home/kdixler/go/src/github.com/pulumi/home/analytics/provider-codegen-errors/.pulumi-aws/upstream-tools/node_modules/tf-patch/cmd/index.js:14833:45)
    at Generator.next (<anonymous>)
    at fulfilled (/home/kdixler/go/src/github.com/pulumi/home/analytics/provider-codegen-errors/.pulumi-aws/upstream-tools/node_modules/tf-patch/cmd/index.js:14805:58)
make: *** [Makefile:131: patch_upstream] Error 1

Affected area/feature

Configurable code replacements

Expose a command to be able to configure custom replacements across a range of files. This is similar to the aws-specific tags_all replacement, but should be configurable by the caller.

Make replacement idempotent

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

Ideally, running the replace process twice should yield the same result as running it once - with the same warnings.

The complicating factor is trying to ensure that the replacements are exactly applied - we report any replacement that’s no longer found as we can then look at what the replacement was and if we need to add a new replacement. E.g. if we were removing a tf specific warning and it’s no longer matched because it was reworded, we probably need to adjust the existing replacement rather than throw it away.

So, on to making it idempotent - If we’ve already replaced something then the new text should be found - so if we don’t find the words to match, we search for the words it should have been replaced with. If the new words aren’t found then we can report it as unmatched.

• If the new words are matched then can we assume it’s already been applied and skip it?
• What if the replacement phrase is quite generic and would match elsewhere in the doc?
• What if the replacement is just whitespace?

There’s nothing to stop a rule matching multiple times in the same file right now IIRC.

Perhaps we should validate the rules better.

• Perhaps assert that a rule matches exactly once and the replacement can’t match something that’s already in the file?
• Or assert that all replacements are unique within a file?
• Perhaps we’d need opt-out to say that a replacement might be the same as existing text and that’s okay?

Affected area/feature

Create architecture diagram for fork-patcher

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

This is a fantastic tool. A quick visual would help place it in the proper context in our tool chain for future success.

Affected area/feature

Might be nice to drop it into the team playbooks, or our internal wiki.