carvel-dev / vendir

Easy way to vendor portions of git repos, github releases, helm charts, docker image contents, etc. declaratively

vendir

• Website: https://carvel.dev/vendir
• Slack: #carvel in Kubernetes slack
• Docs with example workflow and other details
• Install: Grab prebuilt binaries from the Releases page or Homebrew Carvel tap
• Backlog: See what we're up to.

vendir allows you to declaratively state what should be in a directory. It can be used for vendoring software.

$ vendir sync # from a directory that contains vendir.yml

Features:

• Various sources
  • pull Git repositories (examples/git, semver tag resolution in examples/versionselection)
    • including tag semver selection, GPG verification
  • pull Github releases (examples/github-release, semver tag resolution in examples/versionselection)
  • pull HTTP asset (examples/http)
  • pull Docker image contents (examples/image)
  • pull Helm chart contents (examples/helm-chart)
  • ...let us know sources you'd like to see
• Keep only particular portions of pulled content via includePaths/excludePaths or newRootPath
• Override specific directory with a local directory source for quick development
• State which directories are manually managed
• Specify inline content for a directory
• Generates lock file
• Keep common legal files (LICENSE, etc.)

See all examples.

Join the Community and Make Carvel Better

Carvel is better because of our contributors and maintainers. It is because of you that we can bring great software to the community. Please join us during our online community meetings. Details can be found on our Carvel website.

You can chat with us on Kubernetes Slack in the #carvel channel and follow us on Twitter at @carvel_dev.

Check out which organizations are using and contributing to Carvel: Adopter's list

vendir's Issues

show steps vendir is taking

eg show that it's working on particular directory (and may be stream output from git)

Enable selective sync

Today on cf-for-k8s, when I vendir sync --use-directory to replace some remote code with local dev copies, it takes 30s to redownload all the dependencies. I really only need to move the files from the one directory. Could this be made faster?

Allow recursive pulling of imgpkg bundles

Describe the problem/challenge you have
If i have a recursive bundle, I would like to be able to pull it recursively

Describe the solution you'd like
Expose a recursive: true/false option to the imgpkgBundle fetch config

Anything else you would like to add:
Eventually, we may want to also support a filter here using proposed filtering behavior in imgpkg

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

Feature request: Support vendir to update a whole set of dependencies

When having:

- path: k8s/config
  contents:
  - path: A
    git:
      url: https://github.com/config/A
      ref: master
  - path: B
    git:
      url: https://github.com/config/B
      ref: master
- path: k8s/examples
  contents:
  - path: A
    git:
      url: https://github.com/example/A
      ref: master
  - path: B
    git:
      url: https://github.com/example/B
      ref: master
  - path: C
    git:
      url: https://github.com/example/C
      ref: master

If one wants to update only examples, it would need to do it one by one with vendir sync -d k8s/examples/A then B and then C. It would be ideal to support the whole tree under examples when done via vendir sync -d k8s/examples.

And linked to #11 It would be ideal to support this use case for -d that doesn't save vendir.lock file as well as with an option to save the vendir.lock file.

vendir fails when target path does not exist

Error: Syncing directory 'vendor/k8s': Moving staging directory '.vendir-tmp/staging' to final location 'vendor/k8s': rename .vendir-tmp/staging vendor/k8s: no such file or directory

That vendir/k8s directory does not exist, and vendir should probably create it as long as it's relative to the current path.

Cut New Release

Any chance we can get a new release cut? =) We anxiously await use of #64?

We use the release to distribute the packages we depend on.

vendir sync -l -d some_path doesn't filter by some_path

What steps did you take:

vendir sync -l -d somepath

What happened:

I invoked vendir with -l and -d some_path and it downloaded all the artifacts in the lockfile, instead of just the locked dependency I was specifying.

What did you expect:

I was expecting to fetch only the directory specified by -d some_path

Anything else you would like to add:

Not really.

Environment:

• vendir version 0.13.0
• Ubuntu 20.04.2.0

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

Feature Request: Support Flag to Disable Deletion

use-case

We're using vendir in a number of different ways to vendor configuration files in from a centralized source.

1. One way is to "layer" configurations from a centralized source. That is, we import a base of configurations and then add a few minor customizations in new files.
2. The other way is we initialize configurations (then disable and diverge), while maintaining a record of when we divereged in the vendir.yml.

The problem is when we disable a configuration, then vendir sync wants to delete all the files. And if we add files into a vendir'd configuration, and then run vendir sync, it will delete those extraneous files. The current behavior makes sense, however, we're hoping we could introduce something that will address the use-cases above. We'll help implement it as well.

Request A

Support an enabled setting. In the example below, enabled is set to false, so the configuration is skipped. Running vendir sync will not attempt to modify or delete the local copy (it's as though the stanza were deleted from the file). This is useful for when we deliberately want to diverge, but want to keep a record in vendir.yml the point at which we diverged.

apiVersion: vendir.k14s.io/v1alpha1
kind: Config

directories:

  # Terraform components
  - path: components/terraform
    contents:

      - path: account-map
        # Disable this from running
        enabled: false
        git:
          url: https://github.com/cloudposse/terraform-aws-components
          ref: 0.137.0
        newRootPath: modules/account-map
        includePaths:
          - modules/account-map/**/*

      - path: account-settings
        git:
          url: https://github.com/cloudposse/terraform-aws-components
          ref: 0.137.0
        newRootPath: modules/account-settings
        includePaths:
          - modules/account-settings/**/*

Request B

Support an ignorePaths block which specifies paths that should not be deleted (or synchronized), even if they do not exist at the source. This is useful when we layer in configurations. For example, we add additional configuration files that do not exist at the source.

In the example below, the ignorePaths block says to ignore the fact that backend.tf.json does not exist at the source and also, ignore any files in the catalog/* folder.

apiVersion: vendir.k14s.io/v1alpha1
kind: Config

directories:

  # Terraform components
  - path: components/terraform
    contents:

      - path: account-map
        # Ignore files which are generated locally and should not be overridden or deleted.
        ignorePaths: &ignore
        - backend.tf.json
        - catalog/*
        git:
          url: https://github.com/cloudposse/terraform-aws-components
          ref: 0.137.0
        newRootPath: modules/account-map
        includePaths:
          - modules/account-map/**/*

      - path: account-settings
        # Use a YAML anchor to ignore common files
        <<*ignore
        git:
          url: https://github.com/cloudposse/terraform-aws-components
          ref: 0.137.0
        newRootPath: modules/account-settings
        includePaths:
          - modules/account-settings/**/*

Request C

Support a --skip-delete flag. on vendir sync. This is useful during development or as a precaution to avoid accidentally deleting any files.

Running vendir sync --skip-delete will run as usual, but not remove any files (but possibly overwrite).

Typo in project's tags > "vendor" instead of "vendir

Hi, there's a minor typo in the projects tags: vendor instead of vendir

add kubernetes source

support imgpkg bundle download

Describe the problem/challenge you have

want to download imgpkg bundles.

Describe the solution you'd like

imgpkgBundle:
  # image URL; could be plain, tagged or digest reference (required)
  image: gcr.io/repo/image:v1.0.0
  # specifies name of a secret with registry auth details;
  # secret may include 'username', 'password' and/or 'token' keys (optional)
  secretRef:
    # (required)
    name: my-image-auth

would use imgpkg pull -b ...

TBD is is too confusing that we are using image for referring to bundle? mostly interested in doing that because kbld picks up image key automatically for image ref.

Anything else you would like to add:

related: should we update image source to use image key instead of url.

provide a way to specify github oauth token to avoid rate limit

useful when fetching github releases.

missing git tags

What steps did you take:

1. I've created the following vendir.yml:

apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
- path: base
  contents:
  - path: "."
    git:
      url: https://github.com/keycloak/keycloak-operator
      ref: "12.0.2"
    includePaths:
    - deploy/**/*
    newRootPath: deploy/

2. I executed vendir sync:

vendir sync
Fetching: base + . (git from https://github.com/keycloak/[email protected])

  --> git init
  Initialized empty Git repository in /Users/tester/.vendir-tmp/incoming/git/.git/
  --> git config credential.helper store --file /Users/tester/.vendir-tmp/incoming/git-auth/.git-credentials
  --> git remote add origin https://github.com/keycloak/keycloak-operator
  --> git fetch origin
  From https://github.com/keycloak/keycloak-operator
   * [new branch]      10.0.x     -> origin/10.0.x
   * [new branch]      latest     -> origin/latest
   * [new branch]      master     -> origin/master
   * [new tag]         10.0.1     -> 10.0.1
   * [new tag]         7.0.1      -> 7.0.1
   * [new tag]         10.0.0     -> 10.0.0
   * [new tag]         8.0.2      -> 8.0.2
  --> git -c advice.detachedHead=false checkout 12.0.2
  error: pathspec '12.0.2' did not match any file(s) known to git

Error: Syncing directory 'base': Syncing directory '.' with git contents: Fetching git repository: Git [-c advice.detachedHead=false checkout 12.0.2]: exit status 1 (stderr: error: pathspec '12.0.2' did not match any file(s) known to git
)
make: *** [base] Error 1

What happened:
The command git fetch doesn't seem to fetch all tags for the keycloak-operator repo.

What did you expect:
I would expect that all existing tags on upstream could be resolved.

Anything else you would like to add:
I've tried to execute the git commands manually like so:

mkdir keycloak-operator
cd keycloak-operator
git init
git remote add origin https://github.com/keycloak/keycloak-operator
git fetch origin

remote: Enumerating objects: 127, done.
remote: Counting objects: 100% (127/127), done.
remote: Compressing objects: 100% (91/91), done.
remote: Total 10348 (delta 64), reused 65 (delta 32), pack-reused 10221
Receiving objects: 100% (10348/10348), 9.74 MiB | 4.19 MiB/s, done.
Resolving deltas: 100% (5474/5474), done.
From https://github.com/keycloak/keycloak-operator
 * [new branch]      10.0.x     -> origin/10.0.x
 * [new branch]      latest     -> origin/latest
 * [new branch]      master     -> origin/master
 * [new tag]         10.0.1     -> 10.0.1
 * [new tag]         7.0.1      -> 7.0.1
 * [new tag]         10.0.0     -> 10.0.0
 * [new tag]         8.0.2      -> 8.0.2

I get the same behavior as if I use vendir. The 12.0.2 tag doesn't get fetched. I was able to get all tags by using the --tags flag:

git fetch origin --tags

remote: Enumerating objects: 1954, done.
remote: Counting objects: 100% (1330/1330), done.
remote: Compressing objects: 100% (251/251), done.
remote: Total 771 (delta 583), reused 669 (delta 483), pack-reused 0
Receiving objects: 100% (771/771), 211.08 KiB | 1.09 MiB/s, done.
Resolving deltas: 100% (583/583), completed with 158 local objects.
From https://github.com/keycloak/keycloak-operator
 * [new tag]         11.0.0     -> 11.0.0
 * [new tag]         11.0.1     -> 11.0.1
 * [new tag]         11.0.2     -> 11.0.2
 * [new tag]         11.0.3     -> 11.0.3
 * [new tag]         12.0.0     -> 12.0.0
 * [new tag]         12.0.1     -> 12.0.1
 * [new tag]         12.0.2     -> 12.0.2
 * [new tag]         8.0.1      -> 8.0.1
 * [new tag]         9.0.0      -> 9.0.0
 * [new tag]         9.0.2      -> 9.0.2

Environment:

• vendir version (execute vendir --version): 0.13.0
• OS (e.g. from /etc/os-release): Darwin (MacOS BigSur)

support ignorePaths to exclude removal of certain files

Describe the problem/challenge you have

in some cases it's useful to retain certain temp files among fetched directories. (e.g. temp files generated by tools that were operating on top of fetched files.)

Describe the solution you'd like

add ignorePaths key on directory contents. ignorePaths would specify a set of globs that would be used to match in existing directories and copy them over on top of newly fetched sources.

apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
- path: components/terraform
  contents:
  - path: account-map
    git:
      url: https://github.com/cloudposse/terraform-aws-components
      ref: 0.137.0
    newRootPath: modules/account-map
    includePaths:
      - modules/account-map/**/*
    ignorePaths: &ignore
    - backend.tf.json
    - catalog/*

  - path: account-settings
    git:
      url: https://github.com/cloudposse/terraform-aws-components
      ref: 0.137.0
    newRootPath: modules/account-settings
    includePaths:
      - modules/account-settings/**/*
    ignorePaths: *ignore

Details:

• for directories that have been removed in newly fetched content, ignored content wont be retained
• files that already exist in newly fetched content will take precedence over files that copied from original directory.

Feature request: support flag to disable UnpackArchive

Describe the problem/challenge you have
When the target is a zip file, vendir automatically unzip it.
It would be nice to introduce a flag to disable this behavior.

Describe the solution you'd like
Example:

      - path: artifacts/plugins/myplugin.zip
        http:
          url: https://artifacts.example.com/downloads/plugins/myplugin.zip
          unpackArchive: false

Anything else you would like to add:
[Additional information that will assist in solving the issue.]

Just opening this issue to test actions

Foo

Have option for vendir sync to run in different directory.

Describe the problem/challenge you have

When working with packages and where the vendir.yml file is located in a sub directory, it is extra work to ensure that you first cd into the directory before running vendir sync. If running command manually or scripting actions working across multiple packages, means have to use something like the following to ensure left in original directory when command done.

(cd package/foo/bundle; vendir sync)

Describe the solution you'd like

Would like to see a command line option where can tell vendir sync the directory to run in. Eg.,

vendir sync -c package/foo/bundle

The c for option indicative of "change directory".

Thus vendir sync would then itself change the working directory of the process to the specified directory before taking any action.

Anything else you would like to add:

N/A

cc @jorgemoralespou

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

https://kubernetes-charts.storage.googleapis.com/ has been Removed

Tests are failing due to the removal of https://kubernetes-charts.storage.googleapis.com/.

The url is hardcoded in vendir here.

add directory type that can reference source from local directory

similar to manual but points to a local directory. copies over files from there and filters out content per usual includePaths/excludePaths rules.

Bug: Paths don't allow using path names that starts with a string that is used in other path names

what

• Bug: Paths don't allow using path names that starts with a string that is used in other path names

why

We use the following configuration in vendir.yml:

  - path: components/terraform
    contents:
      - path: account-map
        git:
          url: https://github.com/cloudposse/terraform-aws-components
          ref: 0.135.0
        newRootPath: modules/account-map
        includePaths:
          - modules/account-map/**/*

      - path: account
        git:
          url: https://github.com/cloudposse/terraform-aws-components
          ref: 0.135.0
        newRootPath: modules/account
        includePaths:
          - modules/account/**/*

and getting the following error:

Error: Parsing resource config 'vendir.yml': Unmarshaling config: Validating config: Expected to not manage overlapping paths: 'components/terraform/account-map' and 'components/terraform/account'

Note that account_map does not work either, throwing the same validation error.

Moving the folders to the top-level throws the same error as well:

- path: components/terraform/account-map
    contents:
      - path: .
        git:
          url: https://github.com/cloudposse/terraform-aws-components
          ref: 0.135.0
        newRootPath: modules/account-map
        includePaths:
          - modules/account-map/**/*
  - path: components/terraform/account
    contents:
      - path: .
        git:
          url: https://github.com/cloudposse/terraform-aws-components
          ref: 0.135.0
        newRootPath: modules/account
        includePaths:
          - modules/account/**/*

This does not work either:

      - path: accountmap
        git:
          url: https://github.com/cloudposse/terraform-aws-components
          ref: 0.135.0
        newRootPath: modules/account-map
        includePaths:
          - modules/account-map/**/*

      - path: account
        git:
          url: https://github.com/cloudposse/terraform-aws-components
          ref: 0.135.0
        newRootPath: modules/account
        includePaths:
          - modules/account/**/*

throwing the error:

Error: Parsing resource config 'vendir.yml': Unmarshaling config: Validating config: Expected to not manage overlapping paths: 'components/terraform/accountmap' and 'components/terraform/account'

But this works ok:

      - path: a-ccountmap
        git:
          url: https://github.com/cloudposse/terraform-aws-components
          ref: 0.135.0
        newRootPath: modules/account-map
        includePaths:
          - modules/account-map/**/*

      - path: account
        git:
          url: https://github.com/cloudposse/terraform-aws-components
          ref: 0.135.0
        newRootPath: modules/account
        includePaths:
          - modules/account/**/*

So the issue is that the code checks not for the full path, but if a path starts with a string.

@cppforlife can you please take a look at the issue?
We like vendir and want to use it in our architectures.

Thank you

support a way to download subset of github release assets

In cf-for-k8s, we'd like to remove disableAutoChecksumValidation: true from our entry for kpack, for example.

That block looks like this:

  - path: github.com/pivotal/kpack
    githubRelease:
      slug: pivotal/kpack
      tag: v0.0.8
      disableAutoChecksumValidation: true
    includePaths:
    - release-0.0.8.yaml

And when we remove the disableAutoChecksumValidation line, we get the error:

Error: Syncing directory 'config/_ytt_lib': Syncing directory 'github.com/pivotal/kpack' with github release contents: Finding checksums in release notes: Expected to find sha256 checksum for file 'logs-v0.0.8-linux.tgz'

...even though the only file in our include path is release-0.0.8.yaml.

Here's the kpack release page in case that's helpful: https://github.com/pivotal/kpack/releases/tag/v0.0.8

Feature request: support file relocation/rename

I would like vendir to strip/rename paths

- path: vendor/k8s
  contents:
  - path: eduk8s
    git:
      url: https://github.com/eduk8s/eduk8s
      ref: 20.08.04.1
    includePaths:
    - resources/crds-v1/**/*
    - resources/operator/**/*

and this ends up creating:

vendor/k8s/eduk8s/resources/crds-v1
vendor/k8s/eduk8s/resources/operator

but I would want:

vendor/k8s/eduk8s/resources/ 

everything just in this dir.

The use case is:

I’m collecting the k8s resource definitions from multiple sources to then process them with ytt and apply them with kapp, and have a consistent directory structure. When I get stuff from github releases I just get the files I need in the root of the path, when I get it this other way it ends in subdirectories, and hence dir layout is not really consistent

Using newRootPath and legalPaths together result in missing LICENSE file

What steps did you take:
[A clear and concise description steps that can be used to reproduce the problem.]

A typical GitHub project generally includes its license at the root of the project. When specifying the newRootPath, this will adjust the root path of the project after the download and since the legalPath likely exists outside of newRootPath, the legalPath ends up missing.

Consider this config:

---
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
  # vertical pod autoscaler
  - path: source/cnpo.tanzulabs.vmware.com/tenancy/vertical-pod-autoscaler
    contents:
      - path: vendor
        git:
          url: https://github.com/kubernetes/autoscaler.git
          ref: vertical-pod-autoscaler/v0.9.2
        includePaths:
          - vertical-pod-autoscaler/deploy/*.yaml
        excludePaths:
          - vertical-pod-autoscaler/deploy/kustomization.yaml
          - vertical-pod-autoscaler/deploy/vpa-beta-crd.yaml
          - vertical-pod-autoscaler/deploy/vpa-beta2-crd.yaml
          - vertical-pod-autoscaler/deploy/vpa-crd.yaml
        newRootPath: vertical-pod-autoscaler/deploy
        legalPaths:
          - LICENSE
     

What happened:
[A small description of the issue]

Directory was appropriately downloaded, however the legalPaths were missing because (assumption) it existed outside of the newRootPath

What did you expect:
[A description of what was expected]

The legalPaths would be placed within the newRootPath specified.

Anything else you would like to add:
[Additional information that will assist in solving the issue.]

Environment:

• vendir version (execute vendir --version):

vendir --version
vendir version 0.19.0

Succeeded

• OS (e.g. from /etc/os-release): MacOS

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

Symlinks aren't properly copied

What steps did you take:
vendir sync with this config:

apiVersion: vendir.k14s.io/v1alpha1
kind: Config
minimumRequiredVersion: 0.12.0
directories:
- path: config/upstream
  contents:
  - path: serving
    git:
      url: https://github.com/knative/serving
      ref: v0.21.0
    includePaths:
    - config/core/**/*

What happened:
config/upstream/serving/config/core/300-imagecache.yaml is an invalid symlink, pointing to a file that doesn't exist.

What did you expect:
I'm not 100% sure, probably it should collapse the symlink into a regular file with the contents of the file it pointed to. Maybe this needs to be configurable.

Anything else you would like to add:
I looked into this a little and found it was broken in the library that vendir depends on: otiai10/copy#32

Environment:

• vendir version (execute vendir --version): 0.16.0
• OS (e.g. from /etc/os-release): ubuntu focal

vendir cannot access the file because it is being used by another process on windows

What steps did you take:

When you add more than 1 path to directories using the http plugin you get these errors on windows.

---
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
  - path: vendor/one
    contents:
      - path: one
        http:
          url: https://raw.githubusercontent.com/vmware-tanzu/carvel-vendir/develop/examples/http/vendir.yml

  - path: vendor/two
    contents:
      - path: two
        http:
          url: https://raw.githubusercontent.com/vmware-tanzu/carvel-vendir/develop/examples/http/vendir.yml

I suspect that the tmp dir is not unique and/or is shared with both downloads that run in some goroutines in parallel but then they try to delete the dir (or parent dir).

What happened:

Error: Syncing directory 'vendor/two': Deleting tmp dir '.vendir-tmp': remove .vendir-tmp\incoming\vendir-http521295291: The process cannot access the file because it is being used by another process.

Environment:

• vendir version (execute vendir --version): v0.19.0
• OS (e.g. from /etc/os-release): Windows 10

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

Feature request: Automatic Releases on Merge to Master

what

• we rely on the binary releases to ship vendir in our https://github.com/cloudposse/packages
• these are published as alpine, debian (and soon rpm) packages

why

• we are blocked by not having a release for the bug fix contained in #22

references

Fwiw, here's how we do it:

• We automatically cut releases using this action: https://github.com/cloudposse/github-commenter/blob/master/.github/workflows/auto-release.yml
• We automatically build binary release artifacts using https://github.com/cloudposse/github-commenter/blob/master/.github/workflows/go.yml

Vendir should preserve execution permissions on extract

What steps did you take:

- path: terraform.d/plugins
  contents:
  - path: carvel.dev/carvel/k14s/0.6.0
    githubRelease:
      slug: vmware-tanzu/terraform-provider-carvel
      tag: v0.6.0
      assetNames: ["terraform-provider-k14s-binaries.tgz"]
      unpackArchive:
        path: terraform-provider-k14s-binaries.tgz

vendir sync

What happened:
The extracted package binaries will be set as non-executable even though in the archive they are executable.

What did you expect:
The file permissions should be left alone and they should be executable.

Anything else you would like to add:
This is very important for downloading of CLI tools

Environment:

• vendir version (execute vendir --version): 0.13.0
• OS (e.g. from /etc/os-release): Ubuntu 20.04

🎁 Write a post of a vendir usecase 🎁

Context

vendir 's features allow a lot of things, not only around kubernertes. It could be interesting to show how vendir can make things easier, and to make package software installation even smoother.

Action

• Show how vendir can make APOC plugin installation easier in Neo4J (shox how to move away manual tasks)

support verifying signed git commit

originally filed here: carvel-dev/kapp-controller#6. since vendir will be used by kapp-controller (carvel-dev/kapp-controller#12), this feature belongs here.

be more forgiving when trailing slash is specified in the directory path

apiVersion: vendir.k14s.io/v1alpha1
kind: Config
minimumRequiredVersion: 0.8.0
directories:
- path: vendor

and

apiVersion: vendir.k14s.io/v1alpha1
kind: Config
minimumRequiredVersion: 0.8.0
directories:
- path: vendor/

should be equivalent. in v0.10.0 trailing slash causes a fs rename error.

Sort semvers based on build metadata as well

Describe the problem/challenge you have
Currently, semvers are not being sorted by their build metadata, which leaves some room for non determinism of selected versions.

Describe the solution you'd like
Sort the semvers on build metadata. Although the spec says not to rely on it, we should sort build metadata the same way as pre-releases. Spec

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

there's a ton of potential in this, looking forward to trying it out with automated systems

I'm really looking forward to testing this out for use with retrieving manifests, code, and github releases from sources and syncing them. I think there's tons of potential for declarative configuration from this. Will test it out more. Thanks a lot for all you do 👍 not an actual issue just wanted to say thanks

New source idea > java artefacts (ie. java dependencies)

Hi, I wonder it would not be useful to be able to pick jar java artifacts (aka. maven/gradle dependencies).

We could target:

• maven central
• jitpack
• any private JFrog artifactory or Nexus repo

support overriding directories within pulled dependencies via later directory configs

for example:

• pull in config/_ytt_lib/github.com/cloudfoundry/cf-for-k8s
• override config/_ytt_lib/github.com/cloudfoundry/cf-for-k8s/config/_ytt_lib/github.com/pivotal/kpack with something else.

add error hint when vendir.yml is not found

Many people create yaml files with either yml or yaml extensions. Vendir should look for both extensions (prioritizing maybe yml), so that it won't fail when the user creates a vendir.yaml file.

Feature Request: Allow sync (or bumps) for specific dependencies

Context

vendir currently supports the --directory flag:

$ vendir sync -d my_dependency

But it doesn't save the lock file. I would like vendir to be able to isolate sync/bumps to a specific dependency.

`githubRelease` errors on releases without explicit assets

What steps did you take:

I attempted to run vendir sync with a githubRelease for open-policy-agent/gatekeeper. The entry under contents: looks like:

  - path: gatekeeper
    githubRelease:
      slug: open-policy-agent/gatekeeper
      latest: true
      disableAutoChecksumValidation: true

What happened:
I get an error that reads Error: Syncing directory 'config/upstream': Filtering paths in directory 'gatekeeper': Expected to find at least one file within directory. This error is very cryptic. I spent a few troubleshooting cycles creating empty files in config/upstream/gatekeeper or deleting the directory, thinking that the error message was complaining about the local filesystem.

Reading the vendir code, the problem seems to be that there’s no particular asset tied to the release - this project doesn’t provide binaries with their releases. You can download the source code, but that’s not listed in the github API json response as part of the assets array.

the release in question: https://api.github.com/repos/open-policy-agent/gatekeeper/releases/37006810

...
  "assets": [

  ],
  "tarball_url": "https://api.github.com/repos/open-policy-agent/gatekeeper/tarball/v3.3.0",
  "zipball_url": "https://api.github.com/repos/open-policy-agent/gatekeeper/zipball/v3.3.0",
...

https://github.com/vmware-tanzu/carvel-vendir/blob/1da965479755f7b3583e275d337e66f6d7a25950/pkg/vendir/fetch/githubrelease/sync.go#L76 refers specifically to the assets array, and loops over that array, looking for assets to copy. Finding none, it errors out, apparently at https://github.com/vmware-tanzu/carvel-vendir/blob/bf0685a51eae291131fe5d1475ac0a6eceeb3c1e/pkg/vendir/directory/directory.go#L160

What did you expect:
I expected to be able to download the source code, since the Github UI shows it as an available download, and only after looking at the implementation did I understand why the vendir githubResource was not able to download it. If my project chooses to use vendir to track upstream OSS dependencies via githubResources, we probably want to track them uniformly, even if they don't ship any specific assets.

I would expect to be able to choose the tarball_url or zipball_url as options to download, possibly by specifying their paths as they appear when you download them via the github UI: https://github.com/open-policy-agent/gatekeeper/archive/v3.3.0.zip and https://github.com/open-policy-agent/gatekeeper/archive/v3.3.0.tar.gz

Paths within the sourcecode for the release would also be nice, since in that case we'd probably only need metadata from a small number of files within the release.

Environment:

• vendir version (execute vendir --version): 0.16.0
• OS (e.g. from /etc/os-release): MacOS

Feature Request: Add semver support

Context

vendir currently supports tracking specific refs and versions. I'd like to have support for specific version ranges:

  apiVersion: vendir.k14s.io/v1alpha1
  kind: Config
  directories:
  - path: vendor
    contents:
    - path: github.com/cloudfoundry/cf-k8s-networking
      git:
        url: https://github.com/cloudfoundry/cf-k8s-networking
-       ref: master
+       version: ^1.0.0
    includePaths:
    - cfroutesync/crds/**/*
    - install/ytt/networking/**/*
  - path: github.com/cloudfoundry/cc
    manual: {}
  - path: github.com/GoogleCloudPlatform/metacontroller
    manual: {}
  - path: local-dir
    directory:
      path: local-dir

So I can specify valid version ranges for each one of my dependencies.

`go get github.com/vmware-tanzu/carvel-vendir` fails

Looks like there's a bug getting the checksum of carvel-vendir due to a file name aux.go.

❯ go version
go version go1.13.15 linux/amd64

❯ go get github.com/vmware-tanzu/carvel-vendir
go: downloading github.com/vmware-tanzu/carvel-vendir v0.14.0
verifying github.com/vmware-tanzu/[email protected]: github.com/vmware-tanzu/[email protected]: reading https://sum.golang.org/lookup/github.com/vmware-tanzu/[email protected]: 410 Gone

❯ curl https://sum.golang.org/lookup/github.com/vmware-tanzu/[email protected]
not found: create zip: malformed file path "pkg/vendir/config/aux.go": "aux" disallowed as path element component on Windows

Is contents/path not respected?

When using this config:

apiVersion: vendir.k14s.io/v1alpha1
kind: Config
minimumRequiredVersion: 0.8.0
directories:
- path: k8s/eduk8s
  contents:
  - path: upstream/eduk8s
    git:
      url: https://github.com/eduk8s/eduk8s
      ref: 20.08.04.1
    includePaths:
    - resources/crds-v1/**/*
    - resources/operator/**/*
  - path: downstream/hub
    manual: {}

I would expect to end up with this folder structure:

k8s/eduk8s/upstream/eduk8s
k8s/eduk8s/downstream/hub

I understand the second needs to exist, since it's a manual sync, but the former is not and the resources specified in that rule end up in k8s/eduk8s/content/... rather than k8s/eduk8s/upstream/eduk8s/content/....

I have seen the test https://github.com/k14s/vendir/blob/develop/examples/overlapping-dir/vendir.yml which confuses me as it seems to be the expected behavior. What am I missing?

Expected directory contents type to be specified (one of git, manual, etc.)

Hi, could you please clarify what exactly should be specified, and where?

to reproduce:

git clone https://github.com/k14s/vendir
cd vendir/examples/image
vendir sync

output:

Error: Validating config: Validating directory 'vendor' (0): Validating directory contents 'docker.io/dkalinin/consul-helm-naked' (0): Expected directory contents type to be specified (one of git, manual, etc.)

remove manual source in favor of using directory source

If vendir sync fails, manually managed directories might be deleted.

There are two paths in the config: 1) manually managed lib/manifests, 2) Non-existing git lib/bin

apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
- path: lib
  contents:
  - path: manifests
    manual: {}
  - path: bin
    git:
      url: https://github.com/k14s/vendir
      ref: origin/master
    includePaths:
    - bin/non-existing
    newRootPath: bin

Dir structure before sync:

.
├── lib
│  └── manifests
│     └── tpl.yaml
└── vendir.yml

After sync:

.
├── lib
└── vendir.yml

As far as I understand, vendir processes paths in the order they're defined in the configuration. For manually managed paths it moves them into temporary space. If one of the next paths can't be ensured due some error, the path previously moved to the temporary space is not moved back.

If I change the order of the paths, lib/manifests directory stay intact.

It looks like a bug and might lead to data loss (files and staged changes can be restored from git or from IDE history, but still).

Utilize caching when fetching similar repos

Describe the problem/challenge you have
When it comes to using overlapping sync options, it's usually that you would mention the same repo to pull from several times in order to substitute one of the folders with a custom one from the private repo, for instance:

directories:
  - path: components
    contents:
      - path: terraform/tfstate-backend
        git:
          url: [email protected]:cloudpossy/terraform-aws-components.git
          ref: all-new-components
        newRootPath: modules/tfstate-backend
      - path: terraform/account
        git:
          url: [email protected]:cloudpossy/terraform-aws-components.git
          ref: all-new-components
        newRootPath: modules/account
      - path: terraform/account-map
        git:
          url: [email protected]:cloudpossy/terraform-aws-components.git
          ref: all-new-components
        newRootPath: modules/account-map
      - path: terraform/nexus
        git:
          url: [email protected]:privateorg/terraform-aws-components.git
          ref: atmos-novel
        newRootPath: privateorg/modules/nexus

Here where it gets too long to wait when vendir sync is used because it basically tries to react on that planly by downloading [email protected]:cloudpossy/terraform-aws-components.git every time it's mentioned in the spec without attempting to re-use the cached sample.

I'd guess, that it's something of utmost importance to implement this simple functionality and make the tool much more efficient.

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

🚀 No tweet that announces the v0.20.0 release

Hi, I use the release tweet to communicate on the choco package availabiliy, but this time I just can't find the related tweet : can you do one as usual please ?

Thanks in advance for your help.
Friendly yours,
Adrien

Looks like `vendir-windows-amd64.exe` of v0.19.0 returns `0.18.0` (instead of `0.19.0`)

What steps did you take:

1. Download vendir-windows-amd64.exe binary from https://github.com/vmware-tanzu/carvel-vendir/releases/tag/v0.19.0
2. Run vendir-windows-amd64.exe version
3. Check that it answers : 0.18.0 (instead of 0.19.0)

What happened:

• It answers : 0.18.0 instead of 0.19.0

What did you expect:

• Should answer 0.19.0

Environment:

• OS (e.g. from /etc/os-release): Windows

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

release asset size error (due to 404 during asset download)

I'm seeing an issue where vendir throws an error when checking the release asset file size: Checking asset 'file.txt' size: Expected file size to be 74, but was 9. I checked the api response for release assets, it shows the asset size is 74 bytes. When I manually download the file, it is 74 bytes. Any idea what would cause this?

Some details from the API response (some parts are omitted or renamed):

[
  {
    "name": "file.txt",
    "label": "",
    "content_type": "application/text",
    "state": "uploaded",
    "size": 74,
    "download_count": 5,
    "created_at": "2020-08-04T19:58:28Z",
    "updated_at": "2020-08-04T19:58:29Z",
    "browser_download_url": "https://github.com/org/repo/releases/download/v3/file.txt"
  }
]

vendir downloads helm charts to c:\windows

What steps did you take:

I am using the carvel suite with helm on windows. Any helm chart should be able to reproduce the issue.

apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
  - path: vendor
    contents:
      - path: kube-prometheus
        helmChart:
          name: kube-prometheus
          version: 5.0.0
          repository:
            url: https://charts.bitnami.com/bitnami

What happened:

Error: Syncing directory 'vendor': Syncing directory 'kube-prometheus' with helm chart contents: Fetching helm chart: exit status 1 (stderr: Error: failed to untar: mkdir C:\WINDOWS\helm-741980851: Access is denied.)

What did you expect:

It should use the .vendir-tmp dir relative to the current dir or use ~/.cache/vendir or event /tmp or C:\Users\{user}\AppData\Local\Temp

Environment:

• vendir version (execute vendir --version): v0.19.0
• OS (e.g. from /etc/os-release): Windows 10

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible"
👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

Add e2e test for imgpkgBundle

fetching bundles using imgpkg was added, so we should add an e2e test for that behavior

`githubRelease.unpackArchive.path` should support globbing

Describe the problem/challenge you have
Assets in Github releases often include the release version as part of the asset's filename, but vendir's unpackArchive option does not currently support globbing, although the githubRelease.assetNames array does support globbing. Lack of globbing support means that unpackArchive cannot be used as intended for many Github Releases.

Currently, attempting to pass a glob results in a cryptic "Expected known archive type (zip, tgz, tar)" error message.

For example,

this works, but you have to know the filename in advance:

  - path: prometheus
    githubRelease:
      slug: prometheus/prometheus
      latest: true
      disableAutoChecksumValidation: true
      assetNames: ["prometheus-*.linux-amd64.tar.gz"]
      unpackArchive:
        path: 'prometheus-2.25.0.linux-amd64.tar.gz'

this fails:

  - path: prometheus
    githubRelease:
      slug: prometheus/prometheus
      latest: true
      disableAutoChecksumValidation: true
      assetNames: ["prometheus-*.linux-amd64.tar.gz"]
      unpackArchive:
        path: 'prometheus-*.linux-amd64.tar.gz'

“Error: Syncing directory ‘config/upstream’: Syncing directory ‘prometheus’ with github release contents: Expected known archive type (zip, tgz, tar)”

Describe the solution you'd like
Globbing should be supported. If the user-provided glob matches multiple files, an error message would be acceptable, as would be unpacking all of the specified archives.

Support Helm 3 OCI Charts

Describe the problem/challenge you have
I would like to relocate a Helm chart that is stored in an experimental OCI image

Additional troubleshooting: https://kubernetes.slack.com/archives/CH8KCCKA5/p1618514640321100

Describe the solution you'd like
The helm chart is inside the container image, but it has an invalid rootfs so is failing on image pull, no preference on whether that means it's helmChart imgpkgBundle image or even http

Anything else you would like to add: