herrjulz / aviator Goto Github PK

This isn't a real issue. Just wanted to say thank you for your work & making this open source. This is great stuff.

Currently, aviator only supports to write merge results to files. This causes cleanup work if you use temp files.

(To be entirely honest, I'm not 100% sure what the workflow for this would look like yet, just putting an issue here as a way to point this out. Feel free to close this whenever you feel fit.)

• https://developer.ibm.com/blogs/yaml-templating-tool-to-simplify-complex-configuration-management/
• https://github.com/k14s/ytt

I'm investigating using ytt as a way to templatize my concourse pipelines. I thought this might be an interesting project to check out and see if/how it could integrate to aviator.

Thanks for your time!

Hi folks,
testing aviator behaviour and going through the code I noticed that the except piece isn't into account when include_sub_dirs is present. Example:

- base: base.yml
  for_each:
    in: configs
    include_sub_dirs: true
    regexp: ".*.yml"
    except: [test.yml]
  to_dir: ++tmp/base_output/++

This can be solved with #46
@herrjulz can you review pls?

Hi,

Trying out this tool and I have a stupid question. Why doesn't merge merge?

thing1.yaml:

a:
  b: c

thing2.yaml:

a:
  d: e

spruce merge does what I expect:

$ spruce merge thing1.yaml thing2.yaml
a:
  b: c
  d: e

But aviator seems to ignore the second file.

$ cat aviator.yml
spruce:
  - base: thing1.yaml
    merge:
    - with:
      files:
      - thing2.yaml
    to: things.yaml
$ aviator
SPRUCE MERGE:
	thing1.yaml
	to: things.yaml

$ cat things.yaml
a:
  b: c

Why didn't the contents of thing2.yaml get merged into the output?

e.g. change chain to plan

I would like to use aviator in a meta (concourse) pipeline creating other pipelines. For that to be possible fly needs to be run in non-interactive mode (with -n argument).

I would appreciate adding an option along the lines of:

fly:
  non-interactive: true

that would result appending '-n' to fly args

Hi,

I tried to build the project in Windows, but got following error. Does the project support windows?

github.com/JulzDiverse/aviator/vendor/code.cloudfoundry.org/commandrunner/linux_command_runner
..\github.com\JulzDiverse\aviator\vendor\code.cloudfoundry.org\commandrunner\linux_command_runner\linux_command_runner.go:27:11: unknown field 'Setpgid' in struct literal of type syscall.SysProcAttr
..\github.com\JulzDiverse\aviator\vendor\code.cloudfoundry.org\commandrunner\linux_command_runner\linux_command_runner.go:30:18: cmd.SysProcAttr.Setpgid undefined (type *syscall.SysProcAttr has no field or method Setpgid)
..\github.com\JulzDiverse\aviator\vendor\code.cloudfoundry.org\commandrunner\linux_command_runner\linux_command_runner.go:39:11: unknown field 'Setpgid' in struct literal of type syscall.SysProcAttr
..\github.com\JulzDiverse\aviator\vendor\code.cloudfoundry.org\commandrunner\linux_command_runner\linux_command_runner.go:42:18: cmd.SysProcAttr.Setpgid undefined (type *syscall.SysProcAttr has no field or method Setpgid)
..\github.com\JulzDiverse\aviator\vendor\code.cloudfoundry.org\commandrunner\linux_command_runner\linux_command_runner.go:51:11: unknown field 'Setpgid' in struct literal of type syscall.SysProcAttr
..\github.com\JulzDiverse\aviator\vendor\code.cloudfoundry.org\commandrunner\linux_command_runner\linux_command_runner.go:54:18: cmd.SysProcAttr.Setpgid undefined (type *syscall.SysProcAttr has no field or method Setpgid)

I've just upgraded to 1.5.0 and am seeing the following behaviour:

Running aviator, it gets to this part:

AVIATOR EXECUTE:$ fly --target concourse set-pipeline --pipeline mypipeline --config aviator_pipeline.yml

and prints out the correct diff, but then immediately prints bailing out rather than prompting me to accept the change. I can work around this by setting non_interactive: true in aviator.yml. I couldn't see anything related in the recent releases that suggests I've missed some other bit of config, but am quite happy to pointed at it if there was a breaking change along the way. Thanks!

Its there a reason why some methods used in main.go are defined in another file call flags.go?
When trying to generate the binary via the go build I have the following error:

# command-line-arguments
./main.go:14: undefined: setCli

If I defined the methods from flags.go into the main.go, then I do not have a problem anymore.

Regards,
Enrique Encalada

Hey Julz,

The suggestion here is to perform a standard merge, but instead of having to always specify files in the with section, we can directly write the yaml code we would like to merge.

For example, let's say we have the following aviator file:

spruce:
- base: path/to/base.yml
  merge:
  - with_obj:
    foo:
      bar: 42
      baz: true
  to: result.yml

That would merge the base file with the following:

foo:
  bar: 42
  baz: true

Why? Sometimes you want to merge something really tiny (2-3 lines) and creating a separate file for that can be an overkill.

Thanks 🙂

Aa aviator uses the spruce library, there should be an automation (CI) in place to build aviator with newer versions. (Discuss automation details)

Hi,
Not sure if my finding is correct but it looks to me that if fly errors out, then (not every?) exit code from fly gets passed down as aviator exit code.
Since my use case is CI, I care for the pipeline to fail in case of error.

The behavior I am observing:
Add a "put" to pipeline definition with misspelled name, or in any other way not existing among the pipeline resources, say:

   - put: docker-imag
      params:
        build: build-result

Run set-pipeline with fly directly:
fly -t target sp -p pipeline-foo -c pipeline-foo.yml -n
Result:

[...]
apply configuration? [yN]: y
error: invalid configuration:
invalid jobs:
	jobs.pipeline-foo.plan[5].put.docker-imag refers to a resource that does not exist

$ echo $?
1

Doing the same with aviator. My aviator-foo.yml for testing:

fly:
  target: target
  name: pipeline-foo
  config: pipeline-foo.yml
  non_interactive: true

Running:
aviator -f aviator-foo.yml
Result:

[...]
apply configuration? [yN]: y
error: invalid configuration:
invalid jobs:
	jobs.pipeline-foo.plan[5].put.docker-imag refers to a resource that does not exist

$ echo $?
0

For errors from fly, like this one, it would help me if aviator exited with non-zero code. Is it possible to do? Thanks in advance.

Currently, aviator only supports env variables in file paths. They should be supported at any feasible place in the aviator.yml.

Hello,

I'm using aviator (version 1.9.0) to create a complex, or rather lengthy, pipeline.
The pipeline consists of over a dozen resources, and it would be hard to maintain or add to the pipeline if created by hand.

My aviator file is supposed to create an all_resources.yml, but it's not working as intended.
The file contains the full list of resources with their respective repo-names and repo-urls, and additionally it also contains a job, where all those resources are fetched with get as input to a compilation task.
I managed to get the part of the resources: section working and made a really identical part for the plan: section, but SOMEHOW, the - get lines overwrite each other, which ends up with only the last resource in the final file.

I'm really at a dead end here, because I can't see anything I'm doing wrong.
Can you help me out here?

This is my aviator.yml file:

spruce:
  # First we generate all resources to use in the pipeline
  - base: factory/meta-resource-stub.yml
    prune:
      - meta
    for_each:
      in: meta-resources/
      except:
        - example-resource.yml
      regexp: ".*.(yml)"
    to_dir: temp/resources/
  # Merge all resources into one file
  - base: factory/all-resources-stub.yml
    merge:
      - with_all_in: temp/resources/
        regexp: ".*.(yml)"
    to: temp/all-resources.yml

And these are the stubs:
meta-resource.stub.yml

---
jobs:
  - get: (( grab meta.git.repo-name ))
    trigger: (( grab meta.git.trigger ))
    params:
      depth: 1

resources:
  - name: (( grab meta.git.repo-name ))
    type: git
    source:
      uri: (( grab meta.git.http ))
      username: bot-tt
      password: ((sesource.bot-tt-pw))
      branch: (( grab meta.git.branch ))

all-resources-stub.yml

---
jobs:
  -
resources:
  -

When run, it correctly produces the all-resources.yml but inside, it failed to add each resource in the jobs section, instead it overwrites itself, so only the last resource was added.
The resources section correctly produces ALL generated resources.

I don't know what's wrong, since the two sections are near-identical
And also, I know that the resulting file currently is not correct in terms of concourse syntax, but I reduced it to a minimal example to figure out what's wrong.

Hi Julz,

I just updated to v0.18.0 and my aviator file is breaking when doing the fly feature. The following config:

fly:
  config: /tmp/final-pipeline.yml
  name: $PIPELINE
  target: $TARGET
  vars:
  - vars.yml

was working fine under v0.17.0, however when upgrading the version, that same part broke with the following:

YAML Parsing Failed: yaml: unmarshal errors:
  line 15: cannot unmarshal !!seq into map[string]string

Have you seen this behaviour?

Would like to see a new release with latest spruce changes

Sometimes it would useful to be able to render to stdout instead of having to create a file.

Example:

input.yml

foo:
  bar: 2000

avaitor.yml

- base: input.yml
  modify:
    set:
    - path: .foo.baz
       value: [qux]

Expected output:

foo:
  bar: 2000
  baz: [qux]

/cc @gdankov

When triggering aviator -v, I see the following:

Aviator version 0.1.0

But I would expect to see the latest version(which I installed from https://github.com/JulzDiverse/aviator/releases/tag/v0.2.0

https://github.com/JulzDiverse/aviator#the-fly-section-optional

Similar to how the kubectl executor has a --dry-run mode, perhaps it would be beneficial to have the fly executor only do a validate-pipeline and not actually fly up the pipeline.

Thanks for your time and consideration.

for_each is useful for enumerating a list of targets. I use it to loop over multiple Debian and Ubuntu releases, say ["buster", "bullseye", "bionic", "focal"]. So I have a distros/ folder with one yaml file in it per target distribution, each with the tag names and other details that are necessary for building for that distro.

I'd like to add target architectures to that, but it does not seem possible. I want to add a second loop over the target architecutres, say ["aarch64", "powerpc64", "x86_64"], which works the same way: an arches/ folder with one yaml file in it per target architecture, each with the details necessary for building for that architecture.

The desired result would be a nested pair of for-loops, one for distro, one for architecture. Like this pseudo-code:

for arch in ["aarch64", "powerpc64", "x86_64"]:
    for distro in ["buster", "bullseye", "bionic", "focal"]:
        arch_fn   = "arches/"  + arch   + ".yaml"
        distro_fn = "distros/" + distro + ".yaml"
        output = merge(all_inputs, arch_fn, distro_fn)
        out_fn = "outdir/" + distro + "_" + arch + ".yaml"
        write(out_fn, output)

This would produce the cartesian product of files in distros/ and files in arches/; with one merged output file for each pairing. This syntax is not limited to 2 dimensions, but my current use case is 2-dimensional.

One possible aviator.yml syntax might look like this:

spruce:
  - base: resources.yaml
    for_each:
      - in: distros/
      - in: arches/
    merge:
[...]

I can kinda get functionality similar to this now, by generating a third directory full of pre-merged pairings like "focal_aarch64.yaml", "bullseye_x86_64.yaml", etc, and using for_each to loop over those. But to accomplish this, I'm doing ugly shell loops in a Makefile... it would be nice if the tool could automate this.

I am looking to use aviator to build a large number of similar pipelines. The current example shows building multiple apps and merging them into a single pipeline, followed by a call to fly to install that pipeline.

I would like to have a single aviator file that builds multiple pipelines and calls fly on each generated pipeline.

As an example - take the cf sample and remove the last merge - you would have multiple app pipelines that would need to be installed.

https://github.com/JulzDiverse/aviator-cf-apps/blob/master/aviator.yml

I am managing yaml configs with spruce / aviator but also there are some non-yaml static config files that I am copying along. I imagine this applies to some other peoples workflows as well. It would help greatly if I could do things like copy a list of files from place to place. Something like for_each with its properties combined with to_dir but without spruce merge.

Currently, if you set expose: true your pipeline will be hidden. Setting it to false, though, does nothing. I've opened a PR at #34 which enables previously exposed pipelines to be hidden again.

On top of merge aviator should support:

• prune
• eval
• cherry_pick