Version 1.6 values vs global about kontemplate

[dermyrddin]

The simplest (and logical for me) order is:
CLI variables > variables in include section ("values:" in the resource section) > globals

In that case we have one "global" section which could be overridden by resource "values:" section which could be overridden by CLI definitions.

"Defaults" are only the predefined defaults for resource which could be overridded on any level (useful for distributing resources ready for templating).

And "imports" used only for structuring the resource set allowing to export some part of "global" or "values:" section to external file (but should have less priority than variables in section which contains "import").

[tazjin]

If you guys have time, please take a look at the pull request above (specifically the new implementation of value merging in context.go) and test contained release candidate (or build a release yourself if you prefer that!) for 1.7.0.

The only logical change is that values from the import section have lower precedence than globals, based on what @merlineus said.

I think this makes sense because things that are less specific to a particular execution of kontemplate have lower precedence, and import variables are potentially less specific because they may be used in multiple contexts.

To quote my own comment, the order is now (in ascending priority):

1. Default values in resource sets (via default.yaml / default.json)
2. Values imported from files (via import:)
3. Global values in a cluster configuration (via global:)
4. Values set in a resource set's include-section
5. Explicit values set on the CLI (via --var)

[tazjin]

Hey!

From my perspective that commit was actually a bug fix (i.e. the previous behaviour was conceptually wrong). Can you tell me how you use the globals and in what cases you want the resource set defaults to override them?

I think there may be a use-case here I haven't quite understood yet :)

[tommyJimmy87]

Here an example :

---
context: some-context 
global:
  namespace: global-namespace
include:
  - name: some-components
    path: ../some-components
    include:
      - name: component1
        path: component1/apply
        values:
           namespace: component1-namespace
       - name: component2
          path: component2/apply

Within the component, the namespace definition as variable is : {{.namespace}} , the component's content is not actually important.

In this case with the previous version ( until 1.5 ) component1 namespace will be : component1-namespace , instead component2 namespace will be remain : global-namespace.

Now for me this is correct, because i have the globally variables for all my components and if i want to replace one of the globally i just have to add that variable as values inside my component.

In the 1.6 version instead also for the component1 will be remain the global one : global-namespace.

In my case i have a bunch of components defined and not all components have the same global values, and if i'll rollout the new version i have to convert all my global variables in component variables. Anyway I need something replacing the global variables with custom variables. I think this is very useful.

Thanks

[tazjin]

@tommyJimmy87 Thanks for the explanation. I think there is some ambiguity with the precedence of defaults vs. globals and maybe it should simply be a toggleable behaviour. I'll think about this a bit more after work :)

[Arttii]

I think it makes sense to have a way to define global variables to reused across components.

Another very simple example would say if you're using a certificate for all elbs you have and they are different per environment including dozens of components. So having 20 times repeated

cert: arn:aws:acm:eu-central-1:12322312:certificate/4sdsdadawe213232-223238532c

is not super optimal I think. And defaults is not suitable as you have 5 environments with different certificates

[dermyrddin]

I'm also confused with this behaviour after updating to 1.6. My configuration have a global "replicas" variable to set default replicas count (i.e. 1) for all resources. And some resources have an overrided value (for example 3) in "include" section. The same way I've set tag for images - "latest" by default and exact versions for separate resources. So this logic doesn't work anymore :)

[Arttii]

@merlineus Basically the vars in values: are treated as defaults. So for our usecase we need something like:

CLI variables > variables in include section > imports >explicit> globals > defaults or
CLI variables > explicit> variables in include section > imports > globals > defaults

[tazjin]

@tommyJimmy87 Do you consider this issue solved with the recent changes?

[tommyJimmy87]

@tazjin yes, i tried and seems as we discussed, thanks for the quick answer and the support. 👍

[tazjin]

Great, closing issue! I'll do the 1.7.0 release soon (tm)

[tazjin]

@Arttii You can already use variables under the global key in a context file, those will be available to all resource sets imported below.

The discussion here is mostly about the precedence order in which variables with the same name at different levels override each other.

My mental model (so far) has been that it should be like this:

CLI variables > variables in `include` section > imports > globals > defaults

However due to a bug in the logic before 1.6 the order was

CLI variables > variables in `include` section > imports > defaults > globals

But @tommyJimmy87 highlighted a use-case where this is actually the correct order, so it's not as unambiguous as I thought initially.

I'm experimenting with a few different things but there'll probably be some kind of toggle for this eventually.

[Arttii]

@tazjin I see. But to be honest for me global varaibles are there to override defaults, which are there in case no other thing was specified. But maybe its just me.

[tazjin]

@Arttii Aren't we saying the same thing then? 🤔

[Arttii]

Hmm I am also confused now. I will think about it a bit more properly tomorrow when tommy is also in the office.

…

________________________________ From: Vincent Ambo <[email protected]> Sent: Monday, June 25, 2018 5:24:06 PM To: tazjin/kontemplate Cc: Arttii; Mention Subject: Re: [tazjin/kontemplate] Version 1.6 values vs global (#142) @Arttii<https://github.com/Arttii> Aren't we saying the same thing then? 🤔 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#142 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ACEE6CmDWf3FrtU3ZktlHq8qWMATGZFeks5uAQCWgaJpZM4UvSRv>.

[tazjin]

^ Please take a look at the PR above.

[Arttii]

So this i very confusing, our problem was that values supplied in the component explicitly

global:
  namespace: beta
- name: grafana
   path: grafana/apply
   values:
           component: grafana
           namespace: dev

were being overshadowed by Global and not the default being overshadowed by global. I think its completely fine for defaults to be overshadowed by Globals. In this example the grafana component does not even have a default for namespace. Are those default values or what? Because I thoughts defaults are the values specified in default.yaml.

After looking at the code, I think this is where the miscommunication is coming, for me

   values:
           component: grafana
           namespace: dev

are not default but explicit variables. Should we add them as a type? Then the order would be correct I think. I think allowing us to specify these explicit vars is better than having to keep a var file for each component, even though it will hold only one var.

[dermyrddin]

The order

CLI variables > variables in include section > imports > globals > defaults

seem reasonable, if I correctly understand that variables on the right side overriding variables on the left side. Maybe only "imports" and "globals" could be swapped because from my POV (and maybe shadowing logic) included file should have less priority in hierarchy.

But I can't figure out how swapping "globals" and "defaults" in the chain affected "include" section which should override all definitions to the right of it in chain?

[dermyrddin]

@Arttii I thought "values:" are "variables in include section"... Now I'm confused even more :)

[Arttii]

So for my example up top, CLI variables and include variables would be empty. And defaults would be component: grafana,namespace: dev, which will be merged with namespace: beta. And default.yaml is also empty.

Maybe easy fix with be to merge stuff in values: with included vars?

[Arttii]

Yep, that would work for us as well

[tazjin]

The fix for this is included in version 1.7.0.