vidispine / hull Goto Github PK

State

containers and initContainers for a pod can be individually configured and defaulted using _HULL_OBJECT_TYPE_DEFAULT_ and ´sources. Sometimes however the same container is used as an initContainer and a container both and eg. setting many environment variables multiple times feels unnecessary

Feature Idea

Add a sharedContainers field to pod which allows to define properties of containers that are inheritable by containers and initContainers both. The sharedContainers keys are referenceable by single containers and initContainers by key in a field sourceContainer on the container spec.

Feature should be compatible with _HULL_OBJECT_TYPE_DEFAULT_ and ´sources as well:

1. Define sharedContainer in _HULL_OBJECT_TYPE_DEFAULT_ or referenced sources
2. sharedContainers will be merged into individual pod spec
3. sharedContainers can be referenced same way as if defined for pod only

Hello again,

May I bother you with another newbie question? If so - here it goes.

I have a named template that I "call" with hull.util.transformation.include from the values dictionary:

hull:
  objects:
    deployment:
      some-nice-pod:
        pod:
          imagePullSecrets: _HT/secrets/some.nice.utility

The template usually renders a YAML containing a list of entities:

secrets:
- alpha
- bravo
- charlie

It works fine, but there are cases when some.nice.utility must return an empty list (secrets: []). In this case, Helm fails with the following error message: Error: INSTALLATION FAILED: unable to build Kubernetes objects from release manifest: error validating "": error validating data: [apiVersion not set, kind not set]. Perhaps it happens because the manifest structure is maimed, but I can't see exactly how it looks. The --debug parameter doesn't help me see the resulting manifest before it's being validated, so I can't even see what's broken.

Would you be so kind as to provide me with any glues on how to return an empty list to hull.util.transformation.include and prevent Helm from failing?

I really appreciate any help you can provide.

Hello again!

What do you think about allowing the user to set the type of a Secret? It's always Opaque, but a user might need to set another type (e.g., kubernetes.io/dockerconfigjson).

I can try to make a PR if it's appropriate. :-)

Hello!

May I bother you with another question?

Do I understand it right that hull.util.transformation.makefullname should return the static name of an object in the case when the object's staticName is true? If yes, then are there any exceptions?

Here's set of values I'm giving to Helm:

hull:
  objects:
    serviceaccount:
      my-serviceaccount:
        staticName: true
    service:
      my-service:
        staticName: true
    configmap:
      my-configmap:
        staticName: true
    deployment:
      my-deployment:
        annotations:
          hull-test.local/my-service: _HT^my-service
          hull-test.local/my-serviceaccount: _HT^my-serviceaccount
          hull-test.local/my-configmap: _HT^my-configmap

Here are the annotations I get as the result:

hull-test.local/my-configmap: hull-test-4-hull-test-4-my-configmap
hull-test.local/my-service: hull-test-4-hull-test-4-my-service
hull-test.local/my-serviceaccount: hull-test-4-hull-test-4-my-serviceaccount

I didn't find the answer in the documentation, but I see that hull.util.transformation.makefullname calls hull.metadata.fullname and hull.metadata.fullname, obviously, respects the staticName parameter. But it seems that hull.metadata.fullname doesn't have the whole outlook when it's being called by hull.util.transformation.makefullname. Am I right?

Thank you.

Based on @sczech's request:

I'm just wondering if it is possible to set the namespace for the objects inside of values.yaml? Ideally I would like to have a config.general.namespaceOverride key.

Currently the metadata.namespace fields of the objects are not touched or touchable by HULL. They are set by Helm on deployment where the -n or --namespace parameter sets the metadata.namespace for all created objects. This is I think standard Helm procedure.

In that sense having a global/general setting applied to all objects with a config.general.namespaceOverride option would amount to the same as setting a different namespace for helm install/upgrade? But maybe you have a different usecase in mind like one sketched below?

Overriding the namespace on a per object basis could be useful and implementation is rather easy. I can think of having as suggested a namespaceOverride property on the hull.object.base level where you can overwrite this for any given object.

Let's assume a case where you want to have one alternative backup namespace where some of the objects should go to and you want to have a central field to provide that namespace. For this you could then add the namespaceOverride to the objects in question and have them reference a shared field like hull.config.specific.backupNamespace to get the centrally configured namespace. I think that could be your use case?

Hi after creating relatively simple application I'm getting:

Error: create: failed to create: Secret "sh.helm.release.v1.my-release.v1" is invalid: data: Too long: must have at most 1048576 bytes
helm.go:81: [debug] Secret "sh.helm.release.v1.my-release.v1" is invalid: data: Too long: must have at most 1048576 bytes

Any ideas?

Helm lint (Helm version 3.10.0, 3.8.2, 3.5.0) gives the following :

$ helm lint .
==> Linting .
[ERROR] Chart.yaml: appVersion should be of type string but it's of type float64
[INFO] Chart.yaml: icon is recommended

Error: 1 chart(s) linted, 1 chart(s) failed

Indeed in Chart.yaml, we've got appVersion: 1.25. The value should be quoted, as it is currently interpreted as a float.

This is quite a minor detail, but it may break workflows relying on linting to check the validity of the chart.

Hello!

First of all, thank you for sharing your work. The whole idea of generating Helm charts on the fly according to a single values file is fantastic.

I'm trying to use HULL in my work, and I'd like to ask you for some advice.

Is it possible to assign an object's name by a variable set by a user? Something kind of that:

hull:
  objects:
    secret:
      _HT^something.secretName:
<...skipped...>
something:
  secretName: my-secret

The user sets something.secretName, and my chart creates a Secret with this name prefixed by the chart and release names.

Obviously, the straightforward way I showed in my example won't work. So is there a right way to achieve this?

Thanks in advance for any hints.

P.S. Merry Christmas! 🎄

Hello

In pure Helm it's possible to declare template like this, that will create ConfigMap with an entry for every yaml file in mydir folder.

apiVersion: v1
kind: ConfigMap
metadata:
  name: mydir
data:
  {{- (.Files.Glob "mydir/*.yaml").AsConfig | nindent 2 }}

Is it possible to achieve the same with HULL ?
I tried to use _HT!{{ ... }} but without success.

Hello again!

I'm sorry for the disturbance; there's another question if you please.

Here is a minimalistic values.yaml:

hull:
  objects:
    secret:
      my-secret-1:
        data:
          .dockerconfigjson:
            inline: _HT/myUtilities.generateDotDockerConfigJson
      my-secret-2:
        data:
          .dockerconfigjson:
            inline: _HT!{{ include "myUtilities.generateDotDockerConfigJson" . }}

And here's the file with myUtilities.generateDotDockerConfigJson:

{{ define "myUtilities.generateDotDockerConfigJson" }}
{"auths":{"my-registry":{"username":"username","password":"password","email":"email","auth":"dXNlcm5hbWU6cGFzc3dvcmQ="}}}
{{ end }}

Everything seems to be quite straightforward, but in my-secret-1 I'm getting bWFwW2F1dGhzOm1hcFtteS1yZWdpc3RyeTptYXBbYXV0aDpkWE5sY201aGJXVTZjR0Z6YzNkdmNtUT0gZW1haWw6ZW1haWwgcGFzc3dvcmQ6cGFzc3dvcmQgdXNlcm5hbWU6dXNlcm5hbWVdXV0= (which means map[auths:map[my-registry:map[auth:dXNlcm5hbWU6cGFzc3dvcmQ= email:email password:password username:username]]]) and in my-secret-2 I'm getting just "".

It seems that there's a special handling for hull.objects.secret.*.data.*.inline when it looks like a JSON-structure, but I can't find out how to pass the result of myUtilities.generateDotDockerConfigJson as a string. Would you be so kind as to help me out again?

Thank you very much in advance.

State
Currently _HT* can only be used to reference arrays as a whole. It is not possible to dig into an array via indexing.

Feature Idea
Enable indexing of arrays in _HT* to be able to dig into array elements when using _HT*. Syntax should be straight forward:

_HT*hull.config.specific.theArray.0

Hello again!

I hope, it's perfect time for another stupid question. :-)

Would you be so kind as to help me to understand how to access the "global context" from an interpolation placeholder? Here's what I mean:

hull:
  objects:
    configmap:
      my-configmap:
        data:
          my-parameter-1:
            inline: _HT!{{ printf "Hello, world! My name is %s." (index . "$").Values.myName }}
          my-parameter-2:
            inline: Hello, world! My name is {{ (index . "$").Values.myName }}.
myName: Some Name

The my-parameter-1 is being rendered correctly, but I can't figure out how to make my-parameter-2 being rendered too.

Could you help me to comprehend it, please?

Dear maintainers,

First of all, thanks for this library, it's very helpful to reduce the helm boilerplate. It seems there is a small issue with ingress templating.

The hosts section is not correctly rendered. The following snippet is the input

objects:
    ingress:
      frontend:
        enabled: true
        rules:
          frontend:
            host: _HT*hull.config.specific.ingress.host
            http:
              paths:
                standard:
                  path: /
                  pathType: Prefix
                  backend:
                    service: 
                      name: frontend
                      port:
                        name: http
        tls:
          frontend:
            hosts:
              - _HT*hull.config.specific.ingress.host

the output is then

...
 spec:
  rules:
  - host: workspace.qualicoat.test
    http:
      paths:
      - backend:
          service:
            name: release-name-workspace-frontend
            port:
              name: http
        path: /
        pathType: Prefix
  tls:
  - hosts: foo.com
    secretName: release-name-workspace-foo.com

however, the tls section of the output should be

  tls:
  - hosts: 
     - foo.com
    secretName: release-name-workspace-foo.com

Or am I overlooking something?

Hi there!

Please feel free to leave some feedback here e.g. in the comments about this project such as:

• what do you think of the concepts and ideas?
• do plan on using HULL or are actively using it?
• anything particular you would like to see added or improved?

Thanks, any feedback is highly appreciated and helps us in improving the project.

Hello,

First I wanted to thank the devs behind this lib, really enjoying using it 👍

As for this issue, I think fields in jobspec-v1-batch are not being populated in the resulting YAML

I made a small reproduction with this configuration :

hull:
    cronjob:
      _HULL_OBJECT_TYPE_DEFAULT_:
        staticName: true
        schedule: "0 0 * * *"
        successfulJobsHistoryLimit: 1
        failedJobsHistoryLimit: 2
        concurrencyPolicy: Forbid
        job:
          activeDeadlineSeconds: 28800
          backoffLimit: 2
          parallelism: 1
          pod:
            restartPolicy: OnFailure
            containers:
              cron:
                image: 
                  repository: busybox
      foo:
        schedule: "0 0 * * *"
      bar:
        schedule: "0 0 * * *"
        job:
          activeDeadlineSeconds: 28800
          backoffLimit: 2
          parallelism: 1

This would generate this using helm template (removed useless labels/annotations and containers fields) :

---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: bar
spec:
  schedule: 0 0 * * *
  successfulJobsHistoryLimit: 1
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 2
  jobTemplate:
    metadata:
      name: bar
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - image: busybox
            name: cron
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: foo
spec:
  schedule: 0 0 * * *
  successfulJobsHistoryLimit: 1
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 2
  jobTemplate:
    metadata:
      name: foo
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - image: busybox
            name: cron

You can see that activeDeadlineSeconds, backoffLimit and parallelism were not populated.

I also noticed the cronjob schedule field is not pulled from _HULL_OBJECT_TYPE_DEFAULT_ as well, you need to explicitly set it for each cronjob, not sure if it's intended to be that way.

Hello!

It seems that I've found an unexpected (at least by me) scenario when a string is being forcefully converted to an integer by hull.util.transformation.get.

Here's a minimalistic values set:

hull:
  objects:
    job:
      my-job:
        pod:
          containers:
            my-container:
              image:
                registry: my-registry.local
                repository: my-repository
                tag: _HT*image.tag
image:
  tag: "09956e29" # in my "real-world" case this value is just a commit tag, the `image.tag` value is set by the CI pipeline

And here's what I get in spec.template.spec.containers[0].image:

my-registry.local/my-repository:9.956e+32

Can I prevent this behaviour?

Thank you in advance, I'm really sorry for disturbance.

Hello again,

I have another newbie question if you please.

Is it possible to generate properties (not only properties' values) in the values set by transformations?

Something like that:

hull:
  object:
    deployment:
      my-cool-deployment:
        pod:
          _HT_ADD_PROPERTIES: _HT!{{ something_that_generates_a_dictionary }}

The _HT_ADD_PROPERTIES could be some code word to let HULL know that we're going to add some properties.

Thank you for your patience and, again, for a really nice tool!

State
Security features are not enforced by default

Feature Idea
Enforce some easy to implement security features by default:

• automountServiceAccountToken should be set to false by default
• ...

State

At the moment it is possbile to prefix get transformations _HT* with serialization instructions to get string representations of dictionaries and arrays. Allowed prefixes are:

• toJson
• toRawJson
• toPrettyJson
• toYaml
• toString

Feature Idea

Extend prefix usage to include more standard operations on single values (without additional arguments).
Possibilites:

• lower, upper, title, untitle, ... for strings
• int, float64, ... for numbers
• ...

Full list of possible prefixes/functions is here

Future:

Maybe extend to functions with additional arguments but that may stand in contradiction to the idea that _HT* should be simple and easy to use?

Hi. First of all, thank you for this wonderful project.
Hull generates empty values for fields that I don't provide, which is very undesired in the context of ArgoCD. With autosync set the application doesn't ever get synced entirely, given that it removes unused fields and I have no way of disabling these fields in hull's config.
Is it possible to not render empty fields?

Didn't found a way to set hostNetwork.

One of my helm charts let users configure various variable put into hull.config.specific. It works well, except for the imagePullPolicy. Below is part of the chart's value :

hull:
  config:
    specific:
      image:
        pullPolicy: Never
  objects:
    deployment:
      nginx:
        pod:
          containers:
            nginx:
              imagePullPolicy: _HT*hull.config.specific.image.pullPolicy

When doing a helm template, I've got a schema validation error :

Error: values don't meet the specifications of the schema(s) in the following chart(s):
hull:
- (root): Must validate at least one schema (anyOf)
- objects.deployment.nginx: Must validate at least one schema (anyOf)
- objects.deployment.nginx.pod.containers.nginx: Must validate at least one schema (anyOf)
- objects.deployment.nginx.pod.containers.nginx.imagePullPolicy: objects.deployment.nginx.pod.containers.nginx.imagePullPolicy must be one of the following: "Always", "IfNotPresent", "Never"
- objects.deployment.nginx.pod.containers.nginx: Must validate all the schemas (allOf)
- objects.deployment.nginx.pod: Must validate all the schemas (allOf)
- objects.deployment.nginx: Must validate all the schemas (allOf)

I've taken a look at hull's values.schema.json, and there is the following for imagePullPolicy :

"imagePullPolicy": {
    "description": "Image pull policy. One of Always, Never, IfNotPresent. Defaults to Always if :latest tag is specified, or IfNotPresent otherwise. Cannot be updated. More info: https://kubernetes.io/docs/concepts/containers/images#updating-images\n\nPossible enum values:\n - `\"Always\"` means that kubelet always attempts to pull the latest image. Container will fail If the pull fails.\n - `\"IfNotPresent\"` means that kubelet pulls if the image isn't present on disk. Container will fail if the image isn't present and the pull fails.\n - `\"Never\"` means that kubelet never pulls an image, but only uses a local image. Container will fail if the image isn't present",
    "enum": [
        "Always",
        "IfNotPresent",
        "Never"
    ],
    "anyOf": [
        {
            "$ref": "#/definitions/hull.Transformation.Pattern"
        },
        {
            "type": "string"
        }
    ]
}

I'm not 100% knowledgeable, but shouldn't the enum part be inside the anyOf ? I've tried changing this to the following, and it seems to work fine. I'll open a pull request with these changes.

"imagePullPolicy": {
    "description": "Image pull policy. One of Always, Never, IfNotPresent. Defaults to Always if :latest tag is specified, or IfNotPresent otherwise. Cannot be updated. More info: https://kubernetes.io/docs/concepts/containers/images#updating-images\n\nPossible enum values:\n - `\"Always\"` means that kubelet always attempts to pull the latest image. Container will fail If the pull fails.\n - `\"IfNotPresent\"` means that kubelet pulls if the image isn't present on disk. Container will fail if the image isn't present and the pull fails.\n - `\"Never\"` means that kubelet never pulls an image, but only uses a local image. Container will fail if the image isn't present",
    "anyOf": [
        {
            "$ref": "#/definitions/hull.Transformation.Pattern"
        },
        {
            "enum": [
                "Always",
                "IfNotPresent",
                "Never"
            ]
        }
    ]
},

State
When using _HT? the conditional expression must be written in native Go Templating syntax making it unconvenient to access fields or use includes in a short form

Feature Idea
Combine _HT? with _HT* and _HT/ so that a shorter form is sufficient to write the condition.

Alternative 1: Combine symbols like _HT?* and _HT?/
Alternative 2: Combine expressions like _HT?_HT* and _HT?_HT/

In both variants, it must be detected that the condition is a HULL transformation and not a Go templating expression. The HULL transformation must be executed and the result subjected to the boolean check.

I've encountered an error when linting my charts : [ERROR] templates/hull.yaml: unable to parse YAML: invalid Yaml document separator: apiVersion: v1

This happens with helm 3.7+. Helm 3.5 and 3.6 are fine and do not show such an error. Now, generating the template with helm template gives correct results, so this might be a helm issue.

A non-hull helm chart with this basic template succesfullt passes the lint. This is a similar output to helm template with the hull chart. :

templates/test.yaml (there is nothing else in the templates folder) :

apiVersion: v1
data:
  testdata: lorem-ipsum
kind: ConfigMap
metadata:
  name: release-name-test-testconfigmap1
---
apiVersion: v1
data:
  testdata: dolor sic amet
kind: ConfigMap
metadata:
  name: release-name-test-testconfigmap2

However, such a Hull chart fails :

values.yaml :

hull:
  objects:
    configmap:
      testConfigMap:
        data:
          testdata:
            inline: "lorem ipsum dolor"

templates/hull.yaml (there is nothing else in the templates folder) :

{{- include "hull.objects.prepare.all" (dict "HULL_ROOT_KEY" "hull" "ROOT_CONTEXT" $) }}

I don't really understand the cause, and if hull is at fault, or helm itself. I will continue investigating this.

Hello
When using hull I need to access .Chart and .Release objects. Currently I am doing it like this

tag: _HT!{{ (index . "$").Chart.Version }}

Is there simpler/proper/idiomatic way of doing this ?

I'm trying to build a set of deployments based on Hull and ran into an interesting YAML quoting/number formatting issue... I defined a variable tag that I reuse in several places. I populate it with a short Git revision hash. Here's an abbreviated version of the config:

hull:
  config:
    general:
      metadata:
        labels:
          common:
            revision: _HT*hull.config.specific.tag
    specific:
      tag: "3964137"
  objects:
    deployment:
      server:
        pod:
          containers:
            api:
              image:
                registry: us.gcr.io
                repository: my-app/my-api
                tag: _HT*hull.config.specific.tag

However, when I render using helm template and attempt to deploy it via Argo CD (which also calls helm template), I get this (abbreviated, again):

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    revision: "3.964137e+06"
spec:
  template:
    metadata:
      labels:
        revision: "3.964137e+06"
    spec:
      containers:
      - image: us.gcr.io/my-app/my-api:3.964137e+06

As shown above, I explicitly quoted the tag value, i.e. tag: "3964137", and I tried explicitly quoting the shortcut, e.g. tag: "_HT*hull.config.specific.tag". It doesn't seem to have any impact, though.

Is there a way to get my variable value respected as a string and not turned into scientific notation?

Hello again,

It seems that I've found another interesting phenomenon.

Let's assume we have the following values.yaml:

hull:
  objects:
    ingress:
      test:
        rules:
          default:
            host: example.org
            http:
              paths:
                root:
                  path: /
                  pathType: Prefix
                  backend:
                    service:
                      name: test
                      port:
                        name: http
        tls:
          default:
            hosts:
              - example.org
            secretName: tls-certificate

Everything seems to be fine in this case, we get the following Ingress object:

kind: Ingress
metadata:
  annotations: {}
  labels:
    app.kubernetes.io/component: test
    app.kubernetes.io/instance: test
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: hull-test-3
    app.kubernetes.io/part-of: undefined
    app.kubernetes.io/version: 0.0.0
    helm.sh/chart: hull-test-3-0.0.0
  name: test-hull-test-3-test
  namespace: default
spec:
  rules:
  - host: example.org
    http:
      paths:
      - backend:
          service:
            name: test-hull-test-3-test
            port:
              name: http
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - example.org
    secretName: test-hull-test-3-tls-certificate

The spec.tls part is valid.

But when we try to use a transformation in hull.objects.ingress.*.tls.*.hosts, we have something different.

Here's values.yaml:

hull:
  objects:
    ingress:
      test:
        rules:
          default:
            host: example.org
            http:
              paths:
                root:
                  path: /
                  pathType: Prefix
                  backend:
                    service:
                      name: test
                      port:
                        name: http
        tls:
          default:
            hosts:
              - _HT/util.domainNames.getIngressHostname
            secretName: tls-certificate

The templates/_domain_names.tpl module is simple and straightforward:

{{- define "util.domainNames.getIngressHostname" -}}
example.org
{{- end -}}

In this case we have the following result:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations: {}
  labels:
    app.kubernetes.io/component: test
    app.kubernetes.io/instance: test
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: hull-test-3
    app.kubernetes.io/part-of: undefined
    app.kubernetes.io/version: 0.0.0
    helm.sh/chart: hull-test-3-0.0.0
  name: test-hull-test-3-test
  namespace: default
spec:
  rules:
  - host: example.org
    http:
      paths:
      - backend:
          service:
            name: test-hull-test-3-test
            port:
              name: http
        path: /
        pathType: Prefix
  tls:
  - hosts: example.org
    secretName: test-hull-test-3-tls-certificate

The spec.tls structure is wrong: the hosts parameter has become a string instead of being an array.

What do you think, why does it happen?

Thank you!