dhall-lang / dhall-kubernetes

Typecheck, template and modularize your Kubernetes definitions with Dhall

Dhall is a programmable configuration language optimized for maintainability.

You can think of Dhall as: JSON + functions + types + imports

Note that while Dhall is programmable, Dhall is not Turing-complete. Many of Dhall's features take advantage of this restriction to provide stronger safety guarantees and more powerful tooling.

You can try the language live in your browser by visiting the official website:

• https://dhall-lang.org

Getting started

The easiest way to get started experimenting with Dhall is to install the dhall-to-json and/or dhall-to-yaml executables, which enable you to generate JSON and YAML, respectively, on the command line. Platform- and runtime-specific installation instructions can be found in the Dhall documentation.

For other ways to integrate Dhall in your project, read:

• How to integrate Dhall

Tutorials

For a short introduction, read:

• Learn Dhall in Y minutes

To learn more about core language features, read:

• Core language features

For an even longer hands-on tutorial, read:

• Getting started: Generate JSON or YAML

... and for an even longer tutorial, read:

• Language Tour

Finally, we have a cheatsheet for a very condensed overview and quick lookup:

• Dhall Cheatsheet

What is this repository?

The Dhall configuration language has multiple implementations so that Dhall configuration files can be understood natively by several programming languages. You can find the latest list of the language bindings and their respective repositories here:

• How to integrate Dhall - Language support

This repository contains language-independent functionality, such as:

• The grammar and formal semantics

  Dhall is a formally-specified language standard, and language bindings follow the specification in order to ensure portability of Dhall configuration files across language bindings.

• The standard test suite

  This repository contains a test suite that language bindings can use to check compliance against the standard.

• The Prelude

  One Dhall package named the Prelude is versioned with and distributed alongside the language standard. This package contains general-purpose utilities.

• Shared infrastructure for the Dhall ecosystem

  Several services support Dhall developers and this repository contains a NixOps specification of that infrastructure that automatically deploys changes merged to that configuration.

Development status

The current version and versioning policy is detailed in the Versioning document, and you can see the latest changes in the Changelog.

The Dhall configuration language slowly evolves in response to user feedback and if you would like to participate in the language evolution process then you should read:

• CONTRIBUTING.md - How do I change the language

Design philosophy

Programming languages are all about design tradeoffs and the Dhall language uses the following guiding principles (in order of descending priority) that help navigate those tradeoffs:

• Polish

  The language should delight users. Error messages should be fantastic, execution should be snappy, documentation should be excellent, and everything should "just work".

• Simplicity

  When in doubt, cut it out. Every configuration language needs bindings to multiple programming languages, and the more complex the configuration language the more difficult to create new bindings. Let the host language that you bind to compensate for any missing features from Dhall.

• Beginner-friendliness

  Dhall needs to be a language that anybody can learn in a day and debug with little to no assistance from others. Otherwise people can't recommend Dhall to their team with confidence.

• Robustness

  A configuration language needs to be rock solid. The last thing a person wants to debug is their configuration file. The language should never hang or crash. Ever.

• Consistency

  There should only be one way to do something. Users should be able to instantly discern whether or not something is possible within the Dhall language or not.

The Dhall configuration language is also designed to negate many of the common objections to programmable configuration files, such as:

"Config files shouldn't be Turing complete"

Dhall is not Turing-complete. Evaluation always terminates, no exceptions

"Configuration languages become unreadable due to abstraction and indirection"

Every Dhall configuration file can be reduced to a normal form which eliminates all abstraction and indirection

"Users will go crazy with syntax and user-defined constructs"

Dhall is a very minimal programming language. For example: you cannot even compare strings for equality. The language also forbids many other common operations in order to force users to keep things simple

Name

The language is named after a Dustman from the game Planescape: Torment who belongs to a faction obsessed with death (termination). The fountain pen in the logo is the modern analog of Dhall's quill.

The name rhymes with "tall"/"call"/"hall" (i.e. "dɔl" for a US speaker or "dɔːl" for a UK speaker using the International Phonetic Alphabet).

Contributors

Code Contributors

This project exists thanks to all the people who contribute. [Contribute].

Financial Contributors

Become a financial contributor and help us sustain our community. [Contribute]

Individuals

Organizations

Support this project with your organization. Your logo will show up here with a link to your website. [Contribute]

dhall-kubernetes's Issues

Can't use the kubernetes union types

Hi, thanks for this amazing project. Having a strongly typed, safe and normalizing language to configure Kubernetes is amazing!

I'm trying to create something similar to what happens in the helm charts world, where i have a template which accepts a few parameters, and populates the different Kubernetes resources. I was trying to use the typesUnion.dhall and what explained in https://github.com/dhall-lang/dhall-kubernetes#can-i-generate-a-yaml-file-with-many-objects-in-it but i'm getting a "Wrong type of function argument" error and i don't understand why (i tried using --explain as well).

Here is some sample code:

-- chart.dhall
let k8s =
      https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/master/typesUnion.dhall sha256:b0352411b2d25a64261799cbdb4e29c72a4000daaca19a9fa013683fc46964f5

let Config : Type = ./templates/config.dhall

let deployment = ./templates/deployment.dhall

let chart =
        λ(config : Config)
      → 
      { apiVersion =
            "v1"
        , kind =
            "List"
        , items =
            [ k8s.Deployment (deployment config) ]
        }

in chart

--- templates/deployment.dhall
let types =
      https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/master/types.dhall sha256:29d8c4387540826a68775e9439e8d00f226d440a83dedf80ff877c612fa524a2

let defaults =
      https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/master/defaults.dhall sha256:7fb71a223f3aeb4f72824303f7cbfdd038bb2b7a7dc1f76db7e65031dad0523c

let prelude =
      https://raw.githubusercontent.com/dhall-lang/dhall-lang/v8.0.0/Prelude/package.dhall --sha256:0c04cbe34f1f2d408e8c8b8cb0aa3ff4d5656336910f7e86190a6d14326f966d

let Config = ./config.dhall

let kv = prelude.JSON.keyText

let deployment =
        λ(config : Config)
      →   defaults.Deployment
        ⫽ { metadata =
              defaults.ObjectMeta ⫽ { name = config.appName }
          , spec =
                defaults.DeploymentSpec
              ⫽ { selector =
                      defaults.LabelSelector
                    ⫽ { matchLabels = [ kv "app" config.appName ] }
                , replicas =
                    config.nReplicas
                , template =
                      defaults.PodTemplateSpec
                    ⫽ { metadata =
                            defaults.ObjectMeta
                          ⫽ { name =
                                config.appName
                            , labels =
                                [ kv "app" config.appName ]
                            }
                      , spec =
                            defaults.PodSpec
                          ⫽ { containers =
                                [   defaults.Container
                                  ⫽ { name =
                                        "nginx"
                                    , image =
                                        "nginx:1.8"
                                    , ports =
                                        [   defaults.ContainerPort
                                          ⫽ { containerPort = 80 }
                                        ]
                                    }
                                ]
                            }
                      }
                }
          }

in  deployment

--- templates/config.dhall
{ nReplicas : Natural, appName : Text, dnsDomain: Text }

I'm simply trying to invoke dhall <<< './chart.dhall' without even passing any parameter for now and i get this weird error:

Error: Wrong type of function argument

{ spec : - Optional …
         + { … : … }
, …
}

18:               k8s.Deployment (deployment config)

Do you have any suggestion on how to solve it? I thought I was just applying what explained in https://github.com/dhall-lang/dhall-kubernetes#can-i-generate-a-yaml-file-with-many-objects-in-it but i'm missing something :(

Thanks in advance! :)

Cannot build with Nix sandboxing

Using

nix-build release.nix -A dhall-kubernetes --arg src '{rev = "";}'

Fails because dhall-1.18 fails to build. dhall-1.18 test suite makes HTTP requests, which is verboten in Nix, causing the test suite to fail

Fix documentation about --documents

I think we can use typesUnion.dhall together with the --documents flag as well. The docs seem to imply that isn't possible and that you need to use kind: List but I think that's not true.

Dhall 10 first class modules

Hey there :)

What do you think about supporting first class modules now that dhall 10 supports them?
It could go from simply having a package.dhall with

{ types = ./types.dhall,
  defaults = ./defaults.dhall,
  typesUnion = ./typesUnion.dhall
}

or something more complex (but maybe easier to discover?) like for each value we can have

-- package.dhall
{
  Deployment = ./Deployment/package.dhall
}

-- Deployment.dhall
{
    Type = ./Type.dhall,
     default = ./default.dhall
}

What do you think?

Thanks again for this project!

Make `examples` folder

The current state of things wrt usage examples is that there's only two "interactive" examples in the README.

I'd like to have an examples folder, to serve multiple purposes:

• Use it as a place to stash all the usage examples
• Use it to eventually run some tests in CI. E.g. we could test that executing some example always generates the same yaml.

how to use --documents to render many files at once ?

I'd like to exploit dhall-to-yaml --documents to render deployment.dhall and service.dhall in the same output. Is that possible ?

JSONSchemaProps.dhall references itself (Cyclic import)

When trying to create a custom resource definition, dhall errors with

↳ ./../dhall-kubernetes/default/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.CustomResourceDefinition.dhall
  ↳ ./../dhall-kubernetes/types/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.CustomResourceDefinitionSpec.dhall
    ↳ ./../dhall-kubernetes/types/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.CustomResourceValidation.dhall
      ↳ ./../dhall-kubernetes/types/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.JSONSchemaProps.dhall

Cyclic import: ./io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.JSONSchemaProps.dhall

The reason is that JSONSchemaProps references itself (at least I think that's the issue)

Support yaml-to-dhall

I tried running yaml-to-dhall on a configmap like this with dhall-kubernetes/types/io.k8s.api.core.v1.ConfigMap.dhall as SCHEMA:

apiVersion: v1
kind: ConfigMap
metadata:
  name: "eirini"
data:
  key: value

But it failed as it didn't match the schema. The type looks like this:

{ apiVersion :
    Text
, binaryData :
    List { mapKey : Text, mapValue : Text }
, data :
    List { mapKey : Text, mapValue : Text }
, kind :
    Text
, metadata :
    ./io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta.dhall
}

My understanding is that data and binaryData both are optional fields in K8s spec, but that is not true for the dhall type. Why is this so?

For context, I want to use yaml-to-dhall as a begining point to translate my helm chart into a sort of "dhall distribution".

Pin Kube version?

Currently we get the swagger spec from the master branch of Kubernetes.

Now, Kube should not remove any specifications, but they do deprecate some. On our end everything should be fine, as we never remove any types/default, we only add them.

Should we pin the Kube version? If yes, should we support multiple versions?

Update v1.0.0 to support Dhall 9.0.0?

dhall-kubernetes at tag v1.0.0 still uses the old Optional literal syntax, which is no longer permitted in Dhall 9.0.0.

Can we get a v1.1.0 tag which replaces the old Optional literal syntax with Some and None so that we can stay up to date?

Setup CI

From #16 we'll have a script to typecheck all the generated files, and as mentioned in #14 (review) it would be nice to have CI check a set of end-to-end examples so that we don't inadvertently break users.

/cc @Gabriel439 should we use the Hydra instance as all the other repos, and if yes what's missing to make it work?

Support union types

The new API presumes that one runs dhall-to-yaml --omitEmpty. However, there are areas in the API which expect an empty block to be defined, for example,

let Kubernetes = https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/4ad58156b7fdbbb6da0543d8b314df899feca077/defaults.dhall
in
Kubernetes.Volume // { name = "data" , emptyDir = Kubernetes.EmptyDirVolumeSource }

which unfortunately renders as

name: data

instead of:

name: data
emptyDir: {}

The current workaround (in this specific case) is to hard-code the default medium (which is the empty string) so that emptyDir renders and Kubernetes knows that the volume is an emptyDir volume. But it makes me uneasy... I know that there are various areas of the dhall-kops API which work the same way.

Prefer beta over alpha apis

Hi there :)

I just encountered an issue today with Kubernetes 1.14 because the CronJob in defaults.dhall has as api version batch/v2alpha1 instead of batch/v1beta1.

I was able to solve it by overriding locally but I wonder if the default behavior should be preferring stable over beta over alpha in the defaults file.

I didn't see any logic in the generator around that so I assume this was due to alphabetic ordering overriding? (both v2alpha 1 and v1beta1 exist).

Let me know what you think :) I can contribute it as a change if you think it's useful!

Thanks again for this awesome project 👍 and apologies for all the issues I open, if you have a better communication channel for this stuff let me know :)

The deployment.dhall example is broken

I'm using dhall-to-json-1.2.6 (corresponding to dhall-1.20).

$ (cd examples; dhall-to-yaml < deployment.dhall)


↳ ./../api/Deployment/mkDeployment
  ↳ ./../api/Deployment/RawDefaults

Error: Field mismatch

{ deployment    = ../../default/io.k8s.api.apps.v1.Deployment.dhall
, container     = ../../default/io.k8s.api.core.v1.Container.dhall
, containerPort = ../../default/io.k8s.api.core.v1.ContainerPort.dhall
======================================================================
, Int           = intOrString.Int
, String        = intOrString.String
}
/Users/yom/code/dhall-kubernetes/examples/../api/Deployment/RawDefaults:4:1

Transition to new Optional syntax

@f-f I'm sure this is already on your radar, but with the deprecation of the old Optional syntax the generation script will need to be updated do that dhall-kubernetes will continue to parse under the new standard.

Inconsistent types

The current dhall-kubernetes does not give a generic way for the object type of kubernetes E.g. configMap.core.v1.data which is of type object under the kubernetes API documentation is translated to data : List { mapKey : Text, mapValue : Text } which does not support nested map Values (https://github.com/dhall-lang/dhall-kubernetes/blob/master/types/io.k8s.api.core.v1.ConfigMap.dhall)

dhall 9 now supports weakly-typed JSON (#586) which should solve the issue. Is there a plan to support it?

Get the website up and running

There's a template at http://dhall-lang.github.io/dhall-kubernetes/ but has bogus content. I am quite content with the design so far though.

I want to update this issue with wishes for the kind of stuff we want on the website. Please add anything you'd like to see on it to this issue too.

io.k8s.api.core.v1.ObjectFieldSelector has the wrong type

In the swagger doc (v 1.11.0)

"io.k8s.api.core.v1.ObjectFieldSelector": {
    "description": "ObjectFieldSelector selects an APIVersioned field of an object.",
    "required": [
     "fieldPath"
    ],
    "properties": {
     "apiVersion": {
      "description": "Version of the schema the FieldPath is written in terms of, defaults to \"v1\".",
      "type": "string"
     },
     "fieldPath": {
      "description": "Path of the field to select in the specified API version.",
      "type": "string"
     }
    }
   },

But in the dhall version:

{ apiVersion : (Text)
, fieldPath : (Text)
}

both are required; expected type is

{ apiVersion : (Optional (Text))
, fieldPath : (Text)
}

Generator improvements

Hi there, we've been using the generator to generate dhall types/defaults/schemas for other k8s related projects, like argo-workflows and argo-events.

I made a few improvements to the generator:

• Add --skipDuplicates CLI flag and handler.

  • This allows you to just skip types that have the same ModelName but different namespaces; the types are still emmitted but skipped when creating the top-level types/defaults/schema files, etc. Default behaviour is to error at, like current behaviour.
  • Can see the change here

• Add --prefixMap option for specifying external import roots

  • This allows us to generate files for other projects that depend on k8s types and have the generated files import the desired repo or path.
  • Can see the change here
  • For example:

$ dhall-kubernetes-generator --skipDuplicates --prefixMap io.k8s=https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/master/types/. argo-events-swagger.json

Then generates files that look like this:

$ cat types/io.argoproj.common.EventContext.dhall
{ cloudEventsVersion :
    Text
, contentType :
    Text
, eventID :
    Text
, eventTime :
    https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/master/types/io.k8s.apimachinery.pkg.apis.meta.v1.MicroTime.dhall
, eventType :
    Text
, eventTypeVersion :
    Text
, extensions :
    List { mapKey : Text, mapValue : Text }
, schemaURL :
    ./io.argoproj.common.URI.dhall
, source :
    ./io.argoproj.common.URI.dhall
}

If there's any interest in merging these changes back upstream I'd be happy to open one or more PRs.

ConfigMap output incorrectly

Given the following ConfigMap definition:

let types =
      https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/fef6b2328bc8ea8d20dee1d9a0be7bb91655a1a3/types.dhall

let defaults =
      https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/319ed3bc2be0a85029bbe87147a83a2c9b857422/defaults.dhall

let configMap
    : types.ConfigMap
    =   defaults.ConfigMap
      ⫽ { metadata =
            defaults.ObjectMeta ⫽ { name = "test" }
        , data =
            [ { mapKey = "someKey", mapData = "someValue" } ]
        }

in  configMap

And running dhall-to-yaml:

$ dhall-to-yaml --omitEmpty < test.dhall 
apiVersion: v1
kind: ConfigMap
data:
- mapKey: someKey
  mapData: someValue
metadata:
  name: test

The following is output:

apiVersion: v1
kind: ConfigMap
data:
- mapKey: someKey
  mapData: someValue
metadata:
  name: test

However, this is not the expected output which should be:

apiVersion: v1
kind: ConfigMap
data:
  someKey: someValue
metadata:
  name: test

Here is a snippet of the swagger definition from 1.11 for the ConfigMap object:

   "v1.ConfigMap": {
    "id": "v1.ConfigMap",
    "description": "ConfigMap holds configuration data for pods to consume.",
    "properties": {
     "kind": {
      "type": "string",
      "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds"
     },
     "apiVersion": {
      "type": "string",
      "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources"
     },
     "metadata": {
      "$ref": "v1.ObjectMeta",
      "description": "Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata"
     },
     "data": {
      "type": "object",
      "description": "Data contains the configuration data. Each key must consist of alphanumeric characters, '-', '_' or '.'. Values with non-UTF-8 byte sequences must use the BinaryData field. The keys stored in Data must not overlap with the keys in the BinaryData field, this is enforced during validation process."
     },
     "binaryData": {
      "type": "object",
      "description": "BinaryData contains the binary data. Each key must consist of alphanumeric characters, '-', '_' or '.'. BinaryData can contain byte sequences that are not in the UTF-8 range. The keys stored in BinaryData must not overlap with the ones in the Data field, this is enforced during validation process. Using this field will require 1.10+ apiserver and kubelet."
     }
    }
   }

versus the ConfigMap type:

{ apiVersion :
    Text
, binaryData :
    List { mapKey : Text, mapValue : Text }
, data :
    List { mapKey : Text, mapValue : Text }
, kind :
    Text
, metadata :
    ./io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta.dhall
}

defaults.dhall does not contain the correct ingress path

We are using defaults.dhall in order for imports.

When using Ingress we encountered the following typo:
Under defaults.dhall the path is setup as Ingress = ./defaults/io.k8s.api.networking.v1beta1.Ingress.dhall sha256:6d13f39daff533b29f92acb75fd7c405aaf6bd76056c508a833d745e71e0d414

while the correct path sits under io.k8s.api.extensions.v1beta1.Ingress.dhall : https://github.com/dhall-lang/dhall-kubernetes/blob/master/defaults/io.k8s.api.extensions.v1beta1.Ingress.dhall

[RFC]: Resolve defaults recursively

For example, for a deployment:

instead of filling each default with the obvious "empty type" of a field, recursively resolve the default values.

so instead of this:

\(_params : {metadata : (../types/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta.dhall)}) ->
{ apiVersion = ("apps/v1" : Text)
, kind = ("Deployment" : Text)
, metadata = _params.metadata
, spec = ([] : Optional (../types/io.k8s.api.apps.v1.DeploymentSpec.dhall))
, status = ([] : Optional (../types/io.k8s.api.apps.v1.DeploymentStatus.dhall))
} : ../types/io.k8s.api.apps.v1.Deployment.dhall

this:

{ apiVersion = ("apps/v1" : Text)
, kind = ("Deployment" : Text)
, metadata =  ../default/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta.dhall
, spec =  ../default/io.k8s.api.apps.v1.DeploymentSpec.dhall
, status = ../default/io.k8s.api.apps.v1.DeploymentStatus.dhall
} : ../types/io.k8s.api.apps.v1.Deployment.dhall

This makes a "merge" based API where we compose layers of configurations with // way more natural. Similar to https://kustomize.io

Invalid generated files

Some generated files contain invalid Dhall code. This is due to properties named $ref. Dhall does not support properties with $ in the label.

I’d suggest we skip generation of these files to prevent users from using them. I don’t think it would be a good idea to sanitize the labels to something like _dollar_ref or similar since you would not be able to send the output to Kubernetes.

This is the list of invalid files:

./default/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.CustomResourceDefinitionSpec.dhall
./default/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.CustomResourceValidation.dhall
./default/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.JSONSchemaProps.dhall
./default/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.CustomResourceDefinition.dhall
./default/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.CustomResourceDefinitionList.dhall
./default/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.JSONSchemaPropsOrArray.dhall
./default/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.JSONSchemaPropsOrBool.dhall
./default/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.JSONSchemaPropsOrStringArray.dhall

Maybe revert back to using optionals even for emtpy records and lists

In the current master, Container is broken, which is arguably one of the most central pieces of kubernetes. The reason is that readinessProbe (and also livenessProbe) are required in the Container type (see here), but Kubernetes only allows this key to either miss completely or have at least one of the Probe keys defined.

Because of this and also the reason that the generated yaml is very noisy with a lot of {} and [], I propose to go back to using Optional even for Lists and Records that contain only optionals.

Create a nix derivation that stores generated files in nix store

This way, I can depend on dhall-kubernetes's output in my kubernetes project easily without
having to generate the files myself.

Commit bit

@arianvp could you please me add as owner/writer of the repo? For assigning you PRs/bus-factor/availability, etc :)

Add a bit of documentation + some examples

Higher-level API for user

Using the API is still very verbose. As a user I mostly care about a couple of top-level resources from the APIs and their spec field. I think it would be great if this repo would provide a higher-level API built on the generated one that makes it easier to use. For example

let Deployment = ./k8s/api/apps/v1/Deployment
in let metadata = Deployment.defaultMetadata { name = "hello-world" }
in let spec = Deployment.defaultSpec
  { selector = Deployment.defaultSelector { ... }
  , template = Deployment.defaultTemplate
    { metadata = Deployment.defaultMetadata
    , spec = Deployment.defaultPodSpec { ... } : Deployment.PodSpec
    }
  } 
in let deployment = Deployment.default { metadata = metadata, spec = spec }: Deployment.Type

In this example we actually make spec required because having a deployment without it is meaning less. This also reduces some boilerplate with the Some constructor.

You can also get all the default constructors and types from the Deployment record.

Example formatting is intimidating

Are the examples (e.g. https://github.com/dhall-lang/dhall-kubernetes/blob/master/examples/deploymentSimple.dhall ) auto-formatted somehow? These were very offputting to me as the first example I was exposed to, the indentation makes it really hard to parse and implies much more complexity (nesting) than there actually appears to be.

I manually reformatted as https://gist.github.com/timbertson-zd/accefe1f69fd28c77238e43bc80ac3b2. To my (unfamiliar) eyes it looks much more approachable.

Support CRD's

Hi,
I am reading up on dhall to manage our solution. We have few CRD's. Does dhall have the capability to take the CRD spec ( type definition in yaml) and convert it to a type ?
We use kubebuilder to generate the CRD's and the controller for it. I would like to have that as the source of truth. Whenever, we change the CRD type ( add/remove/update fields in the CRD) in that project, I would like the dhall type for that CRD to reflect the changes.
kubebuilder generates a type definition for the CRD , is it possible to take that and generate appropriate dhall type so we can use it to generate yamls to deploy those.

LabelSelector type-checks with empty component Lists

I'm currently trying to fit external-dns into our Dhall-ified pipeline, and I replicated the mistake in their tutorial wherein a DeploymentSpec is created without a LabelSelector (since their Deployment doesn't have any labels in its metadata).

Currently, a DeploymentSpec bundles in its own default LabelSelector, which bundles in empty lists for matchLabels and matchExpressions. Due to --omitEmpty, this succeeds at type-checking, and the enveloping Deployment also type-checks.

However, attempting to actually apply the Deployment results in error: error validating "STDIN": error validating data: ValidationError(Deployment.spec): missing required field "selector" in io.k8s.api.apps.v1.DeploymentSpec.

Not sure what a proper solution is considering dhall-lang#691 precluded language-level support for non-empty Lists.

package.dhall

The Prelude has a package.dhall file (in the root and every subdirectory) that allows you to import the whole package once and access the various components via record selector.

Example:

-- To get the List/map function
let Prelude = https://raw.githubusercontent.com/dhall-lang/Prelude/master/package.dhall
in Prelude.`List`.map

I would like to do something similar for our repo, but there would be different possibilities:

1. package file in the root would be similar to the Prelude one: { types = ./types/package.dhall, default = ./default/package.dhall }.
   In this way to access e.g. a Deployment type and default, one would do

   let kube = https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/master/package.dhall
in let DeploymentType = kube.types.io.k8s.api.apps.v1beta2.Deployment.dhall
in let DeploymentDefault = kube.default.io.k8s.api.apps.v1beta2.Deployment.dhall
in DeploymentDefault : DeploymentType 
-- actually this would not typecheck (DeploymentDefault is a function), but it's for demonstration purposes

2. package file in the root would have keys as the Kube entities, and values as a record with type and default:

{ io.k8s.api.apps.v1beta2.Deployment =
  { type = ./types/io.k8s.api.apps.v1beta2.Deployment.dhall
  , default = ./default/io.k8s.api.apps.v1beta2.Deployment.dhall
  }
, ...
}

In this way to access e.g. a Deployment type and default, one would do

   let kube = https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/master/package.dhall
in let Deployment = kube.io.k8s.api.apps.v1beta2.Deployment
in Deployment.default : Deployment.type 
-- actually this would not typecheck (DeploymentDefault is a function), but it's for demonstration purposes

/cc @arianvp opinions? (I'm leaning towards 2, as it's less lines of boilerplate for the user)

PS: I'm not sure we could use dots in record keys though, so we might need to go back to the nested directories layout, but it's a detail here

Figure out really required keys?

There is some cases in which we mark some fields as required, but they are not. We should somehow figure out if they are actually required only in some situations, and generate the files accordingly.

Some examples:

• in io.k8s.api.apps.v1.DeploymentSpec, selector is marked as required, but Kubernetes is perfectly happy in accepting a Deployment without it (it will populate it based on spec.template.metadata.labels.
• in io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta, we manually mark name as a required field (I think it's required if it's a top-level object), but Kubernetes is perfectly happy in accepting a Deployment whose spec.template.metadata has no name.

Any ideas?

[Question] How useful is dhall-kubernetes (in my context) ?

Honest question.

I am wondering how useful is dhall-kubernetes in the context of a dhall infrastructure setting that would inevitably be using Helm and Terraform as much as possible (instead of plain deployment yaml files).

I just thought of asking this question here directly after replying to dhall-lang/dhall-haskell#97.

Thanks for your reply.

Hydra doesn't build if files don't change

As it happened with #27, if the fileset doesn't change Hydra will not push to Github the build status.

So as suggested by @geigerzaehler in #27 (comment), we should make the commit hash available in the build, and we could do that as suggested in #27 (comment)

Add project references section to the docs?

I just open-sourced dhall-prometheus-operator - I was wondering if there could be a section in the README or the Wiki that links to open-source code which uses dhall-kubernetes.

Thanks!

Comparison with other solutions

As noted in #9, we should probably write down a comparison with other solutions in the readme.
So far on the list:

• Helm
• Jinja

Error while using import

As I play around with Dhall, I tried to recreate the example in this repo. Unfortunately the compilation brought up this error message:

↳ ./deployment.dhall
  ↳ https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/f8be1d55bc641cdb4ad6a1531dbb3e9cad6b5d2e/api/Deployment/mkDeployment

Error: Invalid input

https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/f8be1d55bc641cdb4ad6a1531dbb3e9cad6b5d2e/api/Deployment/mkDeployment:4:12:
  |
4 | in let Some = Prelude.`Optional`.Some
  |            ^
expecting label or whitespace

here is my dhall file:

let default = https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/f8be1d55bc641cdb4ad6a1531dbb3e9cad6b5d2e/api/Deployment/default
in  let Container = https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/f8be1d55bc641cdb4ad6a1531dbb3e9cad6b5d2e/api/Deployment/Container
in  let mkDeployment = https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/f8be1d55bc641cdb4ad6a1531dbb3e9cad6b5d2e/api/Deployment/mkDeployment


in  let config =
    default
    {
        name = "nginx",
        replicas = 2,
        containers = 
            [
                Container
                {
                    name = "nginx",
                    imageName = "nginx",
                    imageTag = "1.15.3",
                    port = [ 80 ] : Optional Natural 
                }
            ]
    }

in mkDeployment config

What did I miss?

Support "format" key

From Kube's Openapi file:

"io.k8s.apimachinery.pkg.util.intstr.IntOrString": {
    "type": "string",
    "format": "int-or-string"
   }

We currently convert this to a Text, while it should instead be a < Int : Natural | String : Text >

So, we should take in account the various values that "format" might take, as they change the semantics.

A quick grep -R '"format"' swagger.json | sed "s/ *//g" | sort | uniq shows that this int-or-string might be the only odd one though ("odd" as in "not present in the specification"):

swagger.json:"format":"byte"
swagger.json:"format":"date-time"
swagger.json:"format":"double"
swagger.json:"format":"int-or-string"
swagger.json:"format":"int32"
swagger.json:"format":"int64"
swagger.json:"format":{

The last line up here is the meta description:

"format": {
 "description": "format is an optional OpenAPI type definition for this column. The 'name' format is applied to the primary identifier column to assist in clients identifying column is the resource name. See https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md#data-types for more.",
 "type": "string"
}

Update to Dhall 1.15

Currently the generated files are compatible with Dhall version < 1.14.

This is due to the breaking change in Natural and Integer grammar that happened in the 1.14 update.

We should

• update the generated Dhall to the latest version
• add a notice about being compatible with version >= 1.14

Change ConfigMap's data field to weakly-typed JSON

Currently ConfigMap's data field's type is List { mapKey : Text , mapValue : Text }. This prevents ConfigMap from hosting nested object structures, which the domain permits.

Can we change the type to the new weakly-typed JSON.object?

Add PR template

Since we have a bunch of things enforced by the CI, it might be a good idea to add a PR template containing a checklist like:

• Run the tests
• Regenerate the README

Generate top-level union type for all apis

Basically, make this auto-generated instead of written by hand: #48 (comment)

IntOrString missing from schemas.dhall

Since the idea behind the schemas.dhall is to import the entire dhall-kubernetes package with one import, the IntOrString type should be added to schemas.dhall so that it can be referenced as:

let Kubernetes = https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/v2.1.0/schemas.dhall
in Kubernetes.IntOrString.String "example"

As things stand, my imports file needs to reference the IntOrString separately, which isn't as simple/easy.

Found as I was trying to port code using the old types.dhall which was able to reference Kubernetes.IntOrString.

Cut a v3.0.0 release?

As we (my employer) prefer to pin to tagged releases and not just GitHub commit hashes, I'd like to cut a v3.0.0 release with the following release notes:

Additions:
* Generate Dhall schemas for multiple Kubernetes versions
* Support `CustomResourceDefinition` in the generator
* `nix-shell` support for `dhall-kubernetes-generator`
* Improve test coverage for `scripts/generate.sh`
* Generator automatically excludes older Kubernetes object versions
* Add `--skipDuplicates` CLI flag and handler to the generator
* Add prefixMap option for specifying external import roots
Cleanup:
* Update omit-empty command in Readme
* Remove top-level `swagger2.json`
* Minimize unnecessary rebuilds

I have the permissions to do it myself, but I'm opening this as an issue in case somebody had some work that they'd like included in a v3.0.0 release and they'd like to hold the release until then, or if somebody has feedback on the language above (maybe I misunderstood something?). So I'll wait 24 hours and then cut the release if there are no objections.

Go back to --omitNull instead of --omitEmpty

... which would imply wrapping all of the non-required fields in Optional.

I'll admit that I originally proposed --omitEmpty as part of #46 but in retrospect I think it may have been a mistake, for a few reasons:

One example is the issue described in #78 caused by using --omitEmpty. There @ari-becker quoted this from the documentation:

A label selector is a label query over a set of resources. … An empty label selector matches all objects. A null label selector matches no objects.

... but when you use omitEmpty there is no way to distinguish a present but empty field (which means to match everything) from an absent field (which means to match nothing). They are semantically distinct!

Similarly, #77 was caused by the same problem: accidentally omitting a field that was a record where transitive fields were all empty (the default EmptyDirVolumeSource), and that was semantically not the same thing.

Also, more generally, Kubernetes is not the only configuration format that has issues like these. For example, I ran into a similar issue when working on encoding the Mergify configuration format in Dhall as part of this chapter of the dhall-manual. It has a delete_head_branch field that stores a value of type {} and with --omitEmpty there is no way to preserve that field.

The reason I'm bringing up non-Kubernetes examples is because I'm trying to slowly come up with best practices as part of writing the Dhall manual and it's becoming increasingly clear to me that --omitEmpty is an anti-pattern that leads to issues like these because it runs the risk of throwing away semantically significant information.

You also don't gain much these days since all that --omitEmpty buys you is omitting a few Somes. Now that we have the :: operator (which has higher precedence than Some) it's actually not that bad to just keep all the Somes for non-required fields. Also if the Somes were to really become problematic we could always revisit that in this issue:

• Optional record fields again

Generate multiple types based on k8s version?

Hi :) would it be possible to support multiple kubernetes (minor) versions at once?
While the versions are backward compatible, i'd like to restrict users of the clusters we own to the subset of the api matching the kubernetes version running to avoid confusion.

Something like creating different directories (1.13 1.14 1.15) would work best i think, so all versions can keep up with the generator at the same time.

What do you think?

Better documentation for contributing / building etc

@jvanbruegge noted they struggled to get things to build. I should spend more time explaining how it works and make sure it isn't actually broken

Pick a license

Current it's not released under and license. And as people seem to be planning to actually use of this perhaps it's wise to add one .

Suggestions are welcome

Port to Haskell?

This allows us to use the Reference implementation AST, to make sure we never generated invalid Dhall.