tweag / tf-ncl

Terraform Configurations with Nickel

Terraform Configurations With Nickel

This repository contains tooling for generating Nickel contracts out of Terraform provider schemas.

It enables configurations to be checked against provider specific contracts before calling Terraform to perform the deployment. Nickel can natively generate outputs as JSON, YAML or TOML. Since Terraform can accept its deployment configuration as JSON, you can straightforwardly export a Nickel configuration, adhering to the right format, to Terraform. Tf-Ncl provides a framework for ensuring a Nickel configuration has this specific format. Specifically, Tf-Ncl is a tool to generate Nickel contracts that describe the configuration schema expected by a set of Terraform providers.

Starting a Tf-Ncl configuration

The easiest way to get started is to use the hello-tf flake template:

nix flake init -t github:tweag/tf-ncl#hello-tf

This will leave you with a flake.nix file containing some glue code for getting a Nickel contract out of tf-ncl, evaluating a Nickel configuration and calling Terraform. It's as easy as

nix develop -c run-terraform init
nix develop -c run-terraform apply

Without Nix it's a bit more complicated. You will need to obtain the Nickel contract using the tools in this repository. Take a look at the working principle for an overview of the process. The most involved step will be calling schema-merge with extracted Terraform provider schemas, see the nix code for inspiration.

Once you have a project set up, you can start writing Nickel configuration code. To quote from the hello-tf example:

let Tf = import "./schema.ncl" in
{
  config.resource.null_resource.hello-world = {
    provisioner.local-exec = [{
      command = m%"
        echo 'Hello, world!'
      "%
    }],
  },
} | Tf.Config

Anything goes! You just need to ensure that your Terraform configuration ends up in the toplevel attribute config and that your entire configuration evaluates to a record satisfying the Tf.Config contract.

To actually turn the Nickel code into a JSON configuration file understood by Terraform, you need to call nickel to export the renderable_config toplevel attribute introduced by the Tf.Config contract:

nickel export <<<'(import "./<your-toplevel-file>.ncl").renderable_config'

This can be useful for inspecting the result of your code. But usually it will be easier to use the wrapper script for Terraform provided in the hello-tf flake template.

For inspiration on what's possible with Nickel, take a look at the examples. Happy hacking!

How?

Unfortunately, Terraform doesn't expose an interface for extracting a machine readable specification for the provider independent configuration it supports. Because of that this repository contains two tools and some glue written in Nix. Maybe this flowchart helps:

flowchart LR
    subgraph Nix
        direction TB
        providerSpec(Required Providers);
        providerSchemasNix(Terraform provider schemas);
        providerSpec -- generateJsonSchema --> providerSchemasNix;
    end
    
    subgraph schema-merge
        direction TB
        providerSchemasGo(Terraform provider schemas);
        merged-json-go(Merged JSON w/ Terraform builtins);
        providerSchemasGo --> merged-json-go;
    end

    subgraph tf-ncl
        direction TB
        merged-json-rust(Merged JSON w/ Terraform builtins);
        nickel-contracts(Monolithic Nickel contract)
        merged-json-rust --> nickel-contracts;
    end

    Nix --> schema-merge
    schema-merge --> tf-ncl

The entire process is packaged up in a Nix function generateSchema which is exposed as a flake output. Also, to generate a Nickel contract for a single provider, there is a flake output schemas:

nix build github:tweag/tf-ncl#schemas.aws

All providers available in nixpkgs are supported. The generateSchema function can also be called manually. For example, to get a monolithic Nickel schema for the aws, github and external Terraform providers, you could use

nix build --impure --expr \
  '(builtins.getFlake "github:tweag/tf-ncl).generateSchema.${builtins.currentSystem} (p: { inherit (p) aws github external; })'

Status

This project is in active development and breaking changes should be expected.

• Automatic contracts for Terraform provider schemas
• Contracts for Terraform state backends #14, #15
• More documentation #13
• Natural handling of field references #12

tf-ncl's Issues

Better introductory documentation

The documentation for first time readers is essentially nonexistant right now. Once the examples crystallize a common approach, we need to write up some tutorials and first use guides.

LSP can't autocomplete for provider "google"

Using the template provided with the TF provider github has autocomplete working.
But with TF provider google autocomplete does not work.

{
  inputs = {
    nixpkgs.url = "nixpkgs/nixos-unstable";
    nickel.url = "github:tweag/nickel";
    tf-ncl.url = "github:tweag/tf-ncl";
    tf-ncl.inputs.nickel.follows = "nickel";
    utils.url = "github:numtide/flake-utils";
  };

  nixConfig = {
    extra-substituters = [ "https://tweag-nickel.cachix.org" ];
    extra-trusted-public-keys = [ "tweag-nickel.cachix.org-1:GIthuiK4LRgnW64ALYEoioVUQBWs0jexyoYVeLDBwRA=" ];
  };

  outputs = inputs: inputs.utils.lib.eachDefaultSystem (system:
    {
      devShell = inputs.tf-ncl.lib.${system}.mkDevShell {
        providers = p: {
          inherit (p) github;
        };
      };
    });
}

{
  inputs = {
    nixpkgs.url = "nixpkgs/nixos-unstable";
    nickel.url = "github:tweag/nickel";
    tf-ncl.url = "github:tweag/tf-ncl";
    tf-ncl.inputs.nickel.follows = "nickel";
    utils.url = "github:numtide/flake-utils";
  };

  nixConfig = {
    extra-substituters = [ "https://tweag-nickel.cachix.org" ];
    extra-trusted-public-keys = [ "tweag-nickel.cachix.org-1:GIthuiK4LRgnW64ALYEoioVUQBWs0jexyoYVeLDBwRA=" ];
  };

  outputs = inputs: inputs.utils.lib.eachDefaultSystem (system:
    {
      devShell = inputs.tf-ncl.lib.${system}.mkDevShell {
        providers = p: {
          inherit (p) google;
        };
      };
    });
}

Note: don't forget to run link-schema in between

Handle nested_type attributes correctly

Terraform provider schemas can contain a nested_type field for an attribute instead of a type field:

// SchemaNestedAttributeType describes a nested attribute
// which could also be just expressed simply as cty.Object(...),
// cty.List(cty.Object(...)) etc. but this allows tracking additional
// metadata which can help interpreting or validating the data.
type SchemaNestedAttributeType struct {
    // A map of nested attributes
    Attributes map[string]*SchemaAttribute `json:"attributes,omitempty"`

    // The nesting mode for this attribute.
    NestingMode SchemaNestingMode `json:"nesting_mode,omitempty"`

    // The lower limit on number of items that can be declared
    // of this attribute type (not applicable to single nesting mode).
    MinItems uint64 `json:"min_items,omitempty"`

    // The upper limit on number of items that can be declared
    // of this attribute type (not applicable to single nesting mode).
    MaxItems uint64 `json:"max_items,omitempty"`
}

This is not currently handled by tf-ncl. This situation appears in the nixpkgs-provided Terraform providers, e.g. the gitlab provider in nixpkgs revision 677ed08a. The surface symptom in tf-ncl is the build failure of nix build .#schemas.x86_64-linux.gitlab with this nixpkgs revision.

Move the nix build infrastructure to crane

The current nix build expressions are very ad-hoc. In particular, lots of work is duplicated because cargo dependencies are not cached. Crane might fix that issue and be more ergonomic at the same time.

Use hashicorp/terraform-schema

Hashicorp is maintaining a schema generation library in Go for Terraform at https://github.com/hashicorp/terraform-schema. Using it would provide us with correct schemas without having to hardcode core Terraform options. An implementation using terraform-schema would automatically resolve #14 and #3 and should ease the implementation of #6.

README instructions don't work

Describe the bug
Follow README instructions, it fails at init.

To Reproduce

$ nix flake init -t github:tweag/tf-ncl#hello-tf
$ nix run .#terraform -- hello-tf.ncl init
error: flake '...' does not provide attribute 'apps.x86_64-linux.terraform', 'packages.x86_64-linux.terraform', 'legacyPackages.x86_64-linux.terraform' or 'terraform'

Expected behavior
Should init.

Don't shadow `terraform` in the development shell

The current flake template creates a devshell which wraps terraform with a custom script, accepting an additional argument which is the Nickel file. However, this script doesn't currently enable any command before we've passed a Nickel file, while many terraform invocation make sense without specifying any file (such as --help, init, etc.). The flake template should bind this script to a different name, so that the plain terraform command is still accessible, e.g. as terraform-ncl or terraform-nickel.

Properly handle computed fields

Computed fields in terraform come in two types. One is never settable by the user configuration and should be handled similarly to the id field hack we have right now. That is, an extra field with forced value should be set.

Others are settable by the user and should be handled similarly, but with a default value.

Tf-Ncl contracts are too monolithic

Schema contracts are huge, monotlithic records at the moment. For example, the AWS Nickel contract is a whopping 30M of code. This quickly becomes unmanageable and slow.

Ideally, these contracts should be built up out of modular pieces, say one generic contract for Terraform builtins and then one contract per provider resource. These should be importable on a piecemeal basis so that tf-ncl library authors can import just those resource contracts that they need.

Tf-Ncl 0.1 tracking issue

• Release tarball tooling to collect a snapshot of contracts
• Document inadequacies of computed field handling

Custom condition checks for data sources

Custom condition checks for Terraform data sources are not supported at the moment. This would need a reasonable encoding for Terraform expressions, similar to #3.

Use non-exported record fields

Computed attributes are the motivating example for tweag/nickel#880.

ncl: ram usage on completion in neovim

Describe the bug

When using nls in neovim (lunarvim) tabbing through completions continually causes ram usage to grow infinitely

To Reproduce
This is a minimal reproducer using tab to cycle through completions, it does the same if you are searching through completions growing infinitely larger.

d=ncl-ram-usage; mkdir $d && cd $d
nix flake init -t github:tweag/tf-ncl#hello-tf

in flake.nix:

use the latest nickel / ncl release

tf-ncl.inputs.nickel.follows = "nickel";

add github profider

nix flake update
git init # required for lvim to autostart lsp
git add flake.nix
echo -e "use flake\nlink-schema" > .envrc
direnv allow

Add syntax highlighting plugin for neovim:

lvim.plugins = {
  {
    "nickel-lang/vim-nickel",
  },
}

Setup nls in lunarvim (neovim) config.lua:

require('lspconfig')["nickel_ls"].setup {}

if using mason to autoconfigure lsp's disable it for nickel

vim.list_extend(lvim.lsp.automatic_configuration.skipped_servers, { "nickel_ls" })

lvim main.ncl

tf-ncl-nls-2024-03-21_18-34-00.webm

notice the difference in ram usage behaviour after I delete the content of the config

Expected behavior
ram usage should not be so high?

Environment

• NixOS 24.05.20240319.b06025f
• Version of the code: nickel-lang-cli nickel 1.5.0 (rev 35a5246)

Terraform backend configuration

Terraform supports different backend configurations: https://developer.hashicorp.com/terraform/language/settings/backends/configuration

These configuration options need to be reflected in the generated contract.

Subblock min-items and max-items should be expressed as a contract

The terraform schema format supports specifying min-items and max-items for subblocks in "list" nesting mode. These constraints should be expressed as a Nickel contract.

Bad interface of `run-terraform`

The default flake template creates a dev shell which wraps terraform with a custom script (cf #37). The CLI UX of this script is a bit limited: it doesn't allow to pass through commands or arguments to the underlying terraform invocation without first specifying a Nickel file (e.g. using -- init --foo --bar), doesn't support --help or --version and the default usage message is missing a basic description of the command.

Provisioner and for_each meta-arguments are missing

The provisioner and for_each meta-arguments supported by Terraform are not implemented in tf-ncl yet.

Provisioners seem to mostly require a lot of extra attributes. Conditions need a proper encoding.

for_each is easily emulated in pure Nickel for statically known iteratees. For dynamically generated collections, we again need a proper encoding.

Terraform wrapper

The flakes in tf-ncl-examples define wrapper scripts for terraform which run nickel export before running terraform. Something like this should go into this repository to be reused.

Similarly, a flake template would be nice.

The flake template should be thin wrapper to a helper call

Currently, initializing the flake template from this repo creates quite a bit of noise, including a wrapper bash script and stuff that is mostly irrelevant to the end user. Ideally, tf-ncl would expose a high-level customizable mkDevShell function or the like, and a basic flake would look like:

{
# ...
  devShell = tf-ncl.lib.mkDevShell {
    providers = mySelectedProviders;
    # + any relevant config/customization option
  }
#...
}

Using tf-ncl with Nomad

This is likely a silly question, but I'm trying to migrate to Hashicorp Nomad for workload orchestration at workplace, and since a great many of my workplace environment's configurations have been done in Nickel, this should be no exception. However I'm not super familiar with HCL, and it looks like there are differences between HCL written for Terraform and those for Nomad. Is there any guidance on using or adapting this project to configuring Nomad?