camratchford / easy-template

Quick and Dirty Jinja templating from the CLI with YAML

Welcome

@camratchford's profile

About Me

• Loves Pizza
• Writes code occasionally

Interests

• Python
• Frontend
• Infrastructre as Code
• PowerShell
• Informing the public about propper clubhouse sandwich composition

easy-template's Issues

Add better file encoding handling for incoming templates.

Since windows task scheduler likes to output UTF-16F for it's XML outputs, and Python defaults to UTF-8, we should check which encoding method the template file uses before attempting to read the file.

• Check the template file
• Add configuration option to have the use select whether they want:
  • The program to output the same encoding as it received
  • The program to convert everything to UTF-8 (default)
• Change the encoding type of the template file we're reading.
• Conditionally change the encoding type of the output file to that which was read.

Add SQL query plugin - available as an alternative install

Config option 'sql_server_type`:

• MSSS
• PGSQL
• SQLite3
• MySQL

Then config options for authentication:

• password
• username
• server
• database
• port

It will populate the globals of the Jinja environment with a list of dicts (one for each row where the key is the column name and the value is the row value). The name of the variable is provided with the SQL query.

Write documentation: Compiling

• Prerequsites
• Instructions
• Troubleshooting

Add Jinja 'blocks' compatibility

• allow user to provide supporting templates
• user selects a directory rather than a single template
• Jinja assembles the output by inserting blocks in specified block locations if the 'include' directive is provided.

Add parameter validation to cli

The cli commands that accept path variables should be using the built-in click.Path type (which adds validation).

Add tests and test

This whole project was published with very little testing.

• Add some unit tests.
• Test multiple platforms (It should work in Windows, MacOS, and Linux without much trouble).
• Add that section to the docs.

Write documentation: Configuration options

• Which variables are required

• What the required variables do

• What jinja options should not be passed through, or wont have any effect

• What logging options are recommended, provide additional examples for formatting, stack trace, different handlers.

• How global variables work, when to use them.

Write documentation: CLI Syntax

• Config-file, what that means
  • What the program does with the folders defined
• var-file, what that means and how the program uses it
• Template-file, how it's used, how many can be used, how the program uses it.

Have variable object ingest environment variables and Python module 'sys' properties

Dynamic variables, like:

• UUID, random B64 string, etc. generation
• Specific API calls (public IP address, random name generators, translators)
  • Should be baked into the code and sanitized to avoid abuse. (Don't just let a URI in the var file to dictate which endpoint is receiving a get request)

Collect the following:

• sys.winver
• sys.platform
• os.sys.getfilesystemencoding()
• os.environ

Add them to a dictionary that is automatically provided in vars:

• Name it something like "local_vars" and "dynamic_vars" sound good.

Remap CLI so it can be used as a C pre-processor

I don't like gnu-cpp, the macros are ugly.

Add ability for user to specify path of outputs

CLI Example

# Where the values provided to the -o argument is the absolute path of the file ezt will output
ext -c ./config.yml -v vars.yml -o /home/cam/projects/new_project/config.py config.py.j2

# With no -o argument provided, the output file will land in the output directory specified in ./config.yml
ext -c ./config.yml -v vars.yml config.py.j2

Write documentation: Advanced usage guide

• Variable precedence
• Jinja2 template language
• From inside another program / script
  • In PowerShell

Add some CI

[ ] Formatting / Linting
[ ] Test runners
[ ] Linux compiling tests with make
[ ] Windows compiling tests with pester (for powershell)

Use arbitrary-imports library for importing external jinja functions

Make directories specified in the config file override-able

• There are 4 directories specified in the config file

  • templates
  • vars
  • outputs
  • logs

• The user should have the option to override the path of templates, vars, and outputs with cli arugments

Proposed args to add:

CLI Argument	Description
--template-dir	Absolute path to the template directory, where relative paths provided in the template argument are relative to
--vars-dir	Absolute path to the vars directory, where relative paths provided in the vars argument are relative to
--output-dir	Absolute path to the outputdirectory, where output files will land if the output path argument is not specified

Add ability for ezt to recurse through a directory tree

• Iterate through filesystem tree
• Identify templates
• Apply templating to templates
• Replicate tree structure in specified output directory

ezt --tree ~/project/input_dir -o ~/project/output_dir --var-file ~/project/vars.yml

Redesign interface with the goal of having ezt be used in shell scripts that generate batches of templated files

For example:

• We want to make a shell script that creates 3 ansible plays based on a jinja template.
• We provide it a variable file, and an inline variable that is the loop variable $i.
• We use the loop variable to name the output file and as select logic from within the variable file.

Shell Script

for i in staging-1 dns-1 dns-2; do
  ezt --var-file=vars.yml -v "host=${i}" -t build_container.yml.j2 -o "plays/build_${i}.yml"
done

vars.yml

hosts:
  staging-1:
    container_host: lxdhost-1
  dns-1:
    container_host: lxdhost-1
  dns-2: 
    container_host: lxdhost-2

build_container.yml.j2

---
- name: Build  {{ host }} container
  hosts: {{ hosts[host].container_host }}
  var_files: /opt/monorepo/ansible/inventories/host_vars/{{ host }}.yml
  tasks:
    - name: Provision LXD container
      include_role:
        name: lxd-lifecycle
        tasks_from: build.yml

Add logging config to documentation

The config object already has the ability to configure logging using logging.dictConfig, it's just missing from the example config.yml file.

Fix built binaries to write to stdout

Untested in Linux, but in Windows the compiled binaries do not have any output on the terminal. Logging (the file handler at least) and error handling otherwise work as expected.

Add AD or LDAP queries to dynamic variables

Query AD/LDAP objects as dynamic variables

• Ether AD or LDAP, not both. Whichever is the least complicated for the user configuration.

  • AD would require a domain controller and credentials (as secure string)
  • LDAP would require LDAP query path, credentials, and other attributes such as LDAP server, port, encryption, etc.

• Can be used to collect User object data such as email address, phone number, group membership, etc.

• Can be used to collect Computer object data as well.