heavyhorst / remco

remco is a lightweight configuration management tool

Remco

remco is a lightweight configuration management tool. It's highly influenced by confd. Remcos main purposes are (like confd's):

• keeping local configuration files up-to-date using data stored in a key/value store like etcd or consul and processing template resources.
• reloading applications to pick up new config file changes

Differences between remco and confd

• Multiple source/destination pairs per template resource - useful for programs that need more than one config file
• Multiple backends per template resource - get normal config values from etcd and secrets from vault
• Pongo2 template engine instead of go's text/template
• Zombie reaping support (if remco runs as pid 1)
• Additional backends can be provided as plugins.
• Create your own custom template filters easily with JavaScript.
• Exec mode similar to consul-template.

Overview

Documentation

See: https://heavyhorst.github.io/remco/

Installation

Building from source

$ go get github.com/HeavyHorst/remco/cmd/remco
$ go install github.com/HeavyHorst/remco/cmd/remco

You should now have remco in your $GOPATH/bin directory

Building from the repository

$ git clone https://github.com/HeavyHorst/remco
$ cd remco
$ make
$ ls bin/
remco

Building a given release

$ export VERSION=v0.12.2
$ git checkout ${VERSION}
$ make release -j
$ ls bin/
remco_0.12.2_darwin_amd64.zip  remco_0.12.2_linux_amd64.zip  remco_0.12.2_windows_amd64.zip  remco_darwin  remco_linux  remco_windows

Using a pre-built release

Download the releases and extract the binary.

$ REMCO_VER=0.12.2
$ wget https://github.com/HeavyHorst/remco/releases/download/v${REMCO_VER}/remco_${REMCO_VER}_linux_amd64.zip
$ unzip remco_${REMCO_VER}_linux_amd64.zip

Optionally move the binary to your PATH

$ mv remco_linux /usr/local/bin/remco

Now you can run the remco command!

Execution

run remco from local dir, configuration is read as default from /etc/remco/config

Command line params:

parameter	description
-config	path to the configuration file
-onetime	flag to one run templating once, overriding "Onetime" flag for all backend resources
-version	print version and exit

Contributing

See Contributing for details on submitting patches.

remco's Issues

Add oneshot option

I want to run remco with oneshot option.
The case is to build shard configuration for terraform on my ci and i only need it one time.
Just build config and exit.
As far as i can see we already have an that option for backend configuration but not for template resource.

Strange metrics from prometheus sink

Hi
I got some time to check recently added prometheus sik support and got pretty strange metrics

that is all of them

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 1.97e-05
go_gc_duration_seconds{quantile="0.25"} 2.233e-05
go_gc_duration_seconds{quantile="0.5"} 3.3328e-05
go_gc_duration_seconds{quantile="0.75"} 4.0256e-05
go_gc_duration_seconds{quantile="1"} 0.000138654
go_gc_duration_seconds_sum 0.000700897
go_gc_duration_seconds_count 18
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 27
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.15.2"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 2.670056e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 4.974308e+07
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.469807e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 818932
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 0.0015231990967114032
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 4.900856e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 2.670056e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 6.2799872e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 3.784704e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 24874
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 6.1947904e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.6584576e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.6103504592878752e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 843806
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 1736
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 58344
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 131072
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 4.194304e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 380569
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 524288
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 524288
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.4007552e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 7
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.31
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1024
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 12
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 2.5690112e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.61035043098e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 7.42998016e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes -1
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 1
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0
# HELP remco_backends_synced_total remco_backends_synced_total
# TYPE remco_backends_synced_total counter
remco_backends_synced_total{name="consul"} 4
remco_backends_synced_total{name="file"} 2
# HELP remco_files_staged_total remco_files_staged_total
# TYPE remco_files_staged_total counter
remco_files_staged_total 12
# HELP remco_files_synced_total remco_files_synced_total
# TYPE remco_files_synced_total counter
remco_files_synced_total 12
# HELP remco_files_template_execution_duration remco_files_template_execution_duration
# TYPE remco_files_template_execution_duration summary
remco_files_template_execution_duration{quantile="0.5"} NaN
remco_files_template_execution_duration{quantile="0.9"} NaN
remco_files_template_execution_duration{quantile="0.99"} NaN
remco_files_template_execution_duration_sum 384.0843696594238
remco_files_template_execution_duration_count 12
# HELP remco_HOSTNAME_runtime_alloc_bytes remco_HOSTNAME_runtime_alloc_bytes
# TYPE remco_HOSTNAME_runtime_alloc_bytes gauge
remco_HOSTNAME_runtime_alloc_bytes 2.609776e+06
# HELP remco_HOSTNAME_runtime_free_count remco_HOSTNAME_runtime_free_count
# TYPE remco_HOSTNAME_runtime_free_count gauge
remco_HOSTNAME_runtime_free_count 818930
# HELP remco_HOSTNAME_runtime_heap_objects remco_HOSTNAME_runtime_heap_objects
# TYPE remco_HOSTNAME_runtime_heap_objects gauge
remco_HOSTNAME_runtime_heap_objects 24723
# HELP remco_HOSTNAME_runtime_malloc_count remco_HOSTNAME_runtime_malloc_count
# TYPE remco_HOSTNAME_runtime_malloc_count gauge
remco_HOSTNAME_runtime_malloc_count 843653
# HELP remco_HOSTNAME_runtime_num_goroutines remco_HOSTNAME_runtime_num_goroutines
# TYPE remco_HOSTNAME_runtime_num_goroutines gauge
remco_HOSTNAME_runtime_num_goroutines 21
# HELP remco_HOSTNAME_runtime_sys_bytes remco_HOSTNAME_runtime_sys_bytes
# TYPE remco_HOSTNAME_runtime_sys_bytes gauge
remco_HOSTNAME_runtime_sys_bytes 7.4007552e+07
# HELP remco_HOSTNAME_runtime_total_gc_pause_ns remco_HOSTNAME_runtime_total_gc_pause_ns
# TYPE remco_HOSTNAME_runtime_total_gc_pause_ns gauge
remco_HOSTNAME_runtime_total_gc_pause_ns 700897
# HELP remco_HOSTNAME_runtime_total_gc_runs remco_HOSTNAME_runtime_total_gc_runs
# TYPE remco_HOSTNAME_runtime_total_gc_runs gauge
remco_HOSTNAME_runtime_total_gc_runs 18
# HELP remco_runtime_gc_pause_ns remco_runtime_gc_pause_ns
# TYPE remco_runtime_gc_pause_ns summary
remco_runtime_gc_pause_ns{quantile="0.5"} NaN
remco_runtime_gc_pause_ns{quantile="0.9"} NaN
remco_runtime_gc_pause_ns{quantile="0.99"} NaN
remco_runtime_gc_pause_ns_sum 700897
remco_runtime_gc_pause_ns_count 18

1. the interesting one is one with HOSTNAME. why do we have HOSTNAME embeded in metric name here ?
2. why do we have such strange remco_runtime_gc_pause_ns for quantiles ?
3. do it differs from go_gc_duration_seconds ?

probably i should mention @syndicut

Strange slow startup

Loving this tool so far but all of the sudden it takes 3 minutes for it to spin up

time="2019-05-08T22:37:08-04:00" level=info msg="Captured interrupt. Exiting..." prefix="remco[20045]"
time="2019-05-08T22:37:08-04:00" level=debug msg="closing client connection" backend=consul prefix="remco[20045]" resource=env
time="2019-05-08T22:37:08-04:00" level=debug msg="closing client connection" backend=vault prefix="remco[20045]" resource=env
time="2019-05-08T22:37:08-04:00" level=debug msg="closing client connection" backend=consul prefix="remco[20045]" resource=env
time="2019-05-08T22:37:08-04:00" level=debug msg="closing client connection" backend=consul prefix="remco[20045]" resource=env
time="2019-05-08T22:37:08-04:00" level=debug msg="closing client connection" backend=vault prefix="remco[20045]" resource=env
time="2019-05-08T22:37:08-04:00" level=debug msg="closing client connection" backend=consul prefix="remco[20045]" resource=env
time="2019-05-08T22:37:08-04:00" level=debug msg="closing client connection" backend=vault prefix="remco[20045]" resource=env
time="2019-05-08T22:37:08-04:00" level=debug msg="closing client connection" backend=vault prefix="remco[20045]" resource=env
time="2019-05-08T22:37:08-04:00" level=debug msg="removing pid file at "/var/run/remco/remco.pid"" prefix="remco[20045]"
^killed  and restarted
time="2019-05-08T22:40:02-04:00" level=error msg="not all templates could be rendered, trying again after 0 seconds" prefix="remco[20324]" resource=env
^3 full minutes before next log message?!

Typing once again

I am doing templating for service that want to get int as input
in consul key i have an int but after {{ config | toYaml }} value became string.
service don't accept string as value :(

Config Generation broken in v0.12

Hello,

I just tried updating to v0.12, but the templates dont render anymore. I was using v0.11.1 before and it still works fine. It throws the following error:

[0007] ERROR ./remco_1.12_linux[40356]: createStageFileAndSync failed: create stage file failed: template execution failed: [Error (where: execution) in templates.tpl | Line 3 Col 4 near 'getv'] key does not exist: /rewrites/0/source resource=redirects
[0007] ERROR ./remco_1.12_linux[40356]: not all templates could be rendered, trying again after 11 seconds resource=redirects

I attached an example below:

template.tpl

{% if lsdir("/rewrites") %}
{% for entry in lsdir("/rewrites") | sortByNumber %}
{{ getv(printf("/rewrites/%s/source", entry)) }}
{% endfor %}
{% endif %}

data.json

{
  "rewrites": [
    {
      "12955": {
        "source": "/(.*)"
      }
    },
    {
      "12956": {
        "source": "/(.*)"
      }
    }
  ]
}

Add replace filter

There is no replace filter in Django template and it actually can be very useful for some basic string manipulation. Due inability to pass multiple variables to Django filer (as I can see from my limited experience) it's tricky to implement flexible port for JavaScript str.replace() function.

confd implement replace function by simply aliasing Go's strings.Replace function.

My not really good example with a basic filter which only replace one character.
JS filer

var pattern = Param.slice(0,1);
var replacement = Param.slice(1,2);
In.replace(pattern, replacement);

Sample data and usage

v = [email protected]
v|replace_char:'@/'|base:'-1'

result: domain.tld

include_dir does not work properly

Hi,

I'm testing remco (built from master) and got a problem with include_dir
It's seems not works properly. Program see a resource file but do nothing. =(

Here is my configs in case I'm missing something.

[root@hostname remco]# /home/alexandrov/remco_linux -config config.toml
[0000]  INFO /home/alexandrov/remco_linux[46543]: loading resource configuration path=/etc/remco/resource.d/test.toml
[0000]  INFO /home/alexandrov/remco_linux[46543]: [creating pid file at "/var/run/remco/remco.pid"]
[0000] DEBUG /home/alexandrov/remco_linux[46543]: [removing pid file at "/var/run/remco/remco.pid"]

config.toml

log_level   = "debug"
log_format  = "text"

include_dir = "/etc/remco/resource.d/"
pid_file    = "/var/run/remco/remco.pid"

[default_backends]
[default_backends.env]
    interval = 10
    keys     = ["/"]

resource.d/test.toml

[[resource]]
  name = 'test'
[[resource.template]]
  src        = "/etc/remco/test.tmpl"
  dst        = "/etc/test_remco"

If I add configuration from test.toml to config.toml everything works fine.

Could you please help to make it works?

[Feature request] Fetch remco.toml and src template files from URL and be watchable

Hi,

Thanks for your tool that seems interesting and promising.

Not sure if it is relevant, but i think it will be a great feature to have the possibility to fetch the template file from an public/private URL and be watchable with interval the same manner as values from backend.
I thinking it will be a perfect feature for remco.toml too (with autoreload and delete config files that are not here anymore)

I ask this because i'm looking for a solution to be able to distribute containers with defaults templates in image but to be able to fetch remco.toml/resources templates from URL/GIT.

What do you think ?

Regards,

use only 1 key to watch

if i set keys = [ "/" ] and watch = true, remco watches all keys and generates config from keys = ["/"] keys. what if i want to use all keys and only generate config on updating /nginx/reload ?

[Question] Are multiple file backends possible? Can i add an additional file-backend for specific resource?

Hi,

i'd like to have a default (file-)backend for all my resources. For a specific resource i'd like to add another, additional (file-)backend with keys specific for this resource. Is this possible? For now, the backend-definition in the resource seems to overwrite the default-backend(s)?

So in my remco.toml i've got this:

[default_backends]
    [default_backends.file]
        watch = true
        interval = 60
        keys = [ "/" ]
        filepath = "default-config.yml"

And in one of my resource-definitions (separate file) i got this:

[backend]
    [backend.file]
        onetime = true
        keys = [ "/" ]
        prefix = "/my-service/"
        filepath = "specific-resource-config.yml"

In the templates for this resource i'd like to be able to access the keys of the default-config and the keys for this specific resource, ideally without having to re-declare the default-file-backend in the resource.

Kind regards!

[Feature request] Support kubernetes authentication in vault backend

Hi,

Thanks for an awesome tool!

It looks exactly as a tool I need(after working with confd), but it lacks k8s support for vault:
confd supports kubernetes authentication for vault backend.

It would be nice to have it in remco as well.

Accesing dictionary option not working as expected

hi, i have such a template

{% set shard = printf("shard_%s", getenv("SHARD", 1)) %}
{% set config = getv("/shard_config") | parseYAML %}
{{ config.shard_1.machine }}
{% set shard_config = config.shard %}
{{ shard_config }}

And such key in consul

shard_1:
  machine: n1-standard-1
shard_2:
  machine: n1-standard-1

the problem is in last two lines of template, i'm expecting that will get data from {{ config.shard_1.machine }} but got empty string instead.

Support making intermediary directories for template dest

When templating to a tmpfs-like filesystem, it's likely that necessary intervening directories may not exist at templating time.

Resources should support a "make_directories" boolean that calls os.makedirs() to make non-existing directories. For consistency, this should probably also force existing directories to obey user/group ownerships.

Use case: daemons which insist on watching a directory rather then a specific file when using remco in persistent mode.

Add support for indent for toYaml

hi!
Please add support to setting starting indent for toYaml.
like this one https://docs.ansible.com/ansible/latest/user_guide/playbooks_filters.html#filters-for-formatting-data

i have to add something like
{{ config | toYaml(indent=2) }}

Ability to exec processes in background mode

Hi,

I'm looking on remco as a configuration manage tool inside containers for our cloud infrastructure.
We are using system OS containers with several applications inside and systemd is used as an init system.

remco is starting as a daemon before any services for which configs should be generated.
But here is a problem - there is not option to detect if remco's resources have been successfully proceed and services could be start.

To resolve this there is exec option (command) but it require foreground mode which is not suitable for systemd services. =(
Also I do not like an idea to run services in foreground mode because they will be killed if remco failed and died. It's ok if remco is used as a supervisor or init system (proc 1) but decrease reliability if it works as configuration management tool only.

I suggest to add additional option and allow to exec processes in background mode.

Template does not reload after etcd key change

config

################################################################

Global configuration

################################################################

log_level = "debug"

log_format = "text"

[[resource]]

name = "haproxy"

[[resource.template]]

src = "/etc/remco/templates/haproxy.tmpl"

dst = "/etc/haproxy/haproxy.cfg"

reload_cmd = "haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf cat /var/run/haproxy.pid"

[resource.backend]

[resource.backend.etcd]

  nodes = ["${ETCD_NODE}"]

  keys = ["/services"]

  watch = true

  interval = 60

  version = 2

dockerfile

FROM haproxy:2.1.7-alpine

ENV REMCO_VER 0.11.1

RUN apk update && apk add ca-certificates wget && update-ca-certificates && apk add openssl

RUN wget --no-check-certificate https://github.com/haproxytech/dataplaneapi/releases/download/v2.0.3/dataplaneapi_2.0.3_Linux_x86_64.tar.gz

RUN mkdir api

RUN tar -C ./api -xzf dataplaneapi_2.0.3_Linux_x86_64.tar.gz

RUN chmod +x ./api/build/dataplaneapi

RUN cp ./api/build/dataplaneapi ./bin

RUN apk --update add --no-cache haproxy bash ca-certificates

RUN wget --no-check-certificate https://github.com/HeavyHorst/remco/releases/download/v${REMCO_VER}/remco_${REMCO_VER}_linux_amd64.zip && \

unzip remco_${REMCO_VER}_linux_amd64.zip && rm remco_${REMCO_VER}_linux_amd64.zip && \

mv remco_linux /bin/remco

RUN ls -ltr ./bin

COPY config /etc/remco/config

COPY haproxy.tmpl /etc/remco/templates/haproxy.tmpl

EXPOSE 8080 9999 443

ENTRYPOINT ["remco"]

everything seems to work well on initial startup. it reads the keys in etcd and create the template correctly , but if a change occurs with the keys it never recreates the template.

please help

Project Status

I apologize in advance if this isn't the best place to be asking this.

The Project Status in README.md states that remco is under active development, and not recommended for production use . However, there has not been a commit since Nov 1st, 2017. Is remco still under active development? Are the lack of recent commit an indicator that remco has reach maturity?

Render yaml config with toYAML

Hi.
I want to render template like config | toYAML but i have to create config in runtime
That is a set that should be compiled from several for and lots of if.
How should i do it with pongo2 and remco ?

As i see there are no simple way how to create empty set and append values to it.
Any ideas ?

Exclude/Raw

Hello,

Just came across this after using confd for a while, potentially silly question coming up...

• How do i exclude sections of a .tmpl file from being interpreted?

I've a config file that has a section of golang/template code that's interpreted by the program using the file. I wish to escape this with some kind of raw/escape sequence. Struggling to find out how through the docs...?

Set ordering

Hi. in #42, you have added helpers to make dicts and lists in runtime.
Recently I get some time to move to them instead of for loop.
But I probably got minor in my use-case but a fundamental problem in some other cases.

my template code is

{% set rules_set = createSet() %}
{%- for project in shard_config.projects -%}
  {%- if project.enabled_rules.team != "false" -%}
      {% set rule = printf("%s/teams/%s/%s", rules_prefix, project.team, rules_suffix )  %}
      {{ rules_set.Append(rule) }}
  {% endif %}
{%- endfor %}
{%- set rulesmap = createMap() %}
{{ rulesmap.Set("rule_files", rulesmap) }}
{{ rulesmap  | toYAML }}

Each time I render a file, I got a different ordering of lines.
I tied to switch to for with sorted according to https://github.com/flosch/pongo2/blob/master/template_tests/for.tpl

rule_files:
{% for line in rulesmap sorted %}
- {{ line }}
{{%endfor %}}

but this produces no lines at all. without sorted too :(

Minimal example for include_dir?

Hey guys,

could you please provide a full (working) minimal example for the include_dir config property? Below is my test setup but it does not work. As a result I expect a destination file (/tmp/test) simply containing the word FOO (without resolving any keys at the moment, just for testing include_dir).

root@0abf41c53fce:/# tree /etc/remco
/etc/remco
|-- config
|-- resource.d
|   `-- test
`-- templates
    `-- test.tmpl

2 directories, 3 files
root@0abf41c53fce:/# cat /etc/remco/config
log_level   = "debug"
log_format  = "text"

pid_file    = "/var/run/remco.pid"
include_dir = "/etc/remco/resource.d/"

[default_backends]
[default_backends.env]
  onetime  = true
  keys     = [""]

root@0abf41c53fce:/# cat /etc/remco/resource.d/test
name = "test"

[[template]]
  src   = "/etc/remco/templates/test.tmpl"
  dst   = "/tmp/test"

root@0abf41c53fce:/# cat /etc/remco/templates/test.tmpl
FOO
root@0abf41c53fce:/# remco -version
remco Version: 0.12.1
UTC Build Time: 2020-10-19-20:38:57
Git Commit Hash: 8d36e4f96f2cf49d9a7953079de025089a5ad2ab+CHANGES
Go Version: go1.15.2
Go OS/Arch: linux/amd64
root@0abf41c53fce:/# remco
[0000]  INFO /usr/local/bin/remco[1008]: creating pid file at "/var/run/remco.pid"
[0000] DEBUG /usr/local/bin/remco[1008]: removing pid file at "/var/run/remco.pid"

Medium security issues in dependencies

Hello,
different scanner pick up warnings in remco with two different libraries used. As i do not understand go and its dependency system it is hard to resolve for me. Especially i do not know how to get a real dependency tree to see what needs fixing...

Snyk Security scanner (much more helpful as it shows dependencies):

✗ Medium severity vulnerability found in gopkg.in/yaml.v2
  Description: Denial of Service (DoS)
  Info: https://snyk.io/vuln/SNYK-GOLANG-GOPKGINYAMLV2-1083943
  Introduced through: github.com/HeavyHorst/easykv/[email protected], github.com/ghodss/[email protected]
  From: github.com/HeavyHorst/easykv/[email protected] > gopkg.in/[email protected]
  From: github.com/ghodss/[email protected] > gopkg.in/[email protected]
  Fixed in: 2.2.8

At least for your own package easykv an update is available in Github but that is not released so far (1.2.5 is latest). Therefor creating a new release 1.2.6 and using it would help partially?

Anchore Security scanner (not that helpful):

+ grype -v -c config.yaml -o table --fail-on high *.telekomcloud.com/project/image:v0.1.0-unscanned
[0000]  INFO grype version: 0.32.0
[0001]  INFO identified distro: Alpine Linux v3.14 from-lib=syft
[0001]  INFO cataloging image from-lib=syft
[0001]  INFO ignoring 2 matches due to user-provided ignore rules
NAME                 INSTALLED                           FIXED-IN                           VULNERABILITY        SEVERITY 
golang.org/x/crypto  v0.0.0-20190701094942-4def268fd1a4  0.0.0-20200220183623-bac4c82f6975  GHSA-ffhg-7mh4-33c4  Medium    
gopkg.in/yaml.v2     v2.2.5                              2.2.8                              GHSA-wxc4-f4m6-wwqv  Medium    

yaml.v2 is the same as above, but new golang.org/x/crypto uses way to old versions in go.sum. But here i do not know what dependency needs an update...

Side note:
your Makefile help mentions "make get-deps" to rebuild vendor directory but it seems the correct command is "make vendor". Right?

Just trying to understand go...

I've gotten most of the setup working with vault but the connection doesn't auto renew the token

Unless I'm doing something wrong, I assuming that if we use app role as the auth type, that prior to the ttl, the token will auto renew, but this isn't happening right now

Add Go template support

Hello thanks for interesting alternative for confd. I think ability to have multiple backends is the 'killer feature'. Currently I need to have multiple Consul backends and in the future Vault will be added to the list but I would not want to be 'vendor locked' by using consul-template.

So I'm trying to switch from confd and during review of remco as candidate my main concern was different template engine. And after actually trying to adapt my templates from Go template to Django temple I can say that it takes a lot of effort. It took less than 2 hours to create new installation and configuration SaltStack formula for remco, but it took more than 4 hours to adapt simple Haproxy template from Go template to Django template.

Main problems with Django template:

• no variables, {% with %} can help, but because it is require end tag it's not very elegant
• very limited amount of filters, there is no basic replace filter like in Jinja2, nor index from Go template, of course there is workaround available out-of-the box with this JavaScript custom filters, but for such basic things you can expect built-in tools
• JS filters feature add a lot of flexibility but it's not portable, you can't share your template without your custom filters, not a problem at first glance but limits widespread of the tool
• JS filters does not throw any errors if they written incorrectly they just simply does not work. I created a deliberately incorrect code and did not receive any error.

And main problem, you can't just switch from confd with some fixes here and there, you need to fully rewrite your templates. I think if Go template became available more people will happily switch to remco because it's flexibility is really impressive.

I understand that adding another template engine is not a piece of cake and probably overly complicate this tool and even more likely, it’s completely not required for you personally. So I won't really expect that Go template will be added it's just my thoughts and experience of using remco.

Allow "onetime" to be forced from command line or allow only specific backends (or both)

The use case for this is having initial/early onetime configuration via environment vars when a container starts (potentially via a containers entrypoint script), with ongoing config updates via etcd/consul etc with remco running as a daemon.

This is currently possible to achieve but it requires having one config file and set of resources for the "oneshot" part and another config for the ongoing part.

It would be great if we could have something similar to the following:

remco -onetime -backends env

The idea being that -backends is a comma separated list of backends to run/allow, with others being ignored, while -onetime would force "onetime" as "true" regardless of the actual resource config.

To be honest the -backends option would fulfil my needs as I would be having my "env" resources with "onetime = true" anyway.

way to run in nodeamon mode

how can i run it with supervisord?

Boolean type

In #33 we got support for aditional types other that strings.
Now i have such a config

...
    enabled_rules:
      team: False
      product: No

and if

  {% if enabled_rules.team | yesno %}
  - {{ enabled_rules.team }}/*.yml
  {% endif %}

And that if is always true. That is frustrating.
diagnosis with

  {{ enabled_rules | toJSON }}
  {{ enabled_rules.team | yesno }}

leave me with

  {"product":"false","team":"false"}
  yes

% remco --version 
remco Version: 0.12.0-beta.2
UTC Build Time: 2019-11-05-15:43:39
Git Commit Hash: 85fa5258e0cc5ad50ab957ea2ea9bcada48b7125+CHANGES
Go Version: go1.13.4
Go OS/Arch: linux/amd64

[backend/file] Ability to specify headers for remote files

The use case is the remote config I'd like to poll isn't publicly accessible and requires authentication.

I'd create this issue for easykv, but I imagine it'll require changes to remco too as we'd need to change the file backend to accept a new setting and pass it along to easykv.

Lemme know if you'd be interested!

remco on multiple servers?

Hello,

I am trying to get a feeling of how a Remco setup could look in production and some help would be gratefully appreciated:

In a production setting would I need install Remco on all my servers where I have .env files?
....or there could be a way Remco can read&write dst config files on remote servers?

Thanks,
Valentin

Support for Runtime Configurator

Hi. it seems that Runtime Configurator from gcp looks like a great source of configs
https://cloud.google.com/deployment-manager/runtime-configurator

can you please add integration with it ?

Macro errors written to template

I'm seeing output like this:

rundeck {
            [Error (where: execution) in /home/greg/scratch/docker-remco/remco/templates/jaas-loginmodule.conf | Line 7 Col 21 near 'getv'] key does not exist: /rundeck/jaas/ldap/providerurl

};

Probably related to this long-standing issue:
flosch/pongo2#54

Support for render template with folder or patterns?

I have bunch of files need to render in the folder, and every time add a file, i have to add the template config too.

Add a filter that returns an array element by index

In Django template It's tricky to access specific array element by index if array is obtained as the result of applying a filter, because it's impossible to directly use index after filter.
I.e. I want only third part of the path

data = /first/second/third/fourth
data|split:'/'|slice:'2:3'|join:''

not very clear and intuitive, slice with single element still returns an array.

Better with custom JS filter, but not very portable, you need to supply your custom filter with your template:

In[Param];

data|split:'/'|index:'2'

In Jinja2 element can be easily accessed by index or slice can be taken without additional filters:

data|split('/')[2]

Go template has an index function.

[feature-request] Reference rendered configuration file in "reload_cmd"

Similar to how we can reference the staged rendered source template in the check_cmd with {{.src}}, it would be convenient to be able to reference the final rendered configuration in the reload_cmd.

For example,

[[resource.template]]
  src = "/etc/remco/templates/template.cfg"
  dst = "/tmp/rendered-template.cfg"
  check_cmd = "echo {{.src}}"
  reload_cmd = "echo {{.dst}"

global BackendConfigs

Could it be possible to have a shared set of default configs for all of the backends, so they wouldn't have to be redeclared for each Resource?

This would allow a setup more like confd. but could be overridden on a per-resource basis

example config.toml:

log_level   = "debug"
log_format  = "json"
include_dir = "/etc/remco/resource.d/"
pid_file    = "/var/run/remco/remco.pid"
log_file    = "" # stdout

[backends.file]
filepath    = "/defaults.yml"
watch       = true
keys        = "/"
# any access to / will use this yml, if not overidden

[[resource]]
  name = "haproxy"
  [[resource.template]]
    src         = "/etc/remco/templates/haproxy.cfg"
    dst         = "/etc/haproxy/haproxy.cfg"
    check_cmd   = "somecommand"
    reload_cmd  = "somecommand"
    mode        = "0644"
    # will be using defaults.yml

   [[resource.template]]
    src         = "/etc/remco/templates/haproxy.cfg"
    dst         = "/etc/haproxy/haproxy-dev.cfg"
    check_cmd   = "somecommand"
    reload_cmd  = "somecommand"
    mode        = "0644"
    [resource.backend]
      [resource.backend.file]
          filepath = "/dev.yml"
          watch = true
          keys = "/"

The key don't start with "/" can not find by remco

config

log_format = "text"

[[resource]]
  name = "disconf"

[[resource.template]]
  src = "/root/remco/templates/t.tmpl"
  dst = "/root/remco/t.yaml"

[resource.backend]
  [resource.backend.etcd]
    nodes = ["x.x.x.x:2379"]
    keys = ["disconf","registry"]
    watch = true
    interval = 60
    username = "xxxxx"
    password = "xxxxx"
    version = 3

t.tmpl

a
{% for i in gets("disconf/*") %}
    key: {{i.Key}}
    value: {{i.Value}}
{% endfor %}
b
{% for i in ls("disconf") %}
   value: {{i}}
{% endfor %}
c

t.yaml

a
b
c

Pongo filters plugin system

Its nice to have backends pluggable, but i wouldlike if the filters also where pluggable.
This is supported in django, and would also like it. This way i don't have to worry about creating a filter that is 10 rows long. but import my custom filter and than do the work in there.

If i look at https://github.com/flosch/pongo2-addons, this will install a package and then I have to manually add a line to import the package. I want the application to this autocratically.

This way I can group multiple filters by functions or use.

Here is an example through i came up with the idea.
Through the consul and env backend plugin i get the following keys:

/database/username
/database/port

What i want is to translate the keys to cfg file, with the followint format:
`DATABASE_USERNAME = ''

I almost got it with the current filters in remco, but i can't find a way to remove the first '_'.

{% for kv in getallkvs() %}
{{ kv.Key|split:"/"|join:"_"|upper }} = '{{ kv.Value }}'{% endfor %}

And what i want is to do the transformation in the plugin, with my own filter.
Than it looks like this:

{% for kv in getallkvs() %}
{{ kv.Key|toEnvVar }} = '{{ kv.Value }}'{% endfor %}

Vault templating

Hello,
In vault (unsealed and accessible via HTTP API using curl).

In my vault in /secret/n2 is stored key=value.
Via the HTTP API I can access it (using the hashicorp doc) using http://127.0.0.1:8200/v1/secret/data/n2.

I saw on the readme array that vault don't support prefix so I did not set prefix.

I didn't success with my remco resource.

[[resource]]
  name = "Vault"
  [[resource.template]]
    src = "/etc/remco/templates/key"
    dst = "/<path>/key"
    make_directories = true
    mode = "0644"
  [resource.backend]
    [resource.backend.vault]
      node = "http://172.0.0.1:8200"
      auth_type = "token" 
      auth_token = "<my_token>"
      watch = true
      keys = ["/v1/secret/data/n2"]
      onetime = false
      interval = 1

Tried many keys (same key as in the code above without v1, or/and data or with just n2)
The key template is just {{ getv("/v1/secret/data/n2") }} (modified in concordance as above).

Thank You in advance.
Regards.

Retry if file was not found

Today we had an issue with Apache Rewrites that were not updated, because the file was not available in that moment remco tried to use it. Is it possible to configure a retry for such cases?

time="2019-07-16T10:39:48Z" level=debug msg="retrieving keys" backend=file key_prefix= prefix="remco_linux[1]" resource=apache-rewrites
time="2019-07-16T10:39:48Z" level=error msg="setVars failed: getValues failed: open /usr/local/rewrites/rewrites.json: no such file or directory" backend=file prefix="remco_linux[1]" resource=apache-rewrites
time="2019-07-16T10:39:48Z" level=error msg="no such file or directory" backend=file prefix="remco_linux[1]" resource=apache-rewrites

Release is old

Took me a bit to figure out why the backend defaults weren't in the docs 😄 Looks like the latest release is from October 7th 2017; before the backend defaults were added.

I can use go get to continue my testing.

Ping

Hola! I saw you mentioned in #18 that you weren't sure if other people were currently using this project. It's very compelling to me as an alternative to confd and I'm considering trying this out for the official Rundeck image I'm working on. It's currently experimental and utilizing confd. However, I have run into some.. friction even trying to get a discussion going about a small change to the env backend, and of course it doesn't support distributing the keyspace out across multiple backends; that feature looks clutch.

Hopefully later in the week or this weekend I can take it for a spin and see how it goes, but first I was curious what your take on the viability of the project is for external use? The potential benefits over confd seem somewhat obvious.. Is there a hot new kid on the block in this space on not seeing?

Thanks for your time in advance if you can reply.

Better/more documentation is needed

Request: make stagefile configurable

remco doesn't seem to like my .env configuration files as the stagefilename is the dest prepended with a '.'
Workaround for now I found that works is to use

 reload_cmd  = "cp /var/www/env /var/www/.env 

Looping over dict/object/map

This seems like a stupid question, but I didn't figure it out with confd, realized that the project was being deprecated, and still haven't figured it out with remco.

I have this in a yaml file:

users:
  - name: admin
     hash: adminhash
  - name: user
    hash: userhash

What is the template code to get the following result:

admin adminhash
user userhash

I assumed it would be something like this:

  {% for user in gets("/users") %}
    {{user.name}} {{user.hash}}
  {% endfor %}

but I've tried quite a few template functions as variations of gets with no success.

Thanks!

get keys from etcd and loop through

Hi.

I'm trying to migrate my configuration from Confd to Remco and I'm stuck on the simple operation which has been described in the documentation.
There is a function "toJson" and I wrote simple configuration:

{% for i in gets("/registry/services/specs/development/filter") | toJson %}
value: {{ i.Value }}
{% endfor %}

The Remco wrote into the log file:

time="2021-01-08T02:44:28+03:00" level=error msg="createStageFileAndSync failed: create stage file failed: set.FromFile(/etc/remco/resource.d/my.tmpl) failed: [Error (where: parser) in /etc/remco/resource.d/my.tmpl | Line 1 Col 67 near 'toJson'] Filter 'toJson' does not exist." prefix="./remco_linux[41390]" resource=nginx

What's wrong in my example and why the documentation so poor to describe the examples with different backends?!

Support GCP secret manager

It seems it has pretty simple api and good integration in all gcp stack
https://cloud.google.com/secret-manager/docs/reference/rest

key does not exists for file backend when value is int or boolean

key does not exists for file backend when value is int or boolean

[0000] INFO remco[6]: set file path backend=file filepath=/opt/better_chef_rundeck/bcr.json
[0000] ERROR remco[6]: createStageFileAndSync failed: create stage file failed: template execution failed: [Error (where: execution) in /etc/remco/templates/run.sh | Line 5 Col 96 near 'getv'] key does not exist: /bcr/port resource=run.sh.toml
[0000] ERROR remco[6]: not all templates could be rendered, trying again after 18 seconds resource=run.sh.toml

Using {{getenv("/bcr/port")}} for value replacement.

My json file bcr.json:
{
"bcr": {
"port": 3000
}
}

Template with [exec] seems to ignore check_cmd for templates

Hi,

i have this resource-configuration in <include_dir>/my-server.toml:

name = "my-server"

[exec]
    command = "my-server -c '/config.conf'"
    kill_signal = "SIGTERM"
    reload_signal = "SIGUSR1"
    kill_timeout = 60
    splay = 30


[[template]]
  src = "/config.conf.tpl"
  dst = "/config.conf"
  mode = "0644"
  check_cmd = "my-server-config-check -c '{{.src}}'"
  reload_cmd = ""

While this starts, terminates and reloads the command defined in [exec] correctly when the values from the backend change and /config.conf is updated by using the configured signals, it seems to ignore the check_cmd. Is this by design or is it a bug (or am i simply missing a typo in the config)?

getv from consul-service

Hi, I want to get a consul service from the consul.
my config

log_level = "debug"
log_file = ""
filter_dir = "/etc/remco/filters.d"

[[resource]]
  name = "test"

  [[resource.template]]
    src = "templates/aaa.tmpl"
    dst = "/tmp/out.txt"
    mode = "0644"

  [resource.backend]
    [[resource.backend.plugin]]
      path = "/etc/remco/plugins/consul-service"
      interval = 60
      keys = ["consul",]
      onetime = false
      [resource.backend.plugin.config]
	 addr = "127.0.0.1:8500"

my template

{{ getallkvs() | toPrettyJSON }}
{{ getv("service-name") }}

But I only get service defined in keys, not service-name. Some debug shows me that getv args is not propagated to pie and did not pass to plugin code
My target template should be something like

{% set all_svc = ?? get_all_svc_with_tag ?? %}
{% for svc in all_svc %}
location /{{svc.name}}/ {
 proxy_pass {{svc.addr}
}
{% endfor %}

consul services bundle ?

Hi, don't you think it is a good idea to bundle in
https://heavyhorst.github.io/remco/plugins/consul-plugin-example/ ?

code here is allredy outdated there is no "github.com/HeavyHorst/remco/backends/plugin"