Coder Social home page Coder Social logo

puppetlabs-concat's Introduction

concat

Table of Contents

  1. Overview
  2. Module Description - What the module does and why it is useful
  3. Usage - Configuration options and additional functionality
  4. Reference - An under-the-hood peek at what the module is doing and how
  5. Limitations - OS compatibility, etc.
  6. License
  7. Development - Guide for contributing to the module

Overview

The concat module lets you construct files from multiple ordered fragments of text.

Module Description

The concat module lets you gather concat::fragment resources from your other modules and order them into a coherent file through a single concat resource.

Beginning with concat

To start using concat you need to create:

  • A concat{} resource for the final file.
  • One or more concat::fragment{}s.

A minimal example might be:

concat { '/tmp/file':
  ensure => present,
}

concat::fragment { 'tmpfile':
  target  => '/tmp/file',
  content => 'test contents',
  order   => '01'
}

Usage

Maintain a list of the major modules on a node

To maintain an motd file that lists the modules on one of your nodes, first create a class to frame up the file:

class motd {
  $motd = '/etc/motd'

  concat { $motd:
    owner => 'root',
    group => 'root',
    mode  => '0644'
  }

  concat::fragment { 'motd_header':
    target  => $motd,
    content => "\nPuppet modules on this server:\n\n",
    order   => '01'
  }

  # let local users add to the motd by creating a file called
  # /etc/motd.local
  concat::fragment { 'motd_local':
    target => $motd,
    source => '/etc/motd.local',
    order  => '15'
  }
}

# let other modules register themselves in the motd
define motd::register (
  $content = "",
  $order   = '10',
) {
  if $content == "" {
    $body = $name
  } else {
    $body = $content
  }

  concat::fragment { "motd_fragment_$name":
    target  => '/etc/motd',
    order   => $order,
    content => "    -- $body\n"
  }
}

Then, in the declarations for each module on the node, add motd::register{ 'Apache': } to register the module in the motd.

class apache {
  include apache::install, apache::config, apache::service

  motd::register { 'Apache': }
}

These two steps populate the /etc/motd file with a list of the installed and registered modules, which stays updated even if you just remove the registered modules' include lines. System administrators can append text to the list by writing to /etc/motd.local.

When you're finished, the motd file will look something like this:

  Puppet modules on this server:

    -- Apache
    -- MySQL

  <contents of /etc/motd.local>

Reference

See REFERENCE.md

Limitations

This module has been tested on all PE-supported platforms, and no issues have been identified.

For an extensive list of supported operating systems, see metadata.json

License

This codebase is licensed under the Apache2.0 licensing, however due to the nature of the codebase the open source dependencies may also use a combination of AGPL, BSD-2, BSD-3, GPL2.0, LGPL, MIT and MPL Licensing.

Development

Acceptance tests for this module leverage puppet_litmus. To run the acceptance tests follow the instructions here. You can also find a tutorial and walkthrough of using Litmus and the PDK on YouTube.

If you run into an issue with this module, or if you would like to request a feature, please file a ticket. Every Tuesday the Content and Tooling Team has office hours in the Puppet Community Slack, where you can ask questions about this and any other supported modules. This session usually runs at, approximately, 15:00 (BST), for about an hour.

If you have problems getting this module up and running, please contact Support.

If you submit a change to this module, be sure to regenerate the reference documentation as follows:

puppet strings generate --format markdown --out REFERENCE.md

Contributors

Richard Pijnenburg (@Richardp82)

Joshua Hoblitt (@jhoblitt)

More contributors.

puppetlabs-concat's People

Contributors

adrianiurca avatar bastelfreak avatar bmjen avatar chelnak avatar cmurphy avatar cyberious avatar daianamezdrea avatar david22swan avatar davids avatar duritong avatar eimlav avatar eputnam avatar florindragos avatar glennsarti avatar gspatton avatar hail9000 avatar hunner avatar j-vizcaino avatar jhoblitt avatar jordanbreen28 avatar lionce avatar lukasaud avatar malikparvez avatar michaeltlombardi avatar pmcmaw avatar ramesh7 avatar ripienaar avatar sanfrancrisko avatar sheenaajay avatar tphoney avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

puppetlabs-concat's Issues

Puppet agent high CPU load with v8

Describe the Bug

I use Foreman with Katello plugin to manage our infrastructure. We sync all modules from the Puppet Forge and use latest versions in our envs. I published a new version of Puppet environment with latest versions of necessary modules and Puppet agents started to get this error and consume CPU:
Error: /Stage[main]/Aide/Concat[/etc/aide.conf]/Concat_file[/etc/aide.conf]: Failed to generate additional resources using 'eval_generate': undefined method 'filter_map' for #<Array:0x00000000060dc270> Did you mean? flat_map

Environment

  • Version [8.0.0]
  • Platform [CentOS Linux release 7.9.2009]
  • Puppet Version [6.28.0]
  • Puppet Server[6.17.0]

Concat doesn't work when there is no group name that matches the user name

If there isn't a group on the system that matches the name of the user name, then concat doesn't work properly, which I believe is due to this fragment in setup.pp:

  $root_group = $id ? {
    root    => 0,
    default => $id
  }

Ideally this should support grabbing the group from a fact or something like that.

The error message is:

err: /Stage[main]/Concat::Setup/File[/home/testuser/puppet]: Could not evaluate: Could not find group testuser

fragment can be local file that might not always exist

In the 1.0 release, I could have a fragment (similar to the example in README)

  concat::fragment { "motd_local":
    target  => 'motd',
    order   => 90,
    ensure  => "/etc/motd.local",
  }

and if that file didn't exist locally, it didn't matter.

Keeping that syntax, I now get:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: validate_re(): "/etc/motd.local" does not match "^$|^present$|^absent$|^file$|^directory$" at /etc/puppet/modules/concat/manifests/fragment.pp:39 on node ____

If I try to use source, I get:

Error: /File[/var/lib/puppet/concat/motd/fragments/90_motd_local]: Could not evaluate: Could not retrieve information from environment devl source(s) file:/etc/motd.local

Please add LICENSES file to your work

I'm looking to include a modify version of your work into a common puppet module I am working on, it would be nice if you could include a LICENSE file for your project. I believe you using the following license:

Copyright [yyyy] [name of copyright owner]

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

order => numeric option doesn't seem to work

As described on the puppet irc channel, order => numeric doesn't seem to work.

This seems to be related to find and the fact that all files found start with fragments/ before the number.

Simple patch (suggested by richardc) seems to be:
diff --git a/files/concatfragments.sh b/files/concatfragments.sh
index 52fd684..884de5d 100644
--- a/files/concatfragments.sh
+++ b/files/concatfragments.sh
@@ -44,7 +44,7 @@ WORKDIR=""
TEST=""
FORCE=""
WARN=""
-SORT1="-z"
+SORT1="-z -t / -k 2"
SORT2=""
FINDARG="-print0"
XARGSARG="-0"`

This works for me at least on an Ubuntu box.

Concat v8 breaks apache v5.6.0

Just FYI, using the newest concat breaks the old apache code:

Failed to generate additional resources using 'eval_generate': undefined method `filter_map' for #Array:0x0000000003db80c8
Did you mean? flat_map
Source:/Stage[main]/Apache/Apache::Vhost[default-ssl]/Concat[15-default-ssl.conf]/Concat_file[15-default-ssl.conf]File:/etc/puppetlabs/code/environments/production/modules/concat/manifests/init.ppLine:157
2023-04-1216:26 Z | err |  

Concat 8.0.0 / Puppet 8.0 - Deferred function calls incompatibility

Describe the Bug

Concat v8 (and also prior) does not play well with Deferred function calls in Puppet 8. They do not get properly evaluated when using them as a content parameter. I first encountered this issue at voxpupuli/puppet-gitlab_ci_runner#167 - but tracked it down to concat / puppet v8.

Expected Behavior

Concat should be able to properly use Deferred function calls.

Steps to Reproduce

With docker.

Filename: Dockerfile-8

FROM debian:bullseye
RUN apt-get update && apt-get install -y wget
RUN wget https://apt.puppetlabs.com/puppet8-release-bullseye.deb
RUN dpkg -i puppet8-release-bullseye.deb
RUN apt-get update && apt-get install -y puppet-agent
RUN /opt/puppetlabs/bin/puppet module install puppetlabs-concat --version 8.0.1
RUN /opt/puppetlabs/bin/puppet apply -e "concat { '/tmp/file': ensure => present, } concat::fragment { 'tmpfile': target  => '/tmp/file', content => Deferred('md5', ['Hello']), order   => '01'}"

Commands:
docker build -f Dockerfile-8 -t concat8_puppet8 . && docker run -it concat8_puppet8 cat /tmp/file

Output:

[+] Building 13.1s (11/11) FINISHED                                                                                                                                                                                                                                                                                                                                        
 => [internal] load build definition from Dockerfile-8                                                                                                                                                                                                                                                                                                                0.0s
 => => transferring dockerfile: 543B                                                                                                                                                                                                                                                                                                                                  0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                                                                     0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/library/debian:bullseye                                                                                                                                                                                                                                                                                                    0.5s
 => [1/7] FROM docker.io/library/debian:bullseye@sha256:63d62ae233b588d6b426b7b072d79d1306bfd02a72bff1fc045b8511cc89ee09                                                                                                                                                                                                                                              0.0s
 => CACHED [2/7] RUN apt-get update && apt-get install -y wget                                                                                                                                                                                                                                                                                                        0.0s
 => CACHED [3/7] RUN wget https://apt.puppetlabs.com/puppet8-release-bullseye.deb                                                                                                                                                                                                                                                                                     0.0s
 => CACHED [4/7] RUN dpkg -i puppet8-release-bullseye.deb                                                                                                                                                                                                                                                                                                             0.0s
 => CACHED [5/7] RUN apt-get update && apt-get install -y puppet-agent                                                                                                                                                                                                                                                                                                0.0s
 => [6/7] RUN /opt/puppetlabs/bin/puppet module install puppetlabs-concat --version 8.0.1                                                                                                                                                                                                                                                                             5.7s
 => [7/7] RUN /opt/puppetlabs/bin/puppet apply -e "concat { '/tmp/file': ensure => present, } concat::fragment { 'tmpfile': target  => '/tmp/file', content => Deferred('md5', ['Hello']), order   => '01'}"                                                                                                                                                          5.9s
 => exporting to image                                                                                                                                                                                                                                                                                                                                                1.0s 
 => => exporting layers                                                                                                                                                                                                                                                                                                                                               1.0s 
 => => writing image sha256:0b0da3d09eb189bc665bcdcb5b56b5a6f23945a64201a4c216a6d5a78e5beed8                                                                                                                                                                                                                                                                          0.0s 
 => => naming to docker.io/library/concat8_puppet8                                                                                                                                                                                                                                                                                                                    0.0s 
#<Puppet::Pops::Evaluator::DeferredValue:0x00007fbc00e1c300>

Environment

  • Version: 8.0.0
  • Puppet Agent 8.0 on Debian Bullseye

Additional Context

I tried the same with puppet 7, which yields the correct result:

Dockerfile-7:

FROM debian:bullseye
RUN apt-get update && apt-get install -y wget
RUN wget https://apt.puppetlabs.com/puppet7-release-bullseye.deb
RUN dpkg -i puppet7-release-bullseye.deb
RUN apt-get update && apt-get install -y puppet-agent
RUN /opt/puppetlabs/bin/puppet module install puppetlabs-concat --version 8.0.1
RUN /opt/puppetlabs/bin/puppet apply -e "concat { '/tmp/file': ensure => present, } concat::fragment { 'tmpfile': target  => '/tmp/file', content => Deferred('md5', ['Hello']), order   => '01'}"

docker build -f Dockerfile-7 -t concat8_puppet7 . && docker run -it concat8_puppet7 cat /tmp/file
Output:

[+] Building 0.5s (11/11) FINISHED                                                                                                                                                                                                                                                                                                                                         
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                                                                     0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                                                                       0.0s
 => [internal] load build definition from Dockerfile-7                                                                                                                                                                                                                                                                                                                0.0s
 => => transferring dockerfile: 543B                                                                                                                                                                                                                                                                                                                                  0.0s
 => [internal] load metadata for docker.io/library/debian:bullseye                                                                                                                                                                                                                                                                                                    0.5s
 => [1/7] FROM docker.io/library/debian:bullseye@sha256:63d62ae233b588d6b426b7b072d79d1306bfd02a72bff1fc045b8511cc89ee09                                                                                                                                                                                                                                              0.0s
 => CACHED [2/7] RUN apt-get update && apt-get install -y wget                                                                                                                                                                                                                                                                                                        0.0s
 => CACHED [3/7] RUN wget https://apt.puppetlabs.com/puppet7-release-bullseye.deb                                                                                                                                                                                                                                                                                     0.0s
 => CACHED [4/7] RUN dpkg -i puppet7-release-bullseye.deb                                                                                                                                                                                                                                                                                                             0.0s
 => CACHED [5/7] RUN apt-get update && apt-get install -y puppet-agent                                                                                                                                                                                                                                                                                                0.0s
 => CACHED [6/7] RUN /opt/puppetlabs/bin/puppet module install puppetlabs-concat --version 8.0.1                                                                                                                                                                                                                                                                      0.0s
 => CACHED [7/7] RUN /opt/puppetlabs/bin/puppet apply -e "concat { '/tmp/file': ensure => present, } concat::fragment { 'tmpfile': target  => '/tmp/file', content => Deferred('md5', ['Hello']), order   => '01'}"                                                                                                                                                   0.0s
 => exporting to image                                                                                                                                                                                                                                                                                                                                                0.0s
 => => exporting layers                                                                                                                                                                                                                                                                                                                                               0.0s
 => => writing image sha256:7cc5bc2cd4ddf414f17111559facd0b8d8c4a54c923475af4a30812a27c73a6e                                                                                                                                                                                                                                                                          0.0s
 => => naming to docker.io/library/concat8_puppet7                                                                                                                                                                                                                                                                                                                    0.0s
8b1a9953c4611296a827abf8c47804d7

motd example incomplete?

err: Failed to apply catalog: Could not find dependent Exec[concat_/etc/motd] for File[/var/lib/puppet/concat/_etc_motd/fragments/10_motd_fragment_Apache] at /etc/puppet/modules/concat/manifests/fragment.pp:51

I tried copy pasting the example from readme - but got the above error when trying to apply it.

Using Puppet: 2.7.11 on Ubuntu 12.04 64bit

concat::fragment does not seem to notify properly

Describe the Bug

When using a notify meta parameter in a concat::fragment, the target resource doesn't get notified.

Expected Behavior

When a concat::fragment's state changes, it should properly notify the target resource.

Steps to Reproduce

Example code:

service { 'foo':
  ensure => 'running',
}

concat { '/tmp/foo':
  ensure => 'present',
}

concat::fragment { '/tmp/foo bar':
  target  => '/tmp/foo',
  content => 'bar',
  order   => '10',
  notify  => Service['foo'],
}
  1. Make sure service foo is running
  2. Make sure file /tmp/foo is not present
  3. Run puppet apply on the above code
Notice: Compiled catalog for localhost in environment production in 0.08 seconds
Notice: /Stage[main]/Main/Concat[/tmp/foo]/File[/tmp/foo]/ensure: defined content as '{sha256}fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9'
Notice: Applied catalog in 0.03 seconds

As you can see, even though the file changed, it didn't trigger restart of the service.

Environment

  • Version 6.2.0
  • Platform Debian 11.4

Module broken on Solaris 10 by addition of -r flag to read in concatfragments.sh

The Solaris 10 read shell builtins have no -r parameter, so puppet-concat is broken on Solaris 10 by commit 03650cc (fragments.concat is always 0 bytes).

Not sure about Solaris 11 but Solaris 10 update 9 has ksh version:

Version M-11/16/88i

The solution is to remove -r. Note that man read(1) incorrectly references /usr/bin/read having a -r option. Don't try to use it as per https://groups.google.com/d/topic/comp.unix.solaris/IlRKIujWmjI/discussion.

Support for content with sensitive type

Use Case

If there is sensitive content included in a file (like a password), I don’t want it to be displayed in the log.

Describe the Solution You Would Like

Support for the Sensitive type in the content:

class secret_within_content {
  $secret = Sensitive('mysecret')
  concat { '/tmp/example.conf':  }
  concat::fragment { 'secret' :
    target  => '/tmp/example.conf',
    content => $secret,
  }
}

class with_inline_epp {
  $secret = Sensitive('mysecret')
  $content = inline_epp("secret = <%= $s %>", { s => $secret })
  concat { '/tmp/example.conf':  }
  concat::fragment { 'secret' :
    target  => '/tmp/example.conf',
    content => $content,
  }
}

Describe Alternatives You've Considered

The alternative for now it to use show_diff => false, which don’t show any diff at all, but I would like to only remove the data that is sensitive, not the entire diff from the logs

Additional Context

I’ve tried adding Sensitive in fragment.pp on line 47:

46  if versioncmp($clientversion, '6.0') >= 0 and versioncmp($_serverversion, '6.0') >= 0 {
47    assert_type(Optional[Variant[String, Deferred, Sensitive]], $content)
48  } else {
49    assert_type(Optional[String], $content)
50  }

But then there is a warning and the diff still show the sensitive content: Warning: /Concat_fragment[secret]: Unable to mark 'content' as sensitive: content is a parameter and not a property, and cannot be automatically redacted. I would like to offer a pull request on this, but i’m unsure where I would go from here... Is a complete provider has to be written for this to work?

Server Error: no parameter named 'create_empty_file'

Describe the Bug

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: no parameter named 'create_empty_file' (file: /etc/puppetlabs/code/environments/production/modules/concat/manifests/init.pp, line: 126) on Concat_file[/usr/local/bin/some_file.sh] (file: /etc/puppetlabs/code/environments/production/modules/concat/manifests/init.pp, line: 126) on node XXXXX

Expected Behavior

should just work

Steps to Reproduce

Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'

Environment

  • Version Git rev: e3fe096
  • Platform OpenBSD 7.2
  • Puppet puppet-7.23.0
  • Puppetserver puppetserver-7.9.5
  • Ruby: ruby-3.1.3

Additional Context

removing the 'create_empty_file' parameter in init.pp in concat_file following line 126 fixes the trouble for me.

feature request: new parameter: order => close

Sometimes one would like to have ordered concats, defining a preamble and a close text within the concatenated file.

It is hard to ensure an appropriate numbering/... for the close if you concat from multiple places/modules. A parameter "order => close" as a reserved value for the order parameter of concat::fragment could be a solution to this situation, if it ensures that this very fragment gets included last to the target.

concat_file.rb is missing the code to run the validate_cmd

Describe the Bug

Unless I examined the code wrongly, the only thing that concact_file does with the validate_cmd
argument is ... validate that the arguments of that command are correct.

However, you seem to have forgotten the code to execute the validate_cmd. Even if you pass the correct
arguments, nothing happens.

Expected Behavior

I'd expect that the validate command be executed and abort the replacement of the file if the command returns 1.

Steps to Reproduce

Steps to reproduce the behavior:

  1. Use your fragment.pp example to create /tmp/concat by running puppet
  2. Modify your example to change line 12 so the content is '3' (or anything else, doesn't matter)
  3. Add a validate_command in line 7 that always returns 1 (the expected failure value) (doesn't matter if you add the ending %).
    validate_cmd => '/usr/bin/false',
  4. Run puppet on the modified example
  5. You'll see that /tmp/concact was replaced with the new content, even if validate_cmd returned an error.

It doesn't matter if validate_cmd returns 0 or 1. There's no code executing validate_cmd or taking action on its result.

Environment

  • Version [9.0.0] (I hope I read it correctly from puppet module list)
  • Platform [Debian 12.5]

Additional Context

You are missing a call to execute validate_cmd. Something along the lines of whatpuppet::filedoes
here

Failling with blanks in path of target file

When using concat module on windows platforms, we get an error when there are some blanks in path of target file as concat command used to concatenate the file at the end does neither escape blanks nor use quotes.

Using File resource not working

Describe the Bug

  file { '/tmp/test':
    ensure  => 'present',
    content => 'test',
    require => Concat['/tmp/test'],
  }

  concat { '/tmp/test':
    ensure => 'present',
  }

Expected Behavior

I would expect the file has the 'test' content. But the file content is empty.

Steps to Reproduce

Apply above code

Environment

  • Puppet. 7
  • Platform Sles 15sp5

Backup issues

A concat setup like this:

concat { $emitter_target:
mode => $ipfilter::config_file_mode,
owner => $ipfilter::config_file_owner,
group => $ipfilter::config_file_group,
notify => Service['ipfilter'],
backup => '.previous',
}

Will even affect the fragments in the puppet-vardir/concat.

After removing a reference to a fragment it vill be removed but the .previous file remain. Next run that file will be removed but a .previous.previous will be created and so on ...

ls concat/_etc_ipf_ipf.conf/fragments

01_ipfilter_header_v4
05_ipfilter_block_v4
10_ipfilter_icmp_v4
15_ipfilter_ssh_v4
20_ipfilter_rule_http
20_ipfilter_rule_snmp-whatsup
20_ipfilter_rule_telnet.previous.previous.previous.previous
20_ipfilter_rule_test.previous.previous.previous.previous.previous.previous.previous
80_output_policy_v4

Automatic clean up of temporary directory

I'm using concat to install public keys in various user folders on a Linux system. At the moment, I'm working through some problems in my deployment and noticed that concat leaves file fragments behind in /var/lib/puppet/concat -- its working directory. These public keys are exposed because that directory is world readable by default.

Is it possible to have concat purge the temporary folder, even in the event of a failure?

Or ... can you ensure the working directory is only readable by the superuser?

Or ... perhaps someone has a suggested alternate approach?

version number incompatible with librarian-puppet

To reproduce create a file "Puppetfile" with the following content

forge "http://forge.puppetlabs.com"
mod 'puppetlabs/concat'

Running "librarian-puppet install" results in:

/usr/lib/ruby/1.9.1/rubygems/version.rb:187:in initialize': Malformed version number string 1.1.0-rc1 (ArgumentError) from /usr/lib/ruby/vendor_ruby/librarian/manifest.rb:12:innew'
from /usr/lib/ruby/vendor_ruby/librarian/manifest.rb:12:in initialize' from /usr/lib/ruby/vendor_ruby/librarian/manifest.rb:125:innew'
from /usr/lib/ruby/vendor_ruby/librarian/manifest.rb:125:in _normalize_version' from /usr/lib/ruby/vendor_ruby/librarian/manifest.rb:109:infetched_version'
from /usr/lib/ruby/vendor_ruby/librarian/manifest.rb:55:in version' from /usr/lib/ruby/vendor_ruby/librarian/manifest.rb:93:insatisfies?'
from /usr/lib/ruby/vendor_ruby/librarian/dependency.rb:117:in satisfied_by?' from /usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:107:inblock in check_manifest'
from /usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:107:in each' from /usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:107:infind'
from /usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:107:in check_manifest' from /usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:57:inblock in recursive_resolve'
from /usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:143:in block (3 levels) in resolving_dependency_map_find_manifests' from /usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:176:inblock in scope_checking_manifest'
from /usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:208:in scope' from /usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:175:inscope_checking_manifest'
from /usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:142:in block (2 levels) in resolving_dependency_map_find_manifests' from /usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:196:inblock in map_find'
from /usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:195:in each' from /usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:195:inmap_find'
from /usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:141:in block in resolving_dependency_map_find_manifests' from /usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:154:inblock (2 levels) in scope_resolving_dependency'
from /usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:168:in block in scope_checking_manifests' from /usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:208:inscope'
from /usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:167:in scope_checking_manifests' from /usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:153:inblock in scope_resolving_dependency'
from /usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:208:in scope' from /usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:152:inscope_resolving_dependency'
from /usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:140:in resolving_dependency_map_find_manifests' from /usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:56:inrecursive_resolve'
from /usr/lib/ruby/vendor_ruby/librarian/resolver/implementation.rb:32:in resolve' from /usr/lib/ruby/vendor_ruby/librarian/resolver.rb:16:inresolve'
from /usr/lib/ruby/vendor_ruby/librarian/action/resolve.rb:26:in run' from /usr/lib/ruby/vendor_ruby/librarian/cli.rb:169:inresolve!'
from /usr/lib/ruby/vendor_ruby/librarian/puppet/cli.rb:63:in install' from /usr/lib/ruby/vendor_ruby/thor/command.rb:27:inrun'
from /usr/lib/ruby/vendor_ruby/thor/invocation.rb:120:in invoke_command' from /usr/lib/ruby/vendor_ruby/thor.rb:363:indispatch'
from /usr/lib/ruby/vendor_ruby/thor/base.rb:439:in start' from /usr/lib/ruby/vendor_ruby/librarian/cli.rb:26:inblock (2 levels) in bin!'
from /usr/lib/ruby/vendor_ruby/librarian/cli.rb:31:in returning_status' from /usr/lib/ruby/vendor_ruby/librarian/cli.rb:26:inblock in bin!'
from /usr/lib/ruby/vendor_ruby/librarian/cli.rb:47:in with_environment' from /usr/lib/ruby/vendor_ruby/librarian/cli.rb:26:inbin!'
from /usr/bin/librarian-puppet:9:in `

'

The docs do not adequately explain the validate_cmd parameters

Requested change

Please update the documentation for concat::concat::validate_cmd and concat_file::validate_cmd to provide sufficient details for users to use it successfully. That could be by copying the docs of File::validate_cmd or by inserting a reference to those docs, for example.

Context / explanation

With the current docs, users have to figure out that

  • the designated command is expected to accept a file name string;
  • a % character must be included in the command string as a placeholder for the filename;
  • the command name in the string must be an absolute pathname; and
  • the command is expected to exit with status 0 to indicate validity, and non-zero to indicate invalidity.

It might be reasonable to expect advanced Puppet users to guess some of that, but why should anyone need to guess? Or to study the module source to figure it out, which is what I had to do?

concat::fragment should accept source => []

Heyo - we're using fragments and the more old-school file source => [list,of,alternates] mechanism, which worked fine in the old setup, but current HEAD is using validate_string() on the source variable.

I agree that each of the elements inside source should be a string if source is an array, but source being an array doesn't seem to break any functionality, either...

My easy answer is just to rip out that validate as there doesn't appear to be a stdlib validate_string_or_maybe_an_array_either_is_all_right_really() - is there some issue you're avoiding, or was this just a case of something getting caught in a strict test case?

v7.3.2 change creates error with running `puppet generate types` on Puppetserver

Describe the Bug

Running puppet generate types with v7.3.2 of this module creates the error

Error: Failed to load custom type 'concat_file' from '/etc/puppetlabs/code/environments/production/modules/concat/lib/puppet/type/concat_file.rb': uninitialized constant Puppet::Type::File

Expected Behavior

Generate types to succeed as it did with previous versions

Steps to Reproduce

Steps to reproduce the behavior:

  1. install module on Puppetserver
  2. run puppet generate types

Environment

  • Puppet version 6 & 7
  • CentOS 7 & 8

Additional Context

Issue seems to be caused by the removal of one of more of these lines from lib/puppet/type/concat_file.rb in the 7.3.2 release

require 'puppet/type/file/owner'
require 'puppet/type/file/group'
require 'puppet/type/file/mode'

Adding require 'puppet/type/file' to lib/puppet/type/concat_file.rb restores functionality

Tags?

Would be nice to have tags corresponding to versions in the Forge.

/var might be mounted with noexec options

Since /var might be mounted with the noexec option there should be a way to set the variable $script_path to a different value. I'm not sure if the following diff is the best way to do it, though:

diff --git a/manifests/setup.pp b/manifests/setup.pp
index 8d36478..0707467 100644
--- a/manifests/setup.pp
+++ b/manifests/setup.pp
@@ -25,7 +25,11 @@ class concat::setup {
default => 'concatfragments.sh'
}

  • $script_path = "${concatdir}/bin/${script_name}"
  • if $::concat_script_path {
  • $script_path = $::concat_script_path
  • } else {
  • $script_path = "${concatdir}/bin/${script_name}"
  • }

$script_command = $::kernel ? {
'windows' => "ruby.exe ${script_path}",

librarian-puppet doesn't like your version number

librarian-puppet doesn't like your version number, and fails to install the module.

/Applications/Vagrant/embedded/lib/ruby/2.0.0/rubygems/version.rb:191:in `initialize': Malformed version number string 1.1.0-rc1 (ArgumentError)

v8 fails with create_empty_file

Describe the Bug

create_empty_file introduced in #766 (v7.4) is broken after #768 (v8):

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: no parameter named 'create_empty_file' (file: /etc/puppetlabs/code/environments/CONCAT/modules/concat/manifests/init.pp, line: 126) on Concat_file[/etc/samba/smb.conf] (file: /etc/puppetlabs/code/environments/CONCAT/modules/concat/manifests/init.pp, line: 126) on node [redacted]

This occurred with this module, concat is called here

Environment

  • module version 7.4 and 8
  • puppet 7.24, Ubuntu 20.04

pluginsync error, even when pluginsync is enabled

Hi,

I am getting the following error when I try to use puppet-concat from Vagrant:

$concat_basedir not defined. Try running again with pluginsync enabled

This is similar to the problem mentioned here: https://groups.google.com/forum/?fromgroups=#!topic/puppet-users/Vm_iQZlITc0

However, I am not running a puppetmaster. I am using Vagrant, which runs puppet in "agent mode". pluginsync is definitely enabled in the [main] section of puppet.conf, and I am even passing the --pluginsync option on the command line (via Vagrant):

  puppet.options = "--verbose --debug --pluginsync"

Any tips on how to get this working with Vagrant? It's such a great idea for a module, and perfectly solves a problem I am currently working on.

Thanks,

Mike

concat::fragment does not append a string to a file

Describe the Bug

I've installed the module puppetlabs-concat version 8.0.1 but when I declare

concat { '/path/to/file':
  ensure => present,
}

concat::fragment { 'tmpfile':
   target  => '/path/to/file',
   content => 'test contents',
   order   => 'ZZ'
}

and after that, when i perform a puppet run, the file gets overridden

Expected Behavior

I wanted to append a string to an existing file

Steps to Reproduce

Steps to reproduce the behavior:
Apply the above mentioned code

Environment

  • Version [8.0.1]
  • Platform [Ubuntu 22.04]
  • Puppet version: 8.6.0

"Could not look up qualified variable 'concat::setup::root_group'; class concat::setup has not been evaluated"

I'm not sure if this is user error or a bug. Puppet3.2 has been throwing tons of warnings for various modules.

I have two simple classes: one exports a concat::fragment, one consumes concat::fragments. when the consumer runs, the error "Could not look up qualified variable 'concat::setup::root_group'; class concat::setup has not been evaluated" is produced on the puppetmaster.

I'm not sure where to dig deeper to resolve this issue.

concat isn't playing nicely with SELinux

It seems that concat builds a config from fragments in a directory with a given SELinux type (puppet_var_lib_t) and then moves the resulting file to the destination.

SELinux will maintain the label for a file on a move, leading to the resulting config file being unreadable to the process that should use it (e.g. postgres).

A copy would workaround this issue.

v9.0.1 breaks compatibility with pre-Puppet 7.17

Describe the Bug

This breaks compatibility with pre-7.17 agents, as this was not implemented until then. For example, when the postgresql module uses it with a Puppet 6 agent:

Error: /Stage[main]/Postgresql::Server::Config/Postgresql::Server::Instance::Config[main]/Concat[/var/lib/pgsql/14/data/pg_hba.conf]/Concat_file[/var/lib/pgsql/14/data/pg_hba.conf]: Failed to generate additional resources using 'eval_generate': uninitialized constant Puppet::Pops::Evaluator::DeferredValue
Did you mean?  Puppet::Pops::Evaluator::DeferredResolver

May want to do something like this instead:
fragment_content = r[:content].respond_to?(:resolve) ? r[:content].resolve : r[:content]

I know the metadata says this version requires Puppet 7 at least, but I think this might be the only bit that breaks Puppet 6.

Expected Behavior

Concat works correctly with pre-Puppet 7.17 agents.

Hey PuppetLabs Module Team, you just broke 700 modules. Is fail in concat::setup really necessary?

Ok, got your attention. I know you guys love metrics and I'll just point out you just broke a ton of modules on github with your latest change to the concat module.

https://github.com/search?l=puppet&p=82&q=%22include+concat%3A%3Asetup%22&ref=searchresults&type=Code
https://github.com/puppetlabs/puppetlabs-concat/blob/master/manifests/setup.pp#L14

So the latest update for concat::setup fail function is a bit harsh:

if $caller_module_name != $module_name {
  fail("Use of private class ${name} by ${caller_module_name}")
}

What's the catastrophic issue that must halt the catalog compile if another module have included this class? (This was the pattern in the original concat module) Why not using warning instead which provides a path for migration?

This triggers a whole bunch of mandatory fixes, and honestly I don't see the value (it's a cleanup issue, not really a catalog compile halt). I wouldn't be screaming about this if it only affected a few people, and heck you guys still haven't fixed all your modules yet:

https://github.com/puppetlabs/puppetlabs-apache/blob/master/manifests/balancer.pp#L45
https://github.com/puppetlabs/puppetlabs-puppet/blob/master/manifests/master.pp#L86

So I'm going to end my rant here. Still love you guys, but think about the impact before you make these breaking changes.

Permission Denied when user/group of the destination file is not root

Image a concat definition like this:

    concat { '/some/path/to/web.xml':
      mode    => '0644',
      owner   => 'tomcat',
      group   => 'tomcat',
    }

Because the concat lists tomcat as user/group, the exec to concat the fragments will also be using those permissions (as that user).

But, the /var/lib/puppet directory is owned 0770 puppet/puppet and won't allow other read/execute rights:

sudo -u tomcat /var/lib/puppet/concat/bin/concatfragments.sh
sudo: unable to execute /var/lib/puppet/concat/bin/concatfragments.sh: Permission denied

Shouldn't the concat bin directory by default then be outside the /var/lib/puppet directory? This setup would prevent any usage of concat{} as a non-root user.

While it's trivial to chmod /var/lib/puppet to 0775, I wouldn't want to give everyone read-rights on that content, as not all files have correct limitations on their file permissions.

Is there a reason why concat can't include concat::setup?

I often find myself writing some manifests only to realise that i've forgotten to include concat::setup.

Then it occurred to me that in alot of my own similar code, the concat define would just include concat::setup. Meaning that I didn't need to remember to include it wherever I want to use concat.

So I wonder if there's a specific reason not to do it like that?

Windows support

AFAICT concat uses shell scripts to produce the final file. That obviously doesn't work on windows. It would be great to be able to use concat on windows.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.