sharpie / puppet-facts_upload

Re-implements the puppet facts upload command and associated Puppet Server endpoint.

Puppet Facts Upload

NOTE: The content in this module has been rendered obsolete as of Puppet 5.5.0 and Puppet Server 5.3.0 which re-added native support for the puppet facts upload command and associated API endpoint. The Puppet 5.x series reached End of Life at the end of January 2021. This module has been archived as it has reached the state of "finished software project".

This project re-implements the puppet facts upload command and part of the associated /puppet/v3/facts endpoint that was removed along with the inventory service in Puppet 4.0. These pieces of functionality were useful for keeping facts updated outside of catalog compilation.

Re-adding the command and endpoint enables two key use cases:

• The ability to run agents off of cached catalogs. Facts become stale in this configuration as they are only updated when a fresh catalog is compiled.

• The ability to submit updated facts at the end of an agent run. This allows facts to reflect any changes made during catalog application and keeps fact data as fresh as possible for PuppetDB queries.

Requirements

This module has been tested against the following versions of Puppet Server and Puppet Enterprise running on the following platforms:

• Versions: Puppet Server 5.1.z and 5.2.z, PE 2017.3.z and PE 2016.4.10 -- PE 2016.4.z
• Platforms: RedHat 6, 7; Ubuntu 14.04, 16.04; SLES 11, 12

The facts_upload module is fairly sensitive to the Puppet Server and PE version numbers as the code hooks into several internal APIs in both Puppet Server and the puppet_enterprise module. Installation of the Java plugins also requires manipulation of Puppet Server's Java classpath for versions older than Puppet Server 5 or PE 2017.3. The module likely will not work if your Puppet Server version or platform is not on the list above.

Puppet Agent support is much less restrictive. The module should work with Puppet versions 4.y and 5.y.

Usage

The facts_upload module currently provides one class: facts_upload::server. This class should be applied to all nodes running the puppetserver or pe-puppetserver services and will configure the Puppet Server to mount a /puppet/v3/facts endpoint and configures auth.conf to allow agents to upload facts for just their own certnames.

The module also provides some Ruby code which is pluginsynced to agents:

• lib/puppet/indirector/facts/rest.rb
• lib/puppet/feature/facts_upload_builtin.rb
• lib/puppet/face/facts/upload.rb

Agents may run puppet facts upload once pluginsync has completed.

Limitations

This module only provides a subset of the functionality that was removed in PUP-2560:

• A /puppet/v3/facts API endpoint for Puppet Server that responds to PUT requests only.

• A Puppet::Node::Facts::Rest indirector terminus that sends requests from agents to the /puppet/v3/facts API endpoint of the server the agent is configured to talk with.

• A puppet facts upload face that fetches facts from Facter and then uploads them to the /puppet/v3/facts API endpoint of the server the agent is configured to talk with.

Notably, the following deprecated features have not been restored:

• Functionality and REST API routes related to the Inventory Service.

• Functionality and indirector terminii related to the store_configs subsystem.

Development

See the output of rake -T for development workflow tasks. More info will be added to this section in a future release.

puppet-facts_upload's Issues

Installation of the facts_upload module fails with unsatisfiable dependencies

Currently, installation of the latests facts_upload module can fail:

# puppet module install sharpie-facts_upload
Notice: Preparing to install into /etc/puppetlabs/code/environments/production/modules ...
Notice: Downloading from https://forgeapi.puppet.com ...
Error: Could not install module 'sharpie-facts_upload' (???)
  No version of 'sharpie-facts_upload' can satisfy all dependencies
    Use `puppet module install --ignore-dependencies` to install only this module

This is caused by the dependencies of the puppetlabs-puppet_authorization module, which is not compatible with version 5.x of puppetlabs-stdlib or puppetlabs-concat. See MODULES-7844 for more details.

Upload fact files

Would it be possible to use this to upload facts files (YAML, JSON) instead of calling facter?

The need would be syncing facts files from an external Puppet Master (in DMZ) and uploading them to the PuppetDB afterwards.

"Could not autoload puppet/type/pe_ini_setting" error when Code Manager is enabled

In order to achieve environment isolation, Code Manager compiles custom types when deploying code. Due to PUP-8002 this causes an error when resource collectors are used:

Error: Could not autoload puppet/type/pe_ini_setting: Attempt to redefine entity 'http://puppet.com/2016.1/runtime/type/pe_ini_setting'. Set at unknown location

"stack level too deep" shows up in agent requests up after uploading facts at least twice

Unfortunately, the use of alias_method can set up the conditions for infinite recursion if the monkeypatch is applied twice. This is because the plurality method ends up aliased to the monkeypatched version that calls the alias. This occurs here:

https://github.com/Sharpie/puppet-facts_upload/blob/1.0.0/src/ruby/ruby/monkeypatches/facts_upload.rb#L8

If puppet facts upload is run twice, non-Fact requests will start failing with "stack level too deep" errors:

# puppet agent -t
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 500 on SERVER: Internal Server Error: org.jruby.exceptions.RaiseException: (SystemStackError) stack level too deep
Info: Retrieving pluginfacts
Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Failed to generate additional resources using 'eval_generate': Error 500 on SERVER: Internal Server Error: org.jruby.exceptions.RaiseException: (SystemStackError) stack level too deep
Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: Error 500 on SERVER: Internal Server Error: org.jruby.exceptions.RaiseException: (SystemStackError) stack level too deep
Info: Retrieving plugin
Error: /File[/opt/puppetlabs/puppet/cache/lib]: Failed to generate additional resources using 'eval_generate': Error 500 on SERVER: Internal Server Error: org.jruby.exceptions.RaiseException: (SystemStackError) stack level too deep
Error: /File[/opt/puppetlabs/puppet/cache/lib]: Could not evaluate: Could not retrieve file metadata for puppet:///plugins: Error 500 on SERVER: Internal Server Error: org.jruby.exceptions.RaiseException: (SystemStackError) stack level too deep
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Internal Server Error: org.jruby.exceptions.RaiseException: (SystemStackError) stack level too deep
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: Error 500 on SERVER: Internal Server Error: org.jruby.exceptions.RaiseException: (SystemStackError) stack level too deep

Since we can't use Ruby 2.0 Module#prepend, the best solution will be to add a guard clause similar to a C once-only header that prevents the patch code from being evaluated twice in a given JRuby instance.

Add guards to prevent module functionality from activating on newer Puppet versions

Now that puppetlabs/puppet#6434 has landed, the functionality patched in by this module will appear in core Puppet and Puppet Server. Guard clauses should be added to disable the patching behavior of this module if the Puppet version is new enough.