facebook / chef-cookbooks Goto Github PK
View Code? Open in Web Editor NEWOpen source chef cookbooks.
License: Apache License 2.0
Open source chef cookbooks.
License: Apache License 2.0
all centos and all debian/ubuntus fail for something equivalent to:
No candidate version available for spl, spl-dkms, zfs-dkms, zfsutils-linux
In chef v17, chef added a lot of built-ins and we are seeing instability with at least one of them
when we call debian?
within a library function, it returns false, so we are now having to do debian? || ubuntu?
to workaround this as we consume fb_helpers.
To put this nicely, is Facebook willing to stop doing this? Or at the very least, stop putting so many of these things in node
and put it to a FB namespace?
Stuff like FB::Fstab.get_unmasked_base_mounts
isn't safe to call from random cookbooks as it relies on node data that may not necessarily be there or be correct. Audit all methods, codemod the internal ones with a _
and document the rest properly in README.
Hi there,
I'm having an issue using fb_fstab on Ubuntu 18.04.
My Node looks like this:
{
"name": "somethingsomething",
"chef_environment": "prod",
"normal": {
"fb_fstab" : {
"enable_remount": true,
"enable_unmount": true,
"mounts" : {
"varlog" : {
"enable_remount": true,
"device": "logs-tmpfs",
"type": "tmpfs",
"opts": "size=1G",
"mount_point": "/var/log"
}
}
},
"tags": [ ]
},
"policy_name": null,
"policy_group": null,
"run_list": [
"recipe[fb_fstab::default]"
]
}
Running fb_fstab::default gives me a NoMethodError.
(the version Numbers are bumped, because we also use a version of this cookbook from like 2 years ago, this run used the cookbooks from d1e06db )
Starting Chef Client, version 14.10.9
resolving cookbooks for run list: ["fb_fstab::default"]
Synchronizing Cookbooks:
- fb_helpers (0.1.1)
- fb_fstab (0.0.2)
Installing Cookbook Gems:
Compiling Cookbooks...
Converging 5 resources
Recipe: fb_fstab::default
* file[/etc/.fstab.chef] action create
- change mode from '0644' to '0444'
* whyrun_safe_ruby_block[validate data] action run
================================================================================
Error executing action `run` on resource 'whyrun_safe_ruby_block[validate data]'
================================================================================
NoMethodError
-------------
undefined method `[]' for nil:NilClass
Cookbook Trace:
---------------
/var/chef/cache/cookbooks/fb_fstab/libraries/default.rb:98:in `get_autofs_points'
/var/chef/cache/cookbooks/fb_fstab/libraries/default.rb:105:in `autofs_parent'
/var/chef/cache/cookbooks/fb_fstab/recipes/default.rb:83:in `block (3 levels) in from_file'
/var/chef/cache/cookbooks/fb_fstab/recipes/default.rb:31:in `each'
/var/chef/cache/cookbooks/fb_fstab/recipes/default.rb:31:in `block (2 levels) in from_file'
Resource Declaration:
---------------------
# In /var/chef/cache/cookbooks/fb_fstab/recipes/default.rb
28: whyrun_safe_ruby_block 'validate data' do
29: block do
30: uniq_devs = {}
31: node['fb_fstab']['mounts'].to_hash.each do |name, data|
32: # Handle only_if
33: if data['only_if']
34: unless data['only_if'].class == Proc
35: fail 'fb_fstab\'s only_if requires a Proc'
36: end
37: unless data['only_if'].call
38: Chef::Log.debug("fb_fstab: Not including #{name} due to only_if")
39: node.rm('fb_fstab', 'mounts', name)
40: next
41: end
42: end
43:
Compiled Resource:
------------------
# Declared in /var/chef/cache/cookbooks/fb_fstab/recipes/default.rb:28:in `from_file'
whyrun_safe_ruby_block("validate data") do
action [:run]
default_guard_interpreter :default
declared_type :whyrun_safe_ruby_block
cookbook_name "fb_fstab"
recipe_name "default"
block #<Proc:[email protected]/var/chef/cache/cookbooks/fb_fstab/recipes/default.rb:29>
block_name "validate data"
end
System Info:
------------
chef_version=14.10.9
platform=ubuntu
platform_version=18.04
ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
program_name=/usr/bin/chef-client
executable=/opt/chef/bin/chef-client
Running handlers:
[2019-02-20T09:21:18+00:00] ERROR: Running exception handlers
Running handlers complete
[2019-02-20T09:21:18+00:00] ERROR: Exception handlers complete
Chef Client failed. 1 resources updated in 04 seconds
[2019-02-20T09:21:18+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2019-02-20T09:21:18+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2019-02-20T09:21:18+00:00] FATAL: NoMethodError: whyrun_safe_ruby_block[validate data] (fb_fstab::default line 28) had an error: NoMethodError: undefined method `[]' for nil:NilClass
this is the generated stacktrace:
Generated at 2019-02-20 09:21:52 +0000
NoMethodError: whyrun_safe_ruby_block[validate data] (fb_fstab::default line 28) had an error: NoMethodError: undefined method `[]' for nil:NilClass
/var/chef/cache/cookbooks/fb_fstab/libraries/default.rb:98:in `get_autofs_points'
/var/chef/cache/cookbooks/fb_fstab/libraries/default.rb:105:in `autofs_parent'
/var/chef/cache/cookbooks/fb_fstab/recipes/default.rb:83:in `block (3 levels) in from_file'
/var/chef/cache/cookbooks/fb_fstab/recipes/default.rb:31:in `each'
/var/chef/cache/cookbooks/fb_fstab/recipes/default.rb:31:in `block (2 levels) in from_file'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/provider/whyrun_safe_ruby_block.rb:25:in `action_run'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/provider.rb:182:in `run_action'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource.rb:578:in `run_action'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:70:in `run_action'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:98:in `block (2 levels) in converge'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:98:in `each'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:98:in `block in converge'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/resource_list.rb:94:in `block in execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/resource_list.rb:92:in `execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:97:in `converge'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:720:in `block in converge'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:715:in `catch'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:715:in `converge'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:754:in `converge_and_save'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:286:in `run'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:321:in `block in fork_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:309:in `fork'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:309:in `fork_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:274:in `block in run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/local_mode.rb:44:in `with_server_connectivity'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:261:in `run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application/client.rb:479:in `sleep_then_run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application/client.rb:468:in `block in interval_run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application/client.rb:467:in `loop'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application/client.rb:467:in `interval_run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application/client.rb:451:in `run_application'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:66:in `run'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/bin/chef-client:25:in `<top (required)>'
/usr/bin/chef-client:74:in `load'
/usr/bin/chef-client:74:in `<main>'
>>>> Caused by NoMethodError: undefined method `[]' for nil:NilClass
/var/chef/cache/cookbooks/fb_fstab/libraries/default.rb:98:in `get_autofs_points'
/var/chef/cache/cookbooks/fb_fstab/libraries/default.rb:105:in `autofs_parent'
/var/chef/cache/cookbooks/fb_fstab/recipes/default.rb:83:in `block (3 levels) in from_file'
/var/chef/cache/cookbooks/fb_fstab/recipes/default.rb:31:in `each'
/var/chef/cache/cookbooks/fb_fstab/recipes/default.rb:31:in `block (2 levels) in from_file'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/provider/whyrun_safe_ruby_block.rb:25:in `action_run'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/provider.rb:182:in `run_action'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource.rb:578:in `run_action'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:70:in `run_action'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:98:in `block (2 levels) in converge'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:98:in `each'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:98:in `block in converge'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/resource_list.rb:94:in `block in execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/resource_list.rb:92:in `execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:97:in `converge'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:720:in `block in converge'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:715:in `catch'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:715:in `converge'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:754:in `converge_and_save'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:286:in `run'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:321:in `block in fork_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:309:in `fork'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:309:in `fork_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:274:in `block in run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/local_mode.rb:44:in `with_server_connectivity'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:261:in `run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application/client.rb:479:in `sleep_then_run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application/client.rb:468:in `block in interval_run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application/client.rb:467:in `loop'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application/client.rb:467:in `interval_run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application/client.rb:451:in `run_application'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:66:in `run'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/bin/chef-client:25:in `<top (required)>'
/usr/bin/chef-client:74:in `load'
/usr/bin/chef-client:74:in `<main>
It seems like I'm missing something. Any Idea how I can get this cookbook to work?
Thanks,
-Markus
When you set splaysecs
the fb_cron
cookbook references the node.get_seeded_flexible_shard()
. This method does not appear to be publicly available.
Its sitting at in_shard?(99)... wanna clean that up?
several shellcheck failures for fb_network_scripts/files/default/ifup-pre-local
kitchen failures for unknown instance:
kitchen failures for grub:
template[grub2_config_efi] action create
Parent directory /boot/efi/EFI/debian does not exist.
various rubocop failures
We should use localhost instead
I'm leaving it out of my testing PR, but it needs defaults for all it's values in attributes...
Thought to be honest, I would actually pull all that configurability of what it's directories are out to match the rest of the cookbooks that let you put whatever you want in the config, but are strongly opinionated about where that config is laid out.
According to both the docs and the logs, '_clowntown_override_file_method' allows you to add additional restrictions to when override files take effect, but if it's not defined, then they always take effect.
Turns out, this isn't true, we always return false. This is a simple bug, but fixing it it probably requires a slow rollout to FB so I haven't sent a PR.
Docs:
For additional safety, you can define a check method in
`node['fb_storage']['_clowntown_override_file_method']`
that will be invoked whenever override files are evaluated.
Actual code where we say we're ignore the method, but then returnf alse.
def override_file_applies?(verb, fname, quiet = false)
if File.exist?(fname)
base_msg = "fb_storage: System has #{fname} file present"
if node['fb_storage']['_clowntown_override_file_method']
...
end
unless quiet
Chef::Log.warn(
"#{base_msg} but the override check method is not defined, " +
'therefore we are ignoring it.',
)
end
return false # this should be `true`
end
return false
end
Debian and ubuntu both allow keys in /etc/apt/trusted.gpg.d
Currently if a key listed in chef is in a file in that directory, which for Ubuntu 18.04 all of the older keys are, then Chef won't find them, it tries to add them, this "succeeds" because it's already there and it tries to do this every time.
To make matters worse, apt-key
does not accept more than one --keyring
option.
I'm not sure of the best solution, but I suspect it's:
/etc/apt/trusted.gpg.d/fb_keyring.gpg
{ubuntu,debian}-keyring-\d{4}-.*.gpg
but should leave that and the main trusted.gpg alone as "the package's pervue":now
is suppsoed to reboot now (if allowed). But it has an undocumented FB-ism, that if you're in firstboot, it switches to managed reboots:
action :now do
# TODO (t15830562) - this action should observe required and override the
# same way as the :deferred action
if node['fb_helpers']['reboot_allowed']
if node.firstboot_any_phase?
set_reboot_override('immediate')
do_managed_reboot
else
That's... not cool.
default file doesn't run tmpreaper for anything by default and only adds new calls based on what people add to the API. The default API does not include anything in directories
(presumably because tmpclean file always does /tmp hardcoded), but the template for tmpreaper drops that. The default file on Ubuntu btw has:
TMPREAPER_TIME=${TMPREAPER_TIME:-7d}
TMPREAPER_PROTECT_EXTRA=${TMPREAPER_PROTECT_EXTRA:-''}
TMPREAPER_DIRS=${TMPREAPER_DIRS:-'/tmp/.'}
nice -n10 tmpreaper --delay=$TMPREAPER_DELAY --mtime-dir --symlinks $TMPREAPER_TIME \
$TMPREAPER_ADDITIONALOPTIONS \
--ctime \
--protect '/tmp/.X*-{lock,unix,unix/*}' \
--protect '/tmp/.ICE-{unix,unix/*}' \
--protect '/tmp/.iroha_{unix,unix/*}' \
--protect '/tmp/.ki2-{unix,unix/*}' \
--protect '/tmp/lost+found' \
--protect '/tmp/journal.dat' \
--protect '/tmp/quota.{user,group}' \
`for i in $TMPREAPER_PROTECT_EXTRA; do echo --protect "$i"; done` \
$TMPREAPER_DIRS
in it.
It's probably better to move away from the templating of calls and instead template /etc/tmpreaper.conf and just populate the right variables.
This diff added a thing to write out status.conf, but it's hardcoded with /etc/httpd/ which breaks on non-CentOS systems. PR incoming eventually
Hi,
The provider code:
if keys && keyring
installed_keys = []
if ::File.exist?(keyring)
cmd = Mixlib::ShellOut.new("LANG=C apt-key --keyring #{keyring} list")
cmd.run_command
cmd.error!
output = cmd.stdout.split("\n")
Chef::Log.debug("apt-key output: #{output.join("\n")}")
installed_keys = output.select { |x| x.start_with?('pub') }.map do |x|
x[%r{pub./(?[A-Z0-9])}, 'keyid']
end
end
Does not detect the debian 9 apt-key output:
pub rsa4096 2016-10-05 [SC]
72EC F46A 56B4 AD39 C907 BBB7 1646 B01B 86E5 0310
uid [ unknown] Yarn Packaging [email protected]
Every chef pass reinstalls all the keys.
References fb_screen
when it should be fb_ethtool
:
"You can opt out of package management by settings node['fb_screen']['manage_packages'] to false."
fb_network_scripts
does some trickery under the hood to in-place modify configs to try and avoid network restarts at much as possible. This is great, but the changes don't get reflected in /var/chef/backups
, making troubleshooting harder at times. I'm not entirely sure if this is something we can sanely improve, but filing this for tracking and visibility.
https://github.com/facebook/chef-cookbooks/blob/master/cookbooks/fb_vsftpd/recipes/default.rb#L69
The disable vsftpd
service resource does not include service_name vsftpd
so it errors when node['fb_vsftpd']['enable'] == false
We should run httpd -t
in a validator when rendering config files in templates
Converging the cookbooks in this repo on a v17+ Chef client result in the following deprecation warning being thrown at the end of the run (using fb_systemd
as an example):
The resource in the fb_systemd cookbook should declare `unified_mode true` at 2 locations:
- /var/chef/cache/cookbooks/fb_systemd/resources/loader_entries.rb
- /var/chef/cache/cookbooks/fb_systemd/resources/reload.rb
See https://docs.chef.io/deprecations_unified_mode/ for further details.
See: https://docs.chef.io/unified_mode/
Anyone using Chef client v17+ will face this deprecation warning until unified_mode true
is appended to the first line of each custom resource file.
Ignore the deprecation warning.
Should be a relatively easy PR (or series of PR's) to craft.
Can we publish these to supermarket.chef.io ?
On Ubuntu 16 and 18 the fb_systemd::udevd
recipe succeeds without issue but on Ubuntu 20 the recipe will fail because the symlink from /sbin/udevadm
to /bin/udevadm
has been removed.
From the udev
filelists provided by Canonical for each release, one can validate the existence of /sbin/udevadm
in Xenial and Bionic Beaver, as well as its non-existence on Focal Fossa.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.