chef / effortless

Automated best practices for Chef Infra and Chef InSpec

Effortless

Project State: Active

Issues Response Time Maximum: 14 days

Pull Request Response Time Maximum: 14 days

Effortless is pattern to better manage Chef and Chef InSpec workloads using Chef Habitat.

Quick Links

• Chef Infra - Chef Infra automates infrastructure configuration, ensuring every system is configured correctly and consistently.

• Chef InSpec - Automate security tests, ensuring consistent standards are enforced in every environment, at every stage of development.

• Chef Habitat - Codify how the application is built, how it runs, and all of its dependencies to free the app from underlying infrastructure and make updates easy.

• Chef Automate - Enterprise dashboard and analytics tool enabling cross-team collaboration with actionable insights for configuration and compliance and an auditable history of changes to environments.

Existing Users

If you're already familiar with the Chef stack, here's a quick rundown of how Effortless works.

1. Effortless uses a build process to pull down all your cookbooks or profiles. The build creates a single, deployable package. For Chef Infra, it contains your cookbooks, an up-to-date Chef Infra client, and the latest best practices. For Chef InSpec, it contains your profiles, an up-to-date Chef InSpec client, and the latest best practices.

2. At runtime, Chef Infra works without Chef Infra Server. It uses Chef Solo mode.

3. At runtime, Chef InSpec works without pulling profiles from Chef Automate. All profiles, including those from Chef Automate, are vendored at build time.

4. Chef Habitat manages Chef Infra and Chef InSpec, and provides a pull-based update strategy for continuous delivery.

5. This workflow is a replacement for the environment and role cookbook patterns or Berkshelf way.

Next Steps

If you are new to the Effortless pattern checkout some of the below examples and walk throughs that will help you understand what you can do with this pattern.

Examples

1. Effortless Audit
2. Effortless Config

effortless's Issues

When building a package errors are thrown because some directories already exist in $pkg_prefix

Describe the problem

When a package is built a new studio is created. This includes the scaffoling files which create the necessary directories. However sometimes these directories may already exist which results in the following errors being thrown.

Software Version

Windows 10 1903

Replication Case

Please see https://github.com/russellseymour/effortless for a repo that when built will throw the above error.

Stacktrace

  workstation: Creating initial bootstrap configuration
   workstation: Creating Chef Infra client configuration
   workstation: Generating config/attributes.json
   workstation: Generating Chef Habitat configuration default.toml
   workstation: Writing configuration
   workstation: Writing hooks
   workstation: Writing default.toml
   workstation: Creating lifecycle hooks
An error occured in the build!
New-Item : An item with the specified name C:\hab\studios\Users--russells--OneDrive--workspaces--turtlesystems--effortless\hab\pkgs\russellseymour\workstation\0.1.1\20190719164359\hooks already exists.
At C:\hab\studios\Users--russells--OneDrive--workspaces--turtlesystems--effortless\hab\pkgs\chef\scaffolding-chef-infra\0.10.0\20190718132935\lib\scaffolding.ps1:43 char:5 
+     New-Item -ItemType directory -Path "$pkg_prefix/hooks"
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : ResourceExists: (C:\hab\studios\...719164359\hooks:String) [New-Item], IOException
+ FullyQualifiedErrorId : DirectoryExist,Microsoft.PowerShell.Commands.NewItemCommand

Possible Solution

I do have a fix for this. It is to add a check to the New-Item cmdlet in the scaffolding-chef-infra/lib/scaffolding.ps1 file. I need to create this issue before I submit the PR.

effortless unable to validate ssl during calls to supermarket through proxy

http_proxy
https_proxy
SSL_CERT_FILE
The above environments variables have been set. Corporate cert gets through all other proxy settings until scaffolding tries to reach supermarket.

We need a feature for hab pkgs to utilize the hab cert chain.

Chef-dk currently cannot reach through a corporate proxy due to it not seeing the valid certs. Hab itself is functioning as expected.

This is the current error

[2019-07-26T14:29:30+00:00] ERROR: SSL Validation failure connecting to host: s3.amazonaws.com - SSL_connect returned=1 errno=0 state=error: certificate verify failed
Error: Failed to generate Policyfile.lock
Reason: (OpenSSL::SSL::SSLError) SSL Error connecting to https://supermarket.chef.io/api/v1/cookbooks/cloudcli/versions/1.2.0/download - SSL Error connecting to https://s3.amazonaws.com/community-files.opscode.com/cookbook_versions/tarballs/21168/original/cloudcli.tgz?1493505782 - SSL_connect returned=1 errno=0 state=error: certificate verify failed

[Tracking] [scaffolding-chef-infra] Windows doesn't honor $pkg_deps set in scaffold

Describe the problem

When using the scaffolding for windows, $pkg_deps is not set, resulting in the artifact not being able to find $scaffold_chef_client.

Software Version

chef/scaffolding-chef 0.6.0 (echohack's build in depot)
hab 0.82.0
hab-sup 0.82.0

Replication Case

Create a minimal plan.ps1 that does not include $pkg_deps, and attempt to execute the built package on Windows.

Stacktrace

Possible Solution

[Tracking] Automatically set data collector endpoint in scaffolding

This is a tracking issue to automatically set the data collector endpoint in scaffolding so that users don't have to worry about this.

[Tracking] [scaffolding-chef-infra] [scaffolding-chef-inspec] Bubble up variable as scaffold_log="level"

When we run into issues there should be a way to specify the log level, in order to debug issues for all underlaying tools, without needing to rebuild the scaffolding.

Windows Chef-Client does not include bundle

Describe the problem

The Windows package being used by the scaffolding (stuartpreston/chef-client-detox) does not include the bundle executable. This results in chef-client failing during Installing Cookbook Gems.

Software Version

Scaffolding 0.6.0
Habitat 0.82.0

Replication Case

Stacktrace

Possible Solution

Use stuartpreston/chef-client instead.

Add examples to docs directory

Describe the problem

To get started one or more examples would be great. Currently the Readme tells a story but it is not easy to follow for first time visitors

Software Version

Any version

Possible Solution

Add three examples:

1. with a new cookbook without dependencies
2. config with a simple cookbook (ntp?) from supermarket
3. air gapped env <-- most complex, maybe this example should have part a) not air gapped and part b) transition to air gapped

Add support for hab plan init

As a chef user with existing cookbooks and a policyfile, I expect hab plan init to automatically detect the repo type and generate all of the files necessary to get started.

[Tracking] [scaffolding-chef-infra] Rubygems environment causes issues with building some packages

When I run the build I get an error that is documented that requires you to downgrade the chef ruby gem as a workaround (Reason: (ArgumentError) "\x80\x00\x00\x00d\x9E\xD3\xD9" is not an octal string)

* https://github.com/rubygems/rubygems/issues/2213 - the current remediation recipe code fails to resolve dependencies due to this rubygems bug. You will see an error during `kitchen converge` like `Reason: (ArgumentError) "\x80\x00\x00\x00d\x9E\xD3\xD9" is not an octal string`. The workaround is to downgrade rubygems:  chef exec gem update --system 2.7.5```

## Software Version
scaffolding-chef-infra 0.8.0 or lesser

## Replication Case
Build a package using https://github.com/chef/cis-rhel as a cookbook in the policyfile and runlist.

## Workaround

You can simply use an older version of the chef-dk to downgrade your rubygems environment. 

pkg_name=cis-rhel
pkg_origin=echohack
pkg_version="0.1.0"
pkg_maintainer="The Habitat Maintainers [email protected]"
pkg_license=("Apache-2.0")
pkg_upstream_url="http://chef.io"
pkg_scaffolding="chef/scaffolding-chef-infra"
scaffold_policy_name="cis-rhel"
scaffold_policyfile_path="$PLAN_CONTEXT"
scaffold_chef_dk="chef/chef-dk/3.0.38"

## Possible Solution
`chef/chef-dk` will need to upgrade it's rubygems environment to version `3.0.5`, which should fix this issue.

This is marked as a tracking issue as the issue may resolve itself without updating code in this repository.

[scaffolding-chef-inspec] Linux scaffolding Inspec_exec_config.json is different than Windows

In updates to the windows inspec_exec_config.json things were changed to make some the default toml attributes easier to reason about. Additionally the compliance section was removed in favor of pulling down the Automate profiles during build. There needs to be consistency between these two files.

[scaffolding-chef-infra] Fix issue with url and `/data-collector/v0`

We need to update the README and the toml configurations to ensure that we are consistent with the Automate URL configuration. I think we need to support both with and without the /data-collector/v0/ so that we have some backward compatibility.

[scaffolding-chef-inspec] Linux scaffolding errors when attempting pull a dependency from Automate

Tested pulling a profile from Automate this morning for Linux and hit an issue with it parsing the values using jq. We have updated the Windows scaffolding to not do this so Linux should have the same behavior. For reference #84

[Tracking] Add example of InSpec supporting different platforms, using top level inspec.yml wrapper

This is a tracking issue to track the work ongoing to create an example in the examples directory of InSpec supporting different platforms, using top level inspec.yml wrapper.

Add support for on-prem depot

As a user of scaffolding-chef, a major blocker for me is the on-prem builder. The main problems are as follows:

• Auth is a bit alpha/beta-y for A2-oauth and that's really what the customer wants to use, but is for now using local users to suffice/get past that issue.
• The sync script that comes with it is very finicky and there are a lot of issues when using self-signed/on-prem CA bundles that require you to go into the script and hack things to make it work.
• The documentation is lacking very straight forward/step-by-step installation and setup instructions (I'm working on that right now).
• Keeping the depot synced with the on-prem script doesn't work properly, and the bldr_package_sync go binary that is maintained by Indellient has a lot of bugs/issues as well.
• We now have things in chef/ for effortless scaffolding, and the script doesn't pull/sync that stuff.

I don't want to use the public depot to host my own packages, and taking 2+ weeks to get the local depot sorted is not delightful at all.

[Tracking] Windows bootstrap script for Habitat Windows Service

Add a powershell script to the bin/bootstrap in the repo that can bootstrap Habitat on Windows.

[scaffolding-chef-infra] Bundle Gems inside the Habitat artifact

When running Effortless Config in a Habitat Artifact, the chef-infra client downloads Gems from rubygems.org are still trying to be downloaded and that download fails in an airgapped environment.

One of the benefits of using an Effortless Config is having artifacts which are immutable and easily portable as harts ability to be used in environments that a regular chef-infra client is harder to use in.

Can these cookbook gems be cached when building and then referenced at runtime without the call-out to rubygems.org?

[Tracking] Update expeditor bot to auto-update changelog

This is a tracking issue to modify the expeditor bot to automatically update the changelog and to create a changelog.

[Tracking] Node size reduction investigation

Scaling Chef Automate can become much easier if the average node size is small.

This is a tracking issue to investigate average node size payload reduction techniques that we can apply in Effortless to improve the scaling experience.

Reference

Scaling Chef Automate

[scaffolding-chef-infra] Use consistent variable names for scaffold variables.

Describe the problem

The Linux build uses scaffold_policyfiles_path (plural) and the Windows build uses $scaffold_policyfile_path (singular). We should standardize on one or the other.

It would also be nice to make both of them support either syntax to catch the existing users out there.

Software Version

Replication Case

Stacktrace

Possible Solution

[scaffolding-chef-infra] Chef-client needs access to binaries outside of Habitat context

Currently the chef-client only uses binaries available to Habitat, and this can result in surprising results, such as using Busybox's version of APT / RPM instead of the one installed on the box.

InSpec has resolved this issue previously and we may adopt a similar solution here. Will need to have some discussion around this issue.

Reference:

inspec/inspec#3645
https://github.com/inspec/inspec/blob/master/habitat/plan.sh#L67
inspec/inspec#3635

[scaffolding-chef-infra] [scaffolding-chef-inspec] Allow report handlers to proxy through Chef Server

Many people use the Chef Infra Server to proxy node data to Chef Automate. Right now scaffolding-chef-infra only supports directly sending to Chef Automate. We should support a configuration option to allow proxying data through a Chef Infra Server.

[scaffolding-chef-inspec] Write error output for failing depends / path

When doing a build and depending on a local set of controls via the inspec.yml, it will throw an error saying that the .tar.gz file could not be found. This isn't the friendliest suggestion to the user. Instead we should suggest that the dependency actually be fixed.

Effortless: Using chef client with chocolatey resource fails

Description

I am trying to use the Effortless pattern to create a workstation cookbook for my Windows 10 machines. I am using the newly merged chef/scaffolding-chef-infra package which has support for Windows now.

The package only installs Chocolatey and then uses the chocolatey_package resource to install two applications:

1. Visual Studio Code
2. Firefox

When Habitat runs the package, Chocolatey is installed correctly and indeed the chocolatey_resource runs, but the following errors are thrown.

workstation.default hook[init]:(HK):     Failures
workstation.default hook[init]:(HK):      - vscode (exited 1) - vscode not installed. An error occurred during installation:
workstation.default hook[init]:(HK):      Item has already been added. Key in dictionary: 'Path'  Key being added: 'PATH'
workstation.default hook[init]:(HK):     STDERR:
workstation.default hook[init]:(HK):     ---- End output of C:\ProgramData\chocolatey/bin/choco.exe install -y vscode ----
workstation.default hook[init]:(HK):     Ran C:\ProgramData\chocolatey/bin/choco.exe install -y vscode returned 1

Indeed the commands that have failed work correctly when run as is in an elevated PowerShell window:

In case the text in the image is too small to see the error is:

I did think that something strange was happening when chef-client was run in the bundled PowerSHell core, however I have installed PSCore on the machine and run the same commands and no errors are thrown.

I have seen something like this a few years ago when I wrote POSHChef. It was down to the way in which items are deserialised. If you use the built in PowerShell cmdlet ConvertFrom-JSON in which "Path" and "PATH" are both valid elements in JSON this same error will be thrown.

However if the [System.Web.Script.Serialization.JavaScriptSerializer]::DeserializeObject() is used then "Path" and "PATH" can be made to be the same thing, although it will not be merged. (https://github.com/POSHChef/POSHChef/blob/master/functions/Exported/ConvertFrom-JSONtoHashtable.ps1).

If VSCode (or any app) is installed manually then the chocolatey_resource correctly detects that the app is installed.

Chef Version

This is being installed into the package using stuartpreston/chef-client package as is version 14.11.21`.

Platform Version

Windows: 10 1903
Habitat: 0.82.0

Replication

To replicate this issue please download by my repo at https://github.com/russellseymour/effortless and build the habitat package.

Then on a Windows 10 machine (use a local one or one in Azure for example):

• Install Habitat
• Run a service
• Habitat install the built package
• Load the service

After the package has initialised it will run chef-client and the issues will be seen as above.

When I first ran this I was only installing Visual Studio Code so thought that it might be something to do with that package. To test this I added another one and it failed as well (as seen above). To try and fix VSCode I added /DontAddToPath as a paramater option to Chocolatey but this had no effect either.

Stacktrace

Stack trace is in a Gist - https://gist.github.com/russellseymour/55d71343c5d34cd6f891af37886df5ed

[Tracking][scaffolding-chef-infra][windows] the scaffold policyfile path can't be overridden in a wrapper plan file.

The Invoke-DefaultBuild and Invoke-DefaultInstall set the $scaffold_policyfile_path at the top of the function to "$PLAN_CONTEXT..\policyfiles". This overrides a user's input from the wrapper plan for this value.

[scaffolding-chef-inspec] Scaffolding should provide a customizable habitat directory, similar to policyfile in scaffolding-chef-infra

Currently scaffolding-chef-inspec is hardcoded to expect the inspec build in the /habitat directory. A user may decide to make an arbitrary directory instead, such as example-app-audit/plan.sh.

Possible Solution

Provide a configuration option to point to this directory.

[Tracking] Test Kitchen Local Dev Workflow

This is a tracking issue to document a test kitchen local dev workflow with Effortless.

[scaffolding-chef-inspec] Accept license during build?

Hey all, in chef/scaffolding-chef-inspec inspec archive is called during build and requires that the license be accepted.

Currently in order to do this you must do CHEF_LICENSE='accept-no-persist' hab pkg build ..

What are your thoughts on a better UX for handing this? Maybe an error in do_build if the env var isn’t set with a descriptive error? Accept the license for the user for inspec archive only? Another habitat-esque way to pass in the license acceptance (maybe via default.toml?)

[Tracking][scaffolding-chef-infra] Add support for self-signed certificates for data collector reporting into Chef Automate

This is a tracking issue to support self-signed certificates for using a data collector token to report runs into Chef Automate.

Resolving this issue may include:

• Modifying the init hook to accept an optional public certificate to be bundled with the package, for the purpose of ssl verification
• Testing with a real customer environment to validate the use case

---

As a work around, I've been using my ssl_cacert_hack cookbook. Basically it expects to be wrapped and the wrapper cookbook to provide a ca-bundle.crt file in my_wrapper/files/default/ca-bundle.crt (for example), and links all of the cert files in /opt and /hab (configurable) to your "system" cabundle (ubuntu and centos have different paths).

Including this first in my runlist works, and we have nodes in A2! It seems a bit hamfisted... but it solved my problem... so "if it's dumb and it works..."?

See also: habitat-sh/core-plans#1799

[scaffolding-chef-inspec] [windows] report_to_stdout needs to be set to default variable if default.toml is overriden

effortless/scaffolding-chef-inspec/lib/scaffolding.ps1

Line 123 in 26b4a90

"stdout": {{cfg.report_to_stdout}}

The value of report_to_stdout does not get set to a default if the default.toml is overriden. This will result in a parsing error where the json provided to inspec is invalid.

Update default.toml example for Effortless Audit server_url

Currently there is an incorrect example for the default.toml in both the Windows and Linux examples folder for Effortless Audit.

This line:

https://github.com/chef/effortless/blob/master/scaffolding-chef-inspec/lib/scaffolding.ps1#L163

Calls for "url": "{{cfg.automate.server_url}}/data-collector/v0/",

And the default.toml shows this as the variable: server_url = "https://<automate_url>/data-collector/v0/"

Which ends up rendering as:

"https://your-a2-server/data-collector/v0//data-collector/v0/"

And should be fixed. I'll put in a PR after the template PR is merged.

[Tracking][scaffolding-chef-infra] Add Windows example for 'infra-windows-policyfile-cookbook'

This is an issue to follow up on the example infra-linux-policyfile-cookbook and to make a Windows counterpart to it.

[scaffolding_chef-inspec]Make verify_ssl a configuration option

Currently the inspec_exec_config.json file has a verify_ssl option in it that is set to false. This should be changed to a configuration option with a more secure default of true.

[design-proposal] Add new API: scaffold_security

scaffold_security API

Chef wants to ensure that users can make smart and simple choices of how security is used in Effortless. This design proposal elects to have a single setting for security, that enables users to choose sane and secure settings for their use-case.

Motivation

As a user of the Effortless pattern,
I want to have simple way to set up security,
so that I can have secure software and make intelligent choices about how I use Effortless.

Specification

The scaffold_security API will be implemented as a setting in a user plan. This is similar to existing settings such as scaffold_cacerts or scaffold_data_bags_path.

A user plan is a plan.sh or plan.ps1 file that consumes either chef/scaffolding-chef-infra or chef/scaffolding-chef-inspec using the pkg_scaffolding= setting.

This API is design to grow with the evolving needs of this project. The underlaying software will update and change over time, and this API is designed in such a way to be able to change those underlying settings, while maintaining the original intent of the user. For example, Chef InSpec may change the format of config.json, or Chef Infra may force SSL in some future version. In these cases, the API we provide does not need to change, enabling users to remain up to date with secure settings for their use-case.

This initial implementation proposes three settings, two of which are required to accept this design proposal, and one which can be implemented later.

Settings

secure

scaffold_security="secure"

This setting is required to be implemented to accept this proposal.

secure is the default setting, and will set all security settings to their maximum. At the time of this proposal, this includes settings such as: (This is not an exhaustive list)

1. Using best openssl settings
2. Using latest core/cacerts for all platforms
3. Setting SSL_CERT_FILE and SSL_CERT_DIR environment variables at runtime
4. Setting ssl_verify_mode :verify_peer in Chef Infra's client.rb
5. Setting "verify_ssl": true in Chef InSpec's config.json

evaluate

scaffold_security="evaluate"

This setting is required to be implemented to accept this proposal.

evaluate is an optional setting for the purposes of evaluating Chef's software in environments where enabling security is extremely costly. Some environments can take days (or weeks! or months!) to distribute certificates or have time-expensive, and non-automated audits that prevent security from being turned on. In these cases, it's useful to have a setting to evaluate the software without security constraints. Where possible, security settings will still automatically be enabled. evaluate should never be used in a production environment, or even in other lower level environments, it should only be constrained to a test lab for evaluation purposes only.

evaluate may disable some security settings, generate temporary (self-signed) certificates, or use other security settings that are not desirable. At the time of this proposal, this includes settings such as: (This is not an exhaustive list)

1. Setting ssl_verify_mode :verify_none in Chef Infra's client.rb
2. Setting "verify_ssl": false in Chef InSpec's config.json

fips

scaffold_security="fips"

This setting is optional to accept this proposal and may be implemented at another time.

fips is an optional setting for enabling users that must meet fips requirements within their organization. Where possible, this still uses the maximum secure settings. However, as many security researchers will note, fips is a less secure option in many cases.

fips may rebuild or change other security settings to a fips mode. At the time of this proposal this may include settings such as: (This is not an exhaustive list)

1. Using the fips canister for openssl dependant Habitat packages.
2. Setting fips in Chef Infra's client.rb

Downstream Impact

There should be no negative downstream impacts. This only impacts how users of Effortless interface with underlying settings.

Appendix

Acceptance and implementation of this design proposal closes #97

Build Badge display issues

Describe the problem

Build badge isn't being displayed correctly.

For anyone who isn't signed into the chef buildkite, the badge isn't being displayed correctly.

When visiting the link, getting prompted with the buildkite chef okta login.

[Tracking] Build Initial CI/CD for Linux and Windows

• Build expeditor workflow for releasing artifacts
• Build automated linting into PR workflow
• Research and discover BATS features necessary to do Linux testing
• Build Linux workflow, including scaffolding and downstream user package build, and a few valid test cases for Linux
• Test CACerts for Linux
• Research and discover Pester features necessary to do Windows testing
• Build Windows workflow, including scaffolding and downstream user package build, and a few valid test cases for Windows
• Test CACerts for Windows

---

CI/CD Platform Support

• Linux - scaffolding-chef-infra
• Linux - scaffolding-chef-inspec
• Windows - scaffolding-chef-infra
• Windows - scaffolding-chef-inspec

[scaffolding-chef-infra] Intigrate cookstyle cops in to the build test.

There has been a lot of work done to help users transition to Chef 15. One thing that helps with this is running cookstyle against a cookbook to see if it has bad patterns that would hinder a move to Chef 15. We should integrate cookstyle into the build so that we fail a build if we hit a bad pattern.

[scaffolding-chef-infra] Primary policyfile lockfile being deleted when `include_policy`

Originally from habitat-sh/core-plans by @qubitrenegade

Hello,

I am having an issue with the scaffolding-chef scaffolding, where it appears to be deleting the base.lockfile.json when including the base policy from another policyfile.

I've created a demo repo. The branch 2019-04-28-pt1-multiple-policy-files has been created with these changes.

I have a fairly standard plan.sh file.

And my policy file is:

# habichef_demo/policyfiles/us-east-1.rb
name 'us-east-1'
include_policy 'base', path: '.'
default['resolver']['domain'] = 'us-east-1.qubitrenegade.com'

When running a build, this fails with:

[2][default:/src:0]# CHEF_POLICYFILE=us-east-1 build
... snip ...
Building policy us-east-1
Error: Failed to generate Policyfile.lock
Reason: (ChefDK::LocalPolicyfileLockNotFound) The provided path ./base.lock.json does not exist.

If I build it first with:

[1][default:/src:0]# build
... snip ...

  habichef_demo-base: Installed Path: /hab/pkgs/qbrd/habichef_demo-base/0.1.0/20190428185221
   habichef_demo-base: Artifact: /src/results/qbrd-habichef_demo-base-0.1.0-20190428185221-x86_64-linux.hart
   habichef_demo-base: Build Report: /src/results/last_build.env
   habichef_demo-base: SHA256 Checksum: 4a126ec5c47dba1643ef767c7ccc02acd36ea780d2f88110db9c964c4598c2b9
   habichef_demo-base: Blake2b Checksum: 7fc45fcf07bebe57f90301c0c83df6700f772fc8925ec0162b1efbfa85b89b6a
   habichef_demo-base:
   habichef_demo-base: I love it when a plan.sh comes together.
   habichef_demo-base:
   habichef_demo-base: Build time: 0m5s
[2][default:/src:0]#

It creates a base.lockfile.json, but running CHEF_POLICYFILE=us-east-1 build again fails with the same error.

This appears to be the result of PR #2515, L99

 rm -f "${scaffold_policyfiles_path}"/*.lock.json

Thanks

• QBRD

[scaffolding-chef-infra] - Provide method to run InSpec on new chef-infra package

Summary

We have a chef-base package (hooked in with compliance and audit cookbooks) that utilizes the scaffolding-chef core-plan. We noticed an odd issue where our compliance nodes stopped reporting back to our A2 instance whenever we would increment the chef-base package version (e.g., 0.1.3 -> 0.1.4), and release it to stable channel. This manifested on both Windows and Linux machines that had been reporting compliance data to A2 previously (both machine types just stopped once the new chef-base package was released to stable, and the supervisor pulled it down for update on a local machine).

Repro

• Habitat Linux v0.71.0/20181218014130, Habitat Windows v0.71.0/20181218014218
• Create chef-base package using scaffolding-chef. Start at v0.1.3 for testing purposes for both Windows and Linux packages.
• Build & upload chef-base package for both Windows and Linux. Release to stable channel.
• Provision a Linux and Windows machine(s), installing the chef-base package on each local node.
• hab config apply our own TOML file with variables set to configure compliance data_collector:

interval = "60"
splay = "60"
log_level = "info"

[data_collector]
enable = "true"
token = "dead-beef"
server_url = "https://automate.example.com/data-collector/v0"

• Make sure each Linux and Windows node(s) report compliance data back to A2 successfully (they are).
• Review the census data from each node for chef-base.default (which matches our TOML above, as expected):

[centos@ip-10-10-1-164 svc]$ curl --silent -X GET http://10.10.1.180:9631/census -H 'Authorization: Bearer ea7-beef' | jq -r '.census_groups | ."chef-base.default" | .service_config'
{
  "incarnation": 1548279642,
  "value": {
    "data_collector": {
      "enable": "true",
      "server_url": "https://automate.example.com/data-collector/v0",
      "token": "dead-beef"
    },
    "interval": "60",
    "log_level": "info",
    "splay": "60"
  }
}

• Increment the plan.{sh,ps1} pkg_version to 0.1.4 for each Linux and Windows package, build both.
• Upload each new pkg version to builder, release to stable channel.
• Wait ~60-120 seconds for packages to get deployed on each node. Look at census data now on the same node:

[centos@ip-10-10-1-164 hooks]$ curl --silent -X GET http://10.10.1.180:9631/census -H 'Authorization: Bearer ea7-beef' | jq -r '.census_groups | ."chef-base.default" | .service_config'
{
  "incarnation": 1548281703,
  "value": {
    "chef_client_ident": "kmott/chef-base/0.1.4/20190123221359"
  }
}

When topology first starts, the packages run hook gets rendered to disk from the scaffolding, with an actual timestamp (at a single point in time) value like this:

echo "chef_client_ident = \"kmott/chef-base/0.1.4/20190123221359\"" | hab config apply chef-base.default 1548281703

That run hook then gets invoked, which works fine the first time around. I then do a hab config apply chef-base.default $(date +%s) compliance.toml of my own TOML data on top of that, which exposes the data_collector info, and things work fine--it works fine as long as the run hook doesn't get re-rendered to disk--even if that run hook gets re-invoked, it has an old timestamp, so the config will never be updated (which isn't really a problem, though it is a bit inconsistent).

Now, the new chef-base version is released, which causes the run hook to get re-rendered to disk (with a newer timestamp than before), and invoked. Since it has a new timestamp, my configuration that I set previously gets wiped out (I am assuming) by L71 of the above snippet.

Notes

There may be more going on here, but that's what I've come up with thus far. Is there something we can do to avoid this, short of not using the scaffolding-chef? Should we write our own scaffolding-chef? Or maybe do something else? Any help you can provide would be great!

[design-proposal] Add method for modifying attributes based on channels

Previous patterns of using Chef allowed for setting attributes based on environment/policy_group. In the Effortless pattern though both constructs cease to exist.

We need a way to configure attributes based on environments and/or channesl. For example, setting a syslog server for Production to be different than in Dev.

One way we could do this is keying off a known attribute at run time (e.g. node['effortless']['channel'] or node.effortless_channel). Then setting that via user.toml.

Let's discuss this and other options!

NOTE: Feel free to edit this to make more sense/add a TL;DR

[Tracking][scaffolding-chef-infra][scaffolding-chef-inspec] Configure reporting to Chef Automate Applications view

As part of the support for effortless, it would be nice to be able to have a way to configure reporting to the Automate Applications View. This would allow users to set a setting in the user.toml or defautl.toml that would report the health and status of a package build using the effortless scaffoldings.

[Tracking]Windows issue with build config when config folder present in user plan

This is an issue with the Invoke-BuildDefaultConfig. Right now we are building the config files as part of the Invoke-BuildDefaultService. We are creating a config directory in the expected place but when Habitat's build service runs it attempts to run a Copy-Item "$PLAN_CONTEXT/config" $pkg_prefix -Recurse and there is already a config directory at $pkg_prefix. There are two ways to fix this in the user plan. The first is to delete the config folder. The second is to override the Invoke-BuildConfig and only copy the individual config files that are in the user's config directory. Instead, it would be better to move the config build to the Invoke-BuildDefaultConfig so that there is no conflict and still preform the recursive copy.

[Tracking] Write CONTRIBUTING.md

This is a tracking issue to write a complete CONTRIBUTING.md!

[scaffolding-chef-inspec] Support external source of automate credentials during pkg build

Latest version of scaffolding-chef-inspec only accepts automate credentials defined in the plan. Credentials should not be stored in SCM.

Users should be able to use $HAB_STUDIO_SECRET_ environment variables to set these values at runtime (build time).

[scaffolding-chef-infra] CACerts in scaffolding may be override bundled Ruby certs

The scaffolding is currently pulling its own CACert bundle in, which may be undesired behavior because it may be overwriting the bundled CACerts in Ruby that the chef-client package delivers.

We need to investigate this further to prove it is true and create a replication case.

If this is true, we may be able to remove this: https://github.com/chef/scaffolding-chef/blob/master/scaffolding-chef/lib/scaffolding.sh#L81

[Tracking] [scaffolding-chef-infra] Windows paths in the non-docker studio cause chef export to fail

For Windows builds, long paths cause chef export to fail during the build.

A path might look like this:
C:/Users/ADMINI~1/AppData/Local/Temp/chefdk-export-3928-2019072317143620190723-3928-nlvye1/cookbook_artifacts/lul_win_security-5ce041cdafceb750d9ca318396c4c4fa411b9b71/files/default/cSecurityOptions/3.1.2/DSCResources/AccountAndBasicAuditing/AccountAndBasicAuditing.psm1

You can use a Docker studio to avoid this issue for now.

[scaffolding-chef-inspec] Add scaffold_cacerts option to inspec scaffolding

Just as we do in effortless-chef-infra, it would be nice to be able to append custom ca certs.

[design-proposal] Add support for testing

The effortless pattern should support testing of the components and be included as a best practice.

Describe the problem

Currently when building out an effortless project testing is an exercise left up to the user. There should be a best practice decided on and included in this repository to improve this experience because it's pretty important to end users.

Software Version

N/a

Possible Solution

Currently when testing effortless packages we do the following:

1. Build the habitat package
2. Use kitchen-vagrant to create a VM(s) with the results directory synced
3. Install the package on the VMs
4. Run the inspec tests

Open to other suggestions!

[scaffolding-infra-chef] Allow injecting SSL certs for `chef` commands

For example, allowing the use of a custom SSL cert file for fetching from an on prem Artifactory.

[scaffolding-chef-infra] [scaffolding-chef-inspec] Proxy issues with Effortless Build tools in Studio

As a user behind a corporate firewall and proxying requests to the internet, I need to be able for Effortless to proxy all requests from build tools (such as Chef-DK) to use a custom CACert chain at build time.

[Tracking] [scaffolding-chef-inspec] Builds fail in Windows docker studio due to InSpec Windows packaging shananigans

This is a tracking issue. No fix may be required in this repo, but further investigation is warrented.

To repro: enter a Windows docker studio and build any package that uses scaffolding-chef-inspec on Windows

hab studio enter -D
build

Sample Output

Traceback (most recent call last):
        21: from C:/hab/pkgs/stuartpreston/inspec/3.7.11/20190328052257/bin/inspec:23:in `<main>'
        20: from C:/hab/pkgs/stuartpreston/inspec/3.7.11/20190328052257/bin/inspec:23:in `load'
        19: from C:/hab/pkgs/stuartpreston/inspec/3.7.11/20190328052257/lib/ruby/gems/2.6.0/gems/inspec-3.7.11/bin/inspec:10:in `<top (required)>'
        18: from C:/hab/pkgs/stuartpreston/inspec/3.7.11/20190328052257/lib/ruby/gems/2.6.0/gems/inspec-3.7.11/bin/inspec:10:in `require_relative'
        17: from C:/hab/pkgs/stuartpreston/inspec/3.7.11/20190328052257/lib/ruby/gems/2.6.0/gems/inspec-3.7.11/lib/inspec.rb:12:in `<top (required)>'
        16: from C:/hab/pkgs/stuartpreston/inspec/3.7.11/20190328052257/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        15: from C:/hab/pkgs/stuartpreston/inspec/3.7.11/20190328052257/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        14: from C:/hab/pkgs/stuartpreston/inspec/3.7.11/20190328052257/lib/ruby/gems/2.6.0/gems/inspec-3.7.11/lib/inspec/profile.rb:17:in `<top (required)>'
        13: from C:/hab/pkgs/stuartpreston/inspec/3.7.11/20190328052257/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        12: from C:/hab/pkgs/stuartpreston/inspec/3.7.11/20190328052257/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        11: from C:/hab/pkgs/stuartpreston/inspec/3.7.11/20190328052257/lib/ruby/gems/2.6.0/gems/inspec-3.7.11/lib/inspec/profile_context.rb:8:in `<top (required)>'
        10: from C:/hab/pkgs/stuartpreston/inspec/3.7.11/20190328052257/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
         9: from C:/hab/pkgs/stuartpreston/inspec/3.7.11/20190328052257/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
         8: from C:/hab/pkgs/stuartpreston/inspec/3.7.11/20190328052257/lib/ruby/gems/2.6.0/gems/inspec-3.7.11/lib/inspec/control_eval_context.rb:4:in `<top (required)>'
         7: from C:/hab/pkgs/stuartpreston/inspec/3.7.11/20190328052257/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
         6: from C:/hab/pkgs/stuartpreston/inspec/3.7.11/20190328052257/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
         5: from C:/hab/pkgs/stuartpreston/inspec/3.7.11/20190328052257/lib/ruby/gems/2.6.0/gems/inspec-3.7.11/lib/inspec/dsl.rb:6:in `<top (required)>'
         4: from C:/hab/pkgs/stuartpreston/inspec/3.7.11/20190328052257/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
         3: from C:/hab/pkgs/stuartpreston/inspec/3.7.11/20190328052257/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
         2: from C:/hab/pkgs/stuartpreston/inspec/3.7.11/20190328052257/lib/ruby/gems/2.6.0/gems/inspec-3.7.11/lib/inspec/plugin/v2.rb:30:in `<top (required)>'
         1: from C:/hab/pkgs/stuartpreston/inspec/3.7.11/20190328052257/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
C:/hab/pkgs/stuartpreston/inspec/3.7.11/20190328052257/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require': Invalid argument @ rb_readlink - C:/src (Errno::EINVAL)
   effortless-a2-profile-test: Clean the cache


    Directory: C:\hab\cache\src


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----        7/23/2019  10:20 AM                effortless-a2-profile-test-0.1.0
   effortless-a2-profile-test: Setting env:LIB=
   effortless-a2-profile-test: Setting env:INCLUDE=
   effortless-a2-profile-test: Preparing to build
   effortless-a2-profile-test: Building

Cannot fetch compliance://admin/stig-windowsserver2016-i-missioncriticalpublic because your compliance token has not been
configured.

Please login using

    inspec compliance login https://your_compliance_server --user admin --insecure --token 'PASTE TOKEN HERE'
   effortless-a2-profile-test: Installing
Copy-Item : Cannot find path 'C:\hab\cache\src\effortless-a2-profile-test-0.1.0\effortless-a2-profile-test-0.1.0.tar.gz' because it does not exist.
At C:\hab\pkgs\echohack\scaffolding-chef-inspec\0.10.0\20190723101649\lib\scaffolding.ps1:109 char:5
+     Copy-Item "$HAB_CACHE_SRC_PATH/$pkg_dirname/$pkg_name-$pkg_versio ...
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : ObjectNotFound: (C:\hab\cache\sr...st-0.1.0.tar.gz:String) [Copy-Item], ItemNotFoundException
+ FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.CopyItemCommand