devblackops / watchmen

Infrastructure test runner using Pester and OVF modules

Infrastructure test runner and notification system using Operation Validation Framework (OVF) PowerShell modules and Pester.

Overview

Watchmen is a PowerShell module to make executing Pester tests contained in OVF modules easier using a simple PowerShell-based Domain Specific Language (DSL). It also provides the ability to execute a number of actions (notifiers) upon failing (or successful) infrastructure tests. Watchmen can also dynamically install OVF modules from public or private PowerShell repositories like the PowerShell Gallery should the module not be found on the system.

More information about creating Watchmen files and executing notifiers can be found at http://watchmen.readthedocs.io/en/latest/

Installation

The easiest and prefered way to install Watchmen is via the PowerShell Gallery. To use the PowerShell Gallery, you must be on Windows 10, have PowerShell 5, or PowerShell 3 or 4 with the PowerShellGet module. See the PowerShell Gallery Getting Started page for more information. Run the following command to install Watchmen and the two dependent modules PSSlack and Posh-SYSLOG.

Install-Module -Name Watchmen -Repository PSGallery

As an alternative, you can clone this repository the a location on your system and copy the subfolder Watchmen to C:\Program Files\WindowsPowerShell\Modules\. If you downloaded the repository as a ZIP file, extract it, and copy the subfolder Watchmen to C:\Program Files\WindowsPowerShell\Modules\. If you copied Watchmen to the Modules folder manually, you must ensure both the depended modules PSSlack and Posh-SYSLOG are installed there as well.

What is OVF?

The Operation Validation Framework is a PowerShell module used to execute Pester tests that validate the operation of a system rather than test the behavior of code. This could mean running tests that validate certain Windows services are running and IIS is responding on port 443.

An OVF module is a PowerShell module that includes Pester tests in a certain folder structure:

• ModuleBase\
  • ModuleName.psd1
  • Diagnostics\
    • Simple\
      • simple.tests.ps1 simple tests (e.g., service checks, endpoint checks)
    • Comprehensive\
      • comprehensive.tests.ps1 comprehensive scenario tests

Example infrastructure tests

• Service 'W3SVC' is running
• Localhost responds on port 443 with HTTP code 200.
• All volumes have a minimum of 5GB free space.

Why OVF?

Pester tests packaged into a PowerShell module gain the immediate benefit of being versionable just like any other PowerShell module. They also are easily publishable to public or private NuGet-based repositories like the PowerShell Gallery. This facilitates high quality test modules that validate common infrastructure to be shared and improved upon by the broader community.

Example Watchmen File

The example Watchmen file below will execute Pester tests contained inside the MyAppOVF module installed on the location machine. Upon any failing (or optionally successful) tests, Watchmen will then execute a number of notifiers such as sending an email, writing to the eventlog, appending to a log file, executing an arbitrary PowerShell script block or script, sending a message to a Slack channel, and send a message to a syslog server.

myapp.watchmen.ps1

# Global notifiers that are executed upon any failing test
WatchmenOptions {
    notifies {
        When 'OnFailure'
        email @{
            fromAddress = '[email protected]'
            smtpServer = 'smtp.mydomain.tld'
            port = 25
            subject = 'Watchmen alert - #{computername} - [#{test}] failed!'
            to = '[email protected]'
        }
        eventlog @{
            eventid = 1
            eventtype = 'error'
        }
        eventlog @{
            eventid = 100
            eventtype = 'information'
        } -when 'onsuccess'
        logfile '\\fileserver01.mydomain.tld\monitoringshare\#{computername}.log'
        powershell {
            Write-Host "Something bad happended! $args[0]"
        }
        powershell '\notifier.ps1'
        slack @{
            Token = '<webhookurl>'
            Channel = '#Watchmen'
            AuthorName = $env:COMPUTERNAME
            PreText = 'Everything is on :fire:'
            IconEmoji = ':fire:'
        }
        syslog 'syslog.mydomain.tld' -when 'always'
    }
}

# Execute the 'Storage.Capacity' tests in version 1.0.0 of the 'MyAppOVF' module
WatchmenTest 'MyAppOVF' {
    version 1.0.0                   # Execute tests from a specific version of the module. Default is latest 
    testType 'Simple'               # Valid values 'simple', 'comprehensive', 'all'. Default is 'all'
    test 'Storage.Capacity'         # Name of test to execute. Default is '*'
    fromSource 'PSGallery'          # Name of PowerShell repository to install module from if not found on system.
    parameters {                    # Parameters that are passed into the Pester script to change the behaviour of the test.
        FreeSystemDriveThreshold = 40000
    }
    notifies {                      # Notifiers to execute for this test in addition to ones defined in 'WatchmenOptions'
        logfile '\\fileserver01.mydomain.tld\monitoringshare\#{computername}.log' -when 'always'
    }
}

# Execute all tests in the SystemOVF module and install module from the 'PSPrivateGallery' repository if not installed on the system.
# Global notifiers will be executed upon any failing tests.
WatchmenTest 'SystemOVF' {
    fromSource 'PSPrivateGallery'
}

Using Watchmen

A Watchmen file is a PowerShell script that can be read by calling Get-WatchmenTest. The object(s) returned represent the OVF tests to execute and the associated notifiers to call upon any failing tests. Running Invoke-WatchmenTest will execute the tests and call any notifiers as appropriate.

Getting watchmen tests

$tests = Get-WatchmenTest -Path '.\myapp.watchmen.ps1'

Executing watchmen tests

$tests | Invoke-WatchmenTest -Verbose -IncludePesterOutput

watchmen's Issues

Watchmen variable used to track state is in $global scope

Current Behavior

Currently the Watchmen variable used to track state as it parses the watchmen file is in the $global scope.

Expected Behavior

Variable should be in $script scope.

Possible Solution

Replace all references to $global:Watchmen with $script:Watchmen

Context

Modules putting things in $global scope is messy. This variable is only used internally in the module and should not be exposed to the outside world.

Your Environment

• Module version used: 1.0.0
• Operating System and PowerShell version:
  • OS: Windows 10
  • PS version: 5.1.14366.0

Add capability to optionally execute notifier(s) on successful Pester test runs

There may be cases where you want certain notifiers to be called even for successful Pester tests. I propose that a new parameter be added to each notifier as well as a global option that tells Watchmen when the notifier should be executed.

Expected Behavior

A new option would be added to tell Watchmen under what conditions the notifier(s) would be executed.

Current Behavior

Watchmen only executes notifiers on test failures.

Possible Solution

Add new parameter called When to each notifier function as well as a global option under Notifies. The default option for this parameter would be OnFailure. Omitting this parameter would not change how Watchmen currently behaves.

WatchmenOptions {
    notifies {
        when 'onFailure'
        eventlog @{
            eventid = '1'
            eventtype = 'information'
        } -when 'OnSuccess'
        syslog 'syslog.mydomain.tld' -when 'always'
    }
}

Context

I'm thinking about adding a new notifier to send Pester test results to InfluxDB in the form of metrics. These metrics would be of the form:

watchmen.test,hostname=myhost,environment=dev,test=My\ Test value=0i 1472528009

The value of the metric would be 0 for test passes and 1 for test failures. Once these types of metrics are in InfluxDB, a dashboard in Grafana can be setup to show the Pester test status as a series of red/green badges.

Docs a little unclear for novices like me.

I'm looking forward to using your module but I some things clarified.

1. Where does 'myapp.watchmen.ps1' go? Does that go in the root of the MyAppOVF module folder?
2. What is the purpose of WatchmenTest 'SystemOVF' at the end of the example file? Is that demonstrating some kind of dependency? Or just simply a demonstration of how one Watchmen definition file can be used to test multiple OVF modules? If so, this doesn't seem very PowerShell-y to me. For example, I would expect something more like Invoke-Watchmen -OVFModule MyAppOVF -WatchmenDefinition MyApp.Watchmen.ps1.

Again, really looking forward to using this! I hope my feedback is valuable to you.

Fix a number of issues with Watchmen (PR #10)

PR for #10

Adds a ConvertFrom-PlaceholderString function to clean up redundant code and makes the
placeholder system more useful. Resolves a number of issues with Email and Logfile notifiers.

Issues:

• UseSSL switch for EMail notifier was hardcoded ($true)
• Custom Message attribute for EMail notifier didn't get sent
• Placeholders in LogFile notifier didn't get replaced (contrary to the project's README.md)

Fix Typo

Descibe -> Describe