intuit / cloudraider

A resiliency tool that automates Failure mode effect analysis tests, simplifying complex testing with a behavior-driven development and testing approach. Provides a programmatic way to execute controlled failures in AWS and a BDD way to write test cases, allowing test plans themselves to become test cases that can be executed as is.

Introduction to Cloud-Raider

CloudRaider is a new Testing Framework to perform "Failure Mode Effect Analysis" (FMEA) testing in AWS. Cloud Raider also provide behavior driven testing approach via Cucumber framework. Cloud Raider provides a programmatic way to execute controlled failures, and also recover quickly back to normal. As it provides BDD way to write test cases, your FMEA test plan can itself become your test cases that can be executed as is.

Chaos Engineering

The Cucumber Client also provides functionality for running automated FMEA tests through the principles of Chaos Engineering. To learn more, Please review here

Getting Started

Install JDK

Install Java 8 JDK JDK 8 Doc: http://docs.oracle.com/javase/8/docs/ JDK 8 Download: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html Java Version: JDK 1.8.0_51 You can check this by typing in the command line: java –version (Java 8 is required for the build) Set the JAVA_HOME system environment variable to /System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home Add $JAVA_HOME to the PATH system environment variable

Install Maven

Maven 3.0.x is available by default in Mac OS X(3.0.3 in Lion,3.0.2 in Snow Leopard after Java for Mac OS X 10.6 update 4) Verify this using mvn –version from the command line To upgrade to Maven 3.2.1, do the following Download the Maven 3.2.1 binary zip from http://maven.apache.org/download.html. Unzip the downloaded file (apache-maven-3.2.1-bin.zip) to the desired location (e.g., /Applications). Set the M2_HOME system environment variable to the installation directory (e.g., /Applications/apache-maven-3.2.1). Prepend $M2_HOME/bin to the PATH system environment variable. From the command line, verify that Maven has been installed correctly: mvn --version

AWS Credentials

A prerequisite to running any of the test cases above is, setting up AWS credentials. For details on how to setup credentials please follow http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html For testing with credentials profile files, update ~/.aws/credentials and ~/.aws/config files. The validator uses the standard as credentials resolving chain. This means that we can use environment variables such as AWS_DEFAULT_PROFILE or command line such as aws.profile. see aws docs for all options and syntax

Test Client

Please use the example test module to get familiar about how to create and execute test cases

Credentials can be provided in config, Environment variables or the standard .aws/config file

Cucumber Example

Please review here

Java Example

Please review here

Installing Cucumber Plugin

IntelliJ

https://www.jetbrains.com/help/idea/cucumber.html

Eclipse

https://shankargarg.wordpress.com/2016/05/15/integrating-eclipse-with-cucumber-natural-plugin/

AWS Simple System Manager

CloudRaider supports execution of commands on EC2 hosts via AWS Simple System Manager (SSM) You can find more information about SSM here

cloudraider's Issues

Documentation update for first time users

Describe the bug

First time users are unable to figure out the mvn commands to build the library and run the scenarios with the client

Expected behavior

A precise step by step guide for first time users to execute their first failure

Port block scenario fails when run with ssm

Describe the bug

ssm servers are blocked when performing network failure scenarios, hence the scenario fails

To Reproduce

Run a network failure scenario which blocks ssm messaging

Expected behavior

SSM has to be bypassed from the failure as failing the communication channel breaks the test.

Add Onboarding Guide for First Time Users

The documentation needs to be structured with the basic steps including the following:

• First Time Users contributing to the app
• First Time Users/Services using Cucumber Client
• First Time Users/Services using Java Client

These individual documentations can exist in their respective directories.

Fix PR build

Describe the bug

PR build is broken

To Reproduce

create a PR

Expected behavior

PR build succeeds on tests passing

Screenshots

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Additional context

travis install dependency fails

Validate the aws account number which is worked on

Is your feature request related to a problem? Please describe.

Ppl often work with multiple aws accounts and often run scenarios or api calls inadvertently on the wrong aws account.

Describe the solution you'd like

Take acc no in the config and validate before performing any operations

Describe alternatives you've considered

This is usefull for a generic scanario where the names of ELB/ec2/tags are same across accounts

Include test coverage measurement & reporting tools

Add required tools to report and measure test coverage and put it back as label/hook on README.

Automate running FMEA test suite using cucumber client

Write a configurable script (shell, etc.) to run fmea tests using cucumber client.
Script should accept the required parameters of AWS environment and should be able to run FMES tests on the service with minimal efforts required from first time consumer of Cloud Raider.

Put all images under common directory and refine directory structure

Refine overall directory structure of the repo and put all images under single directory.

Automate setup of mock service on AWS and run FMEA suite on that service

We want to setup a mock service on AWS. Looking for ways to automate this entire setup.

On this mocked service setup, we want to run FMEA suite using cloud raider.

Explore and Setup CI/CD pipeline to run core library code

Following things should be achieved:

• Explore way to run Cloud Raider Core Library in Pipeline
• Setup CI/CD pipeline

Update the Java Version of the Core Cloud Raider Library

Make the core Cloud Raider library compatible with the latest Java version.

Documentation for Cloud Raider Core Library

Is your feature request related to a problem? Please describe.

Create an exhaustive documentation of all the features given by cloud raider core library.

Describe the solution you'd like

Documentation can be created under this specific folder as markdown file.

Blocking port script fails

Describe the bug

When trying to block port the command fails to do what is expected and errors out

To Reproduce

run the port block feature

expected port block but got this message:

iptables v1.4.21: invalid port/service {0}' specified Try iptables -h' or 'iptables --help' for more information.
iptables v1.4.21: Port "{0}" does not resolve to anything.

Support having multiple config files

Is your feature request related to a problem? Please describe.

We work with multiple aws accounts and the ability to have multiple config files is really helpfull with each config file for each account

Describe the solution you'd like

support passing config files in command line as mvn param

Describe alternatives you've considered

Having a single config file for multiple accounts in confusing and error prone.