Coder Social home page Coder Social logo

chaostoolkit-ibm's Introduction

This project is a collection of actions and probes, gathered as an extension to the Chaos Toolkit. It targets the IBM Cloud platform.

Build Status

This project is a collection of actions and probes, gathered as an extension to the Chaos Toolkit.

The following guidelines for chaostoolkit-ibmcloud

  • All resources can be located with information available directly on the IBM cloud portal, such as Instance ID, Volume ID, try to avoid ID that is hard to locate such as (Volume attachment ID)
  • Instance ID is unique and available directly on IBM Cloud; try to use it to located other resources; for example, if you need an IP address, try to extract it from IP or floating IP.
  • Do not delete resources unless it is the only way, try to stop, change the configuration to achieve the task, for example
    • Stop Virtual Instances instead of deleting it.
    • detach the volume
    • update LoadBalancer backend pool or listener instead.
  • Ensure to provide the rollback action, for any action, for example
    • Start Server after stopped it
    • Attach the volume after detaching it.
  • Tag any resources after executing the action with the following tags "chaostoolkit", "action-name"
  • Add the required busy-wait in action itself as an option instead of depending on Pause only.
  • Assume the default settings of Cloud environment as default parameters, still more advanced parameters can be added as optional, for example
    • Each Virtual Instance, by default, has one Local interface Loopback and one NIC for external access

Not all those principles are applied yet to the current code but still work on Progress to apply them

Probes

Category probe Description
VPC get_virtual_servers check the status of Virtual Instance
Cloud Pak probe_mq put and get message IBM MQ

Actions

Category Action Description
VPC create_instance_action based on the type you can start, stop , restart Server
VPC delete_load_balancer Delete Loadbalancer
VPC remove_volume_from_instance De-attach Volume from virtual Instance
VPC add_volume_to_instance Attach Volume to virtual Instance
VPC start_multiple_instances turn on one or more Virtual Machines
VPC stop_multiple_instances turn off one or more virtual machines
VPC delete_load_balancer_members Delete Load Balancer Members
VPC cordon_subnet Isolate a complete subnet via adding ACL with denyall
VPC uncordon_subnet restoring the orignal ACL should run after cordon_subnet
Linux add_network_latency Add Network Latency
Linux add_drop_packet drop packets sent to specific IP
Container detach_volume_from_worker De-attach volume from VPC IKS/ROKS Worker Nodes
Container attach_volume_to_worker attach from to a VPC IKS/ROKS Worker node

Install

This package requires Python 3.5+

To be used from your experiment, this package must be installed in the Python environment where chaostoolkit already lives.

$ pip install -U chaostoolkit-ibmcloud

Usage

To use the probes and actions from this package, add the following to your experiment file:

{
    "version": "1.0.0",
    "title": "What is the impact of stopping Server?",
    "description": "This a dummy experiment.",
    "tags": [
        "tls"
    ],
    "configuration": {
        "api_key": "<api-key>",
        "generation": "2",
        "service_url": "https://jp-tok.iaas.cloud.ibm.com/v1"
    },
    "steady-state-hypothesis": {
        "title": "Application responds",
        "probes": [
            {
                "type": "probe",
                "name": "Check the a machine is running",
                "tolerance": {
                    "type": "jsonpath",
                    "path": "$[0].status",
                    "expect": "running"
                },
                "provider": {
                    "type": "python",
                    "module": "ibmcloud.vpc.probes",
                    "func": "get_virtual_servers",
                    "arguments": {
                        "name": "windows-jump"
                    }
                }
            }
        ]
    },
    "method": [
        {
            "type": "action",
            "name": "Stop Windows Jump Server",
            "provider": {
                "type": "python",
                "module": "ibmcloud.vpc.actions",
                "func": "create_instance_action",
                "arguments": {
                    "instance_id": "02g7_af1ee383-a6eb-4ed5-b9b7-fdbce23a25c2",
                    "type": "stop",
                    "force": false
                }
            },
            "pause": {
                "after": 30
            }
        },
        {
            "ref": "Check the a machine is running"
        }

    ],
    "rollbacks": [
        {
            "type": "action",
            "name": "Re-start Windows Jump Server",
            "provider": {
                "type": "python",
                "module": "ibmcloud.vpc.actions",
                "func": "create_instance_action",
                "arguments": {
                    "instance_id": "02g7_af1ee383-a6eb-4ed5-b9b7-fdbce23a25c2",
                    "type": "start",
                    "force": true
                }
            },
            "pause": {
                "after": 60
            }
        }
    ]
}

That's it!

Please explore the code to see existing probes and actions.

Configuration

Contribute

If you wish to contribute more functions to this package, you are more than welcome to do so. Please, fork this project, make your changes following the usual PEP 8 code style, sprinkling with tests and submit a PR for review.

Develop

If you wish to develop on this project, make sure to install the development dependencies. But first, create a virtual environment and then install those dependencies.

$ pip install -r requirements-dev.txt -r requirements.txt

Then, point your environment to this directory:

$ pip install -e .

Now, you can edit the files and they will be automatically be seen by your environment, even when running from the chaos command locally.

Test

To run the tests for the project execute the following:

$ pytest

chaostoolkit-ibm's People

Contributors

ahmadsayed avatar haythamelkhoja avatar stevemar avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

chaostoolkit-ibm's Issues

Action instances failed with status code 400: Provided API key could not be found

Hi, 

I'm trying to stop&start instance on internal IBMCloud staging environment. But facing error for API key not found. 
{
  "version": "1.0.0",
  "title": "What is the impact of stopping Server?",
  "description": "This a dummy experiment.",
  "tags": [
    "tls"
  ],
  "configuration": {
    "api_key": "QNu-ASQ-#########################",
    "generation": "2",
    "service_url": "https://us-south-stage01.iaasdev.cloud.ibm.com/v1"
  },
  "method": [
    {
      "type": "action",
      "name": "Stop chaos test server",
      "provider": {
        "type": "python",
        "module": "ibmcloud.vpc.actions",
        "func": "create_instance_action",
        "arguments": {
          "instance_id": "0716_782c4c57-ebb8-4bc4-ab75-#########",
          "type": "stop",
          "force": false,
          "tag": true
        }
      },
      "pauses": {
        "after": 30
      }
    }

  ]
}

Log Output

(chaos-test) root@jinesh-test-instance:~/chaostoolkit-ibm/example/vpc# chaos run shutdown-start.json
[2022-01-06 13:52:10 WARNING]
    There is a new version (1.11.0) of the chaostoolkit available.
    You may upgrade by typing:

    $ pip install -U chaostoolkit

    Please review changes at https://github.com/chaostoolkit/chaostoolkit/blob/master/CHANGELOG.md

[2022-01-06 13:52:10 INFO] Validating the experiment's syntax
[2022-01-06 13:52:10 INFO] Experiment looks valid
[2022-01-06 13:52:10 INFO] Running experiment: What is the impact of stopping Server?
[2022-01-06 13:52:10 INFO] Steady-state strategy: default
[2022-01-06 13:52:10 INFO] Rollbacks strategy: default
[2022-01-06 13:52:10 INFO] No steady state hypothesis defined. That's ok, just exploring.
[2022-01-06 13:52:10 INFO] Playing your experiment's method now...
[2022-01-06 13:52:10 INFO] Action: Stop chaos test server
[2022-01-06 13:52:10 ERROR] Action instances failed with status code 400: Provided API key could not be found
[2022-01-06 13:52:10 INFO] Pausing after activity for 30s...
[2022-01-06 13:52:41 INFO] Let's rollback...
[2022-01-06 13:52:41 INFO] Rollback: Re-start chaos test Server
[2022-01-06 13:52:41 INFO] Action: Re-start chaos test Server
[2022-01-06 13:52:41 ERROR] Action instances failed with status code 400: Provided API key could not be found
[2022-01-06 13:52:41 INFO] Pausing after activity for 60s...
[2022-01-06 13:53:41 INFO] Experiment ended with status: completed

Please help me out . Thank you

Setup instructions failing ... How to resolve?

Not sure if docs need update or I'm doing something wrong, I'm not able to install using pip:

(chaostk) stevens-mbp:chaostoolkit-ibm smc2$ pip install -U chaostoolkit-ibmcloud
ERROR: Could not find a version that satisfies the requirement chaostoolkit-ibmcloud (from versions: none)
ERROR: No matching distribution found for chaostoolkit-ibmcloud
WARNING: You are using pip version 21.1.3; however, version 21.3.1 is available.
You should consider upgrading via the '/Users/smc2/.venvs/chaostk/bin/python3 -m pip install --upgrade pip' command.
(chaostk) stevens-mbp:chaostoolkit-ibm smc2$

also can't install the pre-reqs for the dev environment:

(chaostk) stevens-mbp:chaostoolkit-ibm smc2$ pip --version
pip 21.1.3 from /Users/smc2/.venvs/chaostk/lib/python3.9/site-packages/pip (python 3.9)
(chaostk) stevens-mbp:chaostoolkit-ibm smc2$ pip install -r requirements-dev.txt -r requirements.txt
Collecting coverage
  Downloading coverage-6.2-cp39-cp39-macosx_10_9_x86_64.whl (179 kB)
     |████████████████████████████████| 179 kB 1.4 MB/s 
Collecting pycodestyle
  Downloading pycodestyle-2.8.0-py2.py3-none-any.whl (42 kB)
     |████████████████████████████████| 42 kB 2.2 MB/s 
Collecting pytest>=2.8
  Downloading pytest-6.2.5-py3-none-any.whl (280 kB)
     |████████████████████████████████| 280 kB 6.7 MB/s 
Collecting pytest-cov
  Downloading pytest_cov-3.0.0-py3-none-any.whl (20 kB)
Collecting pytest-sugar
  Downloading pytest-sugar-0.9.4.tar.gz (12 kB)
Requirement already satisfied: requests in /Users/smc2/.venvs/chaostk/lib/python3.9/site-packages (from -r requirements-dev.txt (line 6)) (2.26.0)
Collecting requests_mock
  Downloading requests_mock-1.9.3-py2.py3-none-any.whl (27 kB)
Collecting ibm-vpc==0.7.0
  Downloading ibm-vpc-0.7.0.tar.gz (212 kB)
     |████████████████████████████████| 212 kB 31.1 MB/s 
Collecting jsonpath2==0.4.4
  Downloading jsonpath2-0.4.4-py3-none-any.whl (34 kB)
Collecting logzero==1.5.0
  Downloading logzero-1.5.0-py2.py3-none-any.whl (14 kB)
Collecting paramiko==2.7.2
  Downloading paramiko-2.7.2-py2.py3-none-any.whl (206 kB)
     |████████████████████████████████| 206 kB 20.1 MB/s 
Collecting ibm_platform_services==0.19.1
  Downloading ibm-platform-services-0.19.1.tar.gz (216 kB)
     |████████████████████████████████| 216 kB 25.0 MB/s 
Collecting ibm-cloud-sdk-core==3.10.1
  Downloading ibm-cloud-sdk-core-3.10.1.tar.gz (40 kB)
     |████████████████████████████████| 40 kB 14.1 MB/s 
Collecting mysql-connector-python~=8.0.23
  Downloading mysql_connector_python-8.0.27-py2.py3-none-any.whl (341 kB)
     |████████████████████████████████| 341 kB 19.2 MB/s 
Collecting py~=1.10.0
  Downloading py-1.10.0-py2.py3-none-any.whl (97 kB)
     |████████████████████████████████| 97 kB 13.6 MB/s 
Collecting colorama~=0.4.4
  Downloading colorama-0.4.4-py2.py3-none-any.whl (16 kB)
Collecting cryptography~=3.4.6
  Downloading cryptography-3.4.8-cp36-abi3-macosx_10_10_x86_64.whl (2.0 MB)
     |████████████████████████████████| 2.0 MB 20.7 MB/s 
Collecting wheel~=0.36.2
  Using cached wheel-0.36.2-py2.py3-none-any.whl (35 kB)
Collecting attrs~=20.3.0
  Downloading attrs-20.3.0-py2.py3-none-any.whl (49 kB)
     |████████████████████████████████| 49 kB 13.1 MB/s 
Collecting six~=1.15.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting PyJWT~=2.0.1
  Downloading PyJWT-2.0.1-py3-none-any.whl (15 kB)
Collecting requests
  Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
     |████████████████████████████████| 61 kB 18.9 MB/s 
Collecting toml~=0.10.2
  Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)
Requirement already satisfied: PyYAML~=5.4.1 in /Users/smc2/.venvs/chaostk/lib/python3.9/site-packages (from -r requirements.txt (line 18)) (5.4.1)
Collecting pluggy~=0.13.1
  Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting iniconfig~=1.1.1
  Downloading iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Collecting atomicwrites~=1.4.0
  Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting numpy~=1.20.1
  Downloading numpy-1.20.3-cp39-cp39-macosx_10_9_x86_64.whl (16.1 MB)
     |████████████████████████████████| 16.1 MB 1.1 MB/s 
Collecting certifi~=2020.12.5
  Downloading certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
     |████████████████████████████████| 147 kB 29.7 MB/s 
Collecting chardet~=4.0.0
  Downloading chardet-4.0.0-py2.py3-none-any.whl (178 kB)
     |████████████████████████████████| 178 kB 33.3 MB/s 
Requirement already satisfied: urllib3~=1.26.3 in /Users/smc2/.venvs/chaostk/lib/python3.9/site-packages (from -r requirements.txt (line 25)) (1.26.7)
Collecting coverage
  Downloading coverage-5.5-cp39-cp39-macosx_10_9_x86_64.whl (207 kB)
     |████████████████████████████████| 207 kB 37.7 MB/s 
Collecting python-dateutil~=2.8.1
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting bcrypt~=3.2.0
  Downloading bcrypt-3.2.0-cp36-abi3-macosx_10_9_x86_64.whl (31 kB)
Collecting idna~=2.10
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting pyparsing~=2.4.7
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting pycparser~=2.20
  Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
     |████████████████████████████████| 118 kB 37.3 MB/s 
Collecting websocket-client~=0.48.0
  Downloading websocket_client-0.48.0-py2.py3-none-any.whl (198 kB)
     |████████████████████████████████| 198 kB 28.0 MB/s 
Collecting setuptools~=40.8.0
  Downloading setuptools-40.8.0-py2.py3-none-any.whl (575 kB)
     |████████████████████████████████| 575 kB 24.2 MB/s 
Collecting click~=7.1.2
  Using cached click-7.1.2-py2.py3-none-any.whl (82 kB)
Collecting chaostoolkit~=1.9.0
  Downloading chaostoolkit-1.9.6-py2.py3-none-any.whl (19 kB)
Collecting packaging~=20.9
  Downloading packaging-20.9-py2.py3-none-any.whl (40 kB)
     |████████████████████████████████| 40 kB 14.6 MB/s 
Collecting termcolor~=1.1.0
  Using cached termcolor-1.1.0.tar.gz (3.9 kB)
Collecting zipp~=3.4.0
  Downloading zipp-3.4.2-py3-none-any.whl (5.3 kB)
Collecting pymqi~=1.12.0
  Downloading pymqi-1.12.0.tar.gz (75 kB)
     |████████████████████████████████| 75 kB 11.9 MB/s 
    ERROR: Command errored out with exit status 1:
     command: /Users/smc2/.venvs/chaostk/bin/python3 -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/s4/cr5tk5s150j8t4362bbmp6s40000gp/T/pip-install-59dy79i4/pymqi_cd9419ba7c284ec79c28feee1ea4b677/setup.py'"'"'; __file__='"'"'/private/var/folders/s4/cr5tk5s150j8t4362bbmp6s40000gp/T/pip-install-59dy79i4/pymqi_cd9419ba7c284ec79c28feee1ea4b677/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/s4/cr5tk5s150j8t4362bbmp6s40000gp/T/pip-pip-egg-info-a3430w3d
         cwd: /private/var/folders/s4/cr5tk5s150j8t4362bbmp6s40000gp/T/pip-install-59dy79i4/pymqi_cd9419ba7c284ec79c28feee1ea4b677/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/s4/cr5tk5s150j8t4362bbmp6s40000gp/T/pip-install-59dy79i4/pymqi_cd9419ba7c284ec79c28feee1ea4b677/setup.py", line 165, in <module>
        raise Exception('MQ libraries could not be found')
    Exception: MQ libraries could not be found
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/55/33/c567a624e7c09a5ab50352dc2aa586fbdf75fbe185d55a2d1d027fe685b2/pymqi-1.12.0.tar.gz#sha256=ca18659242f34ff3484b454fd872036a87b285c214e0907353c815e12e46cb75 (from https://pypi.org/simple/pymqi/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement pymqi~=1.12.0 (from versions: 1.0.1, 1.1, 1.2, 1.3, 1.4, 1.5, 1.5.2, 1.5.3, 1.5.4, 1.7.0, 1.7.1, 1.7.2, 1.7.2.post1, 1.8.0, 1.9.0, 1.9.0.post1, 1.9.1, 1.9.2, 1.9.3, 1.10, 1.10.1, 1.11.0, 1.11.1, 1.12.0)
ERROR: No matching distribution found for pymqi~=1.12.0
WARNING: You are using pip version 21.1.3; however, version 21.3.1 is available.
You should consider upgrading via the '/Users/smc2/.venvs/chaostk/bin/python3 -m pip install --upgrade pip' command.
(chaostk) stevens-mbp:chaostoolkit-ibm smc2$ 

implement unit testing stack

even though all actions as JSON and example file, still needed to be bundled through the unit testing platform major decision needs to be considered

  1. Should the execute the unit testing over actual IBM cloud directly
  2. Or mock IBM API to execute the unit testing.

The current suggestion is to Mock when you could and using the IBM cloud only when it must, give an option to execute the unit testing without the test cases that need the actual IBM cloud.

add network latency through ssh key

the current implementation for add network latency is very basic by ssh to the machine directly login using username and password consider supporting using ssh key

Manipulate ROKS, IKS volume

as of now VPC gen 2 does not provide direct access to the machine, but still for some cases attaching additional/removing volume is an API and CLI available to deploy SDS (Software Defined Storage) solution such as OCS, portworx, longhorn, storage.

The target is to manipulate this volume, it can be useful to test the effectiveness of the SDS storage solution to accommodate a loss of the disk

Bring this extension to Chaos Toolkit documentation?

Hey all,

I'm Sylvain, the maintainer of Chaos Toolkit. I wanted to thank you for this extension and effort. I wondered if you'd be interested for me to reference it into the main CTK documentation at https://chaostoolkit.org/drivers/overview/ ?

I'd be happy to add it myself. The only thing that may be required from you is ensuring is that the generated Python packages are properly done with the right metadata so the automation can read the readme and other bits and pieces accordingly during doc build.

Cheers,

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.