chaostoolkit / chaostoolkit-addons Goto Github PK

Fantastic day everyone!

On the readme file it specifies that the python requirements is 3.5+
https://github.com/chaostoolkit/chaostoolkit-addons/blob/master/README.md#install

However the init.py file includes an import that is only available from python 3.8+

If someone will try to use it or import the safeguards.py for example while they are lower than 3.8, they will get an error:
ModuleNotFoundError: No module named 'importlib.metadata'

Hey, seems like the safeguard control gets stuck in the after_experiment_control when exit_gracefully is called before reaching the call to wait on the now_all_done Barrier. I can reliably get the following experiment to hang forever unless I actually configure a pause for the probe, but I'm guessing there could be a more elegant solution:

---
title: safeguard test
description: safeguard test

controls:
- name: safeguard
  provider:
    type: python
    module: chaosaddons.controls.safeguards
    arguments:
      probes:
        - name: safeguard
          type: probe
          provider:
            type: process
            path: date
          tolerance: 1
          # pauses:
          #   after: 1

[2022-06-22 17:54:48 DEBUG] [cli:113] Running command 'run'
[2022-06-22 17:54:48 DEBUG] [cli:117] Using settings file '/Users/jeal/.chaostoolkit/settings.yaml'
[2022-06-22 17:54:49 DEBUG] [__init__:399] No controls to apply on 'loader'
[2022-06-22 17:54:49 DEBUG] [__init__:399] No controls to apply on 'loader'
[2022-06-22 17:54:49 DEBUG] [caching:24] Building activity cache...
[2022-06-22 17:54:49 DEBUG] [caching:35] Cached 3 activities
[2022-06-22 17:54:49 INFO] [experiment:58] Validating the experiment's syntax
[2022-06-22 17:54:49 DEBUG] [configuration:63] Loading configuration...
[2022-06-22 17:54:49 DEBUG] [secret:78] Loading secrets...
[2022-06-22 17:54:49 DEBUG] [secret:104] Done loading secrets
[2022-06-22 17:54:49 DEBUG] [python:196] Control 'validate_control' loaded from '/Users/jeal/.pyenv/versions/3.9.9/envs/chaostoolkit/lib/python3.9/site-packages/chaosaddons/controls/safeguards.py'
[2022-06-22 17:54:49 DEBUG] [python:192] Control module '/Users/jeal/.pyenv/versions/3.9.9/envs/chaostoolkit/lib/python3.9/site-packages/chaostoolkit_rappi-0.1.0-py3.9.egg/chaos_rappi/controls/state_sharing.py' does not declare 'validate_control'
[2022-06-22 17:54:49 INFO] [experiment:109] Experiment looks valid
[2022-06-22 17:54:49 DEBUG] [caching:42] Clearing activities cache
[2022-06-22 17:54:49 DEBUG] [caching:24] Building activity cache...
[2022-06-22 17:54:49 DEBUG] [caching:35] Cached 3 activities
[2022-06-22 17:54:49 DEBUG] [configuration:63] Loading configuration...
[2022-06-22 17:54:49 DEBUG] [secret:78] Loading secrets...
[2022-06-22 17:54:49 DEBUG] [secret:104] Done loading secrets
[2022-06-22 17:54:49 DEBUG] [configuration:155] Loading dynamic configuration...
[2022-06-22 17:54:49 INFO] [run:320] Running experiment: journal output test
[2022-06-22 17:54:49 DEBUG] [__init__:52] Initializing controls
[2022-06-22 17:54:49 DEBUG] [__init__:61] Initializing control 'safeguard'
[2022-06-22 17:54:49 DEBUG] [python:196] Control 'configure_control' loaded from '/Users/jeal/.pyenv/versions/3.9.9/envs/chaostoolkit/lib/python3.9/site-packages/chaosaddons/controls/safeguards.py'
[2022-06-22 17:54:49 DEBUG] [__init__:61] Initializing control 'get metadata'
[2022-06-22 17:54:49 DEBUG] [python:192] Control module '/Users/jeal/.pyenv/versions/3.9.9/envs/chaostoolkit/lib/python3.9/site-packages/chaostoolkit_rappi-0.1.0-py3.9.egg/chaos_rappi/controls/state_sharing.py' does not declare 'configure_control'
[2022-06-22 17:54:49 INFO] [run:344] Steady-state strategy: default
[2022-06-22 17:54:49 INFO] [run:348] Rollbacks strategy: default
[2022-06-22 17:54:49 INFO] [run:353] No steady state hypothesis defined. That's ok, just exploring.
[2022-06-22 17:54:49 DEBUG] [__init__:409] Applying before-control 'safeguard' on 'experiment'
[2022-06-22 17:54:49 DEBUG] [python:196] Control 'before_experiment_control' loaded from '/Users/jeal/.pyenv/versions/3.9.9/envs/chaostoolkit/lib/python3.9/site-packages/chaosaddons/controls/safeguards.py'
[2022-06-22 17:54:49 DEBUG] [__init__:409] Applying before-control 'safeguard' on 'activity'
[2022-06-22 17:54:49 DEBUG] [python:192] Control module '/Users/jeal/.pyenv/versions/3.9.9/envs/chaostoolkit/lib/python3.9/site-packages/chaosaddons/controls/safeguards.py' does not declare 'before_activity_control'
[2022-06-22 17:54:49 DEBUG] [process:52] Running: ['/bin/date']
[2022-06-22 17:54:49 DEBUG] [__init__:409] Applying after-control 'safeguard' on 'activity'
[2022-06-22 17:54:49 DEBUG] [python:192] Control module '/Users/jeal/.pyenv/versions/3.9.9/envs/chaostoolkit/lib/python3.9/site-packages/chaosaddons/controls/safeguards.py' does not declare 'after_activity_control'
[2022-06-22 17:54:49 CRITICAL] [safeguards:290] Safeguard 'safeguard' triggered the end of the experiment
[2022-06-22 17:54:49 INFO] [run:607] Playing your experiment's method now...
[2022-06-22 17:54:49 DEBUG] [safeguards:198] Safeguard 'safeguard' finished normally
[2022-06-22 17:54:49 WARNING] [run:420] Received the exit signal: 20
[2022-06-22 17:54:49 INFO] [run:458] Experiment ended with status: interrupted
[2022-06-22 17:54:49 DEBUG] [__init__:409] Applying after-control 'safeguard' on 'experiment'
[2022-06-22 17:54:49 DEBUG] [python:196] Control 'after_experiment_control' loaded from '/Users/jeal/.pyenv/versions/3.9.9/envs/chaostoolkit/lib/python3.9/site-packages/chaosaddons/controls/safeguards.py'
^C[2022-06-22 17:54:53 DEBUG] [__init__:91] Cleaning up controls
[2022-06-22 17:54:53 DEBUG] [__init__:100] Cleaning up control 'safeguard'
[2022-06-22 17:54:53 DEBUG] [python:192] Control module '/Users/jeal/.pyenv/versions/3.9.9/envs/chaostoolkit/lib/python3.9/site-packages/chaosaddons/controls/safeguards.py' does not declare 'cleanup_control'
[2022-06-22 17:54:53 DEBUG] [__init__:100] Cleaning up control 'get metadata'
[2022-06-22 17:54:53 DEBUG] [python:192] Control module '/Users/jeal/.pyenv/versions/3.9.9/envs/chaostoolkit/lib/python3.9/site-packages/chaostoolkit_rappi-0.1.0-py3.9.egg/chaos_rappi/controls/state_sharing.py' does not declare 'cleanup_control'
[2022-06-22 17:54:53 DEBUG] [caching:42] Clearing activities cache

Aborted!

p.s.: I'm running Python 3.9.9 (from homebrew) on macOS 12.4, though I don't think it has anything to do with it.

I am not sure if there is any standard way for defining configurations that would be used by multiple chaos experiments. For example, the aws chaos extension requires to define aws_region in each experiment. I wanted this value to be dynamic so I defined it in the configuration block of each experiment. Now that the number of experiments is growing, it is getting real difficult to manage this parameter on the experiment level. I came up with a solution by writing a control which would load the configurations defined on the settings.yaml file before each chaos run.

• Few things to note
  • Variable that is defined on the settings.yaml file will be used by default
  • Variable defined in experiment top level will override the settings.yaml variable
  • Variable passed from the command line argument chaos run --var key=value will override the experiment defined variable

def configure_control(configuration: Configuration = None,
                      secrets: Secrets = None, settings: Settings = None,
                      experiment: Experiment = None):
    """
    Load configuration values from settings file
    Settings value can be overwritten by defining it in the experiment
    Usage: On settings file
        controls:
        global_values:
            provider:
            type: python
            module: custommodule.utils.control
    """
    settings_config = settings['configuration']
    for (setting_key, setting_value) in settings_config.items():
        if (configuration.get(setting_key)):
            logger.info("Using %s:%s from local experiment",setting_key,setting_value)
        else:
            logger.info("Using %s:%s from settings file",setting_key,setting_value)
            configuration[setting_key] = setting_value

First of all, let me say that safeguard working very well.

My team and I encounter a problem.
When we are running the safeguard on interval mode, we want the ability to count the number of times that we are executing each probe.
The reason for that is because we have a logging mechanism in each probe, and when we are executing those functions every few seconds, it's flooding the system with those logs.
But, we also want to log at least once.
So, our solution for that issue is if you will get as an argument (in the before_activity function) the amount of the execution of each probe, it will solve that problem.

Here is an example of a simple experiment:

{
  "version": "1.0.0",
  "title": "test",
  "description": "test",
  "configuration": {
    "myarg": "exp.json",
    "another_arg": "another.json"
  },
  "tags": [
    "network",
    "lala"
  ],
  "controls": [
    {
      "name": "Safeguards",
      "provider": {
        "type": "python",
        "module": "chaosaddons.controls.safeguards",
        "arguments": {
          "probes": [
            {
              "name": "Non existing probe",
              "description": "Probe that doesn't exist",
              "type": "probe",
              "provider": {
                "type": "python",
                "module": "blablabla",
                "func": "bla",
                "arguments": {}
              },
              "tolerance": 2
            }
          ]
        }
      }
    }
  ],
  "steady-state-hypothesis": {
    "title": "Check something",
    "probes": [
      {
        "name": "My probe",
        "description": "My best probe",
        "tolerance": {
          "type": "regex",
          "pattern": "exp.json",
          "target": "stdout"
        },
        "type": "probe",
        "provider": {
          "type": "process",
          "path": "echo",
          "arguments": "${myarg}"
        }
      }
    ]
  },
  "method": [
    {
      "name": "My probe",
      "description": "My best probe",
      "tolerance": {
        "type": "regex",
        "pattern": "exp.json",
        "target": "stdout"
      },
      "type": "probe",
      "provider": {
        "type": "process",
        "path": "echo",
        "arguments": "${myarg}"
      }
    }
  ],
  "rollbacks": [
    {
      "name": "My probe",
      "description": "My best probe",
      "tolerance": {
        "type": "regex",
        "pattern": "exp.json",
        "target": "stdout"
      },
      "type": "probe",
      "provider": {
        "type": "process",
        "path": "echo",
        "arguments": "${myarg}"
      }
    }
  ]
}```

Notice the probe with module and function as "blabla"
Obviously it doesn't exists. But if you run the experiment, it just runs normally by ignoring that probe.
If I put the real probe there, it works fine.

I think the expected result would be some kind of an exception and abort everything.
We've been running experiments for a long time without knowing that we've had a non working probe.

Hello,

RUNTIME :

(chaos-env) $ chaos info extensions
NAME VERSION LICENSE DESCRIPTION
chaostoolkit-addons 0.1.0 UNKNOWN Addons for your Chaos Toolkit experiments

(chaos-env) $ chaos info core
NAME VERSION
CLI 1.9.0
Core library 1.19.0

I'm working on an internal network without internet connection. Working with python 3.7.0
I have manual installed chaostoolkit with setup.py, and also installed all requirements (chaostoolkit-librairy and python modules with .whl).

I have installed chaostoolkit-addons with setup.py. Now, I get an error when I want to discover chaostoolkit-addons (using 0.1.0 because I got an setuptools-scm error with higher version of addons).

Here is my error when I want to discover :

$ chaos discover --no-install chaostoolkit-addons Traceback (most recent call last): File "/home/user/chaos-env/bin/chaos", line 33, in <module> sys.exit(load_entry_point('chaostoolkit==1.9.0', 'console_scripts', 'chaos')()) File "/home/user/chaos-env/lib/python3.7/site-packages/click/core.py", line 829, in __call__ return self.main(*args, **kwargs) File "/home/user/chaos-env/lib/python3.7/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/home/user/chaos-env/lib/python3.7/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/user/chaos-env/lib/python3.7/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "/home/user/chaos-env/lib/python3.7/site-packages/click/core.py", line 610, in invoke return callback(*args, **kwargs) File "/home/user/chaos-env/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func return f(get_current_context(), *args, **kwargs) File "/home/user/chaos-env/lib/python3.7/site-packages/chaostoolkit-1.9.0-py3.7.egg/chaostoolkit/cli.py", line 440, in discover File "/home/user/chaos-env/lib/python3.7/site-packages/chaostoolkit_lib-1.19.0-py3.7.egg/chaoslib/discovery/discover.py", line 33, in discover package = load_package(package_name) File "/home/user/chaos-env/lib/python3.7/site-packages/chaostoolkit_lib-1.19.0-py3.7.egg/chaoslib/discovery/package.py", line 49, in load_package package = importlib.import_module(name) File "/home/user/tools/python/lib/python3.7/importlib/__init__.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1003, in _gcd_import File "<frozen importlib._bootstrap>", line 942, in _sanity_check ValueError: Empty module name

There is another possibility to install addons without internet connection ?
Actually, when I create my experiment, I don't have any method

Thank you in advance

Because of the unreliable nature of the signals, they don't always stop the experiment. If a signal was thrown, some random try and except closure can catch and swallow it. Causing the experiment to continue running.

We would like to suggest a quick workaround.
In safeguards we would like to set an environment variable "CHAOS_STOP" whenever the safeguard probe fails.
And then in controls, we can verify if the environment is set before every activity and stop the experiment by preventing the activity from running. Perhaps by running exit gracefully again or creating a proper way of stopping the experiments in controls.

My team and I found that the safeguard is working only on the second attempt when the tolerance does not met.
We even did a small test, to check it on a small experiment with just one probe and one action (not including the safeguard),
and it seems that every time we made the tolerance to be different than the output, only when it fails twice then and only then, the safeguard stopping the experiment.

Can you please check why this is happening?

As per the documentation, when using idle_for utils, the type can be empty. But chaostoolkit expects the type to be either action or probe when used within method. Is this idle_for function not meant to be used within method?

Code (method in an experiment):

method:
  - type: ""
    name: idle-for
    provider:
      type: python
      module: chaosaddons.utils.idle
      func: idle_for
      arguments:
        duration: 300

Error:

[2022-10-18 12:40:41 INFO] Validating the experiment's syntax
[2022-10-18 12:40:41 ERROR] an activity must have a type