netflix-skunkworks / diffy Goto Github PK

Upload diffy to pypi and ensure that is installable via pip.

Allow users to capture a local baseline of their system, store it, and then compare system state to that baseline at a later date. Useful for demo and test purposes. One example would be to baseline a VM, infect it in someway, and then run analysis to confirm detection of the infection.

Jinja < 2.10.1 has a security vulnerability. We pull this in via flask, we should roll our web-requirements.in and pickup a newer version for the library with the vulnerability fixed.

The Error is

`compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

----------------------------------------

Command "/home/harry/diffy/env/bin/python3 -u -c "import setuptools, tokenize;file='/tmp/pip-install-xihzs_85/python-levenshtein/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-record-0losak7n/install-record.txt --single-version-externally-managed --compile --install-headers /home/harry/diffy/env/include/site/python3.6/python-levenshtein" failed with error code 1 in /tmp/pip-install-xihzs_85/python-levenshtein/`

Today diffy attempts to send all commands within the request window. For larger clusters completion time may exceed a reasonable request limit.

The api should background analysis requests such that it does not block additional web requests. It should return a worker ID that can be queried for status.

The TrustedBinaryCollectionPayloadPlugin is aware of the location of a single trusted binary, or set of trusted binaries, which can be used as an alternative to osqueryi, to profile an instance.

If no baseline is available, diffy will attempt to cluster the hosts. Diffy should issue a warning that the results of this cluster may not be meaningful when the number of hosts is low, e.g. <30.

When creating a baseline, additional targets are ignored (https://github.com/Netflix-Skunkworks/diffy/blob/master/diffy/core.py#L72). However, some TargetPlugin derivatives will submit multiple targets. We should be able to set a dictionary key in target_plugin["options"] indicating multiple targets, pass that using **kwargs, and add decision logic in core.py that acts on multiple or single targets based on the value.

It looks like diffy only supports creating baselines based on auto-scaling groups.

Unfortunately, I'm not using auto-scaling groups, and it's not an option for me to do so... but I do have tags.

Would it be possible/desirable to allow diffy to work based on tags in addition to ASGs?

I'm writing some PoC code to experiment with this but would love your feedback.

Thanks!

We allow multiple commands to be specified by:
https://github.com/Netflix-Skunkworks/diffy/blob/master/diffy/config.py#L199

and
https://github.com/Netflix-Skunkworks/diffy/blob/master/diffy/config.py#L194

These are then issued in series on the host. However when multiple commands are passed the output is not properly collected in a valid json format. Moreover we have no way to separate out multiple outputs.

We should allow for multiple collection commands is a format like:

{
    "<command1>": {"some": "output"},
    "<command2>": {"some": "output"}
}

This will likely mean that instead of having the commands executed directly we will need some script to execute and collect the output of multiple commands in the above format.

Snyk informed me that PyYAML is vulnerable to a security issue as well.

I really would like to understand how Diffy works, and noticed that you gave a talk at OSDFCon. I've checked out the slides, just wanted to know if a video will be released.

Thanks

Create documentation for the prerequisites assumed for each plugin. What does each plugin require, to run?