Entropy based scanning about tell-me-your-secrets

[valayDave]

Can you suggest more ideas on how you think the integration could be done ?

[valayDave]

Do you think Adding a Remote check feature would be very helpful? Or An Ansible like integration to run this on multiple instances and retrieve the leaks from everywhere?

[inverse]

No ideas right now - I didn't dig into how this was achieved with other products but perhaps we could adapt what they are doing.

What do you mean remote check?

[valayDave]

SSH Based multi-machine Key scan with this module.

[inverse]

Here is some examples: http://blog.dkbza.org/2007/05/scanning-data-for-entropy-anomalies.html

From trufflehog

def shannon_entropy(data, iterator):
    """
    Borrowed from http://blog.dkbza.org/2007/05/scanning-data-for-entropy-anomalies.html
    """
    if not data:
        return 0
    entropy = 0
    for x in iterator:
        p_x = float(data.count(x))/len(data)
        if p_x > 0:
            entropy += - p_x*math.log(p_x, 2)
    return entropy

[inverse]

Or even existing libraries like: https://github.com/lyft/high-entropy-string

[inverse]

@valayDave started implementing it in #24 but I think the whole topic could become quite complex, especially around false positives and other things.

Perhaps this project should just focus on rule based scanning, like what it's doing.

https://github.com/Yelp/detect-secrets - does it quite well from what I could see, although I didn't try the tool out that much.

@x-way what do you think?