checkmarx / 2ms Goto Github PK
View Code? Open in Web Editor NEWToo many secrets (2MS) helps people protect their secrets on any file or on systems like CMS, chats and git
License: Apache License 2.0
Too many secrets (2MS) helps people protect their secrets on any file or on systems like CMS, chats and git
License: Apache License 2.0
@jossef @cx-monicac the idea here is to support this output to make it easier to ingest by security tools, one of those is CxOne
Update supporting services, how to use, how to contribute.
optional, when specified, instead of cli args, values of args should be loaded from a config file
I give you the freedom to decide the file format (json, yaml, properties, ...)
and its structure
Description:
The Checkmarx 2MS tool is a powerful secret leakage detection tool that helps developers identify sensitive data and other secrets that may have been unintentionally leaked within their code repositories. To integrate this tool effectively into the development workflow, we need to add support for GitHub Actions to Checkmarx 2MS.
Technical Details:
To add GitHub Actions support to Checkmarx 2MS, we will create a custom action that can be used within GitHub workflows. This action will leverage the Checkmarx 2MS tool to scan a specified code repository for potential secret leakage issues and provide detailed results to the user. The action should be configurable, allowing users to specify the repository to scan, the API key to use for authentication, and any other relevant options.
Once the custom action is created, we can add it to the GitHub Marketplace, making it easily accessible for users. Additionally, we will provide documentation on how to integrate this action into existing workflows and best practices for using the Checkmarx 2MS tool for secret leakage detection within the GitHub ecosystem.
when provided, the value is expected to be a regular expression.
if provided and valid input, this regex should be converted to a rule and included in the rule set.
usage
./2ms --regex "((25[0-5]|(2[0-4]|1\d|[1-9]|)\d)\.?\b){4}"
After #34, we want to CD to DockerHub: https://hub.docker.com/r/checkmarx/2ms.
The release shouldn't be for each commit, but we should build a process of manually (by label or by workflow_dispatch
) releasing, to include more content in each release, and to avoid many small releases.
repository.go is quite straight-forward implementation.
suggesting to use Gitleaks engine to scan historical commits if they contain secrets
With the demand for GPT related keys we should add support to detect those as well.
This means developing the detection and contributing it to gitleaks, so they'll add it to their big list of key already detected. See https://github.com/gitleaks/gitleaks/tree/master/cmd/generate/config/rules
Please talk to me before submitting the PR to them.
Reference (in Hebrew) about GPT stolen keys at https://www.geektime.co.il/gpt-4-pirates-are-rampant/
CC @Alexaro1cx
2ms currently only has flags for the scan. We want to create commands to support different operations.
Reference to the concept can be seen at https://kubernetes.io/docs/reference/kubectl/ and https://docs.kics.io/latest/commands/
list - list plugins / engines / rules
scan - (default) scan for secrets
help - show cli help (same as -h today)
global flags
-v version
-h help
scan options
--log-level (same as today)
--all-plugins (default)
--plugin-include string, comma separated (add to plugins list, starts from empty)
--plugin-exclude string, comma separated (remove from all plugin list, starts from full)
--all-rules (default)
--rule-include string, comma separated
--rule-exclude string, comma separated
--config (file path)
list sub commands
plugins - list plugins
rules - list rules
config - list configs detected (env variables / files)
Notice each plugins brings its own flags to the scan command
e.g. confluence has
--confluence-url (mandatory)
--confluence-username (optional, for private instances)
--confluence-token(optional, for private instances)
--confluence-spaces (comma separated list, or all if not present)
if time allows:
scan sub commands
(plugin name) - scan with just one plugin (e.g. 2ms scan confluence)
confluence
Description:
The goal of this issue is to containerize the 2MS tool for easy deployment and management in any infrastructure. This will involve generating a Dockerfile for the project and adding it to Docker Hub.
Technical Details:
We will be following the best practices recommended by KICS to ensure that the containerized 2MS tool is secure and efficient.
2ms should support scanning of bitbucket (and similarly structured git based) repositories
When using 2ms filesystem --path somedir/
on a non empty dir, sometimes there's a crash (possible data-race).
mkdir testdir
echo "someinfo" > testdir/testfile.txt
2ms filesystem --path testdir
Expecting the command to succeed and report no secrets.
Sometimes ( around 1 / 3 times ) the program panics with this error stack:
panic: sync: WaitGroup is reused before previous Wait has returned
goroutine 1 [running]:
sync.(*WaitGroup).Wait(0x0?)
/usr/local/go/src/sync/waitgroup.go:118 +0x7f
github.com/checkmarx/2ms/cmd.postRun(0xc00012a600?, {0xa79f14?, 0x2?, 0x2?})
/home/<redacted>/Downloads/2ms/cmd/main.go:171 +0x3a
github.com/spf13/cobra.(*Command).execute(0xc00012a600, {0xc0001242c0, 0x2, 0x2})
/home/<redacted>/go/pkg/mod/github.com/spf13/[email protected]/command.go:936 +0xa2b
github.com/spf13/cobra.(*Command).ExecuteC(0xef3ce0)
/home/<redacted>/go/pkg/mod/github.com/spf13/[email protected]/command.go:1044 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
/home/<redacted>/go/pkg/mod/github.com/spf13/[email protected]/command.go:968
github.com/checkmarx/2ms/cmd.Execute()
/home/<redacted>/Downloads/2ms/cmd/main.go:107 +0x5e5
main.main()
/home/<redacted>/Downloads/2ms/main.go:22 +0x1fa
exit status 2
I removed my information from the stack trace and replaced with
The idea is a directory in the project, combined with .go files, and each is like a "plugin"
for ease of maintenance and future contribution
Lines 152 to 153 in 18ea1c4
Couln't understand why but without the sleep we can't get all elements.
Maybe it would make sense to spend some time investigating if it's realated to issues on goroutines management or it's related to some kind of limitations.
Originally posted by @joaopedrocsilva in #52 (comment)
use gitleaks with common config
suggest args to expose for user
similar to what you can do in .gitignore
but as optional args
add a number argument that is going to limit the historical scans when using the git
plugin
Today we run the full list of rules as defined at secrets/secrets.go#L111
Add the following sub-commands to the list command to all the user know which rules are available:
rules - list rules
Add the following flags for the scan command to allow the user to control which rules are being used:
--all-rules (default)
--rule-include string, comma separated (adds to list, starts from empty)
--rule-exclude string, comma separated (remove from list, starts from full list)
Only if appear on the same file - when results corresponding to the same page they should appear once with all the versions where the secret is present.
In #81, I added a validation pipeline to make sure the PR title is following the semantic-release
conventions and will bump the SemVer.
But semantic-release
is using a small subset of Angular Conventional Commit, and I need to research it to find how to control both commitlint
and semantic-release
in a simple way.
Running 2ms without parameters gives this result:
$ ./2ms
Summary:
- Total items scanned: 0
- Total items with secrets: 0
Detailed Report:
Should the confluence switches be mandatory if the confluence plugin is enabled/running?
Or at least give a warning 2ms is missing some info for a confluence scan.
to receive and read cli arguments
Implement the Paligo scanner from inside Checkmarx and run it in parallel with the current implementation in Checkmarx to compare the results and replace it.
We need to create a config file for golangci-lint
and add these linters:
Refactor the return type of getItems func. Replace the entire json structure for the body.value field (the valid one).
Get contents for older versions, also consider to add an option to scan it or no
running the command: ./2ms repository --path "<file_to_scan>"
where <file_to_scan>
is this file.
2ms finds a hardcoded secret at line 8
See here what the condition should be to identify if the PR is from a fork or not.
With your PR, update the pull_request_template.md
and remove the comment about adding the label manually.
I've tried
--confluence https://checkmarx.atlassian.net/wiki/
the actual value passed: https//checkmarx.atlassian.net/wiki/
params - discord token?
TBD
add options to filter out specific secrets results by their id (in case known issues or false positives)
Provide an example call with flags set when printing help -h / --help
./2ms --confluence [https://<MyConfluence>.atlassian.net/wiki](https://%3Cmyconfluence%3E.atlassian.net/wiki) --confluence-spaces <MySpace> --confluence-username <MyUsername> --confluence-token <MyToken>
this should be generic as plugins will be added
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.