sirwart / ripsecrets Goto Github PK
View Code? Open in Web Editor NEWA command-line tool to prevent committing secret keys into your source code
License: MIT License
A command-line tool to prevent committing secret keys into your source code
License: MIT License
Origin: https://gitlab.com/gitlab-org/gitlab/-/issues/342327
Docs: https://docs.gitlab.com/ee/user/admin_area/settings/account_and_limit_settings.html#personal-access-token-prefix
Pattern: "glpat-[A-Za-z0-9_/-]{20,}"
I write a simple code like
package main
func main() {
clientSecretKey := "alkfjlaf^*flkajlfkay7782085ljafg"
println(clientSecretKey)
}
and hope ripsecrets can tell me 'you hardcode the secret in source files', but there's nothing output
Say my .secretsignore
should ignore everything:
$ cat .secretsignore
*
It doesn't unfortunately work:
$ pre-commit run ripsecrets --all-files --verbose
Found existing alias for "pre-commit run". You should use: "prcr"
ripsecrets...............................................................Failed
- hook id: ripsecrets
- duration: 0.01s
- exit code: 1
credentials.toml:3:password = 'uJSU7Kxquv5FXDRLF7SCBaksmo9o2Zp8'
Adding an empty [secrets]
section makes it work though:
$ echo "[secrets]" >> .secretsignore
$ pre-commit run ripsecrets --all-files --verbose
Found existing alias for "pre-commit run". You should use: "prcr"
ripsecrets...............................................................Passed
- hook id: ripsecrets
- duration: 0.01s
This seems to be due to missed creation of the Gitignore
object in one branch in https://github.com/sirwart/ripsecrets/blob/main/src/ignore_info.rs
I'm not sure exactly what's entailed by this, but I don't think it would be too difficult
The only official reference to syntax I could find was this image
from https://access.redhat.com/documentation/en-us/red_hat_quay/3/html/use_red_hat_quay/use-quay-manage-repo
Pretty sure it's just "[A-Z0-9]{64}"
though.
Hi there! First off, thanks for this awesome tool!
I'm seeing some unexpected / inconsistent behavior where secrets files are being sometimes being ignored, based on whether the [secrets]
section is present in the .secretsignore
file.
Setup:
> ripsecrets --version
ripsecrets 0.1.3
# this is the v0.1.5 git tag, but it has 0.1.3 as the Cargo.toml version
# create some secret file with TOTALLY FAKE AWS access key
> echo "aws_access_key_id = AKIATHAAGaZc4krwNWdc" > test.txt
# create a secrets ignore file
> echo "test.txt" > .secretsignore
> ripsecrets && echo "no secrets found"
no secrets found
> ripsecrets --strict-ignore && echo "no secrets found"
no secrets found
> ripsecrets --strict-ignore test.txt && echo "no secrets found"
test.txt:1:aws_access_key_id = AKIATHAAGaZc4krwNWdc
So when using normally, with no positional file/directory specified, it comes back exit code 0. But when the file is provided as a positional argument, it comes back with a failure.
But, if we add a [secrets]
tag to the .secretsignore
file, then the command comes back with exit code 0.
> echo "[secrets]" >> .secretsignore
> cat .secretsignore
test.txt
[secrets]
> ripsecrets --strict-ignore test.txt && echo "no secrets found"
no secrets found
I think this is because in src/find_secrets.rs
, it conditionally ignores the explicitly provided files only if there's a ignore_matcher.is_some()
(permalink), which looks like it's coming from ignore_info.rs
behind this conditional.
Is this expected behavior? If not, I'm happy to contribute a fix!
Thanks again!
Cheers!
I happened upon https://github.com/OWASP/SEDATED/blob/master/config/regexes.json today. I think it has some key formats that ripsecrets doesn't. It also looks for variable names, which is not desirable for ripsecrets, so those can be ignored.
I just tested secrets
on my repo, and it did a really good job! It discovered a couple of (test mode) unencrypted private keys, and flagged a bunch of public key fingerprints that definitely look like secrets. Kudos on the tool!
In case you're looking at tweaking the heuristics, I wanted to call out this line it flagged, which is clearly a false positive:
if ((filterMode == FilterMode.OFFLINE_STORAGE && key.isPersistent()) || (filterMode == FilterMode.API_RETURN_VALUE && key.isReturnable())
Thanks!
On the heels of this wild news about Microsoft leaking 38 TB of data because of a committed SAS token, maybe ripsecrets could audit for that, too.
Here are some examples from that doc:
{
"inputs": [
{
"storageType": "File",
"source": {
"sourceUrl": "https://my.blob.core.windows.net/source-en/source-english.docx?sv=2019-12-12&st=2021-01-26T18%3A30%3A20Z&se=2021-02-05T18%3A30%3A00Z&sr=c&sp=rl&sig=d7PZKyQsIeE6xb%2B1M4Yb56I%2FEEKoNIF65D%2Fs0IFsYcE%3D"
},
"targets": [
{
"targetUrl": "https://my.blob.core.windows.net/target/try/Target-Spanish.docx?sv=2019-12-12&st=2021-01-26T18%3A31%3A11Z&se=2021-02-05T18%3A31%3A00Z&sr=c&sp=wl&sig=AgddSzXLXwHKpGHr7wALt2DGQJHCzNFF%2F3L94JHAWZM%3D",
"language": "es"
},
{
"targetUrl": "https://my.blob.core.windows.net/target/try/Target-German.docx?sv=2019-12-12&st=2021-01-26T18%3A31%3A11Z&se=2021-02-05T18%3A31%3A00Z&sr=c&sp=wl&sig=AgddSzXLXwHKpGHr7wALt2DGQJHCzNFF%2F3L94JHAWZM%3D",
"language": "de"
}
]
}
]
}
Looks like the presence of sv
with an ISO date and sig
query params, and sig is base64 encoded.
Adding these URLs to the end of test/one_per_line/azure
reflects that ripsecrets doesn't already catch them.
it's very slow compared to rg (with all the extras removed) - I only checked for 3 piped rgxs
Feature request
Allow global .secretsignore (in user home directory). Alternatively, allow providing path to the .secretsignore
file.
Hi!
Love the tool! The performance gains are awesome! Curious if you had any plans to make adding additional secret patterns to the tool more friendly (Maybe an external TOML file with user specified patterns?). If you could forsee this being part of the tool, would love to pull together a PR if we can agree on an implementation.
Otherwise - would be happy to PR some additional secret patterns!
Long secrets are not ignored.
See the example below. The first two secrets are ignored, while the last two are not.
.secretsignore
[secrets]
96etKOmnte-bpLDSIcwdhXYlC82gF8x-ERPqZ7oo1Ug
8AOiCMgwF1eg5yLDgw9D1eymTSOp21PJwr4zdQRQyYQ
eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ0b2tlbl90eODlIjoiYWNjZXNzIiwiZXhwIjoxNjUxMTQxMzc3LCJpYXQiOjE2NTExNDA0ODAsImp0aSI6ImQzAAJmYzBiNzI2NDRjMjY5ODI0NGFiMTQ2OTc1N2YyIiwidXNlcl9pZCI6MX0.87aml-57DmEUo4LrlZwnDw4iVfiWVNA90xxCi01M2h0
eyJ0eXAiOiJKV1QiCCJhbGciOiJIUzI1NiJ9.eyJ0b2tlbl90eWPlIjoiYWNjZXNzIiwiZXhwIjoxNjUxMTQxMzgxLCJpYXQiOjE2NTExNDEwODEsImp0aSI6Ijk1YjRjMDA2ODZjNTRkYTU4OTE1NWYzOTgzZjcxNmJiIiwidXNlcl9pZCI6MX0.zs-3zv1eCSu9JeRBJgFw6CBoZUA4B2R3z6gl2vNYwdA
Easily reproducible like this:
ripsecrets .secretsignore
The release CI only handles building an OCI image. I speculate that the release artifacts are handled locally. The ripgrep release.yml might be a good example to follow.
I'm happy to do this up if you greenlight it.
When we find "random string" secrets, we use a regex that matches patterns along the lines of secret_key = "<key>"
. However, the matcher should return the <key>
as the match range. This isn't a problem now since we don't highlight matched secrets, but it's preventing this PR from being merged in: #18.
According to this PR: #2:
Once this is merged, someone (me?) could put a PR on https://github.com/pre-commit/pre-commit.com/blob/main/all-repos.yaml that adds https://github.com/sirwart/secrets to it.
Someone needs to add https://github.com/sirwart/ripsecrets to the pre-commit repository.
➜ temp cat src/source.java
password=123
username=333
pwd=344
passwort=3333333
benutzername=32
➜ temp ripsecrets
➜ temp
I looked at clap_mangen and I think it's the right thing to use, but it looks like it might need some massaging to get the ripsecret::main::Args into it.
When trying to install ripsecrets via cargo I noticed that it now tries to download the entire sentry module because it's listed as a submodule, which takes a long time because it's a large repo. Either the configuration needs to change or removed as a submodule.
@sirwart Are there any specific issues you want fixed or pending features you want implemented before releasing 1.0.0?
Cargo installation fails citing could not find _unix_ in _os_
Compiling ripsecrets v0.1.5 (C:\Users\dsieradski\.cargo\git\checkouts\ripsecrets-0fa9e8d7534dec57\ac370a9)
error[E0433]: failed to resolve: could not find `unix` in `os`
--> src\pre_commit.rs:6:14
|
6 | use std::os::unix::fs::PermissionsExt;
| ^^^^ could not find `unix` in `os`
error[E0599]: no method named `set_mode` found for struct `Permissions` in the current scope
--> src\pre_commit.rs:79:11
|
79 | perms.set_mode(perms.mode() | 0o100);
| ^^^^^^^^ method not found in `Permissions`
error[E0599]: no method named `mode` found for struct `Permissions` in the current scope
--> src\pre_commit.rs:79:26
|
79 | perms.set_mode(perms.mode() | 0o100);
| ^^^^ method not found in `Permissions`
Some errors have detailed explanations: E0433, E0599.
For more information about an error, try `rustc --explain E0433`.
error: could not compile `ripsecrets` (bin "ripsecrets") due to 3 previous errors
error: failed to compile `ripsecrets v0.1.5 (https://github.com/sirwart/ripsecrets?branch=main#ac370a97)`, intermediate artifacts can be found at `C:\Users\DSIERA~1\AppData\Local\Temp\cargo-installNnxbIE`
Splitting off from #66 (comment) and #66 (comment), it'd be great if this was uploading to crates.io on release.
https://doc.rust-lang.org/cargo/reference/publishing.html
https://users.rust-lang.org/t/does-anyone-use-github-actions-to-run-cargo-publish/92374
Just found a bunch of additional alternatives to add to the README:
Was Looking at https://github.com/sirwart/secrets/blob/main/src/find_secrets.rs#L24= and found reference to the AKIA-like set of access key IDs. I don't think these are secrets, however the regex could be replaced for the true format of AWS Secrets which would be tied to these identifiers (as listed under "Secret keys"): https://summitroute.com/blog/2018/06/20/aws_security_credential_formats/
Like this https://bind9.readthedocs.io/en/v9_16_5/advanced.html#loading-a-new-key
See also discussion in #35
While secrets
is basically "get secrets" and expect none to be returned, the name is, well, not very S.E.O. friendly. @sirwart, are you open to considering some other names while this tool is still in its infancy?
got a false positive on https://ola.hallengren.com/scripts/MaintenanceSolution.sql
./MaintenanceSolution.sql:84:)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
i went to ignore just this "secret" in the .secretsignore
file like this:
[secrets]
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
but couldn't get it to ignore... had to add the whole file to .secretsignore
MaintenanceSolution.sql
[secrets]
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
this file's like 9100 lines long, so not ideal 😄
this is using secrets 0.1.2
Since there are already pre-built binaries for each release, it would be nice if we could utilize them in the pre-commit hook instead of requiring a dependency on cargo install
, which isn't necessarily available in all environments you would want to check for secrets.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.