LogSecretHunter is a powerful tool designed to scan log files stored in Amazon S3 for sensitive information, such as secrets, API keys, and passwords. By leveraging the detect-secrets
tool, LogSecretHunter ensures your log files are free from sensitive data leaks, helping you maintain security and compliance.
- Automated Scanning: Asynchronous downloads and scans log files using asyncio and aiobotocores from your specified S3 bucket based on a prefix and time window.
- Comprehensive Detection: Utilizes the robust 'detect-secrets' engine to identify a wide range of sensitive information.
- Easy Integration : Seamlessly integrates with your existing AWS environment and workflows.
- Detailed Reporting: Provides detailed reports on detected secrets for easy remediation.
- Prerequisites:
- Python 3.7+
- AWS credentials configured (e.g., via
aws configure
or environment variables)
- Usage:
- Clone this repository:
git clone https://github.com/issabayevmk/logsecrethunter.git
cd logsecrethunter
- Activate python virtual environment, e.g. for Linux, Mac
python3 -m venv venv
source venv/bin/activate
- Install dependencies
pip install -r requirements.txt
- Run the tool:
python log_secret_hunter.py <bucket_name> <prefix> <start_time> <end_time> <download_dir> <result_file> [--profile_name <profile_name>] [--log_level <log_level>]
Example
python download_s3.py my-bucket my-prefix 2024-01-01T00:00:00 2024-01-31T23:59:59 /path/to/download /path/to/results.txt --profile_name my-aws-profile --log_level INFO
bucket_name
: The name of the S3 bucket.prefix
: The prefix for the S3 objects.start_time
: The start of the time window (ISO 8601 format, e.g., 2024-01-01T00:00:00).end_time
: The end of the time window (ISO 8601 format, e.g., 2024-01-31T23:59:59).download_dir
: The directory to download the files to.results_file
: The file to save the scan results.
--profile_name
: The AWS profile name to use.--log_level
: Set the logging level (DEBUG, INFO, WARNING, ERROR, CRITICAL). Default is WARNING.
The script uses Python's logging
library to provide detailed logs of its operations. By default, the log level is set to WARNING
, but this can be adjusted using the --log_level
argument.
- Download Files: The script downloads files from the specified S3 bucket and prefix that were modified within the specified time window.
- Process Files:
- Scan for Secrets: Each file is scanned for secrets using detect-secrets.
- Decompress Files: If a file is compressed (.gz or .zip), it is decompressed.
- Scan Decompressed Files: Decompressed files are also scanned for secrets.
- Cleanup: Both the original and decompressed files are deleted after scanning.
- Save Results: The found secrets are saved to the specified result file.
Contributions are welcome! Please open an issue or submit a pull request.
- Fork the repository.
- Create a new branch (git checkout -b feature-branch).
- Make your changes.
- Commit your changes (git commit -m 'Add new feature').
- Push to the branch (git push origin feature-branch).
- Open a pull request.
This project is licensed under the MIT License - see the LICENSE file for details.
For questions or feedback, feel free to open an issue.