eth0izzle / bucket-stream

Find interesting Amazon S3 Buckets by watching certificate transparency logs.

Bucket Stream

Find interesting Amazon S3 Buckets by watching certificate transparency logs.

This tool simply listens to various certificate transparency logs (via certstream) and attempts to find public S3 buckets from permutations of the certificates domain name.

Be responsible. I mainly created this tool to highlight the risks associated with public S3 buckets and to put a different spin on the usual dictionary based attacks. Some quick tips if you use S3 buckets:

1. Randomise your bucket names! There is no need to use company-backup.s3.amazonaws.com.
2. Set appropriate permissions and audit regularly. If possible create two buckets - one for your public assets and another for private data.
3. Be mindful about your data. What are suppliers, contractors and third parties doing with it? Where and how is it stored? These basic questions should be addressed in every info sec policy.
4. Try Amazon Macie - it can automatically classify and secure sensitive data.

Thanks to my good friend David (@riskobscurity) for the idea.

Installation

Python 3.4+ and pip3 are required. Then just:

1. git clone https://github.com/eth0izzle/bucket-stream.git
2. (optional) Create a virtualenv with pip3 install virtualenv && virtualenv .virtualenv && source .virtualenv/bin/activate
3. pip3 install -r requirements.txt
4. python3 bucket-stream.py

Usage

Simply run python3 bucket-stream.py.

If you provide AWS access and secret keys in config.yaml Bucket Stream will attempt to access authenticated buckets and identity the buckets owner. Unauthenticated users are severely rate limited.

usage: python bucket-stream.py

Find interesting Amazon S3 Buckets by watching certificate transparency logs.

optional arguments:
  -h, --help            Show this help message and exit
  --only-interesting    Only log 'interesting' buckets whose contents match
                        anything within keywords.txt (default: False)
  --skip-lets-encrypt   Skip certs (and thus listed domains) issued by Let's
                        Encrypt CA (default: False)
  -t , --threads        Number of threads to spawn. More threads = more power.
                        Limited to 5 threads if unauthenticated.
                        (default: 20)
  --ignore-rate-limiting
                        If you ignore rate limits not all buckets will be
                        checked (default: False)
  -l, --log             Log found buckets to a file buckets.log (default:
                        False)
  -s, --source          Data source to check for bucket permutations. Uses
                        certificate transparency logs if not specified.
                        (default: None)
  -p, --permutations    Path of file containing a list of permutations to try
                        (see permutations/ dir). (default: permutations\default.txt)

F.A.Qs

• Nothing appears to be happening

  Patience! Sometimes certificate transparency logs can be quiet for a few minutes. Ideally provide AWS secrets in config.yaml as this greatly speeds up the checking rate.

• I found something highly confidential

  Report it - please! You can usually figure out the owner from the bucket name or by doing some quick reconnaissance. Failing that contact Amazon's support teams.

Contributing

1. Fork it, baby!
2. Create your feature branch: git checkout -b my-new-feature
3. Commit your changes: git commit -am 'Add some feature'
4. Push to the branch: git push origin my-new-feature
5. Submit a pull request.

License

MIT. See LICENSE

bucket-stream's Issues

Buckets checked , 0 found .. in hours

Sadly, I don't have a credit card, to test it with the subscription, but, running without, for hours, not a single bucket is found.
Perhaps, Amazon done some changes?
Thanks!

When using AWS Keys it won't scrub domains other than ".s3.amazonaws.com"

I noticed when running this script without AWS Keys it catches buckets from multiple AWS locations, such as US-East- and US-West-. However, when using AWS Key and Secret it only logs domain at ".s3.amazonaws.com"

Possible error in code?

Result of python3 bucket-stream.py --ignore-rate-limiting is:

Traceback (most recent call last):
File "/home/platt-sam/bucket-stream/bucket-stream/bucket-stream.py", line 336, in
main()
File "/home/platt-sam/bucket-stream/bucket-stream/bucket-stream.py", line 315, in main
THREADS.extend([CertStreamThread(q)])
File "/home/platt-sam/bucket-stream/bucket-stream/bucket-stream.py", line 71, in init
self.c = CertStreamClient(
TypeError: CertStreamClient.init() missing 1 required positional argument: 'url'

If I'm just being dumb and not understanding something just let me know! 🙏

ImportError: cannot import name 'queue'

python bucket-stream.py -h
Traceback (most recent call last):
File "bucket-stream.py", line 11, in
from queue import queue as Queue
ImportError: No module named queue

No buckets found now

Hi
I'm not sure if this is still being maintained or not, but for some reason bucket-stream has stopped finding any buckets at all. If I use -l option, the log file just stays empty, even after a number of days.
I think it stopped finding anything from around 21st Jan 2022 I think.
Regards
Xnl

Default vs Extended

I haven't tested this thoroughly yet but I wanted to get some feedback.

It feels like I'm getting less buckets found when using the extended.txt versus when I use the default.txt which contains less permutations. Does this make sense?

Slight permutation changes?

First off, Great script, appreciate the work. I have used in the past a different script, called Slurp. Primarily for the permutations, it allows the usage of "-" delimiters and even blanks, etc, He has a pretty large list also. But, it's not as fast, and you can not use AWS access creds, nor can it log to any where. Plus, I am really not a fan of the Go Lang. Setting it up in any enviornment then having it spit out the compiled app is just like.. oh, there it is.. way over there, all the time.. ok..

Anyways, I took the time, being bored, to give you the equivlent of what your script would look like if you used the permutations that his does. It's very much larger, duplicate are removed and everything. Probably would want to use an external file to load them.. It would add a LOT more detection on your scripts side, but you'd need to do some obvious changes to the permutation parsing / generating.

https://pastebin.com/RpUkywbV

Thanks again! Hopefully you'll think about it !

Feature request

Hey dude,

This is awesome can you add a feature to log buckets it has found please?

if i get time ill see if i can throw a PR request.

Documentation for --skip-lets-encrypt

I looked through the README, codebase, and past issues, but I cannot determine a reason to enable the --skip-lets-encrypt feature flag. What are the pros/cons of this feature flag?

I can update the documentation with any explanation given in this issue. Thanks!

Quitting doesn't work

Once running, the only way to stop checking for buckets seems to be to mash Ctrl+C 3-4 times quickly.

Hitting it once yields the following:

Tested using Python 3.6.4

503 when checking for buckets

Hey,

I think there might be a rate limit needed on when checking buckets.

feature request -- write discovered domains to a log file

I don't have the skills to actually write the PR myself, but I'd like to request the option to log discovered domains to a file.

I run today like:

python3 bucket-stream.py --skip-lets-encrypt -t 50 &>> out.txt

but it doesn't actually write to the file until I interrupt the app with a control+C ...so not sure if you're doing some buffering in the app that prevents real time writing to a file, but yeah -- feature request.

Finding buckets but just XML pages

Hi

This script is great, thank you.

However, I am finding occasional buckets, but when I open the link all I get is XML pages everytime, is this correct or is there a way to view the bucket contents of these publicly accesible buckets.

I am researching on behalf of my company as they are looking to invest in AWS

Wont catch buckets open to "Any Authenticated AWS User"

With s3 buckets there is an option to allow "Any Authenticated AWS User" right now this tool would not find these open buckets:

Example:

#open to everyone (catches this one)
$ aws s3 ls  s3://flaws.cloud/ --no-sign-request --region us-west-2
2017-03-14 04:00:38       2575 hint1.html
2017-03-03 05:05:17       1707 hint2.html
2017-03-03 05:05:11       1101 hint3.html
2017-03-25 21:58:29       2877 index.html
2017-02-27 02:59:28         46 robots.txt
2017-02-27 02:59:30       1051 secret-dd02c7c.html

#closed to anyone not authenticated to AWS (does not matter key)
$ aws s3 ls  s3://level2-c8b217a33fcf1f839f6f1f73a00a9ae7.flaws.cloud/ --no-sign-request --region us-west-2

An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied

#with AWS key (but not one on the account) would not catch this one
$ aws s3 ls  s3://level2-c8b217a33fcf1f839f6f1f73a00a9ae7.flaws.cloud/  --region us-west-2
2017-02-27 03:02:15      80751 everyone.png
2017-03-03 04:47:17       1433 hint1.html
2017-02-27 03:04:39       1035 hint2.html
2017-02-27 03:02:14       2786 index.html
2017-02-27 03:02:14         26 robots.txt
2017-02-27 03:02:15       1051 secret-e4443fc.html

this can be fixed with something like the following:

"""
This will look for the s3 bucket
Requires:
    aws credentials set
Takes: 
    a bucket name
Returns: 
    True if bucket has directory listing, false otherwise
"""
def try_s3(bucket_name):
    s3 = boto3.client('s3')
    try:
        result = s3.list_objects(Bucket=bucket_name)
        print bucket_name
        return True
    except:
        return False

possible to filter certstream for certain tlds

bucket-stream looks like a very nice application.

I'm interesting in being quite specific and only checking certain domains from the certstream. An example would be only .org tlds or a list of specific domains eg domain1.com, domain2.com.

I see some code

def get_permutations(domain, subdomain=None):
    perms = [
        "%s" % domain,
        "www-%s" % domain,
        "%s-www" % domain,
    ]

but I don't this this does that.

I'm thinking a command line argument such as:

-l limit to tld or only from list
-l .org

or 

-l domain-list.txt

Is this currently possible?

Thanks.

"No such bucket"

Hi,
maybe I'm just stupid, but when using bucket-stream, I get eg. bucketname.s3.amazonaws.com, with ACLs = AllUsers: READ, READ_ACP | AuthenticatedUsers: (none). But, when I try to list the contents with aws2 CLI, like aws2 s3 ls s3://bucketname.s3.amazonaws.com, I get

An error occurred (NoSuchBucket) when calling the ListObjectsV2 operation: The specified bucket does not exist

So what's going on? Am I doing something wrong?

Attribute Error?

Thanks in advance if you get time to reply. Im getting a attribute error;

[1m�[33mWaiting for Certstream events - this could take a few minutes to queue up...�[0mTraceback (most recent call last):

File "C:\bucket-stream\bucket-stream.py", line 279, in
main()
File "C:\bucket-stream\bucket-stream.py", line 271, in main
signal.pause() # pause the main thread
AttributeError: module 'signal' has no attribute 'pause'

Any thoughts?

RROR:root:Error connecting to CertStream - Connection is already closed

So.. running without aws keys.

(bucketvenv) bob@awweb1:~/bucket-stream$ python3 bucket-stream.py
It is highly recommended to enter AWS keys in config.yaml otherwise you will be severely rate limited!  You might want to run with --ignore-rate-limiting
No AWS keys, reducing threads to 5 to help with rate limiting.
Waiting for Certstream events - this could take a few minutes to queue up...
ERROR:root:Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
Waiting for Certstream events - this could take a few minutes to queue up...
ERROR:root:Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
Waiting for Certstream events - this could take a few minutes to queue up...
ERROR:root:Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
Waiting for Certstream events - this could take a few minutes to queue up...
ERROR:root:Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
Waiting for Certstream events - this could take a few minutes to queue up...
ERROR:root:Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
Waiting for Certstream events - this could take a few minutes to queue up...
ERROR:root:Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
Waiting for Certstream events - this could take a few minutes to queue up...

keep getting the following error when installing

Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'

any idea? first two steps worked fine

File not found and Incomplete Format Error

HI,
I am running bucket-stream in Debian 9 environment with Python 3.5.3 installed on system. The script is being run as a root on a test environment. Whenever I run the script, a file not found error is encountered for default.txt.

root@test:/home/bucketstream/bucket-stream# python3 bucket-stream.py --ignore-rate-limiting 
It is highly recommended to enter AWS keys in config.yaml otherwise you will be severely rate limited!You might want to run with --ignore-rate-limiting
No AWS keys, reducing threads to 5 to help with rate limiting.
Traceback (most recent call last):
  File "bucket-stream.py", line 328, in <module>
    main()
  File "bucket-stream.py", line 300, in main
    .format(ARGS.threads, len([x for x in get_permutations("")])), "green")
  File "bucket-stream.py", line 241, in get_permutations
    perms.extend([line.strip() % domain for line in open(ARGS.permutations)])
FileNotFoundError: [Errno 2] No such file or directory: 'permutations\\default.txt'

If we explicitly specify the path of default.txt, then an incomplete format error is generated. Am I missing something or does the script requires a specific environment setup?

root@test:/home/bucketstream/bucket-stream# python3 bucket-stream.py --ignore-rate-limiting -s keywords.txt --permutations default.txt
It is highly recommended to enter AWS keys in config.yaml otherwise you will be severely rate limited!You might want to run with --ignore-rate-limiting
No AWS keys, reducing threads to 5 to help with rate limiting.
Traceback (most recent call last):
  File "bucket-stream.py", line 328, in <module>
    main()
  File "bucket-stream.py", line 300, in main
    .format(ARGS.threads, len([x for x in get_permutations("")])), "green")
  File "bucket-stream.py", line 241, in get_permutations
    perms.extend([line.strip() % domain for line in open(ARGS.permutations)])
  File "bucket-stream.py", line 241, in <listcomp>
    perms.extend([line.strip() % domain for line in open(ARGS.permutations)])
ValueError: incomplete format

Thanks!

doesn't work at all