x0rz / phishing_catcher

Phishing catcher using Certstream

Phishing Catcher

Catch possible phishing domains in near real time by looking for suspicious TLS certificate issuances reported to the Certificate Transparency Log (CTL) via the CertStream API. "Suspicious" issuances are those whose domain name scores beyond a certain threshold based on a configuration file.

This is just a working PoC. Feel free to contribute and tweak the code to fit your needs. 👍

Installation

The script should work fine using Python2 or Python3. In either case, install the requirements after cloning or downloading the source code:

pip install -r requirements.txt

Configuration

Phishing Catcher uses a simple YAML configuration file to assign a numeric score for strings that can be found in a TLS certificate's common name or SAN field (i.e., a cert's domain name). The configuration file, suspicious.yaml, ships with sensible defaults, but you can adjust or add to both the strings it contains and the score assigned to each string by editing an override file, external.yaml.

Both the default suspicious.yaml and the user-modifiable external.yaml configuration files contain two YAML dictionaries: keywords and tlds. The keys of the dictionaries are the strings and the values are the scores to assign if that string is found in the domain name for an issued certificate. For example:

keywords:
    'login': 25

Here, a score of 25 is added to the generic keyword login when it is found in a TLS certificate domain name. Increasing this value will raise the level of suspicion against domains with the string login in them, thus allowing you to subject these certificate issuances to increased scrutiny.

However, in order to be reported as suspicious by Phishing Catcher, the score assigned to a given certificate must meet or exceed (>=, "greater than or equal to") the following thresholds:

Score	Reported as
65	Potential
80	Likely
90	Suspicious

💡 See the score_domain() function in the source code for details regarding the scoring algorithm.

Usage

Once configured to your liking, usage is as simple as running the script:

$ ./catch_phishing.py

Example phishing caught

Phishing catcher in Docker container

If you running MacOs or having a different OS version that would make the installation of phishing_catcher difficult, then having the tool dockerized is one of your options.

docker build . -t phishing_catcher

License

GNU GPLv3

If this tool has been useful for you, feel free to thank me by buying me a coffee.

phishing_catcher's Issues

Whitelist google domains

Awesome script u wrote, although I might have a small improvement idea on it.

Maybe it's possible to whitelist all known google domains. There are various public list that state all legit google domains , for instance: https://ipfs.io/ipfs/QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1mXWo6uco/wiki/List_of_Google_domains.html

My output gets a lot of legit google domains, so it's time for a clean up :)

IDNA does not round trip error

Seems to be an error with certstream and the encoding and decoding within the confusables.py file.

Installation Error

I'm getting an error when trying to install. When I go to the link to get Microsoft Visual
C++ Build Tools, I got a 404 Not Found!
Complete output from command c:\users\aalborzfard\appdata\local\programs\pyt
hon\python36\python.exe -u -c "import setuptools, tokenize;file='C:\Users\
AALBOR1\AppData\Local\Temp\pip-install-rhefjs4r\entropy\setup.py';f=getat
tr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close
();exec(compile(code, file, 'exec'))" install --record C:\Users\AALBOR1\App
Data\Local\Temp\pip-record-2otn21ya\install-record.txt --single-version-external
ly-managed --compile:
running install
running build
running build_ext
building 'entropy' extension
error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual
C++ Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools

ImportError: No module named setuptools

@x0rz

I get the below error while installing . Please assist

phishing_catcher # pip install -r requirements.txt 
Collecting termcolor==1.1.0 (from -r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/8a/48/a76be51647d0eb9f10e2a4511bf3ffb8cc1e6b14e9e4fab46173aa79f981/termcolor-1.1.0.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ImportError: No module named setuptools
    
    ----------------------------------------

Feed down?

Have not been able to connect since 11/29.

certificate_update: 0cert [00:00, ?cert/s]Error connecting to CertStream - Handshake status 521 None -+-+- {'date': 'Tue, 12 Dec 2023 00:16:04 GMT', 'content-type': 'text/plain; charset=UTF-8', 'content-length': '15', 'connection': 'keep-alive', 'report-to': '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=47bybxwiYVWIOCgXfDQJsZ386aYaL1CUIcGsKym5Sel6GuLwI2XV7iZdJxa4pTyq6DKJLSBlxJzpc8KyY3mWXv7q0a3GQku2Ahdp7qISSF9qK92bQhIknydjz5KhZ3Lw%2BKQI0VRmJdM%3D"}],"group":"cf-nel","max_age":604800}', 'nel': '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}', 'x-frame-options': 'SAMEORIGIN', 'referrer-policy': 'same-origin', 'cache-control': 'private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0', 'expires': 'Thu, 01 Jan 1970 00:00:01 GMT', 'server': 'cloudflare', 'cf-ray': '8341c6c99c4736fc-YYZ', 'alt-svc': 'h3=":443"; ma=86400'} -+-+- b'error code: 521' - Sleeping for a few seconds and trying a

Slow certs

Started seeing this last week. Cert is running really slow, less that 50, down from 150+. Anyone else seeing this? I have tried through multiple connections on different environments, samething.

Error in arguments

Below error prompts when running the script

certificate_update: 0cert [00:00, ?cert/s][ERROR:websocket] 2020-12-20 15:23:01,269 - error from callback <bound method CertStreamClient._on_error of <certstream.core.CertStreamClient object at 0xb65c70ac>>: _on_error() takes exactly 3 arguments (2 given)

netflix domain is generating too many false positive

Netflix domain is generating too many false positive alerts

homoglyph: unconfuse not working

1. unconfuse not working

from confusables import unconfuse
....
domain = unconfuse(domain)

It seems that now this is not a great idea. It does not work for me

2. it’s completely unclear how national domains are handled
   xn--d1acufc.xn--p1ai

disconnecting - reconnecting

Hey!
Could someone help me with that the phishing_catcher is disconnecting and reconnecting in every minutes?

certificate_update: 386526cert [42:33, 119.09cert/s][ERROR:root] 2024-01-28 09:55:53,383 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
[INFO:root] 2024-01-28 09:55:58,669 - Connection established to CertStream! Listening for events...
certificate_update: 395220cert [43:19, 238.83cert/s][ERROR:root] 2024-01-28 09:56:38,713 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
[INFO:root] 2024-01-28 09:56:44,068 - Connection established to CertStream! Listening for events...
certificate_update: 404788cert [44:33, 205.77cert/s][ERROR:root] 2024-01-28 09:57:53,020 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
[INFO:root] 2024-01-28 09:57:58,294 - Connection established to CertStream! Listening for events...
certificate_update: 418396cert [45:52, 186.19cert/s][ERROR:root] 2024-01-28 09:59:12,524 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
[INFO:root] 2024-01-28 09:59:17,842 - Connection established to CertStream! Listening for events...

I have updated the certstream module with pip, but I have the same issue, but less output of it:

certificate_update: 16887cert [01:23, 143.12cert/s]Error connecting to CertStream - Connection to remote host was lost. - Sleeping for a few seconds and trying again...
certificate_update: 26518cert [02:27, 107.49cert/s]Error connecting to CertStream - Connection to remote host was lost. - Sleeping for a few seconds and trying again...
certificate_update: 38765cert [03:35, 203.87cert/s]Error connecting to CertStream - Connection to remote host was lost. - Sleeping for a few seconds and trying again...
certificate_update: 51049cert [04:44, 145.87cert/s]Error connecting to CertStream - Connection to remote host was lost. - Sleeping for a few seconds and trying again...
certificate_update: 58365cert [05:34, 198.15cert/s]Error connecting to CertStream - Connection to remote host was lost. - Sleeping for a few seconds and trying again...

Also, downgrading the websocket-client is not a solution anymore as i get this error:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
certstream 1.12 requires websocket-client>=0.58.0, but you have websocket-client 0.52.0 which is incompatible.

Is this intended behaviour?

Thank you in advance!

https://pypi.org/simple/entropy/ package not found

When I installing phishing_catcher, I got the following error:

Could not install packages due to an EnvironmentError: 404 Client Error: Not Found for url: https://pypi.org/simple/entropy/

I tried finding on Google if there's any similar packages, but there are too many (results: https://pypi.org/search/?q=entropy) and I'm not too sure which package to use.

May I know what should this package be replaced with?

Error from websocket

[ERROR:websocket] 2021-09-03 06:59:20,009 - error from callback <bound method CertStreamClient._on_message of <certstream.core.CertStreamClient object at 0x7f083221f8e0>>: name 'suspicious' is not defined

use frequency of significant parts of domain for scoring

While playing around with the script and the stream i often saw something like this:

[!] Likely    : catfinder-beta.corp.amazon.com (score=84)
[!] Likely    : catfinder-test.corp.amazon.com (score=84)
[!] Likely    : catfinder.corp.amazon.com (score=83) 
[!] Likely    : cctracker.corp.amazon.com (score=81) 
[!] Likely    : cefeedback.corp.amazon.com (score=83)
[!] Likely    : cepromotions.corp.amazon.com (score=82)
[!] Likely    : contractcentral-gamma.corp.amazon.com (score=82)
[!] Likely    : contractcentral.amazon.com (score=80)
[!] Likely    : cornerstone.amazon.com (score=81)    
[!] Likely    : cosmos-dashboard.corp.amazon.com (score=82)
[!] Likely    : cube-dub.corp.amazon.com (score=83)  
[!] Likely    : cube-metrics.corp.amazon.com (score=84)
[!] Likely    : cube-pdx.corp.amazon.com (score=84)  
[!] Likely    : cube-preview.corp.amazon.com (score=84)
[!] Likely    : cube-showcase.corp.amazon.com (score=84)
[!] Likely    : cube.amazon.com (score=80)           
[!] Likely    : daenerys-beta.corp.amazon.com (score=84)
[!] Likely    : dvatools.corp.amazon.com (score=82)  
[!] Likely    : dxa-dashboard.corp.amazon.com (score=83)
[!] Likely    : fleet-widget.corp.amazon.com (score=85)
[!] Likely    : fm-console.corp.amazon.com (score=83)
[!] Likely    : fua.corp.amazon.com (score=81)       
[!] Likely    : gcxgiftfindertools-eu.corp.amazon.com (score=86)
[!] Likely    : gcxgiftfindertools-fe.corp.amazon.com (score=86)

Therefore i thought that one could find the last part acting like the tld (.com or .co.uk - compare #38 ) and ignore that. The afterwards rightmost part - in this example corp.amazon - is them checked for how often it appeared in the stream in the last say hour (or day,...) and based on that, its score is computed: the highrr this number, the higher the score...

Not working with Python 2.7

Not working with python 2 whereas working fine with python 3

Error connecting to certstream

Could you explain why I get the following error when trying to run the script?

[ERROR:root] 2017-12-03 09:15:13,384 - Error connecting to CertStream - [SSL: TLSV1_ALERT_INTERNAL_ERROR] tlsv1 alert internal error (_ssl.c:579) - Sleeping for a few seconds and trying again...

ImportError: No module named certstream

Getting the Following Error while trying to run ./catch_phishing.py

Traceback (most recent call last):
  File "./catch_phishing.py", line 15, in <module>
    import certstream
ImportError: No module named certstream

I have installed all the requirements using pip3 install -r requirements.txt

DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
WARNING: Ignoring invalid distribution -qdm (/usr/local/lib/python3.9/site-packages)
WARNING: Ignoring invalid distribution -qdm (/usr/local/lib/python3.9/site-packages)
Requirement already satisfied: termcolor==1.1.0 in /usr/local/lib/python3.9/site-packages (from -r requirements.txt (line 1)) (1.1.0)
Requirement already satisfied: certstream==1.10 in /usr/local/lib/python3.9/site-packages (from -r requirements.txt (line 2)) (1.10)
Requirement already satisfied: tqdm==4.19.4 in /usr/local/lib/python3.9/site-packages (from -r requirements.txt (line 3)) (4.19.4)
Requirement already satisfied: tld==0.7.9 in /usr/local/lib/python3.9/site-packages (from -r requirements.txt (line 4)) (0.7.9)
Requirement already satisfied: python_Levenshtein==0.12.0 in /usr/local/lib/python3.9/site-packages (from -r requirements.txt (line 5)) (0.12.0)
Requirement already satisfied: websocket-client==0.48.0 in /usr/local/lib/python3.9/site-packages (from -r requirements.txt (line 6)) (0.48.0)
Requirement already satisfied: PyYAML==5.1 in /usr/local/lib/python3.9/site-packages (from -r requirements.txt (line 7)) (5.1)
Requirement already satisfied: six>=1.9 in /usr/local/lib/python3.9/site-packages (from tld==0.7.9->-r requirements.txt (line 4)) (1.16.0)
Requirement already satisfied: setuptools in /usr/local/lib/python3.9/site-packages (from python_Levenshtein==0.12.0->-r requirements.txt (line 5)) (62.3.2)
WARNING: Ignoring invalid distribution -qdm (/usr/local/lib/python3.9/site-packages)
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
WARNING: Ignoring invalid distribution -qdm (/usr/local/lib/python3.9/site-packages)
WARNING: Ignoring invalid distribution -qdm (/usr/local/lib/python3.9/site-packages)
WARNING: Ignoring invalid distribution -qdm (/usr/local/lib/python3.9/site-packages)
WARNING: There was an error checking the latest version of pip.

I am Using a 2019 Intel Based MacBook Pro; macOS Monterey(12.4)

How to Resolve this?

I have also tried running the docker file but it is also showing an error( Don't know if its because of improper docker installation ). Will Create a new issue for that.

Instagram

Docker Container showing Exit(2) status immediately after starting the container

Running Docker Version 4.10.1 (82475)
System : macOS Monterey(12.4); MacBook Pro Intel Based

I have built docker container by running docker build . -t phishing_catcher from the phishing_catcher file. There were no errors during this time

When ever I try to run docker container by clicking the play button in 2-3 seconds the docker containers quits and Exit(2) is shown in status

How to resolve this issue.

Feature: simuation mode from previously selected evil domains

Hey,

in order to be able to tweak an external.yaml and assess its efficiency, I feel there is a good need of a simulation mode where the code will read domains from a file instead of a certstream server.

PR #58 is an attempt to implement that idea.

Cheers,

--
Mathieu

use number of short parts of domains for scoring

I Propose to score domains higher if they got more than two short parts towards the end so that

some-domain.co.uk

is ok but something like

some-domain.com.info.co.uk.com

is punished.

This pattern happens surprisingly often - i played around with this project just for two days and this is something that got my attention...

Put the lists in a set

For performance improvement, quick change from lists to sets.

Instagram followers

Check domains with special characters

Hello,
Your repo can help a lot of people, but you should check for domains that have special characters like the ỵ , ṙ .
Let me know about updates!

A request

Hello
I was wondering...
Maybe a option to search into certstream log for search specific tags and parameters, will be useful in Osint investigations.

No handlers could be found for logger "certstream"

Just out of nowhere, phishing_catcher stopped showing certificates and upon executing throws the following error:

No handlers could be found for logger "certstream"

I've tried to pip install logging, but that didn't fix the issue. I've also downloaded a fresh version from Github and it doesn't work either.

Any ideas what could have happened? I'm running Python 2.7.18 or 3.9.2 on Debian 11.7.

Your help is much appreciated. Thank you.

Same domains printed multiple times

when I run the script, some domains are printed more that one time. some domains are printed again after a few minutes. what is the reason and how to prevent this?

support for 'include' arrays

Create support to read custom include configs for suspicious_keywords, highly_suspicious and suspicious_tld arrays so that every commit avoids stomping on local changes.

Getting Error on default installation

I cloned the project installed , pip install -r requirements.txt
then while running python catch_phishing I am getting error:
maximum recursion depth exceeded while calling a Python object

~/phishing_catcher$ ./catch_phishing.py
certificate_update: 0cert [00:00, ?cert/s][ERROR:websocket] 2017-11-08 17:56:53,509 - error from callback <bound method CertStreamClient._on_error of <certstream.core.CertStreamClient object at 0x7fb9b7901bd0>>: maximum recursion depth exceeded while calling a Python object
[ERROR:websocket] 2017-11-08 17:56:59,220 - error from callback <bound method CertStreamClient._on_error of <certstream.core.CertStreamClient object at 0x7fb9b7901e10>>: maximum recursion depth exceeded while calling a Python object
^C[INFO:root] 2017-11-08 17:57:01,067 - Kill command received, exiting!!

ImportError: No module named 'certstream'

Hi, Trying to get this phishing catcher set up but keep running into this error. I have followed the set up and have installed certstream, entropy and tqdm.

Every time I try running './catch_phishing.py' it comes up with this error message:

Traceback (most recent call last):
File "./catch_phishing.py", line 17, in
import certstream
ImportError: No module named 'certstream'

Know what the fix is?

CertStreamClient error?

Great project!
I've installed it via pip and are getting the following error;

$ ./catch_phishing.py
certificate_update: 0cert [00:00, ?cert/s][ERROR:websocket] 2018-08-14 17:03:04,419 - error from callback <bound method CertStreamClient._on_open of <certstream.core.CertStreamClient object at 0x7fa525fc73d0>>: _on_open() takes exactly 2 arguments (1 given)
[ERROR:websocket] 2018-08-14 17:03:06,919 - error from callback <bound method CertStreamClient._on_message of <certstream.core.CertStreamClient object at 0x7fa525fc73d0>>: _on_message() takes exactly 3 arguments (2 given)

Tried updating all required components to the latest but still the same error,
tried python3 and python2.7.12, no difference
Guess its a certstremclient error..?

Anyone have any clues on what I should change?

Continuous Integrations + code quality?

Everything is in the title.

What do you think about adding some travis CI as much as checks for PEP-8 compliancy, sorted imports and so on?

'NoneType' object is not iterable

I tried to search for a phishing site by turning off the override_suspicious.yaml: true instead of false, but it generated this line instead [ERROR:websocket] 2023-07-11 11:39:51,676 - error from callback <bound method CertStreamClient._on_message of <certstream.core.CertStreamClient object at 0x7f86db4c8040>>: 'NoneType' object is not iterable

Did I do something wrong?