This causes dnstwist to examine the same domain n times for each invocation of core.py

2 easy ways to fix this:

1. modify sql schema so that CREATE TABLE statement includes UNIQUE (email address, domain) constraint. should probably modify existing tables in the wild during build or add.
2. modify python so that it SELECTs (email, domain) pair before attempting to INSERT. should probably remove duplicates from existing tables in the wild during build or add.

This would summarize the total number of domains seen and new ones found over a given period, e.g. a day. This could be run as another cron job, serving the dual purpose of reassuring the recipient that the service is healthy even when not finding new phishing domains.

It might be helpful to retain some more notes about the status of found domain variants over time.

For example, some organizations may initiate take-down requests with domain registrars, report domains to Google for flagging in Chrome, etc.

Most domain variants in my experience are boring squatted domains, and could be easily differentiated with a simple HTTP request to http://example.com/.

This could be further extended to support customization by the user to look for the presence or absence of certain tell-tale strings in webpages and for other URL patterns, e.g. to detect commonly known phishing kit installations.

Something I noticed when suffering from #9 is that the number of domain variants within a short period of time is inconsistent. This is probably some DNS quirk but might merit more research.

Here's output from a single invocation:

2017-01-16 19:21:16,065:INFO:Now checking [email protected] - example.com
2017-01-16 19:21:17,431:INFO:DNSTwist found 71 variant domains from example.com.
2017-01-16 19:21:17,433:INFO:Found 0 new domain variants from example.com
2017-01-16 19:21:17,433:INFO:Now checking [email protected] - example.com
2017-01-16 19:23:30,415:INFO:DNSTwist found 68 variant domains from example.com.
2017-01-16 19:23:30,416:INFO:Found 0 new domain variants from example.com
2017-01-16 19:23:30,416:INFO:Now checking [email protected] - example.com
2017-01-16 19:23:30,888:INFO:DNSTwist found 70 variant domains from example.com.
2017-01-16 19:23:30,890:INFO:Found 0 new domain variants from example.com
2017-01-16 19:23:30,890:INFO:Now checking [email protected] - example.com
2017-01-16 19:23:46,608:INFO:DNSTwist found 62 variant domains from example.com.
2017-01-16 19:23:46,610:INFO:Found 0 new domain variants from example.com
2017-01-16 19:23:46,610:INFO:Now checking [email protected] - example.com
2017-01-16 19:25:22,057:INFO:DNSTwist found 73 variant domains from example.com.
2017-01-16 19:25:22,059:INFO:Found 0 new domain variants from example.com
2017-01-16 19:25:22,059:INFO:Now checking [email protected] - example.com
2017-01-16 19:25:22,570:INFO:DNSTwist found 73 variant domains from example.com.
2017-01-16 19:25:22,571:INFO:Found 0 new domain variants from example.com
2017-01-16 19:25:22,572:INFO:Now checking [email protected] - example.com
2017-01-16 19:25:32,913:INFO:DNSTwist found 72 variant domains from example.com.
2017-01-16 19:25:32,915:INFO:Found 0 new domain variants from example.com
2017-01-16 19:25:32,915:INFO:Now checking [email protected] - example.com

Some unit tests would be helpful. This is easiest with database-related stuff; network behavior is slower and thus less convenient to unit test.

Hi, for those who have issues using gmail sender with SSL on port 465 I suggest to switch on TLS on 587 adapting try loop on core.py accordingly, it works and I solved.
Thanks!

A simple 2to3-3.6 would suffices to fix it.

Had a request from an associate to add a feature where we check a bunch of TLDs in addition for the one assigned for the domain we are monitoring. While I'm inclined not to change the DNSTwist code, we can actually do this with the util.py add function. Suggest we add an optional parameter that can be a filename containing a CSV of all additional TLDs we'd like to check.

Example

Domain:
example.com

CSV File Contents:
tk
xyz
club

Result:
example.com
example.tk
example.xyz
example.club

If an alert email fails to send, the newly found domains will be marked as "found" and won't be considered for an alert during the next invocation.

Some possible solutions:
-track not only whether domains were previously found, but also their alert delivery status
-(easy) just log the domains found if the email alert fails.

Note that the email can fail to be delivered even if not rejected by the SMTP server (e.g. filtered as spam later in the delivery process) but there's not much we can do about that.

The application should log stuff, either in a system log or an application-specific one for debugging.

Hi, I was wondering whether your scripts keep updated while dnstwist get more capabilities in terms of typosquatting detection.