scrapinghub / wappalyzer-python

UNMAINTAINED Python wrapper for Wappalyzer (utility that uncovers the technologies used on websites)

wappalyzer-python -- UNMAINTAINED

Python wrapper for Wappalizer (utility that uncovers the technologies used on websites)

Warning: this package is not maintained anymore.

Scrapinghub and Javier Casas, the original author, have no plans to support wappalyzer-python in the foreseeable future (this includes fixing bugs, supporting upgraded dependencies like PyV8 etc.)

If you are interested in continuing the work, please get in touch via [email protected] so that we can discuss transferring ownership of this repository.

How to use it

>>> from wappalyzer import Wappalyzer
>>> w = Wappalyzer()

>>> w.analyze('http://wikipedia.org')
{u'Apache': {u'confidence': 100, u'version': u'', u'categories': [u'web-servers']},
u'Varnish': {u'confidence': 100, u'version': u'', u'categories': [u'cache-tools']}}

>>> w.analyze('http://tripadvisor.com')
{u'Apache': {u'confidence': 100, u'version': u'', u'categories': [u'web-servers']},
u'Google Analytics': {u'confidence': 100, u'version': u'', u'categories': [u'analytics']},
u'comScore': {u'confidence': 100, u'version': u'', u'categories': [u'analytics']}}

>>> w.analyze('http://facebook.com')
{u'reCAPTCHA': {u'confidence': 100, u'version': u'', u'categories': [u'captchas']}}

You can specify the User-Agent to use:

>>> w.analyze('http://www.google.com', user_agent='your_user_agent')

Or analyze from already downloaded pages (in this case you'll need to have the url and response headers too):

>>> w.analyze_from_data(url=the_url, html=the_html, headers=the_response_headers)

Apps and Categories are available as dict objects:

>>> w.apps
{u'Google Wallet': {u'website': u'wallet.google.com', u'cats': [41], u'script': [u'checkout\\.google\\.com',
u'wallet\\.google\\.com']}, u'Lockerz Share': ...}

>>> w.categories
{u'42': u'tag-managers', u'48': u'network-storage', u'43': u'paywalls', u'49': u'feed-readers', u'24':
u'rich-text-editors', u'25': u'javascript-graphics', u'26': u'mobile-frameworks', ...}

Data can be also updated with the latest version available from the Wappalyzer Github repo:

>>> from wappalyzer import updater
>>> updater.update_all()

By default app icons will be updated to the data/icons folder, in case you need them somewhere else you can specify the destination folder:

>>> from wappalyzer import updater
>>> updater.update_all(icons_folder='your_icons_folder')

Or update them individually:

>>> updater.update_icons(icons_folder='your_icons_folder')

Requirements

• Requests
• PyV8
• lxml

Note for macos users: If you have problems installing PyV8 you can use PyV8-OS-X:

pip install -e git://github.com/brokenseal/PyV8-OS-X#egg=pyv8

Install

Using setup:

python setup.py install

Using pypi:

pip install wappalyzer-python

wappalyzer-python's Issues

Technology Version

It seems that the library doesn't extract the version of the identified technology (e.g. 8.5 is missing for 'IIS 8.5', being shown in the browser add-on).

no module names 'analyze'

AttributeError: 'module' object has no attribute 'analyze'

is there any solution ?

Set Timeout to analyze

Hello

if you try to reveal the technologies behind http://www.ticketmaster.com , it will take a very long time.
try with ur self:
w.analyze('http://www.ticketmaster.com')

is there is a way to set a timeout period for analyze?

PyV8 dependency fails

The last time I tried using wappalyzer-python it failed with the PyV8 dependency.

Hangs after a certain period of time

On a 4GB memory machine, if I keep giving wappalyzer input to analyse, it will hang after reaching about 2.6GB of memory and stay stuck. I tried pre-scraping sites using scrapy and then giving it as input to the analyse_from_data function but this hangs in the exact same way.

Parse the output

Hello
I want to thank you for this great wrapper
but I have a question:
when you try to analyze some website, there are a lot of "u" in the output, is there is a way to remove it, cause I need to parse the output then store it in a database.

w.analyze('http://wikipedia.org')
{u'Apache': {u'confidence': 100, u'version': u'', u'categories': [u'web-servers']},
u'Varnish': {u'confidence': 100, u'version': u'', u'categories': [u'cache-tools']}}

Regards
Ibrahim

how update json

How may I update the json file for improve the identify proccess by adding new fingerprints?