web-profiler's Introduction

Web Profiler

Tools for gathering statistics about contents and performance of Web sites. There are two parts:

Python "loader" classes that load Web pages (using various backends, like PhantomJS, Chrome, and Firefox) and produce summary statistics, HAR files, and/or screenshots.
Scripts that use these loaders to gather particular statistics (e.g., compare the load times of the HTTP and HTTPS versions of a site).

Loaders

The loaders are Python classes that encapsulate the logic for loading Web pages and saving statistics. Each one implements a common interface (specified by the Loader superclass in loader.py) and currently supports the following backends:

PhantomJS (PhantomJSLoader in phantomjs_loader.py)
Chrome (ChromeLoader in chrome_loader.py)
Firefox (FirefoxLoader in firefox_loader.py)
Python Requests (PythonRequestsLoader in pythonrequests_loader.py)
Curl (CurlLoader in curl_loader.py)
NodeJS (NodeJsLoader in nodejs_loader.py)

API documentation available here.

Dependencies:

PhantomJS

Needed by the PhantomJSLoader.

If the loader can't find PhantomJS, try harding-coding the path to PhantomJS at the top of phantomjs_loader.py.
Xvfb

Needed to run Firefox or Chrome in headless mode.
chrome-har-capturer

Needed by the ChromeLoader.

Set path to binary at top of chrome_loader.py. To install globally:
```
  sudo npm install -g chrome-har-capturer
```
To record HTTP versions, you currently need to use this fork. To install globally:
```
  git clone [email protected]:dtnaylor/chrome-har-capturer.git
  cd chrome-har-capturer
  sudo npm install -g .
```
node.js

Needed by the ChromeLoader. (On Ubuntu, be sure to install nodejs package and not node.)
Python Requests

Used to test the availability of sites of HTTP or HTTPS and by the PythonRequestsLoader.
node-http2

Needed by the NodeJsLoader and ZombieJsLoader. Location of the module must be hardcoded in webloader/nodejs_loader.py and webloader/zombiejs_loader.py. z

Recommend Projects