kernelsvm / pollitt

An open-source library for identifying and rendering outliers and anomalies in data sets — designed for use with Google documents and libraries.

pollitt

An open-source library for identifying and rendering outliers and anomalies in data sets — designed for use with Google documents and libraries.

Overview

Pollitt was created as a tool for parsing small to medium (1-2000 row) data sets with an eye toward parsing and rendering:

• The scale and frequency of values within columns/variables
• Correlation between columns/variables
• Variables/columns that significantly overindex or underindex within a population
• Most-typical, least typical, and key outlier records in a data set

It was designed to be used by non-technical administrators, primarily for the mining of survey data, sales data and customer information by strategists, researchers and internal teams — although it could certainly be adapted for other applications.

To that end, Pollitt was built to work with Google Spreadsheets rather than SQL databases, enabling end-to-end deployment with no-cost tools at no (or minimal) cost to the end user.

Configuration

There are two parts to the configuration of Pollitt:

Source spreadsheet configuration

1. Turn on sharing for your Google spreadsheet. You will want to ensure that 'anyone with the link can view' has been selected in the sharing options. There is no need to 'publish to the web'.
2. Your top row should contain column names, without spaces. If you use leading caps, Pollitt will display them with spaces inserted correctly (i.e. 'MidAtlantic' becomes 'Mid Atlantic'). At this time, failure to remove spaces in column names will result in an error.
3. The second row should specify any categorization you would like to apply to the reporting. Leaving this row blank will result in this column being excluded from the report. In our sample data set, columns 'NewEngland' and 'MidAtlantic' are both labled 'Location'.
4. While your data set may include several uniqueIDs, it is important to designate one of them the primary key by categorizing that column 'ID'.

To see these examples in action, reference the sample Running Data 2012 spreadsheet.

Page configuration

All of the following settings reference the configuration.js file located in Pollit's root directory:

1. Point your instance of Pollitt to the Google Spreadsheet you've created by updating the value of googleSpreadsheetURL. You will need to add '/gviz/tq' to the end of the URL if it's not already included, in order to have the query work correctly.
2. Update the page title and header by updating pageTitle_str
3. You can adjust thresholds for identifying a variable as underindexed or overindexed by changing the values of overIndexThreshold_num and underIndexThreshold_num.
4. As of the current build, sample size is not calculated automatically. To adjust the minimum required sample size, adjust the value of sampleSizePercentageThreshold_num. Assigning it a value of 3, for example, is equivalent to declaring any value that represents less than 3% of the overall records in the set too small to work with.

Deployment

Pollitt can be deployed either to a web server or run locally through the browser by opening the index.html file. Running the file off a server will likely deliver a better, more predictable experience, as local instances will be subject to restrictions in place from, for example, company firewall policies.

Pollitt requires a web connection, both for access to the data in a Google spreadsheet and for the loading of external libraries.

About

Pollitt is built on the Bootstrap 3.2.0 framework, hosted by MaxCDN. It makes use of jQuery 2.1.1, and of both Christian Bach's tablesorter class and Steve Gardner's work on code for calculating Pearson correlation coefficients.

It was developed by Ian Fitzpatrick in August 2014, and released for the 2014 Planning-ness conference in Portland, Oregon.