geekydevu / hnpickup

This is an educational example of a data mining web application: when is good time to post on HN

How to build a data mining web app?

You will need some basic programming and statistical skills. Web Development, jQuery, Python, and Machine Learning skills are a plus. If you can look at new data and immediately see where data mining adds new value, then you are definitely overqualified to use this source code.

The first step is to get your own data. Is there any websites that you visit every day? I'm sure they produce fresh content every day: new articles, new stats, new numbers. How about you start collecting them? If you have your own data you decide what interesting question it may answer.

Next step. Does your favorite website have any trends? More articles are published in summer than in winter? More people are willing to "like" articles in spring than in autumn. Is it possible to predict which article will create more web traffic, thus, more revenue from advertisements?

Finally, once you mine the answer you should display it. Do it so that it's pleasurable for the eye. Colorful time series or multidimensional scaling should do the trick. Describe your graph so the people not familiar with your project can understand it and enjoy it.

Does it seem like a lot of work? Well, here is a source code that deploys your app with one command on Google App Engine. You just need to focus on where to get the data (ETL), what to do with it (DM), and how to display it (VISUALIZATION). The source code has example that you can swap with an idea of your own.

Little by little you will master how to add monetary value to your data, sell it, or build a business model.

START

Create an account and new app placeholder at:

http://appengine.google.com/

Please install Google App Engine SDK from:

http://code.google.com/appengine/downloads.html

Getting started with Python web app development is here:

http://code.google.com/appengine/docs/python/gettingstarted/

For lunching the app you can use this nice GUI:

http://code.google.com/p/google-appengine-mac-launcher/
http://code.google.com/p/google-appengine-wx-launcher/

It's also good to have Google Analytics account:

http://www.google.com/analytics/

You can also check webmaster tools to make sure your website is properly indexed by Google:

https://www.google.com/webmasters/tools

Don't forget to rename your app in the app.yaml file:

application: hnpickupdev -> application: yourapp

Deploy your frontend with one command:

appcfg.py update ./

Deploy backend with one command:

appcfg.py backends ./ update

Current source code requires at least six data points. That means you have to run "/etl_process" webpage at least six times and then "/dm_process" at least once before you see a graph.

INTRO

This is example of a simple data mining application. Here Hacker News aggregator is our source of data. The data mining objective is to figure out when is good time to post an article or a story on Hacker News website so other people will up-vote it and it will get to from the "newest" page to "news" page.

MODEL

This app can serve as a very simple business model where you claim is that your DATA MINING application brings better EXPERIENCE, OUTCOME, and VALUE to existing products. How come? If you start adding new knowledge to existing data you will see the pattern: large data can be abstracted to a small chunk of information that is more valuable than the large dataset. That's how you sell your service. Example? Every day you observe cars; that's a lot of data, however, you notice that around 8 am there are many more cars than at other hours; this is your small chunk of information. This small chunk will save you 30 min of stuck in traffic: better experience, outcome and value.

CODE

Most data mining application will have very similar information flow:

1.ETL -> 2.DM -> 3.VISUALIZATION

Which means:

1.GET THE DATA -> 2.ADD VALUE -> 3.USE IT

That's why the code is organized into three sections:

1. ETL = Extract, Transform, Load (GET THE DATA)
2. DM = Data Mining (ENRICH THE DATA, ADD VALUE)
3. VISUALIZATION = Data presentation in a format that 
can support decision making process (USE IT, SHOW IT)

You can think of the code as a "Hello, World!" web data mining example. You shouldn't be surprised that most of the code went into visualization. That's how you get your customers to buy-in. Data for visualization is obtained using JSON serialization.

More on data sets (ETL) with user recommendations/ratings is here https://gist.github.com/1653794

More on data mining (DM) algorithms is here http://mloss.org/software/

More on data visualization using JS is here https://gist.github.com/1515418.

WARNING

This app shows some raw data. For more complicated projects it might not be good idea to show the raw data. Too much data on the user interface will clog the decision making process.

FUTURE

The hope is that early stage start-ups can use this code to quickly organize their thoughts and prototype their idea. Google App Engine can run this app for free, giving opportunity to demonstrate a working version of their idea.

If you are ambitious and you want to add value to your data using sophisticated statistical methods I suggest you watch Stanford on-line classes at http://www.ml-class.org (if you don't want to login to get the video lectures, here is the hack: http://news.ycombinator.com/item?id=3335753)

OTHER

Similar data mining app:

http://stephenholiday.com/articles/2011/stack-overflow-by-day/

Remember, time series analysis is just a small portion of data mining.

BUSINESS

Here is example of a business model that does exactly that:

http://seatgeek.com/

You will notice that the website has these three components:

1. ETL - events + cost + seats (API pipelines or scraper?)
2. DM - personalized information extraction (the COLUMBUS event engine)
3. VISUALIZATION - calendar view + "theatre" view + price options

Another example of a working buisness model is SHOPOBOT:

http://www.shopobot.com/

They data mine prices on Amazon marketplace.

The hardest thing is finding a value in data. One solution would be applying "time is money" and "money is value" rule to your data. Often times search does exactly that. Both SEATGEEK and SHOPOBOT search for good shopping deals and save you time and money. The same thinking might be applied to many other areas of daily activities beside electronics and tickets. This is good start. But as soon as you master finding places where your web app saves time and money you should move to more advanced areas where domain expertise is needed. Medicine, finance, litigation, and manufacturing are four big markets. Each is producing tons of data every day. Find an expert, a medical doctor or an accountant, ask about a mundane task that one performs every day but could be automated. You just earned your first $1 mln!

BOOKS

There are books that cover the topic of finding business value in data, just one of them:

http://www.amazon.com/Mining-Web-Transforming-Customer-Value/dp/0471416096