hadley / data-movies

Download data from IMDB movies and parse into useful form

The internet movie database, imdb.com, is a website devoted to collecting movie data supplied by studios and fan. It claims to be the biggest movie database on the web and is run by amazon. More about information imdb.com can be found online, including information about the data collection process.

IMDB makes their raw data available. Unfortunately, the data is divided into many text files and the format of each file differs slightly. To create one data file containing all the desired information these ruby scripts extract the relevant information and store in a database. Finally, this data is exported to csv to make it easier to import into data analysis packages.

The following text files were downloaded and used:

• business.list. Total budget
• genres.list. Genres that a movie belongs to (eg. comedy and action)
• movies.list. Master list of all movie titles with year of production.
• mpaa-ratings-reasons.list. MPAA ratings.
• ratings.list. IMDB fan ratings.
• running-times.list. Movie length in minutes.

Movies were selected for inclusion if they had a known length and had been rated by at least one IMDB user. The final output contains the following fields:

• title. Title of the movie.
• year. Year of release.
• budget. Total budget (if known) in US dollars
• length. Length in minutes.
• rating. Average IMDB user rating.
• votes. Number of IMDB users who rated this movie.
• r1-10. Distribution of votes for each rating, to mid point of nearest decile: 0 = no votes, 4.5 = 1-9$%$ votes, 14.5 = 11-19$%$ of votes, etc. Due to rounding errors these may not sum to 100.
• mpaa. MPAA rating.
• action, animation, comedy, drama, documentary, romance, short. Binary variables representing if movie was classified as belonging to that genre.

data-movies's Issues

Running this program

Hello! What call do I make to this program to run it? (New to Ruby)

Invalid byte sequence in US-ASCII (ArgumentError)

I get "02-import.rb:36:in `match': invalid byte sequence in US-ASCII (ArgumentError)", what sould I do with it?

No Genre or Actors/Actresses

Following the answer in #3, I was able to get this to run all the way through. The only issue is, as mentioned in #3, the 'Genres' and 'Ratings' tables are not working, and the Actors/Actresses are not actually added to the 'Movies' table. Here are screenshots of all three tables (in DB Browser for SQLite). I'm aware that the creator of this abandoned this project a long time ago, but I'm wondering if there is anybody out there who knows how to fix this? I'm a beginner to SQL and Ruby, but maybe it has something to do with the syntax of the 'INSERT INTO's in the Ruby script? Any help is greatly appreciated. Thank you very much.

Order movies by rating

Just ran across this project. I didn't realize they exposed the raw data. Did this a while ago (depends on xidel/screen scraper):

Pass a list of named directories to https://gist.github.com/shaiguitar/5eda74e4bfb817d7f88a

Output looks like:

https://gist.githubusercontent.com/shaiguitar/4891a11d51bb54cf9a3e/raw/a608fa264fbf9420a9eb0a04e0bfdeb9171eb6c8/gistfile1.txt

Jgg