mengtingwan / goodreads

code samples for the goodreads datasets

Goodreads Datasets

NOTE: Our datasets have been moved! Please see our new webpage about how to download these datasets.

The datasets were collected in late 2017 from goodreads. Details of the datasets are described in the dataset website

We collected these datasets for academic use only! Please do not redistribute them or use for commercial purposes.

Citations

If you are using our dataset, please cite the following papers:

• Mengting Wan, Julian McAuley, "Item Recommendation on Monotonic Behavior Chains", in RecSys'18. [bibtex]
• Mengting Wan, Rishabh Misra, Ndapa Nakashole, Julian McAuley, "Fine-Grained Spoiler Detection from Large-Scale Review Corpora", in ACL'19. [bibtex]

Notebooks/Code Samples

We've created several notebooks (in python 3.7) to illustrate how to download/read these datasets, and provide some basic explorations of the data.

• download.ipynb: If you prefer to download datasets without GUI. This notebook will show how to download files in bash/python.
• samples.ipynb: This notebook will show how to read '.json.gz' files line-by-line and display sample records of each file.
• statistics.ipynb: This notebook will calculate some basic statistics of the datasets (except the largest complete interaction file 'goodreads_interactions.csv'). Running this notebook may take a while.
• distributions.ipynb: This notebook will operate on the complete interaction file 'goodreads_interactions.csv' and provide some explorations of the distributions of these interactions. Note: Run this notebook only when you have LARGE memory (recommend 32g+)!!
• reviews.ipynb: This notebook will calculate some statistics of the review datasets.

goodreads's Issues

Is this data set not available for download?

Sorry to bother you, I am a student in school, looking for a detailed data set of book user interaction, if possible, could you please send me a data set? Thank you very much indeed!!

Goodreads dataset no longer available?

Hello! I'm a university student who has been using your goodreads dataset to experiment with some ML techniques and learn. I noticed since yesterday that the dataset is no longer available. Would this change be permanent? If so, would there be any chance I could get a compressed version of the data for my own learning purposes? Thanks!

In any case, I want to thank you for providing the very useful data! I have learnt a lot from it.

How were the users chosen?

I see that there are 876,145 total users in the dataset, but goodreads has 90 million users (as of july 2019). I was wondering how were those 876,145 users selected. Was there a minimum number of ratings?

Is the interaction records sequentially ordered?

Thank you for developing and sharing this great dataset. I am currently trying to create a sequence model for reading records. However, I am not sure if each user's interaction records are sequentially ordered.

I noticed that, for each user, the book_id is monotonically decreasing. Does it imply that the records are sequentially ordered in a reverse manner? That the user rates the last row of records first, then rates the second last row of records, and rates the first row at the end.

Thank you for your time!

gdown in colab gives access denied error

Trying to run download.ipynb in colab, I get the following error:

Access denied with the following error:

 	Too many users have viewed or downloaded this file recently. Please
	try accessing the file again later. If the file you are trying to
	access is particularly large or is shared with many people, it may
	take up to 24 hours to be able to view or download the file. If you
	still can't access a file after 24 hours, contact your domain
	administrator. 

You may still be able to access the file from the browser:

	 https://drive.google.com/uc?id=1zmylV7XW2dfQVCLeg1LbllfQtHD2KUon 

I tried

!pip install --upgrade --no-cache-dir gdown --pre
print(gdown.__version__) # 4.6.0

As an alternative to

file_id_map = dict(zip(file_ids['name'].values, file_ids['id'].values))

def download_by_name(fname, output=None, quiet=False):
    if fname in file_id_map:
        url = 'https://drive.google.com/uc?id='+file_id_map[fname]
        gdown.download(url, output=output, quiet=quiet)
    else:
        print(f'The file: {fname} can not be found!')

I tried to use a single line command !gdown 'https://drive.google.com/uc?id=1zmylV7XW2dfQVCLeg1LbllfQtHD2KUon', but still the same error!

Is there more efficient way to obtain data in colab than manually download them and put them in working directory?

cheers,

Academic Use allowed?

Hi Team, Can you confirm if it is legal to use this Goodreads dataset for academic research. I wanted to use it for my research?

Item-Specificity Statistics

In your academic paper "Fine-Grained Spoiler Detection from Large-Scale Review Corpora" (ACL 2019) you plotted a statistic called DF-IIF that shows the importance of a word for a specific item.

I have two questions about it:

1. What are you doing in case of OOV token (let's say for sentences coming from the test set)?
2. Can you please share your code for that calculation as well?

Thanks.

dataset links uncreachable

all the links to the datasets listed in the datasets section of the webpage return a 404 error.

Any plans for an entire goodreads user review dataset?

The API has been discontinued, but it's actually faster to collect reviews from RSS feeds.

If there's interest, I started a script here

https://colab.research.google.com/drive/1uOyVlKaT4QFtce9yQpKj9hRtj5z8Uyta

It still needs work in confirming it has gotten all the books from a user (I think there might be timeouts) and issues with books that have several versions/editions. But the biggest bottleneck is collecting reviews from all 100 million users