lizhaodong / cortex_opendata

High quality open source dataset.

Cortex Dataset

We only take large-scale open source data sets into our list.

Video

1. Youtube8m :High-quality machine-generated annotations from a diverse vocabulary of 3,800+ visual entities which comes with precomputed audio-visual features.

Translation

1. WMT Translation Dataset: Machine Translation Dataset with different language.

Information Retrieval

1. Mirosoft Open Dataset:Dual Word Embeddings Trained on Bing Queries

Question Answering

1. Stanford Question Answering Dataset (SQuAD). Question answering about Wikipedia articles.
2. Deepmind Question Answering Corpus. Question answering about news articles from the Daily Mail.
3. Amazon question/answer data. Question answering about Amazon products.

Speech Recognition

1. TIMIT Acoustic-Phonetic Continuous Speech Corpus. Not free, but listed because of its wide use. Spoken American English and associated transcription.
2. VoxForge. Project to build an open source database for speech recognition.
3. LibriSpeech ASR corpus. Large collection of English audiobooks taken from LibriVox.

Audio Content Analysis

1. Google AudioSet : Google's large-scale dataset of manually annotated audio events that consists of an expanding ontology of 632 audio event classes and a collection of 2,084,320 human-labeled 10-second sound clips drawn from YouTube videos.
2. ExtraSensory Dataset : Behavioral context recognition over 300k examples (minutes) from 60 users in wild.
3. Lakh Pianoroll Dataset a collection of 174,154 multi-track piano-rolls derived from theLakh MIDI Dataset (LMD).

Document Summarization

1. Legal Case Reports Data Set. A collection of 4 thousand legal cases and their summarization.
2. TIPSTER Text Summarization Evaluation Conference Corpus. A collection of nearly 200 documents and their summaries.
3. The AQUAINT Corpus of English News Text. Not free, but widely used. A corpus of news articles.

Depth Estimation

1. NYU Depth Dataset V2 : a comprised of video sequences from a variety of indoor scenes as recorded by both the RGB and Depth cameras from the Microsoft Kinect.

Image Classification

1. CIFAR 10 & CIFAR 100
2. ImageNet :Data set for large-scale image classification. ImageNet dataset is a benchmark for many deep learning research.
3. PASCAL VOC
4. Street View House Numbers (SVHN) Dataset : Digits in real-world, an enhanced edition of MNIST.
5. MS COCO
6. Visual Genome
7. WebVision: Images crawled from the Flickr website and Google Images search. Large Scale with huge domain bias.
8. AI Challenger-图像属性数据集
9. AI Challenger-图像中文描述

Fine-Grained Visual Classification

1. Fine-Grained Visual Classification of Aircraft (FGVC-Aircraft)
2. iNaturalist Challenge at FGVC 2017 : Dataset with 5,089 species with extremely difficult for human to accurately classify due to their visual similarity.
3. iMaterialist Challenge at FGVC 2017 : Dataset designed for automatic product recognition for very similar visual object.

Semantic Segmentation

1. KITTI Data Set : This dataset interests are stereo, optical flow, visual odometry, 3D object detection and 3D tracking using a standard station wagon with two high-resolution color and grayscale video cameras.

Object Detection and Recognition

1. MS-COCO : MS-COCO dataset provided a dataset to solve recognition, segmentation and caption problem. The main purpose for this dataset is to push the progress of scene understanding.

   MS-COCO has the following features.

   • segmentation

   • Recognition in context

   • Superpixel stuff segmentation

   • 330K images (>200K labeled)

   • 1.5 million object instances

   • 80 object categories

   • 91 stuff categories

   • 5 captions per image

   • 250,000 people with keypoints

2. Cityscapes : Large scale city space semantic, instance-wise, dense pixel annotations of 30 classes.

3. Google Open Images Dataset : Open Images is a dataset of ~9 million images that have been annotated with image-level labels and object bounding boxes.

4. CORe50 : A new Dataset and Benchmark for Continuous Object Recognition, it provided a possible entry point for on-line learning.

5. Caltech-256 : 256 object categories containing a total of 30607 images. and the SIFT10M is SIFT features of Caltech-256 dataset.

Facial recognition

1. MegaFace : The largest publicly available facial recognition dataset.
2. MSR Image Recognition Challenge : Large Scale(10M) celebrity face images for the top 100K celebrities, which can be used to train and evaluate both face identification and verification algorithms.

Facial Landmark

1. 300-W :provides annotations for 3837 face images with 68 landmarks.
2. 300-WV : 300 Videos in the Wild (300-VW) Facial Landmark Tracking

Scene Classification

1. Place365 : the goal is to build a system for high-level visual understanding tasks.
2. LSUN :Scene classification and multitasking assistance (room layout estimation, saliency prediction, etc.).

Face Verification by Age

1. LAG: Large Age Gap :  a dataset containing variations of age in the wild, with images ranging from child/young to adult/old. The dataset contains 3,828 images of 1,010 celebrities. For each identity at least one child/young image and one adult/old image are present.
2. IMDB-WIKI : To the best of our knowledge this is the largest publicly available dataset of face images with gender and age labels for training. This work propose a method to solve the expectation of using single image to estimate the age of the figure in the image.

Stereo and 3D Reconstruction

1. Middlebury Stereo Vision Page : provides several multi-frame stereo data sets for comparing the performance of stereo matching algorithms.
2. Middlebury multi-view stereo (MVS) : a calibrated multi-view image dataset with registered ground truth 3D models for the comparison of MVS approaches.
3. TUD MVS :provides 124 different scenes that were recorded in controlled laboratory environment.

Motion Tacking

1. Real World Activity Recognition Dataset
2. Heterogeneity Activity Recognition Dataset : dataset from Smartphones and Smartwatches is a dataset devised to benchmark human activity recognition algorithms (classification, automatic data segmentation, sensor fusion, feature extraction, etc.) in real-world contexts.
3. THUMOS Challenge 2015 : Large video dataset for action classification.
4. MOTChallenge: The Multiple Object Tracking Benchmark : Large Scale Multiple Object Tracking dataset with images and videos.

Key-Point

1. MS-COCO Keypoint Detection Dataset
2. CrowdHuman : A Benchmark for Detecting Human in a Crowd.

Speech Recognition

1. LibriSpeech ASR corpus :Large-scale (1000 hours) corpus of read English speech
2. VoxForge: Voice dataset with ACCENT. (Use to test robust)
3. CHIME:A speech recognition challenge dataset containing ambient noise. The data set contains real, analog and clean voice recordings. Specifically, it includes nearly 9000 recordings of 4 speakers in 4 noisy environments. The analog data is combined with multiple environments and recorded in a noisy environment. The data.
4. TED LIUM : TED Talks speech dataset, with 1495 TED recordings and speech manuscript.
5. Spoken Language Dataset : speech samples of English, German and Spanish languages with equally balanced between languages, genders and speakers.

Environment Sound

1. UrbanSound : Labeled sound recordings of sounds like air conditioners, car horns and children playing. This dataset provide an entry point for environment sound recognition.

Recommendation System

1. Netflix Challenge

2. MovieLens : Movie comments. Baseline for collaborative filtering.

3. Million Song Dataset

4. Last.fm:Music recommendation data set with access to underlying social networks and other metadata.

   

5. Amazon Co-Purchasing and Amazon Reviews

6. Friendster Social Network Dataset