studian / cvnd_p2_image_captioning

Udacity second project for CVND: image captioning. (GPU version)

Udacity Computer Vision Nanodegree Project 2

• This is my image captioning model for project 2 of the Computer Vision Nanodegree.
• I used the model architecture from: https://arxiv.org/abs/1411.4555 in order to implement the system.
• I used Titan XP (12 GB) x 4ea GPU.

Install pytorch 0.4.0

• Cuda : 8.0
• CuDNN : 6.0
• pytorch : 0.4.0

pip install http://download.pytorch.org/whl/cu80/torch-0.4.0-cp36-cp36m-linux_x86_64.whl

Install COCO API and Download COCO Dataset

1. Clone this repo: https://github.com/cocodataset/cocoapi

git clone https://github.com/cocodataset/cocoapi.git  

2. Setup the coco API (also described in the readme here)

cd cocoapi/PythonAPI  
make  
cd ..

3. Download some specific data from here: http://cocodataset.org/#download (described below)

• Under Annotations, download:

  • 2014 Train/Val annotations [241MB] (extract captions_train2014.json and captions_val2014.json, and place at locations cocoapi/annotations/captions_train2014.json and cocoapi/annotations/captions_val2014.json, respectively)
  • 2014 Testing Image info [1MB] (extract image_info_test2014.json and place at location cocoapi/annotations/image_info_test2014.json)

• Under Images, download:

  • 2014 Train images [83K/13GB] (extract the train2014 folder and place at location cocoapi/images/train2014/)
  • 2014 Val images [41K/6GB] (extract the val2014 folder and place at location cocoapi/images/val2014/)
  • 2014 Test images [41K/6GB] (extract the test2014 folder and place at location cocoapi/images/test2014/)

4. The project is structured as a series of Jupyter notebooks that are designed to be completed in sequential order (0_Dataset.ipynb, 1_Preliminaries.ipynb, 2_Training.ipynb, 3_Inference.ipynb).