afcarl / scenenetv1.0

Still a work in progress and adding code..

#SceneNet

All the necessary source code for SceneNet SceneNet: Understanding Real World Indoor Scenes with Synthetic Data will be available here soon.

#Updates

This code enables depth and annotation rendering given a 3D model and trajectory. We provide a sample 3D model in the data folder and a trajectory in data/room_89_simple_data folder. More details will be updated soon. Things to do

• Simulated Annealing
• RGB Rendering to be merged with the code
• Releasing Trajectories
• Converting Depth to DHA format
• Emphasise on Negative Focal Length

#Dependencies

• [Pangolin] (https://github.com/ankurhanda/Pangolin-local) Local copy of https://github.com/stevenlovegrove/Pangolin
• CVD (It will not be a dependency any more in future!)
• TooN (It will be replaced by Eigen3 in future!)
• ImageUtilities
• SceneGraphRendering (will be merged within soon!)
• OpenCV/OpenCV2
• libnoise (from synaptic)

#Build

mkdir build
cd build
cmake .. -DCUDA_PROPAGATE_HOST_FLAGS=0
make -j8

#Demo in your build, run

./opengl_depth_rendering ../data/room_89_simple.obj

You should have annotated images in the folder data/room_89_simple_data that should look like these

#Adding noise to the depth maps

Video

#SceneNet Basis Models (Work In Progress)

• Bedrooms
• Bedroom Layouts
• Livingrooms
• Livingroom Layouts
• Offices
• Office Layouts
• Kitchens
• Kitchen Layouts
• Bathrooms
• Bathroom Layouts

#Sample Texture Library

Download the sample texture library to texture the models from here. Textured bedrooms are shown below.

#Website

More information is available here at our website robotvault.bitbucket.org

#Labels

Label name and number mapping is from Eigen et al. arXiv 2015, ICCV 2015

Label Number	Label Name
1	Bed
2	Books
3	Ceiling
4	Chair
5	Floor
6	Furniture
7	Objects
8	Picture
9	Sofa
10	Table
11	TV
12	Wall
13	Window

#Conversion from SUN RGB-D/NYUv2 37/40 Labels

Get the 40 class mapping for NYUv2 from http://www.cs.berkeley.edu/~sgupta/cvpr13/. SUN RGB-D already provide the 37 class mapping in their meta-data files.

Label name and number are from SUN RGB-D/ NYUv2. Their corresponding SceneNet/Eigen et al. mapping is in the last column.

SUN RGB-D/NYUv2Label Number	Label Name	Eigen et al./SceneNet Mapping
1	Wall	12
2	Floor	5
3	Cabinet	6
4	Bed	1
5	Chair	4
6	Sofa	9
7	Table	10
8	Door	12
9	Window	13
10	BookShelf	6
11	Picture	8
12	Counter	6
13	Blinds	13
14	Desks	10
15	Shelves	6
16	Curtain	13
17	Dresser	6
18	Pillow	7
19	Mirror	7
20	Floor-mat	5
21	Clothes	7
22	Ceiling	3
23	Books	2
24	Refrigerator	6
25	Television	11
26	Paper	7
27	Towel	7
28	Shower-curtain	7
29	Box	7
30	Whiteboard	7
31	Person	7
32	NightStand	6
33	Toilet	7
34	Sink	7
35	Lamp	7
36	Bathtub	7
37	Bag	7
38	Other-structure	7
39	Other-furniture	6
40	Other-prop	7

#Accuracy Script

Compute the global/per-class accuracy with getAccuracyNYU.m script provided in the repository.

#Latex Code

\usepackage[table]{xcolor}
\definecolor{bedColor}{rgb}{0, 0, 1}
\definecolor{booksColor}{rgb}{0.9137,0.3490,0.1882}
\definecolor{ceilColor}{rgb}{0, 0.8549, 0}
\definecolor{chairColor}{rgb}{0.5843,0,0.9412}
\definecolor{floorColor}{rgb}{0.8706,0.9451,0.0941}
\definecolor{furnColor}{rgb}{1.0000,0.8078,0.8078}
\definecolor{objsColor}{rgb}{0,0.8784,0.8980}
\definecolor{paintColor}{rgb}{0.4157,0.5333,0.8000}
\definecolor{sofaColor}{rgb}{0.4588,0.1137,0.1608}
\definecolor{tableColor}{rgb}{0.9412,0.1373,0.9216}
\definecolor{tvColor}{rgb}{0,0.6549,0.6118}
\definecolor{wallColor}{rgb}{0.9765,0.5451,0}
\definecolor{windColor}{rgb}{0.8824,0.8980,0.7608}

\begin{table*}
\begin{tabular}{l}
\textbf{13 class semantic segmentation: NYUv2} \\
\end{tabular}
\centering
\begin{tabular}{|l|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|p{0.5cm}|}
\hline
Training  & \cellcolor{bedColor}\rotatebox{90}{bed} & \cellcolor{booksColor}\rotatebox{90}{books}  & \cellcolor{ceilColor}\rotatebox{90}{ceil.} & \cellcolor{chairColor}\rotatebox{90}{chair}  & \cellcolor{floorColor}\rotatebox{90}{floor}  & \cellcolor{furnColor}\rotatebox{90}{furn}   & \cellcolor{objsColor}\rotatebox{90}{objs.} & \cellcolor{paintColor}\rotatebox{90}{paint.} & \cellcolor{sofaColor}\rotatebox{90}{sofa}   & \cellcolor{tableColor}\rotatebox{90}{table}  & \cellcolor{tvColor}\rotatebox{90}{tv}     & \cellcolor{wallColor}\rotatebox{90}{wall}   & \cellcolor{windColor}\rotatebox{90}{window} \\ \hline
NYU-DHA & 67.7 & 6.5 & 69.9 & 47.9 & \textbf{96.2} & 53.8 & 46.5 & 11.3 & 50.7 & 41.6 & 10.8 & 85.0 & 25.8 \\ \hline
SceneNet-DHA & 60.8 & 2.0 & 44.2 & 68.3 & 90.2 & 26.4 & 27.6 &  6.3 & 21.1 & 42.2 & 0 & \textbf{92.0} & 0.0 \\ \hline
SceneNet-FT-NYU-DHA & 70.8 & 5.3 & 75.0 & 58.9 & 95.9 & 63.3 & 48.4 & 15.2 & 58.0 & 43.6 & 22.3 &  85.1  & 29.9 \\ \hline
NYU-DO-DHA & 69.6 & 3.1 & 69.3 & 53.2 & 95.9 & 60.0 & 49.0 & 11.6 & 52.7 & 40.2 & 17.3 & 85.0 & 27.1 \\ \hline
SceneNet-DO-DHA & 67.9 & 4.7 & 41.2 & 67.7 & 87.9 & 38.4 & 25.6 &  6.3 & 16.3 & 43.8 & 0 & 88.6 & 1.0 \\ \hline
SceneNet-FT-NYU-DO-DHA & \textbf{70.8} & 5.5 & 76.2 & 59.6 & 95.9 & \textbf{62.3} & \textbf{50.0} & 18.0 & \textbf{61.3} & 42.2 & 22.2 & 86.1 & 32.1 \\ \hline
Eigen \textit{et al.} (rgbd+normals) \cite{Eigen:etal:ICCV2015} & 61.1 & \textbf{49.7} & 78.3 & \textbf{72.1} & 96.0 & 55.1 & 40.7 &\textbf{58.7} & 45.8 &\textbf{44.9}& \textbf{41.9} & 88.7 & \textbf{57.7}  \\ \hline
Hermans \textit{et al.}(rgbd+crf)\cite{Hermans:etal:ICRA2014} & 68.4 & N/A & \textbf{83.4} & 41.9 & 91.5 & 37.1 & 8.6 & N/A & 28.5 & 27.7 & 38.4 & 71.8 & 46.1 \\ \hline
\end{tabular}
\vspace{0.5mm} \vspace{0.5mm}
\caption{Results on NYUv2 test data for 13 semantic classes. We see a similar pattern here --- adding synthetic data helps immensely in improving the performance of nearly all functional categories of objects using DHA as input channels. As expected, accuracy on \textit{books}, \textit{painting}, \textit{tv}, and \textit{windows}, is compromised highlighting that the role of depth as a modality to segment these objects is limited. Note that we recomputed the accuracies of \cite{Eigen:etal:ICCV2015} using their publicly available annotations of 320$\times$240 and resizing them to 224$\times$224. Hermans \textit{et al.} \cite{Hermans:etal:ICRA2014} use ``\textit{Decoration}" and ``\textit{Bookshelf}" instead of \textit{painting} and \textit{books} as the other two classes. Therefore, they are not directly comparable. Also, their annotations are not publicly available but we have still added their results in the table. Note that they use 640$\times$480. Poor performance of SceneNet-DHA and SceneNet-DO-DHA on \textit{tv} and \textit{windows} is mainly due to limited training data for these classes in SceneNet.}
\label{table: CA breakdown for 13 classes}
\end{table*}

Relevant Documents

The whole process of data generation and experiments are in the following papers. If you find our work valuable and helpful, please consider citing the relevant document. Thank you.

SceneNet: an Annotated Model Generator for Indoor Scene Understanding, ICRA 2016
Ankur Handa, Viorica Patraucean, Simon Stent, Roberto Cipolla

@inproceedings{Handa:etal:ICRA2016,
  author    = {Ankur Handa and 
               Viorica P{\u a}tr{\u a}ucean and
               Simon Stent and
               Roberto Cipolla},
  title     = {SceneNet: an Annotated Model Generator for Indoor Scene Understanding},
  booktitle = {ICRA},
  year      = {2016}
}

Understanding Real World Indoor Scenes With Synthetic Data, CVPR 2016
Ankur Handa, Viorica Patraucean, Vijay Badrinarayanan, Simon Stent, Roberto Cipolla

@inproceedings{Handa:etal:CVPR2016,
  author    = {Ankur Handa and 
               Viorica P{\u a}tr{\u a}ucean and
               Vijay Badrinarayanan and 
               Simon Stent and
               Roberto Cipolla},
  title     = {Understanding Real World Indoor Scenes With Synthetic Data},
  booktitle = {CVPR},
  year      = {2016}
}

An up-to-date version is maintained on the arXiv

SceneNet: Understanding Real World Indoor Scenes with Synthetic Data arXiv link
Ankur Handa, Viorica Patraucean, Vijay Badrinarayanan, Simon Stent, Roberto Cipolla

@inproceedings{Handa:etal:arXiv2016,
  author    = {Ankur Handa and 
               Viorica P{\u a}tr{\u a}ucean and
               Vijay Badrinarayanan and 
               Simon Stent and
               Roberto Cipolla},
  title     = {SceneNet: Understanding Real World Indoor Scenes With Synthetic Data},
  booktitle = {arXiv},
  year      = {2015}
}

#License

All the code and data are released under a creative commons license which is purely for research purposes only. Please view the summary here http://creativecommons.org/licenses/by-nc/4.0/