frobertpixto / hand-drawn-shapes-dataset

Hand-drawn Shapes (HDS) Dataset. Images like Rectangles, Ellipses and Triangles

Hand-drawn Shapes (HDS) Dataset

Summary

Dataset of Hand-drawn Images of shapes with vertices.

I have created the HDS Dataset for my app Mix on Pix.

See the complete DataSheet (as described in https://arxiv.org/pdf/1803.09010.pdf) for the HDS Dataset here.

Table of Contents

• Directory Structure
• The Images
• Processing
• Vertices
• Direct augmentation of the data
• People who drew the images
• Shape of type Other
• To do
• Communication
• Licenses

Directory Structure

• data: Contains all the HDS Dataset, with Images and Vertices .
  • 1 directory per user
    • For each sample, there is:
      • A file with an image like: images/ellipse/ellipse.aly.0001.png
      • A file with the vertices coordinates, like: vertices/ellipse/ellipse.aly.0001.csv
• processing: Contains notebooks for ETL and modelisation.
  • classify: Classification of shapes.
  • find_vertices: Regression to find the vertices.

The Images

One shape per image. Drawings exist for 4 shapes:

• Rectangle
• Ellipse
• Triangle
• Other

The Dataset contains images (70px x 70px x 1 gray channel) distributed as:

Total	Other	Rectangle	Ellipse	Triangle
27292 images	7287	6956	6454	6595

The shapes have been size-normalized and centered in a fixed-size image.

Processing

Classification

Hand Drawn Images of shapes are classified to determine the type of each shape.
More details here.

Regression

Regression is used to determine the exact intended size and angle of each drawn shape.
The vertices are the oranges points superimposed on the ellipses in the examples below.

More details here.

Vertices

Quick Geometry refresher:

• Vertices in shapes are the points where two or more line segments or edges meet (like a corner for a rectangle).
• Vertices of an ellipse are the 4 corner points at which the ellipse takes the maximum turn. Technically, an ellipse will have 2 vertices and 2 covertices. We will call them all vertices here.
• The singular of vertices is vertex.

Coordinates of vertices are interesting as they are much more precise than just a surrounding box used in Object detection.
Vertices allow us to determine the angle of the shape and it exact size.

Labelling of vertices

Labelling was done by me using a tool I created in Mix on Pix. For each image, the tool also generated a csv file with 1 line per vertex. Each Vertex has:

• a x coordinate betwwen 0 and 1
• a y coordinate betwwen 0 and 1

Where:

• (0,0) is the top left corner of the image
• (1,1) is the bottom right corner of the image

Note that the vertices are in no particular order. I sort them clockwise in the Extract-Transform-Load (ETL) processing.

Usefulness of vertices

Aside from drawing shapes on images like in Mix on Pix, another real-life example could be to determine the direction of a car (rectangle) or a ship (ellipse) in a direct overhead view.

Direct augmentation of the data

• 3 variations were generated per image
  1. Normal
  2. 1.5 to 3.0 wider
  3. 1.5 to 3.0 narrower
• One advantage is that I realized that:
  • People tend to make equilibrated shapes (Circle, Square, Equilateral triangle).
  • Most elongated images were interesting and sometime presented a different challenge than the original.
• This processing was not done for type Other.
• I validated them all manually (or we could say visually) and removed the generated images that were not interesting.
• This is different than the Augmentation done during Training (like horizontal and vertical flips, rotations) because:
  • It applies to all images including Validation set and Test set.
  • Being generated before being drawn provided images of a better quality.

I then used these images to train models that are used in Mix on Pix Auto-Shapes feature.

People who drew the images

Images were mostly generated by asking people I knew to draw Ellipses, Rectangles and Triangles in Mix on Pix.
People were aged from 7 to 87. I asked them to draw the way they wanted, slow or fast. In particular, I did not ask to draw nice shape. It was the intent while drawing that interested me.

Shape of type Other

The first ML models had no Other shape. I was initially happy with the excellent accuracy when the drawing was actually a rectangle, triangle or ellipse. But when testing internally with Mix on Pix, drawings of shapes like stars, hearts or crosses would always be recognized as one of Ellipse, Rectangle or Triangle. It is not a great user experience.

---

To do

Vertices

• Show video of classification and finding vertices in actions.

Later if requested

• Explain labelling rules for Vertices like Biggest shape that fits completely.
• Explain pipeline
• Add requirements.txt for Notebooks

---

Communication

• If you want to contribute, submit a pull request.
• If you found a bug, have suggestions or need help, please, open an issue.

---

Licenses

• The data: Hand-drawn Shapes (HDS) Dataset © 2022 by Francois Robert is licensed under CC BY 4.0. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
• The code: The code in the notebooks is licensed under the MIT license. See here.

---

by Francois Robert