martinthoma / hasy

HASY dataset

Please refer to the HASY paper for details about the dataset. If you want to report problems of the HASY dataset, please send an email to [email protected] or file an issue at https://github.com/MartinThoma/HASY

Errata are listed in the git repository as well as the actual hasy package.

Contents

The contents of the HASYv2 dataset are:

• hasy-data: 168236 png images, each 32px x 32px
• hasy-data-labels.csv: Labels for all images.
• classification-task: 10 folders (fold-1, fold-2, ..., fold-10) which contain a train.csv and a test.csv each. Every line of the csv files points to one of the png images (relative to itself). If those files are used, then the hasy-data-labels.csv is not necessary.
• verification-task: A train.csv and three different test files. All files should be used in exactly the same way, but the accuracy should be reported for each one. The task is to decide for a pair of two 32px x 32px images if they belong to the same symbol (binary classification).
• symbols.csv: All classes
• README.txt: This file

How to evaluate

Classification Task

Use the pre-defined 10 folds for 10-fold cross-validation. Report the average accuracy as well as the minumum and maximum accuracy.

Verification Task

Use the train.csv for training. Use test-v1.csv, test-v2.csv, test-v3.csv` for evaluation. Report TP, TN, FP, FN and accuracy for each of the three test groups.

hasy package

hasy can be used in two ways: (1) as a shell script (2) as a Python module.

If you want to get more information about the shell script options, execute

$ hasy --help
usage: hasy [-h] [--dataset DATASET] [--verify] [--overview] [--analyze_color]
            [--class_distribution] [--distances] [--pca] [--variance]
            [--correlation] [--count-users] [--analyze-cm CM]

optional arguments:
  -h, --help            show this help message and exit
  --dataset DATASET     specify which data to use (default: None)
  --verify              verify PNG files (default: False)
  --overview            Get overview of data (default: False)
  --analyze_color       Analyze the color distribution (default: False)
  --class_distribution  Analyze the class distribution (default: False)
  --distances           Analyze the euclidean distance distribution (default:
                        False)
  --pca                 Show how many principal components explain 90% / 95% /
                        99% of the variance (default: False)
  --variance            Analyze the variance of features (default: False)
  --correlation         Analyze the correlation of features (default: False)
  --count-users         Count how many different users have created the
                        dataset (default: False)
  --analyze-cm CM       Analyze a confusion matrix in JSON format. (default:
                        False)

If you want to use hasy as a Python package, see

python -c "import hasy.hasy_tools;help(hasy.hasy_tools)"

Changelog

• 14.05.2020, hasy Python package: Major refactoring of this repository
• 24.01.2017, HASYv2: Points were not rendered in HASYv1; improved hasy_tools https://doi.org/10.5281/zenodo.259444
• 18.01.2017, HASYv1: Initial upload https://doi.org/10.5281/zenodo.250239

hasy's Issues

Similar datasets

• MNIST
• http://www.ee.surrey.ac.uk/CVSSP/demos/chars74k/
• EMNIST

One-Shot Learning

Add one-shot learning challenge

Missing symbol classes

• = (equals, http://write-math.com/symbol/?id=1364)

See also:

• https://github.com/MartinThoma/hwrt/blob/master/hwrt/misc/symbols.yml

Report Top-3 accuracy

Due to extremely similar classes, the top-3 accuracy should also be reported.

Licensing Terms

Hi there. I'm in the process of creating a simple 3D simulator for reinforcement learning tasks, and I would like to include some text in various 3D scenes.

I'm wondering what are the licensing terms of the HASYv2 dataset, wrt redistributing the data. Could I bundle some of this dataset with my simulator, which is MIT licensed?

Kindest regards,

• Maxime

Compressability

Add information how "compressible" the information is.

• Raw data: (168233 images (3232) pixel / image * 8 bit / pixel) * (1/8 byte / bit) * (1/10**6 MB / bye) ~~ 172.27 MB
• PNG-encoded: 37.5 MB
• .tar.bz2: 29.3 MB
• Auto-encoder?

Completely white images

Some images are completely white.

The reason for this is an error I made while creating the dataset. It simply does not draw points. This will be fixed in HASYv2

Transfer learning: How well does a HASY-trained model on MNIST?

Color scheme

I use jet, but I should switch to viridis

Add 'Noisy classification' challenge

Up to my knowledge, it is not well-studied how models are influenced by label noise. Thus an classification challenge, where the training data labels are randomly changed could be interesting.

Competition: Symbol classification

When (if) I release a new version of the dataset, I would make a competition to establish a baseline. The best 3 teams + all teams with unique and interesting approaches would be mentioned in the dataset paper.

Who would be interested to be asked when (if) that happens? (Please be aware that it might take a long time until this happens)

Mention accuracy on training set

for TF-CNN

Mention relationship to CROHME

CROHME - https://www.isical.ac.in/~crohme/CROHME_data.html

Stop seeding the random number generator

Hi,

In my opinion the package should not seed any random number generators (in this case python's and numpy's). Otherwise the usage of the package becomes quite limited since its users have to worry about setting them themselves, or worse, not even realize that they are not actually generating random test runs.

If you want to keep its effect maybe move the related statements to the __main__ region.

Best regards,
Ricardo Jesus

Larger dimension was scaled

The paper states:

The smaller dimension of each recordings bounding box was scaled to be 32 px.

This is wrong. It should be "The larger dimension".

Errata

In the following list, "predicted" is what the label should be

Predicted=I or |, true=\bot, hasy-data/v2-120965.png
Predicted=M, true=\fint, hasy-data/v2-90580.png
Predicted=8, true=\circ, hasy-data/v2-85959.png
Predicted=\pi, true=\epsilon, hasy-data/v2-38782.png
Predicted=\sigma, true=\mid, hasy-data/v2-98167.png
Predicted=m, true=], hasy-data/v2-133444.png
Predicted=\Gamma, true=[, hasy-data/v2-133304.png
Predicted=\oint, true=\fint, hasy-data/v2-90604.png
Predicted=\fint, true=\int, hasy-data/v2-56520.png
Predicted=\therefore, true=\because, hasy-data/v2-104173.png
Predicted=\therefore, true=\because, hasy-data/v2-104132.png
Predicted=\because, true=\therefore, hasy-data/v2-103990.png
true=\mathcal{Z}, Predicted=X, hasy-data/v2-15452.png
true=\mathfrak{A}, Predicted=\mathcal{U}, hasy-data/v2-15469.png
true=\mathfrak{A}, Predicted=\mathcal{U}, hasy-data/v2-15472.png
true=\mathfrak{A}, Predicted=\mathcal{U}, hasy-data/v2-15476.png
true=\subset, Predicted=\in, hasy-data/v2-05010.png
true=\propto, Predicted=\infty, hasy-data/v2-05448.png
true=\propto, Predicted=\infty, hasy-data/v2-05486.png
true=\propto, Predicted=\infty, hasy-data/v2-05525.png
true=\cap, Predicted=\sqcap, hasy-data/v2-07313.png
true=\shortrightarrow, Predicted=\mapsto, hasy-data/v2-119825.png
true=\top, Predicted=\bot, hasy-data/v2-123811.png
true=], Predicted=m, hasy-data/v2-133444.png
true=\aleph, Predicted=\mathcal{O}, hasy-data/v2-136624.png
true=\mathbb{1}, Predicted=\mathds{R}, hasy-data/v2-154930.png
true=\mathbb{1}, Predicted=\mathds{R}, hasy-data/v2-155160.png
true=\mathds{C}, Predicted=\sum, hasy-data/v2-155798.png
true=\mathds{N}, Predicted=\sqsubseteq, hasy-data/v2-157190.png
true=\mathds{Z}, Predicted=\mathds{N}, hasy-data/v2-159842.png
true=\astrosun, Predicted=\ast, hasy-data/v2-164260.png
true=\astrosun, Predicted=\ast, hasy-data/v2-164263.png
true=\Bowtie, Predicted=\infty, hasy-data/v2-165617.png
true=\bot, Predicted=|, hasy-data/v2-121173.png
true=\bot, Predicted=|, hasy-data/v2-121333.png
true=\top, Predicted=\perp, hasy-data/v2-123809.png
true=\top, Predicted=\perp, hasy-data/v2-123889.png
true=\top, Predicted=\perp, hasy-data/v2-123938.png
true=\top, Predicted=\bot, hasy-data/v2-123811.png
true=\top, Predicted=\bot, hasy-data/v2-123988.png
true=\in, Predicted=\leftarrow, hasy-data/v2-126736.png
true=\exists, Predicted=\nexists, hasy-data/v2-129588.png
true=\rfloor, Predicted=|, hasy-data/v2-132486.png
true=[, Predicted=\Gamma, hasy-data/v2-133304.png
true=\square, Predicted=\blacksquare, hasy-data/v2-146977.png
true=\square, Predicted=\blacksquare, hasy-data/v2-147615.png
true=\mathcal{A}, Predicted=\forall, hasy-data/v2-149371.png
true=\mathcal{A}, Predicted=\forall, hasy-data/v2-149569.png
true=\Bowtie, Predicted=\infty, hasy-data/v2-165617.png
true=\fint, Predicted=M, hasy-data/v2-90580.png
true=\fint, Predicted=\oint, hasy-data/v2-90604.png
true=\fint, Predicted=\oint, hasy-data/v2-90626.png
true=\approx, Predicted=\simeq, hasy-data/v2-91328.png
true=\not\equiv, Predicted=\div, hasy-data/v2-95283.png
true=\mid, Predicted=\sigma, hasy-data/v2-98167.png
true=\vdash, Predicted=\vDash, hasy-data/v2-99138.png
true=\parallel, Predicted=\mathds{R}, hasy-data/v2-102088.png
true=\therefore, Predicted=\div, hasy-data/v2-103630.png
true=\therefore, Predicted=\because, hasy-data/v2-103990.png
true=\neq, Predicted=\rightarrow, hasy-data/v2-107063.png
true=\neq, Predicted=\Rightarrow, hasy-data/v2-107106.png
true=\downarrow, Predicted=\psi, hasy-data/v2-110636.png
true=\mathscr{S}, Predicted=\varphi, hasy-data/v2-162373.png
true=\mathcal{U}, Predicted=\varphi, hasy-data/v2-153783.png
true=\mathfrak{M}, Predicted=\int, hasy-data/v2-154268.png
true=\S, Predicted=\oint, hasy-data/v2-05825.png
true=\vdash, Predicted=\vDash, hasy-data/v2-09273.png
true=\downarrow, Predicted=\Downarrow, hasy-data/v2-10564.png
true=\mathds{E}, Predicted=\mathbb{1}, hasy-data/v2-15717.png

noise:

hasy-data/v2-45995.png
Predicted=\Theta, true=\div, hasy-data/v2-80369.png
true=\pi, Predicted=\sigma, hasy-data/v2-00455.png
true=\sqrt{}, Predicted=\blacksquare, hasy-data/v2-145547.png
true=\pi, Predicted=\div, hasy-data/v2-22121.png
true=\mathcal{F}, Predicted=\sqcup, hasy-data/v2-150599.png

wrong class (have to look up the right class):

true=\in, Predicted=\twoheadrightarrow, hasy-data/v2-126036.png

Training setup

Report batch size

Baseline for Verification Challenge missing