Questions regarding the data split for UW3 about calamari

[ChWick]

Thanks for this very important information. Our dataset does not contain (maybe the extractor crashed or the download failed) the book dirs 0000 to 0315 in the training set which is exactly this missing count...
The single line in the evaluation should be a line which has a width of 0 when rescaling to a height of 48px and is therefore ignored by Calamari. Most probably this is a line that was rotated by 90 degrees.

[alwc]

Thanks for your prompt response Christoph!

I've tried to count the GT lines again after your information using the following code

uw = Path('./uw-lines/train')
uw_train_gt = glob.glob(str(uw_train / '*/*.gt.txt'))
discarded_data = list(range(0, 316))

num_gt_lines = 0
num_train_chars = 0
train_codes = set()

for gt in uw_train_gt:
    
    if int(gt.split('/')[2]) not in discarded_data:

        with Path(gt).open() as f:
            gt_codes = list(f.readline())
            if gt_codes[-1] == '\n':
                gt_codes = gt_codes[:-1]

            num_gt_lines += 1
            num_train_chars += len(gt_codes)
            train_codes.update(

but my results are still different compared to your numbers:

• GT lines for training (num_gt_lines): 72,907
• Chars for training (num_train_chars): 3,423,754
• Code for training (len(train_codes)): 89

Did I do something wrong with my code?

[ChWick]

This should be correct! I extracted everything anew and got your upper counts for train and evaluation.

Obviously even the 316 dir was not fully extracted (100 lines with 70K Characters are missing). The codec discrepancy of 2 occurs possibly due to our postprocessing of chars (unicode roman digits to latin chars, quotes, ...)

[alwc]

I see, thanks for the info.