frankfletcher / sign1 Goto Github PK

RE https://github.com/cogsci2/Sign1/blob/master/notebooks/Sign2%20-%20Implementing%20Mixup-20210113-best.ipynb

Overall looks good. Just a few comments:

• Any characteristics of the third party dataset (https://www.kaggle.com/grassknoted/asl-alphabet) that will limit or complicate evaluation or model development to reach the success criteria from your project plan?
• Why did you supplement with your own images?
• What are the differences between the dataset you created and the third party data? How many subjects (people) are included? What is consistent (background, lighting, etc) and what varies across the images?
• Is there anything about this data that we need to know or explore to interpret the model results? What does an accuracy rate of 100% mean? Which scenarios would we expect the model to be 100% accurate and when would we expect the model perf to be less or unknown? Basically just looking for clarification on the "in-domain" scenario where your perf metrics can be trusted as-is. The "out-of-domain" scenario (scoring the model on images and classes of images unseen during training and evaluation) will require more effort to convince audience of model performance.
• Related the question above. How is the included evaluation structured? The class balance of ground truth is shown in the table, but does this represent the scoring scenario intend to use the model on (scoring a stranger's hand signs and translating to text)? Are the same test subjects (humans) in train and test sets? How does the model perform on images from people or backgrounds it has not seen before?