fr1ll / clip-plot Goto Github PK

• Change folder for embeddings output to match model name
• Enabled viewer to consume multiple embeddings

Why is the method in the class? Can we remove it?

Date layout output file name is not consistent. Layout is referred to as "date" is multiple locations:

• get_date_layout()
• referenced in manifest.json as "date"
• metadata date output named "date.json"

Yet the layout output is referenced as "timeline".

The main feature of glob2 is that it supports recurvsive ** wildcard expansion.

glob.glob supports this now with recursive=True option. Path.glob supports recursive expansion by default.

glob2 also hasn't been updated since 2019.

Let's replace glob2 with Path.glob to remove an unnecessary dependency and to push us more to use Path objects which make other things more convenient.

• Check if all channels are equal for all images
• If they are, convert embedding model into single channel via timm method
• write out jpgs as L instead of RGB

Unused import:
from tensorflow.keras.applications import imagenet_utils

Can we remove it?

Allow clip-plot to use images with the same basename. Since the images location can be provided as a glob, more than one sub directory can be specified:

--images "jpg_parent_folder/**/*.jpg"

jpg_parent_folder
---> jpg_sub_folder_1
-------> imageA.jpg
-------> imageB.jpg
---> jpg_sub_folder_2
-------> imageA.jpg
-------> imageC.jpg

The although the names are not unique, the image paths are unique:

• jpg_parent_folder/jpg_sub_folder_1/imageA.jpg <---Name not unique
• jpg_parent_folder/jpg_sub_folder_1/imageB.jpg
• jpg_parent_folder/jpg_sub_folder_2/imageA.jpg <---Name not unique
• jpg_parent_folder/jpg_sub_folder_2/imageC.jpg

Inconsistent naming of alphabetical and rasterfairy layouts

The user can switch umap layouts when it has multiple variants. But variants are not labeled in the user interface. User has no idea which variant is being displayed.

To do:

• Refactor/encapsulate get_umap_layout
  • Merge process_multi_layout_umap with process_single_layout_umap
  • Add ability in process_multi_layout_umap to handle labels
• Refactor/encapsulate get_rasterfairy_layout

Replace print() statements with proper logging

Missing data/hotspots/user_hotspots.json error in browser (404)

ValueError: n_samples=10 should be >= n_clusters=12.

get_hotspots(...):
    ...
    z = model.fit(vecs)  # Error here!

Notes:

• HDBSCAN not available; using sklearn KMeans
• CUML not available; using umap-learn UMAP

To do:

• Added two more butterflies to test case and issue no longer happens.
• Need to add error handling.

Output parameters used to generate visualization in the manifest

Getting an error when selecting "Arrange images using lat/long coordinates"

THREE.Object3D.add: object not an instance of THREE.Object3D. null [three.min.js:7429:13]
add http://localhost:5000/tests/smithsonian_butterflies_10/output_test_temp/assets/vendor/dist/three.min.js:7429
show http://localhost:5000/tests/smithsonian_butterflies_10/output_test_temp/assets/js/tsne.js:3471
showHideContext http://localhost:5000/tests/smithsonian_butterflies_10/output_test_temp/assets/js/tsne.js:797
set http://localhost:5000/tests/smithsonian_butterflies_10/output_test_temp/assets/js/tsne.js:708
addEventListeners http://localhost:5000/tests/smithsonian_butterflies_10/output_test_temp/assets/js/tsne.js:661

This issue is most likely caused by the updated ThreeJS library.

• #8

FFCV is a library intending to reduce image bottlenecks during inference: https://github.com/libffcv/ffcv

It would be interesting to see if we can use this instead of our own code to do image validation, copying, and bulk inference.

...because it's actually kind cool.

We should also label the slider properly if we do this though.