Updating the index about chroma

[jeffchuber]

Update here!

• You can update the index by just adding more items with add_items. You might need to check the size of the index first however and resize_index first. No idea what the logic is around how expensive this is.. cons of making it too large, etc.
• You can soft delete using mark_deleted and unmark_deleted. Seems the ideal approach is to tombstone items and then have a background job clean things up. This is what weaviate did for example.

some relevant links

• nmslib/hnswlib#388
• nmslib/hnswlib#4

So the next step here is to proof of concept growing the index. As well as proof of concept around deleting items.

[jeffchuber]

Getting close on this... I need to figure out how to use RETURNING in the context of executemany in python. It should be possible, but it's not working as I expect... Going to run a test in the CLI next.

[jeffchuber]

Not worth implementing until we swap out duckdb.

[jeffchuber]

Moving to backlog