Support for optional documents metadata like "source file", "date created" etc. while indexing, to show them when searching about ragatouille

[adharm]

I'm taking a stab at this here.

I'll finish up the tests in the next day or so, and open a PR.

[LarsAC]

Ouch, saw this issue yesterday and started planning an implementation. Just got started so almost saw your note in time @anirudhdharmarajan ...

[0-hero]

Looking forward for your implementation. @anirudhdharmarajan

[adharm]

PR's up, would love any feedback on the approach I took for this (and a couple other related things that got pulled in 😅

[bclavie]

Live in 0.0.5a thanks to @anirudhdharmarajan's amazing work!

cc @LarsAC @0-hero @tm17-abcgen

[0-hero]

Thankyou, just raised a PR on llama-hub too run-llama/llama-hub#893

[bclavie]

Hey!

We don't currently support this, but it'd be relatively simple to add (though take more space on-disk!).

Right now, when an independent is create, we also export a collection.json, which contains all the documents inside it. This is useful so we can return the document: the index only store the passage ids (they correspond to the indices in the list contained in collection.json)

Apart from limited time, there's nothing stopping us from also storing a metadata.json file, which would be a list of dictionaries (or of a Metadata Pedantic object exported to dict, for easier validation), where the entry at index i would contain the metadata associated with passage_id i.

If you'd like to try to implement this, feel free! Otherwise it's definitely on the list of nice-to-have features that we'll circle to eventually.