Documentation to help support a custom data set about dspy

[davidwynter]

To answer my first question convqa_program mentioned in section 3.4 of the paper looks like it accommodates this requirement. but cannot find a reference to it other than in the 1 instance in the paper. Maybe the convqa_attempt is it, but has no reference to turns.

I asked ChatGPT about sizing required to train a small BERT model, (BERT-Base model has 12 transformer layers and 110 million parameters) it replied 8 cores, 16GB RAM and a NVidia GPU with at least 2GB of RAM. I have such a machine, so can give it a spin to see if it works.

[okhat]

Hey @davidwynter ,

Please see here for indexing your own data for retrieval: https://github.com/stanford-futuredata/ColBERT#running-a-lightweight-colbertv2-server

Before you launch the server, use the ColBERT intro notebook (or the Overview in the ColBERT README) to index your collection

In principle, you shouldn't need to train a new retriever. You can usually use the one we provide (depending on licensing at least; it's trained on MS MARCO, the dataset itself is for non-commercial research use).