Comments (5)
@okhat
Can you suggest something here?
from colbert.
It's not the easiest thing to use, but ColBERT does support pre-filtering:
Here's the chunk I use:
if len(query.conditions) > 0:
results = searcher.search(query.query, k=query.k, filter_fn=lambda pids: torch.tensor(
[index for index in pids.numpy().tolist() if keepResult(query, index)], dtype=pids.dtype))
else:
results = searcher.search(query.query, k=query.k, full_length_search=True)
Note: The query
object contains the filter conditions. The keepResult
function returns a boolean about whether the metadata for the given passage id (index
parameter) matches with the filter in the query
.
from colbert.
Hi @detaos
Thank you for your response
During indexing, how should I index it with metadata? My indexing function is something like
with Run().context(RunConfig(nranks=1, experiment=EXPERIMENT_NAME)): # nranks specifies the number of GPUs to use
config = ColBERTConfig(doc_maxlen=doc_maxlen, nbits=nbits, kmeans_niters=4) # kmeans_niters specifies the number of iterations of k-means clustering; 4 is a good and fast default.
# Consider larger numbers for small datasets.
indexer = Indexer(checkpoint=COLBERT_CHECKPOINT, config=config)
for name, text_list in texts_dict.items():
index_name = f'SEC.Earningcalls.{ticker}.{year}.{name}.{nbits}bits'
indexer.index(name=index_name, collection=text_list, overwrite=True)
How can I pass the metadata information, currently I am just passing the list of texts. Thanks in advance
from colbert.
You don't need to index metadata that won't help the search. For example, lastmod
dates from HTML pages are useful metadata, but no one is searching for a lastmod
date. So, I keep my non-search-related metadata separate. I have a mapping object from passage ID to page ID, then have a metadata object that has the metadata for each page. My keepResult
function uses the mapping for the candidate passage ID to the page ID to get the page's metadata to check against the filter. Essentially: metadata[page_ids[passage_id]]
It's a bit convoluted, but if you store the metadata for each passage, then you end up with a LOT of redundant metadata (presupposing you have many pages that are longer than 1 passage, which I do).
from colbert.
Ok, understood
Thanks @detaos, will implement this in my code
from colbert.
Related Issues (20)
- How to quickly check if installation is working fine?
- ColBert is not failing when Error is encounter during both train and indexing
- How to insert new document into the pre-built index? HOT 1
- Is there a check point of ColBERT that wasn't trained on MSMARCO?
- How to check the centroids and the data in the clusters?
- Extract only embeddings
- Execution fails in colbert.index_objs() with assert classname.endswith('Vector')
- Results on BEIR HOT 1
- unable to open file </root/.cache/huggingface/hub/models--bert-base-uncased/snapshots/86b5e0934494bd15c9632b12f734a8a67f723594/model.safetensors> in read-only mode: No such file or directory (2)
- Add_to_index only work first time
- Tokenization Assumption for Query Marker Replacement is Inconsistent
- GPU crashes when running "D_packed @ Q.to(dtype=D_packed.dtype).T" with no error message HOT 1
- Training script from doc is not working
- ImportError: cannot import name 'packaging' from 'pkg_resources' HOT 1
- Indexing stuck at encoding passages HOT 2
- ImportError: .../torch_extensions/py38_cu117/decompress_residuals_cpp/decompress_residuals_cpp.so: cannot open shared object file: No such file or directory
- How to load the checkpoint of "colbert-ir/colbertv2.0"
- FAISS RuntimError
- pthread.h error
- ValueError: negative shift count
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from colbert.