Comments (5)
Thanks for the question!
Indeed, query augmentation is only applied to queries. The v0.1 implementation (from which your snippet is taken) does this in a short way that could possibly seem confusing at first.
The above snippet appends 103
s (i.e., [MASK]
tokens) to queries and documents and sets the attention_mask
to zero. This attention mask controls attention inside the BERT encoder. This attention is different from ColBERT attention (MaxSim). After the BERT encoder, we mask this padding only for the document side right before MaxSim. When storing the documents to disk, we filter out these padding embeddings too (which are masked anyway).
In contrast, for the query side, the representations of the [MASK]
s is not masked before MaxSim. However, inside the BERT encoder, setting the attention mask set to zero forces the mask tokens to be computed in the last layer, and not to influence the other token representations. Empirically, this has little to no effect on performance given plenty of training data, so the simpler implementation is in this manner. (Keep in mind that query augmentation itself (as described) still contributes substantially to performance, though.)
Let me know if you have further questions!
from colbert.
This relies on an implementation detail of HuggingFace transformers: the attention mask doesn't change how the values are computed. It only affects what values are attended to.
So, to answer your question: in queries, the [MASK] positions are represented by paying attention (in the 12th layer) to all other query tokens. Notice this happens only in the 12th layer by default. If you do it in all 12 layers, there's little to no difference either. You can experiment with it to see how it functions.
Also, you may want to use the v0.2 code, which should be easier to understand and otherwise more extensive.
from colbert.
Thanks for your reply!
I know that doc side doesn't use the mask padding now. However, I am still confused about how mask padding makes a contribution. As I understand it, mask padding doesn't affect other tokens' representation, but will other token's representation affect mask's representation? I am not sure about this.
Besides, did you experiment how the query max_length affect the performance with mask padding?
from colbert.
I still have some doubts but I think query augmentation with mask padding is a great idea, and I will do some further experiments to verify it. Thank you for your reply!
from colbert.
I am having trouble understanding Query Augmentation and why does it add so much to performance. I saw the ablation study and the difference is quite significant.
From your earlier comments, it seems like augmentation happens only in the last layer of ColBERT, which again confuses me about the importance of this step and why would it contribute so much to the performance.
from colbert.
Related Issues (20)
- Request for AMD gpu support
- How to quickly check if installation is working fine?
- ColBert is not failing when Error is encounter during both train and indexing
- How to insert new document into the pre-built index? HOT 1
- Is there a check point of ColBERT that wasn't trained on MSMARCO?
- How to check the centroids and the data in the clusters?
- Extract only embeddings
- Execution fails in colbert.index_objs() with assert classname.endswith('Vector')
- Results on BEIR HOT 1
- unable to open file </root/.cache/huggingface/hub/models--bert-base-uncased/snapshots/86b5e0934494bd15c9632b12f734a8a67f723594/model.safetensors> in read-only mode: No such file or directory (2)
- Add_to_index only work first time
- Tokenization Assumption for Query Marker Replacement is Inconsistent
- GPU crashes when running "D_packed @ Q.to(dtype=D_packed.dtype).T" with no error message HOT 1
- Training script from doc is not working
- ImportError: cannot import name 'packaging' from 'pkg_resources' HOT 1
- Indexing stuck at encoding passages HOT 2
- ImportError: .../torch_extensions/py38_cu117/decompress_residuals_cpp/decompress_residuals_cpp.so: cannot open shared object file: No such file or directory
- How to load the checkpoint of "colbert-ir/colbertv2.0"
- FAISS RuntimError
- pthread.h error
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from colbert.